[
https://issues.apache.org/jira/browse/BEAM-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15553163#comment-15553163
]
Amit Sela commented on BEAM-59:
-------------------------------
We've discussed over the dev list about data locality, and it seems that it
makes sense that if HDFS can provide this information to a runner, we should
use it.
[[email protected]] and [~jbonofre] do you think it should be a part of this
effort ? I've opened BEAM-673 for the Spark runner when it seemed there was no
interest in this as part of the SDK, but the discussion has grown since and
there seems to be an agreement that it should be a part of the HdfsIO or some
HdfsUtils.
WDYT ?
> IOChannelFactory rethinking/redesign
> ------------------------------------
>
> Key: BEAM-59
> URL: https://issues.apache.org/jira/browse/BEAM-59
> Project: Beam
> Issue Type: New Feature
> Components: sdk-java-core, sdk-java-gcp
> Reporter: Daniel Halperin
>
> Right now, FileBasedSource and FileBasedSink communication is mediated by
> IOChannelFactory. There are a number of issues:
> * Global configuration -- e.g., all 'gs://' URIs use the same credentials.
> This should be per-source/per-sink/etc.
> * Supported APIs -- currently IOChannelFactory is in the "non-public API"
> util package and subject to change. We need users to be able to add new
> backends ('s3://', 'hdfs://', etc.) directly, without fear that they will be
> broken.
> * Per-backend features: e.g., creating buckets in GCS/s3, setting expiration
> time, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
