[
https://issues.apache.org/jira/browse/BEAM-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15992406#comment-15992406
]
ASF GitHub Bot commented on BEAM-59:
------------------------------------
GitHub user dhalperi opened a pull request:
https://github.com/apache/beam/pull/2818
[BEAM-59] Delete old restrictions on output file paths
These predate Apache Beam and are no longer relevant now that Text and Avro
are implemented
in the SDK.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dhalperi/beam remove-old-validation
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/2818.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2818
----
commit 50d7ff5380dc038a0797a313414fd56ab291cabb
Author: Dan Halperin <[email protected]>
Date: 2017-05-02T06:36:57Z
[BEAM-59] Delete old restrictions on output file paths
These predate Apache Beam and are no longer relevant now that Text and Avro
are implemented
in the SDK
----
> Switch from IOChannelFactory to FileSystems
> -------------------------------------------
>
> Key: BEAM-59
> URL: https://issues.apache.org/jira/browse/BEAM-59
> Project: Beam
> Issue Type: New Feature
> Components: sdk-java-core, sdk-java-gcp
> Reporter: Daniel Halperin
> Assignee: Daniel Halperin
> Fix For: First stable release
>
>
> Right now, FileBasedSource and FileBasedSink communication is mediated by
> IOChannelFactory. There are a number of issues:
> * Global configuration -- e.g., all 'gs://' URIs use the same credentials.
> This should be per-source/per-sink/etc.
> * Supported APIs -- currently IOChannelFactory is in the "non-public API"
> util package and subject to change. We need users to be able to add new
> backends ('s3://', 'hdfs://', etc.) directly, without fear that they will be
> broken.
> * Per-backend features: e.g., creating buckets in GCS/s3, setting expiration
> time, etc.
> Updates:
> Design docs posted on dev@ list:
> Part 1: IOChannelFactory Redesign:
> https://docs.google.com/document/d/11TdPyZ9_zmjokhNWM3Id-XJsVG3qel2lhdKTknmZ_7M/edit#
> Part 2: Configurable BeamFileSystem:
> https://docs.google.com/document/d/1-7vo9nLRsEEzDGnb562PuL4q9mUiq_ZVpCAiyyJw8p8/edit#heading=h.p3gc3colc2cs
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
