[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270310#comment-16270310 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on issue #4080: [BEAM-2500] Add S3

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270301#comment-16270301 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull request #4080:

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270302#comment-16270302 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull request #4080:

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270303#comment-16270303 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull request #4080:

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270300#comment-16270300 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull request #4080:

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270296#comment-16270296 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull request #4080:

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270088#comment-16270088 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull request #4080:

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270089#comment-16270089 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull request #4080:

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270085#comment-16270085 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull request #4080:

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270087#comment-16270087 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull request #4080:

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270083#comment-16270083 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull request #4080:

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270086#comment-16270086 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull request #4080:

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270082#comment-16270082 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull request #4080:

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270081#comment-16270081 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull request #4080:

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270080#comment-16270080 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull request #4080:

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270076#comment-16270076 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull request #4080:

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270074#comment-16270074 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull request #4080:

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270075#comment-16270075 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull request #4080:

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270069#comment-16270069 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull request #4080:

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269353#comment-16269353 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on issue #4080: [BEAM-2500] Add S3

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269334#comment-16269334 ] ASF GitHub Bot commented on BEAM-2500: -- lukecwik commented on issue #4080: [BEAM-2500] Add S3

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16239745#comment-16239745 ] ASF GitHub Bot commented on BEAM-2500: -- GitHub user jacobmarble opened a pull request:

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16239725#comment-16239725 ] Jacob Marble commented on BEAM-2500: Thanks, Ismael. I have updated my fork and will submit a PR in a

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202062#comment-16202062 ] Ismaël Mejía commented on BEAM-2500: [~jmarble] Hi, I took a look at the implementation, one comment,

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188453#comment-16188453 ] Jacob Marble commented on BEAM-2500: I have started a fork here:

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173410#comment-16173410 ] Jacob Marble commented on BEAM-2500: Hmm, I think there's a bug. Please ignore for now. > Add support

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173403#comment-16173403 ] Jacob Marble commented on BEAM-2500: I see this warning thousands of times when reading from S3: Sep

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171952#comment-16171952 ] Jacob Marble commented on BEAM-2500: Thanks, Steve. Fixed. > Add support for S3 as a Apache Beam

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171891#comment-16171891 ] Steve Loughran commented on BEAM-2500: -- looking at the code, the main thing I'd highlight is that

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170919#comment-16170919 ] Jacob Marble commented on BEAM-2500: Here is a working implementation. I'm going to use it for a work

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167614#comment-16167614 ] Steve Loughran commented on BEAM-2500: -- bq. . So we'll have to have a way to stream bytes into S3

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167608#comment-16167608 ] Steve Loughran commented on BEAM-2500: -- This is how Hadoop does its multipart upload OutputStream

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166841#comment-16166841 ] Jacob Marble commented on BEAM-2500: Luke, thanks! I may have found what you're referencing in

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166823#comment-16166823 ] Luke Cwik commented on BEAM-2500: - The GCS implementation uses a fixed size buffer of 32 or 64mbs (don't

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166803#comment-16166803 ] Jacob Marble commented on BEAM-2500: Multipart upload could get us around the content length

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166741#comment-16166741 ] Jacob Marble commented on BEAM-2500: Chamikara, thanks for your comment. I'll switch my implementation

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166735#comment-16166735 ] Luke Cwik commented on BEAM-2500: - Performing the multipart download/upload will become important as 5GiBs

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166678#comment-16166678 ] Chamikara Jayalath commented on BEAM-2500: -- Thanks for looking into this Jacob. I'll try to answer

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165391#comment-16165391 ] Jacob Marble commented on BEAM-2500: I'm interested in implementing S3 support. Not being familiar Beam

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136759#comment-16136759 ] Ismaël Mejía commented on BEAM-2500: I created BEAM-2790 to move the discussion on the issue with the

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123497#comment-16123497 ] Jean-Baptiste Onofré commented on BEAM-2500: Thanks for the update. On my side, I gonna take a

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123445#comment-16123445 ] Guillaume Balaine commented on BEAM-2500: - Sorry about that, discovered a bug later down the line,

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123094#comment-16123094 ] Steve Loughran commented on BEAM-2500: -- +also, as well as any fixes to S3A, it'd be sweet if Beam

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123047#comment-16123047 ] Guillaume Balaine commented on BEAM-2500: - Thanks, that's fine really, the only trouble was that I

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120232#comment-16120232 ] Steve Loughran commented on BEAM-2500: -- Thanks for the update ":" is a troublespot char in hadoop

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120095#comment-16120095 ] Guillaume Balaine commented on BEAM-2500: - I got s3a to work on a simple aggregation job, I just

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16073967#comment-16073967 ] Steve Loughran commented on BEAM-2500: -- It's not clear that the S3 clients from EMR or Apache (S3A)

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16059757#comment-16059757 ] ASF GitHub Bot commented on BEAM-2500: -- GitHub user lukecwik opened a pull request: