[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270310#comment-16270310 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on issue #4080: [BEAM-25

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270301#comment-16270301 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull requ

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270302#comment-16270302 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull requ

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270303#comment-16270303 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull requ

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270300#comment-16270300 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull requ

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270296#comment-16270296 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull requ

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270088#comment-16270088 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull requ

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270089#comment-16270089 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull requ

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270085#comment-16270085 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull requ

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270087#comment-16270087 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull requ

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270084#comment-16270084 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull requ

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270083#comment-16270083 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull requ

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270086#comment-16270086 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull requ

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270082#comment-16270082 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull requ

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270081#comment-16270081 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull requ

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270080#comment-16270080 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull requ

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270076#comment-16270076 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull requ

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270074#comment-16270074 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull requ

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270075#comment-16270075 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull requ

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270069#comment-16270069 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on a change in pull requ

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269353#comment-16269353 ] ASF GitHub Bot commented on BEAM-2500: -- jacobmarble commented on issue #4080: [BEAM-25

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269334#comment-16269334 ] ASF GitHub Bot commented on BEAM-2500: -- lukecwik commented on issue #4080: [BEAM-2500]

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16239745#comment-16239745 ] ASF GitHub Bot commented on BEAM-2500: -- GitHub user jacobmarble opened a pull request:

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16239725#comment-16239725 ] Jacob Marble commented on BEAM-2500: Thanks, Ismael. I have updated my fork and will su

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16202062#comment-16202062 ] Ismaël Mejía commented on BEAM-2500: [~jmarble] Hi, I took a look at the implementation

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16188453#comment-16188453 ] Jacob Marble commented on BEAM-2500: I have started a fork here: https://github.com/Koc

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16173410#comment-16173410 ] Jacob Marble commented on BEAM-2500: Hmm, I think there's a bug. Please ignore for now.

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16173403#comment-16173403 ] Jacob Marble commented on BEAM-2500: I see this warning thousands of times when reading

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16171952#comment-16171952 ] Jacob Marble commented on BEAM-2500: Thanks, Steve. Fixed. > Add support for S3 as a A

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16171891#comment-16171891 ] Steve Loughran commented on BEAM-2500: -- looking at the code, the main thing I'd highli

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16170919#comment-16170919 ] Jacob Marble commented on BEAM-2500: Here is a working implementation. I'm going to use

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16167614#comment-16167614 ] Steve Loughran commented on BEAM-2500: -- bq. . So we'll have to have a way to stream by

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16167608#comment-16167608 ] Steve Loughran commented on BEAM-2500: -- This is how Hadoop does its multipart upload

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16166841#comment-16166841 ] Jacob Marble commented on BEAM-2500: Luke, thanks! I may have found what you're referen

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16166823#comment-16166823 ] Luke Cwik commented on BEAM-2500: - The GCS implementation uses a fixed size buffer of 32 or

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16166803#comment-16166803 ] Jacob Marble commented on BEAM-2500: Multipart upload could get us around the content l

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16166769#comment-16166769 ] Jacob Marble commented on BEAM-2500: Luke, that put-copy API is called by the Java SDK,

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16166741#comment-16166741 ] Jacob Marble commented on BEAM-2500: Chamikara, thanks for your comment. I'll switch my

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16166735#comment-16166735 ] Luke Cwik commented on BEAM-2500: - Performing the multipart download/upload will become imp

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16166678#comment-16166678 ] Chamikara Jayalath commented on BEAM-2500: -- Thanks for looking into this Jacob. I'

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16165391#comment-16165391 ] Jacob Marble commented on BEAM-2500: I'm interested in implementing S3 support. Not bei

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16136759#comment-16136759 ] Ismaël Mejía commented on BEAM-2500: I created BEAM-2790 to move the discussion on the

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123497#comment-16123497 ] Jean-Baptiste Onofré commented on BEAM-2500: Thanks for the update. On my side,

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123445#comment-16123445 ] Guillaume Balaine commented on BEAM-2500: - Sorry about that, discovered a bug later

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123094#comment-16123094 ] Steve Loughran commented on BEAM-2500: -- +also, as well as any fixes to S3A, it'd be sw

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123091#comment-16123091 ] Steve Loughran commented on BEAM-2500: -- * Do a patch for that &the ByteBuffer & I'll l

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123047#comment-16123047 ] Guillaume Balaine commented on BEAM-2500: - Thanks, that's fine really, the only tro

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120232#comment-16120232 ] Steve Loughran commented on BEAM-2500: -- Thanks for the update ":" is a troublespot ch

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120095#comment-16120095 ] Guillaume Balaine commented on BEAM-2500: - I got s3a to work on a simple aggregatio

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073967#comment-16073967 ] Steve Loughran commented on BEAM-2500: -- It's not clear that the S3 clients from EMR or

[jira] [Commented] (BEAM-2500) Add support for S3 as a Apache Beam FileSystem

[ https://issues.apache.org/jira/browse/BEAM-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059757#comment-16059757 ] ASF GitHub Bot commented on BEAM-2500: -- GitHub user lukecwik opened a pull request: