[jira] [Commented] (FLINK-5706) Implement Flink's own S3 filesystem

[ https://issues.apache.org/jira/browse/FLINK-5706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15895648#comment-15895648 ] Steve Loughran commented on FLINK-5706: --- Stefan, I don't think you appreciate how ha

[jira] [Comment Edited] (FLINK-5706) Implement Flink's own S3 filesystem

[ https://issues.apache.org/jira/browse/FLINK-5706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15895648#comment-15895648 ] Steve Loughran edited comment on FLINK-5706 at 3/4/17 6:24 PM: -

[jira] [Commented] (FLINK-5706) Implement Flink's own S3 filesystem

[ https://issues.apache.org/jira/browse/FLINK-5706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15895808#comment-15895808 ] Steve Loughran commented on FLINK-5706: --- I should add that my current stance with us

[jira] [Commented] (FLINK-5706) Implement Flink's own S3 filesystem

[ https://issues.apache.org/jira/browse/FLINK-5706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15897209#comment-15897209 ] Steve Loughran commented on FLINK-5706: --- If you look at where object stores are most

[jira] [Commented] (FLINK-7589) org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 159764230; received: 64638536)

[ https://issues.apache.org/jira/browse/FLINK-7589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16166527#comment-16166527 ] Steve Loughran commented on FLINK-7589: --- bq. is there any plan to make Flink use AW

[jira] [Commented] (FLINK-10363) S3 FileSystem factory prints secrets into logs

[ https://issues.apache.org/jira/browse/FLINK-10363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16622263#comment-16622263 ] Steve Loughran commented on FLINK-10363: Stephan: we went to a lot of effort to

[jira] [Commented] (FLINK-10363) S3 FileSystem factory prints secrets into logs

[ https://issues.apache.org/jira/browse/FLINK-10363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623330#comment-16623330 ] Steve Loughran commented on FLINK-10363: see WHIRR-642 for this same issue; it's

[jira] [Commented] (FLINK-7365) excessive warning logs of attempt to override final parameter: fs.s3.buffer.dir

[ https://issues.apache.org/jira/browse/FLINK-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119873#comment-16119873 ] Steve Loughran commented on FLINK-7365: --- There's a special log you can turn off to s

[jira] [Commented] (FLINK-7266) Don't attempt to delete parent directory on S3

[ https://issues.apache.org/jira/browse/FLINK-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153910#comment-16153910 ] Steve Loughran commented on FLINK-7266: --- FWIW, in s3a we create a single delete requ

[jira] [Commented] (FLINK-7266) Don't attempt to delete parent directory on S3

[ https://issues.apache.org/jira/browse/FLINK-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16155245#comment-16155245 ] Steve Loughran commented on FLINK-7266: --- if you are using s3a then delete(path, recu

[jira] [Commented] (FLINK-7589) org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 159764230; received: 64638536)

[ https://issues.apache.org/jira/browse/FLINK-7589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16156763#comment-16156763 ] Steve Loughran commented on FLINK-7589: --- well, you are allowed to file bug reports.

[jira] [Commented] (FLINK-19550) StreamingFileSink can't be interrupted when writing to S3

[ https://issues.apache.org/jira/browse/FLINK-19550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17243998#comment-17243998 ] Steve Loughran commented on FLINK-19550: s3a in 3.2.2 and 3.3.1 will ship with 1

[jira] [Commented] (FLINK-19595) Flink SQL support S3 select

[ https://issues.apache.org/jira/browse/FLINK-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17244004#comment-17244004 ] Steve Loughran commented on FLINK-19595: s3a connector supports s3 select (HADOO

[jira] [Commented] (FLINK-9061) add entropy to s3 path for better scalability

[ https://issues.apache.org/jira/browse/FLINK-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16553596#comment-16553596 ] Steve Loughran commented on FLINK-9061: --- It'd still be good to conduct some experim

[jira] [Commented] (FLINK-9525) Missing META-INF/services/*FileSystemFactory in flink-hadoop-fs module

[ https://issues.apache.org/jira/browse/FLINK-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16502161#comment-16502161 ] Steve Loughran commented on FLINK-9525: --- we're actually moving Hadoop off that intr

[jira] [Commented] (FLINK-8888) Upgrade AWS SDK in flink-connector-kinesis

[ https://issues.apache.org/jira/browse/FLINK-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392743#comment-16392743 ] Steve Loughran commented on FLINK-: --- If you are pulling in the shaded SDK, note

[jira] [Commented] (FLINK-7589) org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 159764230; received: 64638536)

[ https://issues.apache.org/jira/browse/FLINK-7589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395479#comment-16395479 ] Steve Loughran commented on FLINK-7589: --- We've been thinking of cutting a shaded ver

[jira] [Commented] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16414262#comment-16414262 ] Steve Loughran commented on FLINK-8794: --- it does if you turn s3guard on with Hadoop

[jira] [Commented] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16414272#comment-16414272 ] Steve Loughran commented on FLINK-8794: --- {quote} writing to local disks would decrea

[jira] [Commented] (FLINK-9061) S3 checkpoint data not partitioned well -- causes errors and poor performance

[ https://issues.apache.org/jira/browse/FLINK-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16414280#comment-16414280 ] Steve Loughran commented on FLINK-9061: --- you can get it on delete requests too, if y

[jira] [Commented] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415491#comment-16415491 ] Steve Loughran commented on FLINK-8794: --- That's amazon EMR's problem. Switch to thei

[jira] [Comment Edited] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415491#comment-16415491 ] Steve Loughran edited comment on FLINK-8794 at 3/27/18 12:06 PM: ---

[jira] [Commented] (FLINK-9061) S3 checkpoint data not partitioned well -- causes errors and poor performance

[ https://issues.apache.org/jira/browse/FLINK-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415501#comment-16415501 ] Steve Loughran commented on FLINK-9061: --- [~StephanEwen]: I knew that, but it's still

[jira] [Commented] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417140#comment-16417140 ] Steve Loughran commented on FLINK-8794: --- bq. Enable consistent-view can cause other

[jira] [Commented] (FLINK-9061) S3 checkpoint data not partitioned well -- causes errors and poor performance

[ https://issues.apache.org/jira/browse/FLINK-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417145#comment-16417145 ] Steve Loughran commented on FLINK-9061: --- The s3a connector will have the same issue,

[jira] [Comment Edited] (FLINK-9061) S3 checkpoint data not partitioned well -- causes errors and poor performance

[ https://issues.apache.org/jira/browse/FLINK-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417145#comment-16417145 ] Steve Loughran edited comment on FLINK-9061 at 3/28/18 10:33 AM: ---

[jira] [Commented] (FLINK-9061) S3 checkpoint data not partitioned well -- causes errors and poor performance

[ https://issues.apache.org/jira/browse/FLINK-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417964#comment-16417964 ] Steve Loughran commented on FLINK-9061: --- [~greghogan] I cut the link as it was just

[jira] [Commented] (FLINK-9061) S3 checkpoint data not partitioned well -- causes errors and poor performance

[ https://issues.apache.org/jira/browse/FLINK-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16422645#comment-16422645 ] Steve Loughran commented on FLINK-9061: --- something less than 8, maybe 5, though it's

[jira] [Commented] (FLINK-8801) S3's eventual consistent read-after-write may fail yarn deployment of resources to S3

[ https://issues.apache.org/jira/browse/FLINK-8801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819324#comment-16819324 ] Steve Loughran commented on FLINK-8801: --- we cant set times against objects as they

[jira] [Commented] (FLINK-8801) S3's eventual consistent read-after-write may fail yarn deployment of resources to S3

[ https://issues.apache.org/jira/browse/FLINK-8801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819326#comment-16819326 ] Steve Loughran commented on FLINK-8801: --- * s3a in hadoop 3.1+ lets you see etags as

[jira] [Commented] (FLINK-15814) Log warning when StreamingFileSink is used with an ambiguous S3 scheme and error when used with s3p

[ https://issues.apache.org/jira/browse/FLINK-15814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17045839#comment-17045839 ] Steve Loughran commented on FLINK-15814: Out of curiosity, what is the problem?

[jira] [Commented] (FLINK-8543) Output Stream closed at org.apache.hadoop.fs.s3a.S3AOutputStream.checkOpen

[ https://issues.apache.org/jira/browse/FLINK-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365506#comment-16365506 ] Steve Loughran commented on FLINK-8543: --- bq. I am running on a Hortonworks based had

[jira] [Commented] (FLINK-8543) Output Stream closed at org.apache.hadoop.fs.s3a.S3AOutputStream.checkOpen

[ https://issues.apache.org/jira/browse/FLINK-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365520#comment-16365520 ] Steve Loughran commented on FLINK-8543: --- Created HADOOP-15239 ; I'll take a patch wi

[jira] [Commented] (FLINK-8543) Output Stream closed at org.apache.hadoop.fs.s3a.S3AOutputStream.checkOpen

[ https://issues.apache.org/jira/browse/FLINK-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371976#comment-16371976 ] Steve Loughran commented on FLINK-8543: --- bq. So all files being accompanied by .vali

[jira] [Commented] (FLINK-8543) Output Stream closed at org.apache.hadoop.fs.s3a.S3AOutputStream.checkOpen

[ https://issues.apache.org/jira/browse/FLINK-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371870#comment-16371870 ] Steve Loughran commented on FLINK-8543: --- There's no truncate operation in the S3 pro

[jira] [Commented] (FLINK-8543) Output Stream closed at org.apache.hadoop.fs.s3a.S3AOutputStream.checkOpen

[ https://issues.apache.org/jira/browse/FLINK-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16372701#comment-16372701 ] Steve Loughran commented on FLINK-8543: --- copying a subset of the old file to the new

[jira] [Commented] (FLINK-10817) Upgrade presto dependency to support path-style access

[ https://issues.apache.org/jira/browse/FLINK-10817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679607#comment-16679607 ] Steve Loughran commented on FLINK-10817: If you are working with non-AWS endpoin

[jira] [Commented] (FLINK-32241) UnsupportedFileSystemException when using the ABFS Hadoop driver for checkpointing in Flink 1.17

[ https://issues.apache.org/jira/browse/FLINK-32241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17728393#comment-17728393 ] Steve Loughran commented on FLINK-32241: this is odd as the buffer to disk code

[jira] [Commented] (FLINK-30450) FileSystem supports exporting client-side metrics

[ https://issues.apache.org/jira/browse/FLINK-30450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17649716#comment-17649716 ] Steve Loughran commented on FLINK-30450: if you can grab the IOStatistics from a

[jira] [Commented] (FLINK-19589) Expose S3 options for tagging and object lifecycle policy for FileSystem

[ https://issues.apache.org/jira/browse/FLINK-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259767#comment-17259767 ] Steve Loughran commented on FLINK-19589: the filesystem createFile(Path path) AP

[jira] [Commented] (FLINK-13602) S3 filesystems effectively do not support credential providers

[ https://issues.apache.org/jira/browse/FLINK-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16995629#comment-16995629 ] Steve Loughran commented on FLINK-13602: not much we can do in S3A to help here.

[jira] [Commented] (FLINK-11187) StreamingFileSink with S3 backend transient socket timeout issues

[ https://issues.apache.org/jira/browse/FLINK-11187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16725942#comment-16725942 ] Steve Loughran commented on FLINK-11187: There's limits to how well you can buff

[jira] [Commented] (FLINK-11187) StreamingFileSink with S3 backend transient socket timeout issues

[ https://issues.apache.org/jira/browse/FLINK-11187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16738131#comment-16738131 ] Steve Loughran commented on FLINK-11187: bq. As far as I can tell, this is just

[jira] [Commented] (FLINK-23722) S3 Tests fail on AZP: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.

[ https://issues.apache.org/jira/browse/FLINK-23722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17402184#comment-17402184 ] Steve Loughran commented on FLINK-23722: FWIW, you can fix this by setting fs.s3

[jira] [Commented] (FLINK-26563) HadoopS3RecoverableWriterITCase hang on azure

[ https://issues.apache.org/jira/browse/FLINK-26563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17504275#comment-17504275 ] Steve Loughran commented on FLINK-26563: you could tune the s3a retry params to