Github user zhangxinyu1 commented on the issue:
https://github.com/apache/flink/pull/6108
@kl0u Great! I look forward for it.
About the bandwidth limitation, we hope jobs can read resource below the
speed x bytes/sec.
---
Github user kl0u commented on the issue:
https://github.com/apache/flink/pull/6108
@zhangxinyu1 as soon as this sink is ready, I believe that the existing
File Source will be able to read the output of the Bucketing Sink. As far as
bandwidth limitations are concerned, could you
Github user zhangxinyu1 commented on the issue:
https://github.com/apache/flink/pull/6108
@kl0u Thanks. Would you please consider to implement a BucketingSource
which we can use it to read data from FileSystems? Besides we also care about
the limit of bandwidth.
---
Github user kl0u commented on the issue:
https://github.com/apache/flink/pull/6108
Thanks for the useful input here @zhangxinyu1 and @StephanEwen. As soon as
I have sth concrete I create the JIRA and post it here.
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6108
@kl0u please link the issue once you created it.
This is currently very early, in design discussions between @kl0u, me, and
@aljoscha.
The main points about the rewrite are
-
Github user zhangxinyu1 commented on the issue:
https://github.com/apache/flink/pull/6108
@StephanEwen Thanks for your reply. BTW, is their any issue about
BucketingSink rewriting? We also want to use the BucketingSink which supports
for parquet and orc.
---
Github user zhangxinyu1 commented on the issue:
https://github.com/apache/flink/pull/6108
@StephanEwen Thanks your reply. Is their any issue about BuckingSink
rewriting? We also want to use the BuckingSink which support for parquet and
orc.
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/6108
Do you have a Hadoop version older than 2.7?
We are currently attempting to rewrite the Bucketing Sink completely for
better compatibility with S3 and with better support for Parquet /
Github user zhangxinyu1 commented on the issue:
https://github.com/apache/flink/pull/6108
@kl0u @joshfg @StephanEwen Could you please take a look at this pr?
---
