[jira] [Commented] (FLINK-8500) Get the timestamp of the Kafka message from kafka consumer(Kafka010Fetcher)

[ https://issues.apache.org/jira/browse/FLINK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488779#comment-16488779 ] yanxiaobin commented on FLINK-8500: --- I prefer to  to expose Kafka's ConsumerRecord

[jira] [Commented] (FLINK-8500) Get the timestamp of the Kafka message from kafka consumer(Kafka010Fetcher)

[ https://issues.apache.org/jira/browse/FLINK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496006#comment-16496006 ] yanxiaobin commented on FLINK-8500: --- So far, it can solve the current problems, but in the long run,

[jira] [Comment Edited] (FLINK-8500) Get the timestamp of the Kafka message from kafka consumer(Kafka010Fetcher)

[ https://issues.apache.org/jira/browse/FLINK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496136#comment-16496136 ] yanxiaobin edited comment on FLINK-8500 at 5/31/18 5:42 AM: [~tzulitai]   

[jira] [Commented] (FLINK-8500) Get the timestamp of the Kafka message from kafka consumer(Kafka010Fetcher)

[ https://issues.apache.org/jira/browse/FLINK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496136#comment-16496136 ] yanxiaobin commented on FLINK-8500: --- [~tzulitai]   thanks, greate.  > Get the timestamp of the Kafka

[jira] [Commented] (FLINK-8500) Get the timestamp of the Kafka message from kafka consumer(Kafka010Fetcher)

[ https://issues.apache.org/jira/browse/FLINK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338958#comment-16338958 ] yanxiaobin commented on FLINK-8500: --- I mean that I want to use Kafka message timestamp as event time! 

[jira] [Updated] (FLINK-8500) Get the timestamp of the Kafka message from kafka consumer(Kafka010Fetcher)

[ https://issues.apache.org/jira/browse/FLINK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yanxiaobin updated FLINK-8500: -- Attachment: image-2018-01-31-10-48-59-633.png > Get the timestamp of the Kafka message from kafka

[jira] [Commented] (FLINK-8500) Get the timestamp of the Kafka message from kafka consumer(Kafka010Fetcher)

[ https://issues.apache.org/jira/browse/FLINK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346180#comment-16346180 ] yanxiaobin commented on FLINK-8500: --- hi,[~aljoscha] , thank you for your reply!Please look at the next

[jira] [Comment Edited] (FLINK-8500) Get the timestamp of the Kafka message from kafka consumer(Kafka010Fetcher)

[ https://issues.apache.org/jira/browse/FLINK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346180#comment-16346180 ] yanxiaobin edited comment on FLINK-8500 at 1/31/18 3:01 AM: hi,[~aljoscha] ,

[jira] [Comment Edited] (FLINK-8500) Get the timestamp of the Kafka message from kafka consumer(Kafka010Fetcher)

[ https://issues.apache.org/jira/browse/FLINK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346180#comment-16346180 ] yanxiaobin edited comment on FLINK-8500 at 1/31/18 3:09 AM: hi,[~aljoscha] ,

[jira] [Commented] (FLINK-5479) Per-partition watermarks in FlinkKafkaConsumer should consider idle partitions

[ https://issues.apache.org/jira/browse/FLINK-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343115#comment-16343115 ] yanxiaobin commented on FLINK-5479: --- I've also met this problem at the moment. > Per-partition

[jira] [Commented] (FLINK-8500) Get the timestamp of the Kafka message from kafka consumer(Kafka010Fetcher)

[ https://issues.apache.org/jira/browse/FLINK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347951#comment-16347951 ] yanxiaobin commented on FLINK-8500: --- Greate！I think it's reasonable.But I think it's best to optimize

[jira] [Comment Edited] (FLINK-8500) Get the timestamp of the Kafka message from kafka consumer(Kafka010Fetcher)

[ https://issues.apache.org/jira/browse/FLINK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347951#comment-16347951 ] yanxiaobin edited comment on FLINK-8500 at 2/1/18 3:28 AM: --- Greate！I think it's

[jira] [Commented] (FLINK-8500) Get the timestamp of the Kafka message from kafka consumer(Kafka010Fetcher)

[ https://issues.apache.org/jira/browse/FLINK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346550#comment-16346550 ] yanxiaobin commented on FLINK-8500: --- hi, [~aljoscha] .  What do you think? > Get the timestamp of the

[jira] [Comment Edited] (FLINK-5479) Per-partition watermarks in FlinkKafkaConsumer should consider idle partitions

[ https://issues.apache.org/jira/browse/FLINK-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343115#comment-16343115 ] yanxiaobin edited comment on FLINK-5479 at 2/1/18 10:30 AM: I've also met this

[jira] [Updated] (FLINK-8500) Get the timestamp of the Kafka message from kafka consumer(Kafka010Fetcher)

[ https://issues.apache.org/jira/browse/FLINK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yanxiaobin updated FLINK-8500: -- Attachment: image-2018-01-30-14-58-58-167.png > Get the timestamp of the Kafka message from kafka

[jira] [Commented] (FLINK-8500) Get the timestamp of the Kafka message from kafka consumer(Kafka010Fetcher)

[ https://issues.apache.org/jira/browse/FLINK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344591#comment-16344591 ] yanxiaobin commented on FLINK-8500: --- As shown below（Kafka010Fetcher）：

[jira] [Commented] (FLINK-8500) Get the timestamp of the Kafka message from kafka consumer(Kafka010Fetcher)

[ https://issues.apache.org/jira/browse/FLINK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376428#comment-16376428 ] yanxiaobin commented on FLINK-8500: --- hi, [~aljoscha] ! I can solve that problem like that，but I still

[jira] [Created] (FLINK-8321) Generate _SUCCESS (map-reduce style) when folder has been written！

yanxiaobin created FLINK-8321: - Summary: Generate _SUCCESS (map-reduce style) when folder has been written！ Key: FLINK-8321 URL: https://issues.apache.org/jira/browse/FLINK-8321 Project: Flink

[jira] [Updated] (FLINK-8321) Generate _SUCCESS (map-reduce style) when folder has been written！

[ https://issues.apache.org/jira/browse/FLINK-8321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yanxiaobin updated FLINK-8321: -- Description: There is no success file generation, and the downstream processing can't determine when

[jira] [Created] (FLINK-8500) Get the timestamp of the Kafka message from kafka consumer(Kafka010Fetcher)

yanxiaobin created FLINK-8500: - Summary: Get the timestamp of the Kafka message from kafka consumer(Kafka010Fetcher) Key: FLINK-8500 URL: https://issues.apache.org/jira/browse/FLINK-8500 Project: Flink

[jira] [Comment Edited] (FLINK-8500) Get the timestamp of the Kafka message from kafka consumer(Kafka010Fetcher)

[ https://issues.apache.org/jira/browse/FLINK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390914#comment-16390914 ] yanxiaobin edited comment on FLINK-8500 at 3/8/18 8:47 AM: --- Sorry to reply so

[jira] [Commented] (FLINK-8500) Get the timestamp of the Kafka message from kafka consumer(Kafka010Fetcher)

[ https://issues.apache.org/jira/browse/FLINK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390914#comment-16390914 ] yanxiaobin commented on FLINK-8500: --- Sorry to reply so late. I see. We really should be careful here. I

[jira] [Commented] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395286#comment-16395286 ] yanxiaobin commented on FLINK-8794: --- I will open a case with AWS for the root cause of the problem. >

[jira] [Commented] (FLINK-8939) Provide better support for saving streaming data to s3

[ https://issues.apache.org/jira/browse/FLINK-8939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398211#comment-16398211 ] yanxiaobin commented on FLINK-8939: --- I am also using BucketingSink API for saving streaming data to s3. 

[jira] [Comment Edited] (FLINK-8939) Provide better support for saving streaming data to s3

[ https://issues.apache.org/jira/browse/FLINK-8939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398211#comment-16398211 ] yanxiaobin edited comment on FLINK-8939 at 3/14/18 8:07 AM: I am also using

[jira] [Updated] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yanxiaobin updated FLINK-8794: -- Issue Type: Improvement (was: Bug) > When using BucketingSink, it happens that one of the files is

[jira] [Commented] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393956#comment-16393956 ] yanxiaobin commented on FLINK-8794: --- About : 1.What I described above is that there will be such a

[jira] [Comment Edited] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393956#comment-16393956 ] yanxiaobin edited comment on FLINK-8794 at 3/10/18 1:53 AM: About : 1.What I

[jira] [Commented] (FLINK-8500) Get the timestamp of the Kafka message from kafka consumer(Kafka010Fetcher)

[ https://issues.apache.org/jira/browse/FLINK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451808#comment-16451808 ] yanxiaobin commented on FLINK-8500: --- hi [~aljoscha] ：What is your idea? I now solve the problem by

[jira] [Commented] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415131#comment-16415131 ] yanxiaobin commented on FLINK-8794: --- The underlying implementation is

[jira] [Commented] (FLINK-8939) Provide better support for saving streaming data to s3

[ https://issues.apache.org/jira/browse/FLINK-8939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415112#comment-16415112 ] yanxiaobin commented on FLINK-8939: --- I'm using AWS S3. This problem has been plaguing me all the time.

[jira] [Commented] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416734#comment-16416734 ] yanxiaobin commented on FLINK-8794: --- thanks! indeed so.  Enable consistent-view can cause other

[jira] [Commented] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417174#comment-16417174 ] yanxiaobin commented on FLINK-8794: --- Other problems have nothing to do with Flink. > When using

[jira] [Commented] (FLINK-8939) Provide better support for saving streaming data to s3

[ https://issues.apache.org/jira/browse/FLINK-8939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16413319#comment-16413319 ] yanxiaobin commented on FLINK-8939: --- thanks! Because S3 follows the *eventually consistent* principle.

[jira] [Commented] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16404584#comment-16404584 ] yanxiaobin commented on FLINK-8794: --- S3 follows the *eventually consistent* principle, so this problem

[jira] [Comment Edited] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384954#comment-16384954 ] yanxiaobin edited comment on FLINK-8794 at 3/4/18 4:13 AM: --- yes. If we allow for

[jira] [Commented] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384954#comment-16384954 ] yanxiaobin commented on FLINK-8794: --- yes. If we allow for different directory that should be already

[jira] [Updated] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yanxiaobin updated FLINK-8794: -- Description: When using BucketingSink, it happens that one of the files is always in the [.in-progress]

[jira] [Created] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

yanxiaobin created FLINK-8794: - Summary: When using BucketingSink, it happens that one of the files is always in the [.in-progress] state Key: FLINK-8794 URL: https://issues.apache.org/jira/browse/FLINK-8794

[jira] [Comment Edited] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383356#comment-16383356 ] yanxiaobin edited comment on FLINK-8794 at 3/2/18 8:40 AM: --- hi, [~pnowojski]

[jira] [Commented] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383356#comment-16383356 ] yanxiaobin commented on FLINK-8794: --- hi, [~pnowojski] ! Thank you for your suggestion！ The downstream

[jira] [Commented] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383209#comment-16383209 ] yanxiaobin commented on FLINK-8794: --- hi, [~aljoscha] Thank you for your reply！   There are the following

[jira] [Updated] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yanxiaobin updated FLINK-8794: -- Description: When using BucketingSink, it happens that one of the files is always in the

[jira] [Updated] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

[ https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yanxiaobin updated FLINK-8794: -- Description: When using BucketingSink, it happens that one of the files is always in the

[jira] [Commented] (FLINK-8500) Get the timestamp of the Kafka message from kafka consumer

[ https://issues.apache.org/jira/browse/FLINK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749861#comment-16749861 ] yanxiaobin commented on FLINK-8500: --- I have changed the title. > Get the timestamp of the Kafka

[jira] [Updated] (FLINK-8500) Get the timestamp of the Kafka message from kafka consumer

[ https://issues.apache.org/jira/browse/FLINK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yanxiaobin updated FLINK-8500: -- Summary: Get the timestamp of the Kafka message from kafka consumer (was: Get the timestamp of the

[jira] [Commented] (FLINK-12992) All host(s) tried for query failed (tried com.datastax.driver.core.exceptions.TransportException: Error writing...)

[ https://issues.apache.org/jira/browse/FLINK-12992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876018#comment-16876018 ] yanxiaobin commented on FLINK-12992: At present, I have upgraded the netty version of the datastax

[jira] [Created] (FLINK-12992) All host(s) tried for query failed (tried com.datastax.driver.core.exceptions.TransportException: Error writing...)

yanxiaobin created FLINK-12992: -- Summary: All host(s) tried for query failed (tried com.datastax.driver.core.exceptions.TransportException: Error writing...) Key: FLINK-12992 URL:

[jira] [Updated] (FLINK-16791) Could not deploy Yarn job cluster when using flink-s3-fs-hadoop-1.10.0.jar

[ https://issues.apache.org/jira/browse/FLINK-16791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yanxiaobin updated FLINK-16791: --- Description: Since aws s3 is needed, so copy the flink-s3-fs-hadoop-1.10.0.jar JAR file to the 

[jira] [Commented] (FLINK-16792) flink-s3-fs-hadoop cannot access s3

[ https://issues.apache.org/jira/browse/FLINK-16792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067389#comment-17067389 ] yanxiaobin commented on FLINK-16792: The issue is related to

[jira] [Created] (FLINK-16792) flink-s3-fs-hadoop cannot access s3

yanxiaobin created FLINK-16792: -- Summary: flink-s3-fs-hadoop cannot access s3 Key: FLINK-16792 URL: https://issues.apache.org/jira/browse/FLINK-16792 Project: Flink Issue Type: Bug

[jira] [Commented] (FLINK-16791) Could not deploy Yarn job cluster when using flink-s3-fs-hadoop-1.10.0.jar

[ https://issues.apache.org/jira/browse/FLINK-16791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067380#comment-17067380 ] yanxiaobin commented on FLINK-16791: I checked that the application does not contain any hadoop

[jira] [Updated] (FLINK-16792) flink-s3-fs-hadoop cannot access s3

[ https://issues.apache.org/jira/browse/FLINK-16792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yanxiaobin updated FLINK-16792: --- Description: When flink-s3-fs-hadoop-1.10.0.jar is placed in the lib directory of Flink

[jira] [Created] (FLINK-16791) Could not deploy Yarn job cluster when using flink-s3-fs-hadoop-1.10.0.jar

yanxiaobin created FLINK-16791: -- Summary: Could not deploy Yarn job cluster when using flink-s3-fs-hadoop-1.10.0.jar Key: FLINK-16791 URL: https://issues.apache.org/jira/browse/FLINK-16791 Project:

[jira] [Commented] (FLINK-16791) Could not deploy Yarn job cluster when using flink-s3-fs-hadoop-1.10.0.jar

[ https://issues.apache.org/jira/browse/FLINK-16791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068293#comment-17068293 ] yanxiaobin commented on FLINK-16791: Hi,  thanks everyone. Just as [~kkl0u] said, It works to use

[jira] [Commented] (FLINK-16791) Could not deploy Yarn job cluster when using flink-s3-fs-hadoop-1.10.0.jar

[ https://issues.apache.org/jira/browse/FLINK-16791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071439#comment-17071439 ] yanxiaobin commented on FLINK-16791: ok, thanks. > Could not deploy Yarn job cluster when using

[jira] [Commented] (FLINK-16791) Could not deploy Yarn job cluster when using flink-s3-fs-hadoop-1.10.0.jar

[ https://issues.apache.org/jira/browse/FLINK-16791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067576#comment-17067576 ] yanxiaobin commented on FLINK-16791: Hi Yang Wang, thanks. I have already put flink-shaded-hadoop

[jira] [Commented] (FLINK-16791) Could not deploy Yarn job cluster when using flink-s3-fs-hadoop-1.10.0.jar

[ https://issues.apache.org/jira/browse/FLINK-16791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067618#comment-17067618 ] yanxiaobin commented on FLINK-16791: It does't work for now. The unshaded