Re: Flink solution to active - active Multi site cloud data ingestion

2019-05-15 Thread Seth Wiesman
In the future, these kinds of questions are more appropriate for the user mailing list (u...@flink.apache.org). Dev is for internal Flink development. On Wed, May 15, 2019 at 12:06 PM Seth Wiesman wrote: > Hi Gregory, > > The easiest solution would be to include the site in your key so that at

Re: User Interface not showing the actual count received and produced

2019-05-15 Thread PoolakkalMukkath, Shakir
Hi Flink team, I am developing a flow which uses * FlinkKafkaConsumer010 to consume message from Kafka and * FlinkKinesisProducer to produce the results to Kinesis. In the user interface, I always see Bytes and Record received from Kafka is zero even though it is receiving events

Re: Flink solution to active - active Multi site cloud data ingestion

2019-05-15 Thread Seth Wiesman
Hi Gregory, The easiest solution would be to include the site in your key so that at query time the rows from each site can be aggregated together. Instead of , the table would be and your query would become Select sum(value) FROM table GROUP BY key; Otherwise, you will need to get all that

Re: User Interface not showing the actual count received and produced

2019-05-15 Thread PoolakkalMukkath, Shakir
Hi Flink team, I am developing a flow which uses * FlinkKafkaConsumer010 to consume message from Kafka and * FlinkKinesisProducer to produce the results to Kinesis. In the user interface, I always see Bytes and Record received from Kafka is zero even though it is receiving

[jira] [Created] (FLINK-12522) Error While Initializing S3A FileSystem

2019-05-15 Thread Manish Bellani (JIRA)
Manish Bellani created FLINK-12522: -- Summary: Error While Initializing S3A FileSystem Key: FLINK-12522 URL: https://issues.apache.org/jira/browse/FLINK-12522 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-12523) "Kafka 0.11 end-to-end test" failed because it cannot find test-scripts

2019-05-15 Thread Bowen Li (JIRA)
Bowen Li created FLINK-12523: Summary: "Kafka 0.11 end-to-end test" failed because it cannot find test-scripts Key: FLINK-12523 URL: https://issues.apache.org/jira/browse/FLINK-12523 Project: Flink

Re: [DISCUSS] Clean up and reorganize the JIRA components

2019-05-15 Thread JingsongLee
Big +1 to add benchmark component. 1.Many of our code changes now require benchmark. Having a benchmark component makes it much easier for us to align. 2.Running benchmark regularly can also prevent performance degradation caused by our code. Best, JingsongLee

Re: [Discuss]: Adding Metrics to StreamingFileSink

2019-05-15 Thread Thomas Weise
+1 to both suggestions It should be possible to extend the connector (we run into the same issues with KinesisConsumer). Metrics are essential to understand the performance, especially for things like S3 writes, error, retries, memory buffers and so on. Thomas On 2019/05/15 07:43:39, Kailash

[jira] [Created] (FLINK-12524) Introduce planner rules about rank

2019-05-15 Thread godfrey he (JIRA)
godfrey he created FLINK-12524: -- Summary: Introduce planner rules about rank Key: FLINK-12524 URL: https://issues.apache.org/jira/browse/FLINK-12524 Project: Flink Issue Type: New Feature

Apply for a Contributor permission

2019-05-15 Thread Zhou, Brian
Hi, I want to contribute to Apache Flink, starting from Chinese translation. Would you please give me the contributor permission? The specific JIRA FLINK-11560: Translate "Flink Applications" page into Chinese. Best Regards, Brian

[jira] [Created] (FLINK-12525) Impose invariant StreamExecutionEnvironment.setBufferTimeout > 0

2019-05-15 Thread Robert Stoll (JIRA)
Robert Stoll created FLINK-12525: Summary: Impose invariant StreamExecutionEnvironment.setBufferTimeout > 0 Key: FLINK-12525 URL: https://issues.apache.org/jira/browse/FLINK-12525 Project: Flink

[Discuss]: Adding Metrics to StreamingFileSink

2019-05-15 Thread Kailash Dayanand
Hello, I was looking to add metrics to the streaming file sink. Currently the only details available is the generic information about for any operator like the number of records in, number of records out etc. I was looking at adding some metrics and contributing back as well as enabling the

Re: [DISCUSS] Improve Queryable State and introduce a QueryServerProxy component

2019-05-15 Thread Aljoscha Krettek
Hi Everyone, I think this is a good discussion and valuable ideas have come up. However, it seems none of the committers and/or PMCs currently have time to work on this subject. Till, who’s focusing on the distributed runtime side, which is touched quite a bit by queryable state, is currently

[jira] [Created] (FLINK-12513) Improve end-to-end (bash based) tests

2019-05-15 Thread Alex (JIRA)
Alex created FLINK-12513: Summary: Improve end-to-end (bash based) tests Key: FLINK-12513 URL: https://issues.apache.org/jira/browse/FLINK-12513 Project: Flink Issue Type: Improvement

Flink solution to active - active Multi site cloud data ingestion

2019-05-15 Thread Melekh, Gregory
Hello Flink Experts. We have Flink job consuming data from Kafka and ingest it to multi-site (Azure-east – Azure-west) replicated Cassandra. Now we have to aggregate data hourly. The problem is that device X can report once on site A and once on site B. This means that some messages for that

[jira] [Created] (FLINK-12514) Refactor the failure checkpoint counting mechanism with ordered checkpoint id

2019-05-15 Thread vinoyang (JIRA)
vinoyang created FLINK-12514: Summary: Refactor the failure checkpoint counting mechanism with ordered checkpoint id Key: FLINK-12514 URL: https://issues.apache.org/jira/browse/FLINK-12514 Project: Flink

Re: [DISCUSS] Improve Queryable State and introduce a QueryServerProxy component

2019-05-15 Thread vino yang
Hi Aljoscha, Thanks for agreeing this is a valuable idea. I know the committers and PMCs are busy before Flink 1.9 even 2.0. We think it's a good improvement for queryable state and even some interactive query scenarios. I don't mind if the timeline will be pulled long. Although, it could not be

Re: [DISCUSS] Clean up and reorganize the JIRA components

2019-05-15 Thread Piotr Nowojski
Hi, I would like to propose two changes: 1. Renaming “Runtime / Operators” to “Runtime / Task” or something like “Runtime / Processing”. “Runtime / Operators” was confusing me, since it sounded like it covers concrete implementations of the operators, like “WindowOperator” or various join

[jira] [Created] (FLINK-12515) Sync flink-shaded-hadoop with dependency management entries

2019-05-15 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-12515: Summary: Sync flink-shaded-hadoop with dependency management entries Key: FLINK-12515 URL: https://issues.apache.org/jira/browse/FLINK-12515 Project: Flink

Re: [VOTE] Release flink-shaded 7.0, release candidate 1

2019-05-15 Thread Chesnay Schepler
I've opened a JIRA: https://issues.apache.org/jira/browse/FLINK-12515 On 13/05/2019 12:03, jincheng sun wrote: Thanks for your vote! @Chesnay Schepler ! When moving the modules, we copy the dependency version of the dependencyManagement in the `flink-parent` pom in the specific sub-module

[jira] [Created] (FLINK-12521) Use minio as AWS S3 alternative in end-to-end tests

2019-05-15 Thread Alex (JIRA)
Alex created FLINK-12521: Summary: Use minio as AWS S3 alternative in end-to-end tests Key: FLINK-12521 URL: https://issues.apache.org/jira/browse/FLINK-12521 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-12520) Support to provide fully-qualified domain host name in TaskManagerMetricGroup

2019-05-15 Thread Yun Tang (JIRA)
Yun Tang created FLINK-12520: Summary: Support to provide fully-qualified domain host name in TaskManagerMetricGroup Key: FLINK-12520 URL: https://issues.apache.org/jira/browse/FLINK-12520 Project: Flink

[jira] [Created] (FLINK-12517) Run network tests with dynamically-linked openSSL

2019-05-15 Thread Nico Kruber (JIRA)
Nico Kruber created FLINK-12517: --- Summary: Run network tests with dynamically-linked openSSL Key: FLINK-12517 URL: https://issues.apache.org/jira/browse/FLINK-12517 Project: Flink Issue Type:

[jira] [Created] (FLINK-12518) Run e2e tests with openSSL

2019-05-15 Thread Nico Kruber (JIRA)
Nico Kruber created FLINK-12518: --- Summary: Run e2e tests with openSSL Key: FLINK-12518 URL: https://issues.apache.org/jira/browse/FLINK-12518 Project: Flink Issue Type: Sub-task

Re: [VOTE] Release flink-shaded 7.0, release candidate 1

2019-05-15 Thread jincheng sun
Thanks a lot for open the JIRA @Chesnay Schepler , I will try to fix it ASAP. :) Chesnay Schepler 于2019年5月15日周三 下午5:40写道: > I've opened a JIRA: https://issues.apache.org/jira/browse/FLINK-12515 > > On 13/05/2019 12:03, jincheng sun wrote: > > Thanks for your vote! @Chesnay Schepler ! > > > >

[jira] [Created] (FLINK-12516) Update Travis base image from trusty to xenial

2019-05-15 Thread Nico Kruber (JIRA)
Nico Kruber created FLINK-12516: --- Summary: Update Travis base image from trusty to xenial Key: FLINK-12516 URL: https://issues.apache.org/jira/browse/FLINK-12516 Project: Flink Issue Type: Bug

Re: [DISCUSS] Clean up and reorganize the JIRA components

2019-05-15 Thread Kurt Young
+1 to add benchmark component. Best, Kurt On Wed, May 15, 2019 at 6:13 PM Piotr Nowojski wrote: > Hi, > > I would like to propose two changes: > > 1. Renaming “Runtime / Operators” to “Runtime / Task” or something like > “Runtime / Processing”. “Runtime / Operators” was confusing me, since it

[jira] [Created] (FLINK-12519) Introduce planner rules about semi/anti join

2019-05-15 Thread godfrey he (JIRA)
godfrey he created FLINK-12519: -- Summary: Introduce planner rules about semi/anti join Key: FLINK-12519 URL: https://issues.apache.org/jira/browse/FLINK-12519 Project: Flink Issue Type: New