Re: Flink operators for Kubernetes

Sounds interesting. Currently Flink can run in standalone mode in a kubernets cluster, details see here.   There is also a JIRA Flink-9953: Active kubernetes integration is target for run Flink on Kubernetes natively. Jin On 10/2/18, 1:30 PM, "Anand Swaminathan" wrote:    

MapState support for operator state

Hi everyone, I would like to ask if there is any conceptual problem with having MapState available for operator states, or is it just a lack of implementation? Best regards, Daniel Berecz

[jira] [Created] (FLINK-10485) Gremlin Support from Flink for Graph Processing

Wei Ding created FLINK-10485: Summary: Gremlin Support from Flink for Graph Processing Key: FLINK-10485 URL: https://issues.apache.org/jira/browse/FLINK-10485 Project: Flink Issue Type: New

[jira] [Created] (FLINK-10484) New latency tracking metrics format causes metrics cardinality explosion

Jamie Grier created FLINK-10484: --- Summary: New latency tracking metrics format causes metrics cardinality explosion Key: FLINK-10484 URL: https://issues.apache.org/jira/browse/FLINK-10484 Project:

[jira] [Created] (FLINK-10483) Can't restore from a savepoint even with Allow Non Restored State enabled

Elias Levy created FLINK-10483: -- Summary: Can't restore from a savepoint even with Allow Non Restored State enabled Key: FLINK-10483 URL: https://issues.apache.org/jira/browse/FLINK-10483 Project: Flink

Flink operators for Kubernetes

Hello All, This is Anand from Lyft. Just wanted to send out a note that we at Lyft are working on building Operators (https://coreos.com/operators/) to support deploying, managing Flink Applications in Kubernetes. The operator is responsible for creating Flink Clusters, start jobs and seamlessly

[jira] [Created] (FLINK-10482) java.lang.IllegalArgumentException: Negative number of in progress checkpoints

Julio Biason created FLINK-10482: Summary: java.lang.IllegalArgumentException: Negative number of in progress checkpoints Key: FLINK-10482 URL: https://issues.apache.org/jira/browse/FLINK-10482

Re: Flink Low level APIs How To’s

Hi Ramesh, Apart from Till recommended, internal classes go with document around the source code define them, and for the "practices" part, maybe you can take a look at their tests. But after all, these are all about Flink dev topics and so unstable to be used in production codes. Best, tison.

Re: Flink Low level APIs How To’s

Hi Ramesh, the JobGraph, JobVertex, JobGraphGenerator, etc. are all internal classes for which the community does not give any stability guarantees. Thus, they can change from one version to another breaking all your programs. Therefore, I would not recommend using these primitives if you are not

[jira] [Created] (FLINK-10481) Wordcount end-to-end test in docker env unstable

Till Rohrmann created FLINK-10481: - Summary: Wordcount end-to-end test in docker env unstable Key: FLINK-10481 URL: https://issues.apache.org/jira/browse/FLINK-10481 Project: Flink Issue

Flink Low level APIs How To’s

Team, would like to create jobs from DAG's with details which is maintained externally(Filters, transform and Stream SQL). Idea is to create JobGraph and submit to Flink. I have started looking into JobGraph, Job, Vertex, JobGenerator from flink low level API's, Looks promising but is there any

Re: [DISCUSS] Move hadoop 2.4 test profiles to cron jobs

+1, sounds good to cut down travis burden. Best, tison. Fabian Hueske 于2018年10月2日周二 下午9:02写道： > +1 > > Am Di., 2. Okt. 2018 um 14:50 Uhr schrieb Till Rohrmann < > trohrm...@apache.org>: > > > Great idea Chesnay. +1 for running Hadoop 2.4 in a cron job. This will > help > > us to cut down our

Re: [DISCUSS] Move hadoop 2.4 test profiles to cron jobs

+1 Am Di., 2. Okt. 2018 um 14:50 Uhr schrieb Till Rohrmann < trohrm...@apache.org>: > Great idea Chesnay. +1 for running Hadoop 2.4 in a cron job. This will help > us to cut down our Travis time by almost 2. > > Cheers, > Till > > On Tue, Oct 2, 2018 at 1:49 PM Chesnay Schepler > wrote: > > >

Re: [DISCUSS] Move hadoop 2.4 test profiles to cron jobs

Great idea Chesnay. +1 for running Hadoop 2.4 in a cron job. This will help us to cut down our Travis time by almost 2. Cheers, Till On Tue, Oct 2, 2018 at 1:49 PM Chesnay Schepler wrote: > Hello, > > As part of FLINK-10453 I would like to move the travis profiles that run > against hadoop 2.4

[DISCUSS] Move hadoop 2.4 test profiles to cron jobs

Hello, As part of FLINK-10453 I would like to move the travis profiles that run against hadoop 2.4 into a cron job. The cron job would be organized as a eparate branch in the apache/flink repository that would be run daily. My reasoning is that most changes made to Flink aren't actually

[jira] [Created] (FLINK-10480) Add Firebase Firestore Connector

Niklas Gögge created FLINK-10480: Summary: Add Firebase Firestore Connector Key: FLINK-10480 URL: https://issues.apache.org/jira/browse/FLINK-10480 Project: Flink Issue Type: New Feature

Re: [DISCUSS] Dropping flink-storm?

+1 for dropping it > On 1. Oct 2018, at 10:55, Fabian Hueske wrote: > > +1 to drop it. > > Thanks, Fabian > > Am Sa., 29. Sep. 2018 um 12:05 Uhr schrieb Niels Basjes : > >> I would drop it. >> >> Niels Basjes >> >> On Sat, 29 Sep 2018, 10:38 Kostas Kloudas, >> wrote: >> >>> +1 to drop it

Re: [DISCUSS] Improvements to the Unified SQL Connector API

Thanks for the proposal! I like the proposed changes a lot, especially support for reading/writing key data of systems that have a key/value split will be very nice to have. > On 2. Oct 2018, at 11:58, Timo Walther wrote: > > Thanks for the feedback Fabian. I updated the document and

[jira] [Created] (FLINK-10479) Add Firebase Firestore Connector

Niklas Gögge created FLINK-10479: Summary: Add Firebase Firestore Connector Key: FLINK-10479 URL: https://issues.apache.org/jira/browse/FLINK-10479 Project: Flink Issue Type: New Feature

Re: [DISCUSS] Improvements to the Unified SQL Connector API

Thanks for the feedback Fabian. I updated the document and addressed your comments. I agree that tables which are stored in different systems need more discussion. I would suggest to deprecate the field mapping interfaces in this release and remove it in the next release. Regards, Timo Am

Re: [DISCUSS] Feature freeze for Flink 1.7

Thanks for the feedback. For FLINK-10122, the proper solution would require the redesigned source interface which will most likely not make it into the release. I think the community is working hard on implementing it as fast as possible, though. Concerning state migration, Gordon is the best to

Running job in detached mode via ClusterClient.

Hey, I've a question regarding submission job in detached mode. We're trying to submit job via YarnClusterClient in detached mode and the first one goes correctly, but when we submit second job we get exception: Multiple environments cannot be created in detached mode at

Re: [DISCUSS] Improvements to the Unified SQL Connector API

Thanks for the proposal Timo! I've done a pass and added some comments (mostly asking for clarification, details). Overall, this is going into a very good direction. I think the tables which are stored in different systems and using a format definition to define other formats require some more

[jira] [Created] (FLINK-10478) Kafka Producer wrongly formats "%" for transaction ID

Obi Tetsuya created FLINK-10478: --- Summary: Kafka Producer wrongly formats "%" for transaction ID Key: FLINK-10478 URL: https://issues.apache.org/jira/browse/FLINK-10478 Project: Flink Issue

[jira] [Created] (FLINK-10477) Add rescale button to web interface

Sander Ploegsma created FLINK-10477: --- Summary: Add rescale button to web interface Key: FLINK-10477 URL: https://issues.apache.org/jira/browse/FLINK-10477 Project: Flink Issue Type: