Re: [DISCUSS] [Contributing] (2) - Review Steps

2018-10-08 Thread Peter Huang
+1 On Mon, Oct 8, 2018 at 7:47 PM Thomas Weise wrote: > +1 > > > On Mon, Oct 8, 2018 at 7:36 PM Tzu-Li Chen wrote: > > > +1 > > > > Jin Sun 于2018年10月9日周二 上午2:10写道: > > > > > +1, look forward to see the change. > > > > > > > On Oct 9, 2018, at 12:07 AM, Fabian Hueske > wrote: > > > > > > > >

Re: [DISCUSS] [Contributing] (2) - Review Steps

2018-10-08 Thread Thomas Weise
+1 On Mon, Oct 8, 2018 at 7:36 PM Tzu-Li Chen wrote: > +1 > > Jin Sun 于2018年10月9日周二 上午2:10写道: > > > +1, look forward to see the change. > > > > > On Oct 9, 2018, at 12:07 AM, Fabian Hueske wrote: > > > > > > Hi everyone, > > > > > > Since we have addressed all comments (please raise your

Re: [DISCUSS] [Contributing] (2) - Review Steps

2018-10-08 Thread Tzu-Li Chen
+1 Jin Sun 于2018年10月9日周二 上午2:10写道: > +1, look forward to see the change. > > > On Oct 9, 2018, at 12:07 AM, Fabian Hueske wrote: > > > > Hi everyone, > > > > Since we have addressed all comments (please raise your voice if not!), I > > would like to move forward and convert the proposal [1]

Re: Sharing state between subtasks

2018-10-08 Thread Elias Levy
Kafka Streams handles this problem, time alignment, by processing records from the partitions with the lowest timestamp in a best effort basis. See KIP-353 for the details. The same could be done within the Kafka source and multiple input stream operators. I opened FLINK-4558

Re: Sharing state between subtasks

2018-10-08 Thread Jamie Grier
I'll add to what Thomas already said.. The larger issue driving this is that when reading from a source with many parallel partitions, especially when reading lots of historical data (or recovering from downtime and there is a backlog to read), it's quite common for there to develop an event-time

Re: Request for Contributor permissions

2018-10-08 Thread Fabian Hueske
Hi Rafi, Welcome to the Flink community! I gave you contributor permissions for JIRA. Best, Fabian Am Mo., 8. Okt. 2018 um 21:18 Uhr schrieb Rafi Aroch : > Hi, > > I would like to assign an issue to myself. Could someone assign contributor > permissions to my user? > My username in JIRA is:

Request for Contributor permissions

2018-10-08 Thread Rafi Aroch
Hi, I would like to assign an issue to myself. Could someone assign contributor permissions to my user? My username in JIRA is: *aroch*. Looking forward to contribute. Thanks, Rafi

Re: [DISCUSS] [Contributing] (2) - Review Steps

2018-10-08 Thread Jin Sun
+1, look forward to see the change. > On Oct 9, 2018, at 12:07 AM, Fabian Hueske wrote: > > Hi everyone, > > Since we have addressed all comments (please raise your voice if not!), I > would like to move forward and convert the proposal [1] into a page for > Flink's website [2]. > I will

[jira] [Created] (FLINK-10515) Improve tokenisation of program args passed as string

2018-10-08 Thread Andrey Zagrebin (JIRA)
Andrey Zagrebin created FLINK-10515: --- Summary: Improve tokenisation of program args passed as string Key: FLINK-10515 URL: https://issues.apache.org/jira/browse/FLINK-10515 Project: Flink

Re: [DISCUSS] [Contributing] (2) - Review Steps

2018-10-08 Thread Fabian Hueske
Hi everyone, Since we have addressed all comments (please raise your voice if not!), I would like to move forward and convert the proposal [1] into a page for Flink's website [2]. I will create a pull request against the website repo [3]. Once the page got merged, we can start posting the review

Re: [DISCUSS] Breaking the Scala API for Scala 2.12 Support

2018-10-08 Thread Aljoscha Krettek
Yes, but I think we would pretty much have to do that. I don't think we can stop doing 2.11 releases. > On 8. Oct 2018, at 15:37, Chesnay Schepler wrote: > > The infrastructure would only be required if we opt for releasing 2.11 and > 2.12 builds simultaneously, correct? > > On 08.10.2018

[jira] [Created] (FLINK-10514) change Tychyon to Alluxio

2018-10-08 Thread yangxiaoshuo (JIRA)
yangxiaoshuo created FLINK-10514: Summary: change Tychyon to Alluxio Key: FLINK-10514 URL: https://issues.apache.org/jira/browse/FLINK-10514 Project: Flink Issue Type: Improvement

Re: [DISCUSS] Breaking the Scala API for Scala 2.12 Support

2018-10-08 Thread Chesnay Schepler
The infrastructure would only be required if we opt for releasing 2.11 and 2.12 builds simultaneously, correct? On 08.10.2018 15:04, Aljoscha Krettek wrote: Breaking the API (or not breaking it but requiring explicit types when using Scala 2.12) and the Maven infrastructure to actually build

Re: [DISCUSS] Breaking the Scala API for Scala 2.12 Support

2018-10-08 Thread Aljoscha Krettek
Breaking the API (or not breaking it but requiring explicit types when using Scala 2.12) and the Maven infrastructure to actually build a 2.12 release. > On 8. Oct 2018, at 13:00, Chesnay Schepler wrote: > > And the remaining parts would only be about breaking the API? > > On 08.10.2018

Re: [DISCUSS] Breaking the Scala API for Scala 2.12 Support

2018-10-08 Thread Chesnay Schepler
And the remaining parts would only be about breaking the API? On 08.10.2018 12:24, Aljoscha Krettek wrote: I have an open PR that does everything we can do for preparing the code base for Scala 2.12 without breaking the API: https://github.com/apache/flink/pull/6784 On 8. Oct 2018, at

[jira] [Created] (FLINK-10513) Replace TaskManagerActions#notifyFinalState with #updateTaskExecutionState

2018-10-08 Thread Till Rohrmann (JIRA)
Till Rohrmann created FLINK-10513: - Summary: Replace TaskManagerActions#notifyFinalState with #updateTaskExecutionState Key: FLINK-10513 URL: https://issues.apache.org/jira/browse/FLINK-10513

Re: [DISCUSS] Breaking the Scala API for Scala 2.12 Support

2018-10-08 Thread Aljoscha Krettek
I have an open PR that does everything we can do for preparing the code base for Scala 2.12 without breaking the API: https://github.com/apache/flink/pull/6784 > On 8. Oct 2018, at 09:56, Chesnay Schepler wrote: > > I'd rather not maintain 2 master branches. Beyond the maintenance overhead

[jira] [Created] (FLINK-10512) Remove legacy REST API docs

2018-10-08 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-10512: Summary: Remove legacy REST API docs Key: FLINK-10512 URL: https://issues.apache.org/jira/browse/FLINK-10512 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] Dropping flink-storm?

2018-10-08 Thread Till Rohrmann
Good point. The initial idea of this thread was to remove the storm compatibility layer completely. During the discussion I realized that it might be useful for our users to not completely remove it in one go. Instead for those who still want to use some Bolt and Spout code in Flink, it could be

[jira] [Created] (FLINK-10511) Code duplication of creating RPC service in ClusterEntrypoint and AkkaRpcServiceUtils

2018-10-08 Thread Shimin Yang (JIRA)
Shimin Yang created FLINK-10511: --- Summary: Code duplication of creating RPC service in ClusterEntrypoint and AkkaRpcServiceUtils Key: FLINK-10511 URL: https://issues.apache.org/jira/browse/FLINK-10511

Re: [DISCUSS] Dropping flink-storm?

2018-10-08 Thread Chesnay Schepler
I don't believe that to be the consensus. For starters it is contradictory; we can't /drop /flink-storm yet still /keep //some parts/. From my understanding we drop flink-storm completely, and put a note in the docs that the bolt/spout wrappers of previous versions will continue to work. On

Re: Enrich testing doc with more unit test examples using AbstractStreamOperator

2018-10-08 Thread Matthieu Bonneviot
Hi I am a new flink developer and I also feel we need to improve the testing side of flink. >From Tony list: 1-stream logic control: we may consider reactiveX as a reference for stream testing: http://reactivex.io/. They have a test scheduler and use marble diagram to write unit test: for

Re: [DISCUSS] Dropping flink-storm?

2018-10-08 Thread Till Rohrmann
Thanks for opening the issue Chesnay. I think the overall consensus is to drop flink-storm and only keep the Bolt and Spout wrappers. Thanks for your feedback! Cheers, Till On Mon, Oct 8, 2018 at 9:37 AM Chesnay Schepler wrote: > I've created https://issues.apache.org/jira/browse/FLINK-10509

[jira] [Created] (FLINK-10510) Bump asm version to 6

2018-10-08 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-10510: Summary: Bump asm version to 6 Key: FLINK-10510 URL: https://issues.apache.org/jira/browse/FLINK-10510 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] Breaking the Scala API for Scala 2.12 Support

2018-10-08 Thread Chesnay Schepler
I'd rather not maintain 2 master branches. Beyond the maintenance overhead I'm wondering about the benefit, as the API break still has to happen at some point. @Aljoscha how much work for supporting scala 2.12 can be merged without breaking the API? If this is the only blocker I suggest to

Re: MapState support for operator state

2018-10-08 Thread Dániel Berecz
Hi Till, sorry for this. I sent this e-mail from my other account, which is not subscribed to this thread and according to the website should have been rejected. Once I realized it I sent another e-mail from the correct account. Bestt, Daniel Till Rohrmann ezt írta (időpont: 2018. okt. 8., H,

Re: [DISCUSS] Dropping flink-storm?

2018-10-08 Thread Chesnay Schepler
I've created https://issues.apache.org/jira/browse/FLINK-10509 for removing flink-storm. On 28.09.2018 15:22, Till Rohrmann wrote: Hi everyone, I would like to discuss how to proceed with Flink's storm compatibility layer flink-strom. While working on removing Flink's legacy mode, I noticed

[jira] [Created] (FLINK-10509) Drop flink-storm

2018-10-08 Thread Chesnay Schepler (JIRA)
Chesnay Schepler created FLINK-10509: Summary: Drop flink-storm Key: FLINK-10509 URL: https://issues.apache.org/jira/browse/FLINK-10509 Project: Flink Issue Type: Improvement

Re: MapState support for operator state

2018-10-08 Thread Till Rohrmann
Hi Daniel, Didn't you post this question before? Let's not spread the discussion out over multiple threads. Cheers, Till On Sun, Oct 7, 2018 at 4:27 PM Dániel Berecz wrote: > Hi everyone, > > I would like to ask if there is any conceptual problem with having MapState > available for operator