RE: [DISCUSS] FLIP-2 Extending Window Function Metadata

Hi, If it is to extend the Context to pass more information between the stages of processing a window (triggering -> process -> eviction), why not adding also a "EvictionInfo"? I think this might actually help with the issues discussed in the tread related to the eviction policy. I could

[jira] [Created] (FLINK-4205) Implement stratified sampling for DataSet

Do Le Quoc created FLINK-4205: - Summary: Implement stratified sampling for DataSet Key: FLINK-4205 URL: https://issues.apache.org/jira/browse/FLINK-4205 Project: Flink Issue Type: New Feature

Re: [DISCUSS] FLIP-1 : Fine grained recovery from task failures

I added a FLIP document in the wiki: https://cwiki.apache.org/confluence/display/FLINK/FLIP-1+%3A+Fine+Grained+Recovery+from+Task+Failures For now, this contains the link to the Google Doc and a link to this discussion thread. Once a Jira is created for this it should also be added there. On

Re: sampling function

Hi Till, I have created the JIRA: https://issues.apache.org/jira/browse/FLINK-4205 Thank you, Do On Tue, Jul 12, 2016 at 6:05 PM, Till Rohrmann wrote: > Stratified sampling would also be beneficial for the DataSet API. I think > it would be best if this method is also

Restructuring Javadoc and Scaladoc for libraries

Hi all, I just noticed some scaladocs (Gelly Scala API, Streaming Scala API, and FlinkML) are missing in scaladoc page but found in javadoc page, even though the APIs are for Scala. Is this intentional? I think we have to move some documentation to scaladoc. Regards, Chiwan Park

Re: [DISCUSS] Enhance Window Evictor in Flink

Hi Aljoscha, I agree, the user will know exactly that they are creating an EventTime based evictor or ProcessingTime based evictor looking at the code. So do you think it will be ok to have multiple versions of TimeEvictor (one for event time and one for processing time) and also a DeltaEvcitor

[jira] [Created] (FLINK-4214) JobExceptionsHandler will return all exceptions

Sumit Chawla created FLINK-4214: --- Summary: JobExceptionsHandler will return all exceptions Key: FLINK-4214 URL: https://issues.apache.org/jira/browse/FLINK-4214 Project: Flink Issue Type: Bug

Re: [DISCUSS] FLIP-1 : Fine grained recovery from task failures

Thanks for this very first proposal! Both the proposed functionality and the way you explained it are super nice. :-) I think that this has been long overdue in Flink. :-) Having worked on both the ExecutionGraph and IntermediateResults before, I agree that these are the relevant components for

[jira] [Created] (FLINK-4207) WindowOperator becomes very slow with allowed lateness

Aljoscha Krettek created FLINK-4207: --- Summary: WindowOperator becomes very slow with allowed lateness Key: FLINK-4207 URL: https://issues.apache.org/jira/browse/FLINK-4207 Project: Flink

[jira] [Created] (FLINK-4206) Metric names should alle special characters

Chesnay Schepler created FLINK-4206: --- Summary: Metric names should alle special characters Key: FLINK-4206 URL: https://issues.apache.org/jira/browse/FLINK-4206 Project: Flink Issue Type:

Re: [DISCUSS] Move Documentation from Wiki to Documentation

+1 I really like the re-organization you just did [1]! :-) Thanks! Is there a way to reflect this in the left-side navigation as well by having all User/Contributors/... child pages grouped together? [1] https://cwiki.apache.org/confluence/display/FLINK/Apache+Flink+Home On Mon, Jul 11, 2016

Re: [DISCUSS] Move Documentation from Wiki to Documentation

I'm not sure, I'll try to figure that out. On Wed, 13 Jul 2016 at 12:15 Ufuk Celebi wrote: > +1 > > I really like the re-organization you just did [1]! :-) Thanks! > > Is there a way to reflect this in the left-side navigation as well by > having all User/Contributors/... child

[DISCUSS] FLIP-2 Extending Window Function Metadata

Hi, this is a proposal to introduce a new interface for the window function to make it more extensible for the future where we might want to provide additional information about why a window fired to the user function:

Re: [DISCUSS] FLIP-2 Extending Window Function Metadata

Sure, I also thought about this but went for the "extreme" initially. If no-one objects I'll update the doc in a bit. On Wed, 13 Jul 2016 at 14:17 Stephan Ewen wrote: > Thanks for opening this. > > I see the need for having an extensible context object for window function >

Re: [DISCUSS] FLIP-1 : Fine grained recovery from task failures

I deliberately kept it initially in the Google Doc because it is easier to comment. I would move it to the wiki once we have passed the initial discussion and have converged (or are close to convergence) on the design. On Tue, Jul 12, 2016 at 8:11 PM, Chesnay Schepler wrote:

Re: [DISCUSS] FLIP-2 Extending Window Function Metadata

Thanks for opening this. I see the need for having an extensible context object for window function invocations, but i think hiding every parameter in the context is a bit unnatural. How about having a function "apply(Key, Values, WindowContext, Collector)" ? It should be possible to write the

[jira] [Created] (FLINK-4210) Move close()/isClosed() out of MetricGroup interface

Chesnay Schepler created FLINK-4210: --- Summary: Move close()/isClosed() out of MetricGroup interface Key: FLINK-4210 URL: https://issues.apache.org/jira/browse/FLINK-4210 Project: Flink

[jira] [Created] (FLINK-4209) Docker image breaks with multiple NICs

Ismaël Mejía created FLINK-4209: --- Summary: Docker image breaks with multiple NICs Key: FLINK-4209 URL: https://issues.apache.org/jira/browse/FLINK-4209 Project: Flink Issue Type: Improvement

Re: [DISCUSS] Enhance Window Evictor in Flink

Hi, I think the way to go here is to add both an EventTimeEvictor and a ProcessingTimeEvictor. The problem is that "isEventTime" cannot really be determined. That's also the reason why there is an EventTimeTrigger and a ProcessingTimeTrigger. It was just an oversight that the TimeEvictor does not

Re: [DISCUSS] Enhance Window Evictor in Flink

Hi Aljoscha, I am thinking of adding a method boolean isEventTime(); in the EvictorContext apart from long getCurrentProcessingTime(); MetricGroup getMetricGroup(); long getCurrentWatermark(); This method can be used to make the Evictor not iterate through all the elements in TimeEvictor. There

[jira] [Created] (FLINK-4208) Support Running Flink processes in foreground mode

Ismaël Mejía created FLINK-4208: --- Summary: Support Running Flink processes in foreground mode Key: FLINK-4208 URL: https://issues.apache.org/jira/browse/FLINK-4208 Project: Flink Issue Type:

Re: [DISCUSS] Enhance Window Evictor in Flink

I still think it should be explicit in the class. For example, if you have this code: input .keyBy() .window() .trigger(EventTimeTrigger.create()) .evictor(TimeTrigger.create()) the time behavior of the trigger is explicitly specified while the evictor would dynamically adapt based on

Re: [DISCUSS] FLIP-1 : Fine grained recovery from task failures

Thank you for writing the proposal! I agree with Chesnay. I thought one of the main motivations of the new FLIP process is to move the discussion onto ASF infra, to record the decision making process on the mailing list and to make ongoing activities more visible to the community. If we want to

Re: [DISCUSS] Enhance Window Evictor in Flink

Hi, I was hoping to use the isEventTime method in the WindowAssigner to set that information in the EvictorContext. What do you think?. Thanks and Regards, Vishnu Viswanath, On Wed, Jul 13, 2016 at 10:09 AM, Aljoscha Krettek wrote: > Hi, > I think the way to go here is to

[jira] [Created] (FLINK-4211) Dynamic Properties not working for jobs submitted to Yarn session

Stefan Richter created FLINK-4211: - Summary: Dynamic Properties not working for jobs submitted to Yarn session Key: FLINK-4211 URL: https://issues.apache.org/jira/browse/FLINK-4211 Project: Flink

Re: [DISCUSS] FLIP-1 : Fine grained recovery from task failures

+1 This was my original motivation. I saw a lot of discussion on the design docs I initiated but none of this was reflected on the ML. On Wed, 13 Jul 2016 at 17:11 Robert Metzger wrote: > Thank you for writing the proposal! > > I agree with Chesnay. I thought one of the