Re: Tagging Flink classes with InterfaceAudience and InterfaceStability

[Robert Metzger]

Thank you Nick. I'll look into the check_compatiblilty.sh script to see
which tools are used.
I think finding breaking API changes immediately is a better process then
reworking the APIs before a release.

As you can see from my email response times (2 days since your email), I'm
probably too overloaded right now to participate in the Yetus project ...
Sadly.


@others: I know its not the most interesting thing to go through my list of
stable interfaces, but keep in mind that we have to maintain the stuff for
probably quite some time, so it would be good to have more than one pair of
eyes looking at it :)


On Mon, Nov 23, 2015 at 6:20 PM, Nick Dimiduk  wrote:

> >
> > Do you know if Hadoop/HBase is also using a maven plugin to fail a build
> on
> > breaking API changes? I would really like to have such a functionality in
> > Flink, because we can spot breaking changes very early.
>
>
> I don't think we have maven integration for this as of yet. We release
> managers run a script $HBASE/dev-support/check_compatibility.sh that
> generates a source and binary compatibility report. Issues are then
> resolved during the period leading up to the release candidate.
>
> I think Hadoop is relying on a "QA bot" which reads patches from JIRA and
> > then does these
> > checks?
> >
>
> The "QA bot" is just a collection of shell scripts used during "Patch
> Available" status when a patch has been attached to JIRA or when a PR has
> been submitted through github. The check_compatibility script could be
> included in that automation, I don't see why not. Maybe you'd like to open
> a YETUS ticket :)
>
> I've pushed a branch to my own GitHub account with all classes I would make
> > public annotated:
> >
> >
> https://github.com/apache/flink/compare/master...rmetzger:interface_stability_revapi?expand=1
> > Since this is really hard to read, I (half-automated) generated the
> > following list of annotated classes:
> >
> >
> https://github.com/rmetzger/flink/blob/interface_stability_revapi/annotations.md
> >
> > Please let me know if you would like to include or exclude classes from
> > that list.
> > Also, let me know which methods (in stable classes) you would mark as
> > experimental.
> >
> >
> >
> >
> > On Wed, Nov 11, 2015 at 10:17 AM, Aljoscha Krettek 
> > wrote:
> >
> > > +1 for some way of declaring public interfaces as experimental.
> > >
> > > > On 10 Nov 2015, at 22:24, Stephan Ewen  wrote:
> > > >
> > > > I think we need anyways an annotation "@PublicExperimental".
> > > >
> > > > We can make this annotation such that it can be added to methods and
> > can
> > > > use that to declare
> > > > Methods in an otherwise public class (such as DataSet) as
> experimental.
> > > >
> > > > On Tue, Nov 10, 2015 at 10:19 PM, Fabian Hueske 
> > > wrote:
> > > >
> > > >> I am not sure if we always should declare complete classes as
> > > >> @PublicInterface.
> > > >> This does definitely make sense for interfaces and abstract classes
> > > such as
> > > >> MapFunction or InputFormat but not necessarily for classes such as
> > > DataSet
> > > >> that we might want to extend by methods which should not immediately
> > be
> > > >> considered as stable.
> > > >>
> > > >>
> > > >> 2015-11-10 21:36 GMT+01:00 Vasiliki Kalavri <
> > vasilikikala...@gmail.com
> > > >:
> > > >>
> > > >>> Yes, my opinion is that we shouldn't declare the Gelly API frozen
> > yet.
> > > >>> We can reconsider when we're closer to the 1.0 release, but if
> > > possible,
> > > >> I
> > > >>> would give it some more time.
> > > >>>
> > > >>> -V.
> > > >>>
> > > >>> On 10 November 2015 at 21:06, Stephan Ewen 
> wrote:
> > > >>>
> > >  I think no component should be forced to be stable. It should be
> an
> > >  individual decision for each component, and in some cases even for
> > >  individual classes.
> > > 
> > >  @Vasia If you think Gelly should not be declared interface-frozen,
> > > then
> > >  this is a good point to raise and this should definitely be
> > reflected.
> > >  There is no point in declaring certain APIs as frozen when we are
> > not
> > > >> yet
> > >  confident they have converged.
> > > 
> > > 
> > > 
> > >  On Tue, Nov 10, 2015 at 8:39 PM, Vasiliki Kalavri <
> > >  vasilikikala...@gmail.com
> > > > wrote:
> > > 
> > > > Hi Robert,
> > > >
> > > > thanks for bringing this up!
> > > >
> > > > I generally like the idea, but I wouldn't rush to annotate the
> > Gelly
> > > > classes yet. Gelly hasn't had that many users and I'm quite sure
> > > >> we'll
> > >  find
> > > > things to improve as it gets more exposure.
> > > > TBH, I think it's quite unfair to force Gelly (also e.g. ML,
> Table)
> > > >> to
> > > >>> a
> > > > "1.0" status (from an API stability point of view) since they're
> > > >> really
> > > > young compared to the other Flink 

[jira] [Created] (FLINK-3076) Display channel exchange mode in the plan visualizer

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-3076:
---

 Summary: Display channel exchange mode in the plan visualizer
 Key: FLINK-3076
 URL: https://issues.apache.org/jira/browse/FLINK-3076
 Project: Flink
  Issue Type: Improvement
  Components: Webfrontend
Affects Versions: 0.10.0
Reporter: Chesnay Schepler
Priority: Minor
 Fix For: 1.0.0


FLINK-2686 extended the JSON plan to include the channel exchange mode. The 
plan visualizer should be adjusted to display it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

could you please add me Contributor list?

[jun aoki]

So that I can assign myself to Flink jiras?

-- 
-jun

Re: could you please add me Contributor list?

[Henry Saputra]

Hi Jun,

What is your JIRA username?

But for now, you can always work on JIRA issue without assigning to
yourself. Just add to comment that you are planning to work on it.

- Henry

On Tue, Nov 24, 2015 at 5:18 PM, jun aoki  wrote:
> So that I can assign myself to Flink jiras?
>
> --
> -jun

Re: withParameters() for Streaming API

[Chesnay Schepler]

should we do the same for IOFormats to be consistent? after FLINK-2351 
they aren't used by our formats.


On 24.11.2015 15:17, Suneel Marthi wrote:

Agree with @till +1 to change this now

On Tue, Nov 24, 2015 at 9:15 AM, Till Rohrmann  wrote:


If not API breaking before 1.0, then probably never?

On Tue, Nov 24, 2015 at 3:06 PM, Stephan Ewen  wrote:


I was also thinking of deprecating that. With that, RichFunctions should
change "open(Configuration)" --> "open()".

Would be heavily API breaking, so bit hesitant there...

On Tue, Nov 24, 2015 at 2:48 PM, Timo Walther 

wrote:

Thanks for the hint Matthias.
So actually the parameter of the open() method is useless? IMHO that

does

not look like a nice API design...
We should try to keep DataSet and DataStream API in sync.
Does it make sense to deprecate withParameters() for 1.0?

Timo


On 24.11.2015 14:31, Matthias J. Sax wrote:


We had this discussion a while ago.

If I recall correctly, "withParameters()" is not encourage to be used

in

DataSet either.

This is the thread:



https://mail-archives.apache.org/mod_mbox/flink-dev/201509.mbox/%3C55EC69CD.1070003%40apache.org%3E

-Matthias

On 11/24/2015 02:14 PM, Timo Walther wrote:


Hi all,

I want to set the Configuration of a streaming operator and access it
via the open method of the RichFunction.
There is no possibility to set the Configuration of the open method

at

the moment, right? Can I open an issue for a withParameters()

equivalent

for the Stremaing API?

Regards,
Timo

Re: Either left() vs left(value)

[Gyula Fóra]

I opened a PR: https://github.com/apache/flink/pull/1402  with my
suggestions, let me know what you think and we can either merge this or
leave it as it is :)

I would also like to hear the opinions of others about this.

Vasiliki Kalavri  ezt írta (időpont: 2015. nov.
23., H, 22:03):

> Either is abstract already ;)
>
> On 23 November 2015 at 21:54, Gyula Fóra  wrote:
>
> > I think it is not too bad to only have the Right/Left classes. You can
> then
> > write it like this:
> >
> > Either e1 = new Left<>("");
> > Either e2 = new Right<>(1);
> >
> > (this would be pretty much like in scala)
> >
> > or we can add static methods like: Left.of(...), Right.of(...) which
> would
> > work exactly as it does now.
> >
> > And then we can live without the static methods in Either (Either would
> > become Abstract).
> >
> > Gyula
> >
> > Vasiliki Kalavri  ezt írta (időpont: 2015.
> nov.
> > 23., H, 21:25):
> >
> > > Ah I see. Well, as I also said in the PR, Left and Right make no sense
> on
> > > their own, they're helper classes for Either. Hence, I believe they
> > should
> > > be private. Maybe we could rename the methods to createLeft() /
> > > createRight() ?
> > >
> > > On 23 November 2015 at 20:58, Gyula Fóra  wrote:
> > >
> > > > I was actually not suggesting to drop the e.left() method but instead
> > the
> > > > Either.left(val).
> > > > Renaming the left(), right() methods might be confusing as than it
> > would
> > > be
> > > > inconsistent with the scala version.
> > > >
> > > > On the other hand we could change the way the user can create the
> Left
> > > > Right classes, maybe directly expose them instead of the static
> method.
> > > (or
> > > > rename the static method)
> > > >
> > > > Gyula
> > > >
> > > > Vasiliki Kalavri  ezt írta (időpont:
> 2015.
> > > nov.
> > > > 23., H, 20:14):
> > > >
> > > > > Hey Gyula,
> > > > >
> > > > > I don't think dropping the method is a good idea. We need a way to
> > > > retrieve
> > > > > left and right values, no?
> > > > > How about renaming to getLeft() / getRight()?
> > > > >
> > > > > -V.
> > > > >
> > > > > On 23 November 2015 at 09:55, Gyula Fóra 
> > wrote:
> > > > >
> > > > > > Hey guys,
> > > > > >
> > > > > > I know this should have been part of the PR discussion but it
> kind
> > of
> > > > > > slipped through the cracks :)
> > > > > >
> > > > > > I think it might be useful to change the method name for
> > > > > Either.left(value)
> > > > > > to Either.Left(value) (or drop the method completely).
> > > > > >
> > > > > > The reason is that it is slightly awkward to use it with java 8
> > > > lambdas.
> > > > > > You cannot use Either::left because of the name clash. Maybe it's
> > > not a
> > > > > > huge issue but a small inconvenience that will come up more often
> > as
> > > we
> > > > > are
> > > > > > gradually moving to java 8 anyways :)
> > > > > >
> > > > > > What do you think?
> > > > > > Gyula
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Null Pointer Exception in tests but only in COLLECTION mode

[Martin Junghanns]

Hi Max,

fixed in https://github.com/apache/flink/pull/1396

Best,
Martin

On 24.11.2015 13:46, Maximilian Michels wrote:

Hi André, hi Martin,

This looks very much like a bug. Martin, I would be happy if you
opened a JIRA issue.

Thanks,
Max

On Sun, Nov 22, 2015 at 12:27 PM, Martin Junghanns
 wrote:

Hi,

What he meant was MultipleProgramsTestBase, not FlinkTestBase.

I debugged this a bit.

The NPE is thrown in

https://github.com/apache/flink/blob/master/flink-java/src/main/java/org/apache/flink/api/java/operators/AggregateOperator.java#L296

since current can be null if the input iterator is empty.

In Cluster Execution, it is checked that the output of the previous function
(e.g. Filter) is not empty in:

https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/operators/AllGroupReduceDriver.java#L144

which avoids going into AggregateOperator and getting a NPE.

However, in Collection Mode, the execution is not grouped (don't know why,
yet). In

https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/operators/base/GroupReduceOperatorBase.java#L207

the copied input data is handed over to the aggregate function which leads
to the NPE.

Checking inputDataCopy.size() > 0 before calling the aggregate solves the
problem.

If someone can confirm that this is not a more generic problem, I would open
an issue and a PR.

Best,
Martin


On 20.11.2015 18:41, André Petermann wrote:


Hi all,

during a workflow, a data set may run empty, e.g., because of a join
without matches.

We're using FlinkTestBase and found out, that aggregate functions on
empty data sets work fine in CLUSTER execution mode but cause a Null
Pointer Exception at AggregateOperator$AggregatingUdf in COLLECTION mode.

Here is the minimal example on 1.0-SNAPSHOT:
https://gist.github.com/p3et/59a65bab11098dd11054

Are we doing something wrong, or is this a bug?

Cheers,
Andre

[jira] [Created] (FLINK-3075) Rename Either creation methods to avoid name clash with projection methods

[Gyula Fora (JIRA)]

Gyula Fora created FLINK-3075:
-

 Summary: Rename Either creation methods to avoid name clash with 
projection methods
 Key: FLINK-3075
 URL: https://issues.apache.org/jira/browse/FLINK-3075
 Project: Flink
  Issue Type: Improvement
Reporter: Gyula Fora
Priority: Minor


Currently the method signatures for creating Either values `Either.left(left)` 
and the projection methods `either.left()` only differ in the parameters. 

This makes it awkward to use with lambdas such as: 
'eitherStream.filter(Either:isLeft).map(Either::left)'
The above code is currently impossible.

I suggest to change the creation methods to `Either.createLeft(left)` and 
`Either.createRight(right)` and also to directly expose the Left, Right classes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3065) Can't cancel failing jobs

[Gyula Fora (JIRA)]

Gyula Fora created FLINK-3065:
-

 Summary: Can't cancel failing jobs
 Key: FLINK-3065
 URL: https://issues.apache.org/jira/browse/FLINK-3065
 Project: Flink
  Issue Type: Bug
  Components: Command-line client, Webfrontend
Affects Versions: 0.10.0, 1.0.0
Reporter: Gyula Fora
Priority: Blocker


It is currently not possible to stop a failing streaming job (if it get's stuck 
while failing for instance).

There is no cancel button in the web interface, also it doesnt show on the list 
of running jobs in the command line.

This means jobs getting stuck while failing will take down the cluster 
eventually.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3066) Kafka source fails on leader change

[Gyula Fora (JIRA)]

Gyula Fora created FLINK-3066:
-

 Summary: Kafka source fails on leader change
 Key: FLINK-3066
 URL: https://issues.apache.org/jira/browse/FLINK-3066
 Project: Flink
  Issue Type: Bug
  Components: Streaming Connectors
Affects Versions: 0.10.0, 1.0.0
Reporter: Gyula Fora


I got the following exception during my streaming job:

16:44:50,637 INFO  org.apache.flink.runtime.jobmanager.JobManager   
 - Status of job 4d3f9443df4822e875f1400244a6e8dd (deduplo!) changed to FAILING.
java.lang.Exception: Failed to send data to Kafka: This server is not the 
leader for that topic-partition.
at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.checkErroneous(FlinkKafkaProducer.java:275)
at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.invoke(FlinkKafkaProducer.java:246)
at 
org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:37)
at 
org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:166)
at 
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:221)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:588)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.common.errors.NotLeaderForPartitionException: This 
server is not the leader for that topic-partition.

And then the job crashed and recovered. This should probably be something that 
we handle without crashing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3067) Kafka source fails during checkpoint notifications with NPE

[Gyula Fora (JIRA)]

Gyula Fora created FLINK-3067:
-

 Summary: Kafka source fails during checkpoint notifications with 
NPE
 Key: FLINK-3067
 URL: https://issues.apache.org/jira/browse/FLINK-3067
 Project: Flink
  Issue Type: Bug
Reporter: Gyula Fora


While running a job with many kafka sources I experienced the following error 
during the checkpoint notifications:

java.lang.RuntimeException: Error while confirming checkpoint
at org.apache.flink.runtime.taskmanager.Task$2.run(Task.java:935)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at 
org.I0Itec.zkclient.ZkConnection.writeDataReturnStat(ZkConnection.java:115)
at org.I0Itec.zkclient.ZkClient$10.call(ZkClient.java:817)
at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
at org.I0Itec.zkclient.ZkClient.writeDataReturnStat(ZkClient.java:813)
at org.I0Itec.zkclient.ZkClient.writeData(ZkClient.java:808)
at org.I0Itec.zkclient.ZkClient.writeData(ZkClient.java:777)
at kafka.utils.ZkUtils$.updatePersistentPath(ZkUtils.scala:332)
at kafka.utils.ZkUtils.updatePersistentPath(ZkUtils.scala)
at 
org.apache.flink.streaming.connectors.kafka.internals.ZookeeperOffsetHandler.setOffsetInZooKeeper(ZookeeperOffsetHandler.java:112)
at 
org.apache.flink.streaming.connectors.kafka.internals.ZookeeperOffsetHandler.commit(ZookeeperOffsetHandler.java:80)
at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.notifyCheckpointComplete(FlinkKafkaConsumer.java:563)
at 
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.notifyOfCompletedCheckpoint(AbstractUdfStreamOperator.java:176)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:478)
at org.apache.flink.runtime.taskmanager.Task$2.run(Task.java:931)
... 5 more
06:23:28,373 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph   
 - Source: Kafka[event.rakdos.log] (1/1) (d79e6b7a25b1ac307d2e0c8

This resulted in the job crashing and getting stuck during cancelling which 
subsequently lead to having to restart the cluster.

This might be a zookeeper issue but we should be able to handle it (catch the 
exception maybe).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Slinding Window Join (without duplicates)

[Aljoscha Krettek]

Hi,
I’m not sure this is a problem. If a user specifies sliding windows then one 
element can (and will) end up in several windows. If these are joined then 
there will be multiple results. If the user does not want multiple windows then 
tumbling windows should be used.

IMHO, this is quite straightforward. But let’s see what others have to say.

Cheers,
Aljoscha 
> On 23 Nov 2015, at 20:36, Matthias J. Sax  wrote:
> 
> Hi,
> 
> it seems that a join on the data streams with an overlapping sliding
> window produces duplicates in the output. The default implementation
> internally just use two nested-loops over both windows to compute the
> result.
> 
> How can duplicates be avoided? Is there any way after all right now? If
> not, should be add this?
> 
> 
> -Matthias
> 

Re: [VOTE] Release Apache Flink 0.10.1 (release-0.10.0-rc1)

[Stephan Ewen]

Hi Slava!

I think the problem with your build is the file handles. It shows in
various points:

Exception in thread "main" java.lang.InternalError:
java.io.FileNotFoundException:
/Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home/jre/lib/ext/localedata.jar
(Too many open files in system)

Caused by: java.io.IOException: Too many open files in system
at sun.nio.ch.KQueueArrayWrapper.init(Native Method)
at sun.nio.ch.KQueueArrayWrapper.(KQueueArrayWrapper.java:98)
at sun.nio.ch.KQueueSelectorImpl.(KQueueSelectorImpl.java:87)
at 
sun.nio.ch.KQueueSelectorProvider.openSelector(KQueueSelectorProvider.java:42)
at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:126)


Can you check this on your system? I'll try to compile with the same flags
on my system as well...



On Tue, Nov 24, 2015 at 11:07 AM, Vyacheslav Zholudev <
vyacheslav.zholu...@gmail.com> wrote:

> I'm having trouble building release-0.10.1-rc1 with parameters:
> mvn clean install -Dhadoop.version=2.6.0.2.2.6.0-2800 -Pvendor-repos
>
> Env: maven 3, JDK 7, MacOS 10.10.5
>
> Attached maven log when it started to produce failing tests.
>
> P.S. I had to kill the build process since it got stuck (probably due to
> some long waiting interval)
>
> mvn.log
> <
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/file/n9315/mvn.log
> >
>
>
>
> --
> View this message in context:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Release-Apache-Flink-0-10-1-release-0-10-0-rc1-tp9296p9315.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>

Re: [VOTE] Release Apache Flink 0.10.1 (release-0.10.0-rc1)

[Vyacheslav Zholudev]

I'm having trouble building release-0.10.1-rc1 with parameters:
mvn clean install -Dhadoop.version=2.6.0.2.2.6.0-2800 -Pvendor-repos

Env: maven 3, JDK 7, MacOS 10.10.5

Attached maven log when it started to produce failing tests. 

P.S. I had to kill the build process since it got stuck (probably due to
some long waiting interval)

mvn.log

  



--
View this message in context: 
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Release-Apache-Flink-0-10-1-release-0-10-0-rc1-tp9296p9315.html
Sent from the Apache Flink Mailing List archive. mailing list archive at 
Nabble.com.

Re: Slinding Window Join (without duplicates)

[Matthias J. Sax]

Stephan is right. A tumbling window does not help. The last tuple of
window n and the first tuple of window n+1 are "close" to each other and
should be joined for example.

From a SQL-like point of view this is a very common case expressed as:

SELECT * FROM s1,s2 WHERE s1.key = s2.key AND |s1.ts - s2.ts| < window-size

I would not expect to get any duplicates here.

Basically, the window should move by one tuple (for each stream) and
join with all tuples from the other stream that are within the time
range (window size) were the ts of this new tuple define the boundaries
of the window (ie, there are no "fixed" window boundaries as defined by
a time-slide).

Not sure how a "session window" can help here... I guess using most
generic window API allows to define slide by one tuple and window size X
seconds. But I don't know how duplicates could be avoided...

-Matthias

On 11/24/2015 11:04 AM, Stephan Ewen wrote:
> I understand Matthias' point. You want to join elements that occur within a
> time range of each other.
> 
> In a tumbling window, you have strict boundaries and a pair of elements
> that arrives such that one element is before the boundary and one after,
> they will not join. Hence the sliding windows.
> 
> What may be a solution here is a "session window" join...
> 
> On Tue, Nov 24, 2015 at 10:33 AM, Aljoscha Krettek 
> wrote:
> 
>> Hi,
>> I’m not sure this is a problem. If a user specifies sliding windows then
>> one element can (and will) end up in several windows. If these are joined
>> then there will be multiple results. If the user does not want multiple
>> windows then tumbling windows should be used.
>>
>> IMHO, this is quite straightforward. But let’s see what others have to say.
>>
>> Cheers,
>> Aljoscha
>>> On 23 Nov 2015, at 20:36, Matthias J. Sax  wrote:
>>>
>>> Hi,
>>>
>>> it seems that a join on the data streams with an overlapping sliding
>>> window produces duplicates in the output. The default implementation
>>> internally just use two nested-loops over both windows to compute the
>>> result.
>>>
>>> How can duplicates be avoided? Is there any way after all right now? If
>>> not, should be add this?
>>>
>>>
>>> -Matthias
>>>
>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [VOTE] Release Apache Flink 0.10.1 (release-0.10.0-rc1)

[Gyula Fóra]

Hi,
Regarding my previous comment for the Kafka/Zookeeper issue, let's discuss
if this is critical enough so we want to include it in this release or the
next bugfix.

I will try to further investigate the reason the job failed in the first
place (we suspect broker failure)

Cheers,
Gyula

Vyacheslav Zholudev  ezt írta (időpont:
2015. nov. 24., K, 11:17):

> Sorry, should have paid attention to the stacktraces. And actually I see
> than
> Java8 got picked up.
> Will try to fix the issue and rerun
>
>
>
> --
> View this message in context:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Release-Apache-Flink-0-10-1-release-0-10-0-rc1-tp9296p9318.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>

Re: Slinding Window Join (without duplicates)

[Stephan Ewen]

Since sessions are built per key, they have groups of keys that are close
enough together in time. They will, however, treat the closeness
transitively...

On Tue, Nov 24, 2015 at 11:33 AM, Matthias J. Sax  wrote:

> Stephan is right. A tumbling window does not help. The last tuple of
> window n and the first tuple of window n+1 are "close" to each other and
> should be joined for example.
>
> From a SQL-like point of view this is a very common case expressed as:
>
> SELECT * FROM s1,s2 WHERE s1.key = s2.key AND |s1.ts - s2.ts| < window-size
>
> I would not expect to get any duplicates here.
>
> Basically, the window should move by one tuple (for each stream) and
> join with all tuples from the other stream that are within the time
> range (window size) were the ts of this new tuple define the boundaries
> of the window (ie, there are no "fixed" window boundaries as defined by
> a time-slide).
>
> Not sure how a "session window" can help here... I guess using most
> generic window API allows to define slide by one tuple and window size X
> seconds. But I don't know how duplicates could be avoided...
>
> -Matthias
>
> On 11/24/2015 11:04 AM, Stephan Ewen wrote:
> > I understand Matthias' point. You want to join elements that occur
> within a
> > time range of each other.
> >
> > In a tumbling window, you have strict boundaries and a pair of elements
> > that arrives such that one element is before the boundary and one after,
> > they will not join. Hence the sliding windows.
> >
> > What may be a solution here is a "session window" join...
> >
> > On Tue, Nov 24, 2015 at 10:33 AM, Aljoscha Krettek 
> > wrote:
> >
> >> Hi,
> >> I’m not sure this is a problem. If a user specifies sliding windows then
> >> one element can (and will) end up in several windows. If these are
> joined
> >> then there will be multiple results. If the user does not want multiple
> >> windows then tumbling windows should be used.
> >>
> >> IMHO, this is quite straightforward. But let’s see what others have to
> say.
> >>
> >> Cheers,
> >> Aljoscha
> >>> On 23 Nov 2015, at 20:36, Matthias J. Sax  wrote:
> >>>
> >>> Hi,
> >>>
> >>> it seems that a join on the data streams with an overlapping sliding
> >>> window produces duplicates in the output. The default implementation
> >>> internally just use two nested-loops over both windows to compute the
> >>> result.
> >>>
> >>> How can duplicates be avoided? Is there any way after all right now? If
> >>> not, should be add this?
> >>>
> >>>
> >>> -Matthias
> >>>
> >>
> >>
> >
>
>

Re: Slinding Window Join (without duplicates)

[Stephan Ewen]

I understand Matthias' point. You want to join elements that occur within a
time range of each other.

In a tumbling window, you have strict boundaries and a pair of elements
that arrives such that one element is before the boundary and one after,
they will not join. Hence the sliding windows.

What may be a solution here is a "session window" join...

On Tue, Nov 24, 2015 at 10:33 AM, Aljoscha Krettek 
wrote:

> Hi,
> I’m not sure this is a problem. If a user specifies sliding windows then
> one element can (and will) end up in several windows. If these are joined
> then there will be multiple results. If the user does not want multiple
> windows then tumbling windows should be used.
>
> IMHO, this is quite straightforward. But let’s see what others have to say.
>
> Cheers,
> Aljoscha
> > On 23 Nov 2015, at 20:36, Matthias J. Sax  wrote:
> >
> > Hi,
> >
> > it seems that a join on the data streams with an overlapping sliding
> > window produces duplicates in the output. The default implementation
> > internally just use two nested-loops over both windows to compute the
> > result.
> >
> > How can duplicates be avoided? Is there any way after all right now? If
> > not, should be add this?
> >
> >
> > -Matthias
> >
>
>

Re: [VOTE] Release Apache Flink 0.10.1 (release-0.10.0-rc1)

[Vyacheslav Zholudev]

Sorry, should have paid attention to the stacktraces. And actually I see than
Java8 got picked up.
Will try to fix the issue and rerun



--
View this message in context: 
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Release-Apache-Flink-0-10-1-release-0-10-0-rc1-tp9296p9318.html
Sent from the Apache Flink Mailing List archive. mailing list archive at 
Nabble.com.

Re: [VOTE] Release Apache Flink 0.10.1 (release-0.10.0-rc1)

[Gyula Fóra]

Hi,

I vote -1 for the RC due to the fact that the zookeeper deadlock issue was
not completely solved.

Robert could find the problem with the dependency management plugin and has
opened a PR:

 [FLINK-3067] Enforce zkclient 0.7 for Kafka
https://github.com/apache/flink/pull/1399

Cheers,
Gyula

Vyacheslav Zholudev  ezt írta (időpont:
2015. nov. 24., K, 11:07):

> I'm having trouble building release-0.10.1-rc1 with parameters:
> mvn clean install -Dhadoop.version=2.6.0.2.2.6.0-2800 -Pvendor-repos
>
> Env: maven 3, JDK 7, MacOS 10.10.5
>
> Attached maven log when it started to produce failing tests.
>
> P.S. I had to kill the build process since it got stuck (probably due to
> some long waiting interval)
>
> mvn.log
> <
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/file/n9315/mvn.log
> >
>
>
>
> --
> View this message in context:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Release-Apache-Flink-0-10-1-release-0-10-0-rc1-tp9296p9315.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>

Re: withParameters() for Streaming API

[Till Rohrmann]

If not API breaking before 1.0, then probably never?

On Tue, Nov 24, 2015 at 3:06 PM, Stephan Ewen  wrote:

> I was also thinking of deprecating that. With that, RichFunctions should
> change "open(Configuration)" --> "open()".
>
> Would be heavily API breaking, so bit hesitant there...
>
> On Tue, Nov 24, 2015 at 2:48 PM, Timo Walther  wrote:
>
> > Thanks for the hint Matthias.
> > So actually the parameter of the open() method is useless? IMHO that does
> > not look like a nice API design...
> > We should try to keep DataSet and DataStream API in sync.
> > Does it make sense to deprecate withParameters() for 1.0?
> >
> > Timo
> >
> >
> > On 24.11.2015 14:31, Matthias J. Sax wrote:
> >
> >> We had this discussion a while ago.
> >>
> >> If I recall correctly, "withParameters()" is not encourage to be used in
> >> DataSet either.
> >>
> >> This is the thread:
> >>
> >>
> https://mail-archives.apache.org/mod_mbox/flink-dev/201509.mbox/%3C55EC69CD.1070003%40apache.org%3E
> >>
> >> -Matthias
> >>
> >> On 11/24/2015 02:14 PM, Timo Walther wrote:
> >>
> >>> Hi all,
> >>>
> >>> I want to set the Configuration of a streaming operator and access it
> >>> via the open method of the RichFunction.
> >>> There is no possibility to set the Configuration of the open method at
> >>> the moment, right? Can I open an issue for a withParameters()
> equivalent
> >>> for the Stremaing API?
> >>>
> >>> Regards,
> >>> Timo
> >>>
> >>>
> >
>

Flink Master on Travis

[Matthias J. Sax]

Hi,

I just observed that a couple of builds failed recently due to
timeout... Is there anything we can do about this?

Two recent build passed but took 1:56:25, 1:50:47 what is close to the
2h time limit, too.

Just increasing the timeout seems no a good idea I guess.


-Matthias



signature.asc
Description: OpenPGP digital signature

Re: withParameters() for Streaming API

[Stephan Ewen]

I was also thinking of deprecating that. With that, RichFunctions should
change "open(Configuration)" --> "open()".

Would be heavily API breaking, so bit hesitant there...

On Tue, Nov 24, 2015 at 2:48 PM, Timo Walther  wrote:

> Thanks for the hint Matthias.
> So actually the parameter of the open() method is useless? IMHO that does
> not look like a nice API design...
> We should try to keep DataSet and DataStream API in sync.
> Does it make sense to deprecate withParameters() for 1.0?
>
> Timo
>
>
> On 24.11.2015 14:31, Matthias J. Sax wrote:
>
>> We had this discussion a while ago.
>>
>> If I recall correctly, "withParameters()" is not encourage to be used in
>> DataSet either.
>>
>> This is the thread:
>>
>> https://mail-archives.apache.org/mod_mbox/flink-dev/201509.mbox/%3C55EC69CD.1070003%40apache.org%3E
>>
>> -Matthias
>>
>> On 11/24/2015 02:14 PM, Timo Walther wrote:
>>
>>> Hi all,
>>>
>>> I want to set the Configuration of a streaming operator and access it
>>> via the open method of the RichFunction.
>>> There is no possibility to set the Configuration of the open method at
>>> the moment, right? Can I open an issue for a withParameters() equivalent
>>> for the Stremaing API?
>>>
>>> Regards,
>>> Timo
>>>
>>>
>

Re: withParameters() for Streaming API

[Suneel Marthi]

Agree with @till +1 to change this now

On Tue, Nov 24, 2015 at 9:15 AM, Till Rohrmann  wrote:

> If not API breaking before 1.0, then probably never?
>
> On Tue, Nov 24, 2015 at 3:06 PM, Stephan Ewen  wrote:
>
> > I was also thinking of deprecating that. With that, RichFunctions should
> > change "open(Configuration)" --> "open()".
> >
> > Would be heavily API breaking, so bit hesitant there...
> >
> > On Tue, Nov 24, 2015 at 2:48 PM, Timo Walther 
> wrote:
> >
> > > Thanks for the hint Matthias.
> > > So actually the parameter of the open() method is useless? IMHO that
> does
> > > not look like a nice API design...
> > > We should try to keep DataSet and DataStream API in sync.
> > > Does it make sense to deprecate withParameters() for 1.0?
> > >
> > > Timo
> > >
> > >
> > > On 24.11.2015 14:31, Matthias J. Sax wrote:
> > >
> > >> We had this discussion a while ago.
> > >>
> > >> If I recall correctly, "withParameters()" is not encourage to be used
> in
> > >> DataSet either.
> > >>
> > >> This is the thread:
> > >>
> > >>
> >
> https://mail-archives.apache.org/mod_mbox/flink-dev/201509.mbox/%3C55EC69CD.1070003%40apache.org%3E
> > >>
> > >> -Matthias
> > >>
> > >> On 11/24/2015 02:14 PM, Timo Walther wrote:
> > >>
> > >>> Hi all,
> > >>>
> > >>> I want to set the Configuration of a streaming operator and access it
> > >>> via the open method of the RichFunction.
> > >>> There is no possibility to set the Configuration of the open method
> at
> > >>> the moment, right? Can I open an issue for a withParameters()
> > equivalent
> > >>> for the Stremaing API?
> > >>>
> > >>> Regards,
> > >>> Timo
> > >>>
> > >>>
> > >
> >
>

[jira] [Created] (FLINK-3068) Add a Table API configuration to TableEnvironment

[Timo Walther (JIRA)]

Timo Walther created FLINK-3068:
---

 Summary: Add a Table API configuration to TableEnvironment
 Key: FLINK-3068
 URL: https://issues.apache.org/jira/browse/FLINK-3068
 Project: Flink
  Issue Type: New Feature
  Components: Table API
Reporter: Timo Walther


Once FLINK-2828 is merged and a TableEnvironment is no longer optional, it 
makes sense provide a way to configure the Table API. This includes timezones, 
charsets, nullability etc. (e.g. FLINK-2961) but many more settings for future 
features such as the SQL API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Null Pointer Exception in tests but only in COLLECTION mode

[Maximilian Michels]

Hi André, hi Martin,

This looks very much like a bug. Martin, I would be happy if you
opened a JIRA issue.

Thanks,
Max

On Sun, Nov 22, 2015 at 12:27 PM, Martin Junghanns
 wrote:
> Hi,
>
> What he meant was MultipleProgramsTestBase, not FlinkTestBase.
>
> I debugged this a bit.
>
> The NPE is thrown in
>
> https://github.com/apache/flink/blob/master/flink-java/src/main/java/org/apache/flink/api/java/operators/AggregateOperator.java#L296
>
> since current can be null if the input iterator is empty.
>
> In Cluster Execution, it is checked that the output of the previous function
> (e.g. Filter) is not empty in:
>
> https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/operators/AllGroupReduceDriver.java#L144
>
> which avoids going into AggregateOperator and getting a NPE.
>
> However, in Collection Mode, the execution is not grouped (don't know why,
> yet). In
>
> https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/operators/base/GroupReduceOperatorBase.java#L207
>
> the copied input data is handed over to the aggregate function which leads
> to the NPE.
>
> Checking inputDataCopy.size() > 0 before calling the aggregate solves the
> problem.
>
> If someone can confirm that this is not a more generic problem, I would open
> an issue and a PR.
>
> Best,
> Martin
>
>
> On 20.11.2015 18:41, André Petermann wrote:
>>
>> Hi all,
>>
>> during a workflow, a data set may run empty, e.g., because of a join
>> without matches.
>>
>> We're using FlinkTestBase and found out, that aggregate functions on
>> empty data sets work fine in CLUSTER execution mode but cause a Null
>> Pointer Exception at AggregateOperator$AggregatingUdf in COLLECTION mode.
>>
>> Here is the minimal example on 1.0-SNAPSHOT:
>> https://gist.github.com/p3et/59a65bab11098dd11054
>>
>> Are we doing something wrong, or is this a bug?
>>
>> Cheers,
>> Andre
>>
>

Re: [VOTE] Release Apache Flink 0.10.1 (release-0.10.0-rc1)

[Vyacheslav Zholudev]

I can confirm that the build works fine when increasing a max number of
opened files. Sorry for confusion.



--
View this message in context: 
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Release-Apache-Flink-0-10-1-release-0-10-0-rc1-tp9296p9327.html
Sent from the Apache Flink Mailing List archive. mailing list archive at 
Nabble.com.

Re: withParameters() for Streaming API

[Matthias J. Sax]

We had this discussion a while ago.

If I recall correctly, "withParameters()" is not encourage to be used in
DataSet either.

This is the thread:
https://mail-archives.apache.org/mod_mbox/flink-dev/201509.mbox/%3C55EC69CD.1070003%40apache.org%3E

-Matthias

On 11/24/2015 02:14 PM, Timo Walther wrote:
> Hi all,
> 
> I want to set the Configuration of a streaming operator and access it
> via the open method of the RichFunction.
> There is no possibility to set the Configuration of the open method at
> the moment, right? Can I open an issue for a withParameters() equivalent
> for the Stremaing API?
> 
> Regards,
> Timo
> 



signature.asc
Description: OpenPGP digital signature

Re: withParameters() for Streaming API

[Timo Walther]

Thanks for the hint Matthias.
So actually the parameter of the open() method is useless? IMHO that 
does not look like a nice API design...

We should try to keep DataSet and DataStream API in sync.
Does it make sense to deprecate withParameters() for 1.0?

Timo

On 24.11.2015 14:31, Matthias J. Sax wrote:

We had this discussion a while ago.

If I recall correctly, "withParameters()" is not encourage to be used in
DataSet either.

This is the thread:
https://mail-archives.apache.org/mod_mbox/flink-dev/201509.mbox/%3C55EC69CD.1070003%40apache.org%3E

-Matthias

On 11/24/2015 02:14 PM, Timo Walther wrote:

Hi all,

I want to set the Configuration of a streaming operator and access it
via the open method of the RichFunction.
There is no possibility to set the Configuration of the open method at
the moment, right? Can I open an issue for a withParameters() equivalent
for the Stremaing API?

Regards,
Timo

Re: [DISCUSS] Include import statements in documentation code examples

[Maximilian Michels]

>Also, I'll not add the import statements to ALL examples, only to those
>where people might copy paste them.

That sounds ok to me for now.

On Thu, Nov 19, 2015 at 6:38 PM, Ufuk Celebi  wrote:
> I think it's confusing to only have a subset of import statements provided.
> But then again, the missing ones will be resolved without confusion
> (hopefully) ;) We can go with this and see what feedback we get.
>
> (Just doing it for some examples sounds reasonable.)
>
> – Ufuk
>
> On Thu, Nov 19, 2015 at 5:29 PM, Robert Metzger  wrote:
>
>> Thank you for the feedback.
>>
>> I was also spending some time thinking about automating this, but I don't
>> have the time right now to bring the required infrastructure in place.
>>
>> For now, I'll just add import statements for classes with the potential of
>> confusion (in particular between the Scala and Java API, Hadoop/Flink
>> classes, ..)
>> Also, I'll not add the import statements to ALL examples, only to those
>> where people might copy paste them.
>> Please -1 me if you are against this, otherwise, I'll soon open a PR.
>>
>>
>> On Wed, Nov 18, 2015 at 5:04 PM, Nick Dimiduk  wrote:
>>
>> > In HBase we keep an hbase-examples module with working code. Snippets
>> from
>> > that module are pasted into docs and referenced. Yes, we do see
>> divergence,
>> > especially when refactor tools are involved. I once looked into a doc
>> tool
>> > for automatically extracting snippets from source code, but that turned
>> > into a rat-hole and didn't pursue it further. Maybe tooling has improved
>> > since then?
>> >
>> > On Wednesday, November 18, 2015, Maximilian Michels 
>> > wrote:
>> >
>> > > Hi Robert.
>> > >
>> > > Good suggestion. Generally, it would be nice to have complete code
>> > > examples available in the documentation. Even better, a way to only
>> > > show excerpts of the complete example with the option of copying the
>> > > complete working example.
>> > >
>> > > For instance:
>> > >
>> > > public Example {
>> > >public static void main(String[] args) {
>> > >ExecutionEnvironment env = ...
>> > >
>> > >// BEGIN: example
>> > >env.fromElements(1,2,3,4)
>> > >env.map(element -> element * 2)
>> > >// END: example
>> > >
>> > >env.print();
>> > >}
>> > > }
>> > >
>> > > This still poses the problem that we need to run those examples in an
>> > > automated way to ensure they are actually working.
>> > >
>> > > Cheers,
>> > > Max
>> > >
>> > > On Wed, Nov 18, 2015 at 12:09 PM, Robert Metzger > > > > wrote:
>> > > > Hi,
>> > > >
>> > > > I helped somebody yesterday on SO [1] who had issues with the Scala
>> API
>> > > > because he was importing the classes from the Java API.
>> > > > Somebody else complained about this issue as well in the comments
>> below
>> > > the
>> > > > documentation [2], and I think both users are right: Its an
>> unnecessary
>> > > > obstacle when learning Flink that users have to figure out which
>> class
>> > to
>> > > > import.
>> > > >
>> > > > How about adding import statements to the examples?
>> > > > Is there a nicer way of solving the problem?
>> > > >
>> > > >
>> > > > Regards,
>> > > > Robert
>> > > >
>> > > >
>> > > >
>> > > > [1] http://stackoverflow.com/a/33721528/568695
>> > > > [2]:
>> > > >
>> > >
>> >
>> https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/streaming_guide.html#comment-2365998014
>> > >
>> >
>>

[jira] [Created] (FLINK-3069) Make state materialization asynchronous

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-3069:
---

 Summary: Make state materialization asynchronous
 Key: FLINK-3069
 URL: https://issues.apache.org/jira/browse/FLINK-3069
 Project: Flink
  Issue Type: Improvement
  Components: Streaming
Affects Versions: 0.10.0
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.0.0


The checkpointing technique can work with asynchronous state materialization. 
We should add the option to create state handles that are asynchronously 
materialized, so that stream processing can continue while a checkpoint is 
happening.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

withParameters() for Streaming API

[Timo Walther]

Hi all,

I want to set the Configuration of a streaming operator and access it 
via the open method of the RichFunction.
There is no possibility to set the Configuration of the open method at 
the moment, right? Can I open an issue for a withParameters() equivalent 
for the Stremaing API?


Regards,
Timo

Union a data stream with a product of itself

[Vasiliki Kalavri]

Hi squirrels,

when porting the gelly streaming code from 0.9 to 0.10 today with Paris, we
hit an exception in union: "*A DataStream cannot be unioned with itself*".

The code raising this exception looks like this:
stream.union(stream.map(...)).

Taking a look into the union code, we see that it's now not allowed to
union a stream, not only with itself, but with any product of itself.

First, we are wondering, why is that? Does it make building the stream
graph easier in some way?
Second, we might want to give a better error message there, e.g. "*A
DataStream cannot be unioned with itself or a product of itself*", and
finally, we should update the docs, which currently state that union a
stream with itself is allowed and that "*If you union a data stream with
itself you will still only get each element once.*"

Cheers,
-Vasia.

[jira] [Created] (FLINK-3073) Activate streaming mode by default

[Robert Metzger (JIRA)]

Robert Metzger created FLINK-3073:
-

 Summary: Activate streaming mode by default
 Key: FLINK-3073
 URL: https://issues.apache.org/jira/browse/FLINK-3073
 Project: Flink
  Issue Type: Improvement
  Components: TaskManager
Reporter: Robert Metzger
 Fix For: 1.0.0


Currently, TaskManagers are still started in the batch mode.

I have the impression that more users are actually using Flink for stream 
processing, and, the streaming mode also allows batch workloads.

It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3074) Make ApplicationMaster/JobManager akka port configurable

[Robert Metzger (JIRA)]

Robert Metzger created FLINK-3074:
-

 Summary: Make ApplicationMaster/JobManager akka port configurable
 Key: FLINK-3074
 URL: https://issues.apache.org/jira/browse/FLINK-3074
 Project: Flink
  Issue Type: Improvement
  Components: YARN Client
Reporter: Robert Metzger
Assignee: Robert Metzger
 Fix For: 1.0.0


Similar to the BlobServer, the YARN ApplicationMaster should allow starting it 
on a specified list or range of ports.

In cases where only certain ports are allowed by a firewall, users can specify 
a range of ports where they want the AM to allocate its RPC port



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)