@Robert Thanks! I think the PR will be ready to merge soon :) @Matthias I fixed the finite-source issue on my branch, now every example and ITCase runs and stopps without throwing an exception. Also, in case of finite sources, the spout wrapper will not loop infinitely. I will study your branch and make comments in the afternoon.
Peter 2015-06-01 21:46 GMT+02:00 Robert Metzger <rmetz...@apache.org>: > Great to see that you two are working together on the storm compatibility > layer. > > Please let the other Flink committers know when Matthias PR is in a state > that we can review it again (= when you think its ready). > Given the feedback from Peter and the long list of missing features and the > current rework, I would suggest to merge the storm compatiblilty layer > after the 0.9 release. > The issues from the document Stephan send around two weeks ago are making > good progress, so the release will probably forked-off rather soon. > > On Fri, May 29, 2015 at 2:10 PM, Szabó Péter <nemderogator...@gmail.com> > wrote: > > > Thank you very much, this explains a lot of things :) > > I'm aware of that currently the support of TopologyContext is limited, > so I > > do not expect it to work smoothly. However, there was another issue with > > the grouping by the "id" field, which seemed very strange. Anyway, I will > > live the SimpleJoin example to you, then. I will write if I find out > > something important. > > > > I think the Strom compatibility layer is a really nice extension of the > > Flink streaming API. It would be great if you could add more > functionality > > to it in the future. > > By the way, I'm not completely sure if you know: I was asked by Marton to > > prepare your pull request for merging into the master. I restructured > your > > commits, cleaned up the code and rebased the branch on the current > master. > > Actually, to follow the changes in the behaviour of the Flink streaming > > sources, I'm doing a second refactor right now. If you are working on > > something, and have not rebased yet, you can use my storm branch to > follow > > up. > > > > Here are the links to my storm branches: > > - storm-backup (exmaples and other experiments; currently a little bit > > outdated, because it is supposed to be a backup branch): > > https://github.com/mbalassi/flink/tree/storm-backup > > - storm (last clean state of my work on the flink-storm-compatibility > pull > > request, including code cleanup & refactor and one or two simple > examples): > > https://github.com/mbalassi/flink/tree/storm > > > > Peter > > > > > > 2015-05-29 10:36 GMT+02:00 Matthias J. Sax < > mj...@informatik.hu-berlin.de > > >: > > > > > Hi Peter, > > > > > > I started to look into the issue. However, I could not find the > > > following classes in the git repository: > > > > > > org.apache.flink.stormcompatibility.util.AbstractStormSpout > > > org.apache.flink.stormcompatibility.util.OutputFormatter > > > org.apache.flink.stormcompatibility.util.StormBoltFileSink > > > org.apache.flink.stormcompatibility.util.StormBoltPrintSink > > > org.apache.flink.stormcompatibility.util.TupleOutputFormatter > > > > > > Thus, I cannot compile and run the code. Can you please update the git. > > > > > > However, I had a quick look into the code, and see a few issues right > > > away. For example the code uses "tuple.select(...)" what is not > > > supported so far. Right now, attributes can only be accessed via index. > > > Furthermore, the example uses a lot of meta information that cannot be > > > provided easily in Flink (ie, TopologyContext is only supported very > > > limited). > > > > > > To add those things, I will need much more time. I don't think I should > > > be part of the first pull request, but be added later. I will integrate > > > the SingleJoinBolt example as a ITCase if the functionality is there. > It > > > seems to be a good idea, to add more examples from storm-starter to > > > flink-storm-examples. > > > > > > > > > -Matthias > > > > > > > > > On 05/28/2015 09:37 AM, Szabó Péter wrote: > > > > Hi Matthias, > > > > > > > > Of course, here is the package that contains the example's source > > > classes. > > > > > > > > > > https://github.com/mbalassi/flink/tree/storm-backup/flink-staging/flink-streaming/flink-storm-examples/src/main/java/org/apache/flink/stormcompatibility/singlejoin > > > > It is mostly a copy-paste of SimpleJoin from storm-starter, though, I > > > > separated the spouts and the join bolt from the rest of the topology. > > > > I would be happy, if you could fix it. Probably I'm overlooking > > > something. > > > > > > > > Peter > > > > > > > > 2015-05-27 17:13 GMT+02:00 Matthias J. Sax < > > > mj...@informatik.hu-berlin.de>: > > > > > > > >> Hi Peter, > > > >> > > > >> Thanks a lot for your feedback. It's exiting to see, that somebody > > uses > > > >> the layer already. :) > > > >> > > > >> The current prototype is going to be merged soon. However, I am more > > > >> than happy to extend the functionality of the layer. Can you please > > > >> share your example with me, so I can see what the problem is and fix > > it? > > > >> > > > >> I am pretty sure, that the fix will be merged later on, too. There > are > > > >> many other limitation in the layer. Right now, it is still in beta > > > state. > > > >> ;) > > > >> > > > >> -Matthias > > > >> > > > >> > > > >> On 05/27/2015 03:48 PM, Szabó Péter wrote: > > > >>> Hey everyone, > > > >>> > > > >>> I experimented with the Storm compatibility layer Matthias wrote, > and > > > ran > > > >>> some Storm examples on Flink. I found that Storm's SimpleJoin > example > > > >> does > > > >>> not work. I suppose it is because of the multiple input streams. > I'm > > > >>> willing to add another example instead. > > > >>> Right now, I'm getting it through Aljoscha's streaming refactor. > > > >>> > > > >>> Peter > > > >>> > > > >> > > > >> > > > > > > > > > > > > >
