Re: [VOTE] Accept Stateful Functions into Apache Flink

[Vasiliki Kalavri]

+1 (binding) from me. I hope this is not too late :)

Thank you for this great contribution!

On Wed, 30 Oct 2019 at 14:45, Stephan Ewen  wrote:

> Thank you all for voting.
>
> The voting period has passed, but only 13 PMC members have voted so far,
> that is less than 2/3rd of the PMCs (17 members).
>
> I will take a few days to ping other members to vote, after that we will
> gradually lower the threshold as per the process to account for inactive
> members.
>
> Best,
> Stephan
>
>
>
>
> On Tue, Oct 29, 2019 at 6:20 PM Seth Wiesman  wrote:
>
> > +1 (non-binding)
> >
> > Seth
> >
> > > On Oct 23, 2019, at 9:31 PM, Jingsong Li 
> wrote:
> > >
> > > +1 (non-binding)
> > >
> > > Best,
> > > Jingsong Lee
> > >
> > >> On Wed, Oct 23, 2019 at 9:02 PM Yu Li  wrote:
> > >>
> > >> +1 (non-binding)
> > >>
> > >> Best Regards,
> > >> Yu
> > >>
> > >>
> > >>> On Wed, 23 Oct 2019 at 16:56, Haibo Sun  wrote:
> > >>>
> > >>> +1 (non-binding)Best,
> > >>> Haibo
> > >>>
> > >>>
> > >>> At 2019-10-23 09:07:41, "Becket Qin"  wrote:
> >  +1 (binding)
> > 
> >  Thanks,
> > 
> >  Jiangjie (Becket) Qin
> > 
> >  On Tue, Oct 22, 2019 at 11:44 PM Tzu-Li (Gordon) Tai <
> > >> tzuli...@apache.org
> > 
> >  wrote:
> > 
> > > +1 (binding)
> > >
> > > Gordon
> > >
> > > On Tue, Oct 22, 2019, 10:58 PM Zhijiang <
> wangzhijiang...@aliyun.com
> > > .invalid>
> > > wrote:
> > >
> > >> +1 (non-binding)
> > >>
> > >> Best,
> > >> Zhijiang
> > >>
> > >>
> > >> --
> > >> From:Zhu Zhu 
> > >> Send Time:2019 Oct. 22 (Tue.) 16:33
> > >> To:dev 
> > >> Subject:Re: [VOTE] Accept Stateful Functions into Apache Flink
> > >>
> > >> +1 (non-binding)
> > >>
> > >> Thanks,
> > >> Zhu Zhu
> > >>
> > >> Biao Liu  于2019年10月22日周二 上午11:06写道：
> > >>
> > >>> +1 (non-binding)
> > >>>
> > >>> Thanks,
> > >>> Biao /'bɪ.aʊ/
> > >>>
> > >>>
> > >>>
> >  On Tue, 22 Oct 2019 at 10:26, Jark Wu  wrote:
> > 
> >  +1 (non-binding)
> > 
> >  Best,
> >  Jark
> > 
> >  On Tue, 22 Oct 2019 at 09:38, Hequn Cheng  > >>>
> > >> wrote:
> > 
> > > +1 (non-binding)
> > >
> > > Best, Hequn
> > >
> > > On Tue, Oct 22, 2019 at 9:21 AM Dian Fu <
> > >> dian0511...@gmail.com>
> > >> wrote:
> > >
> > >> +1 (non-binding)
> > >>
> > >> Regards,
> > >> Dian
> > >>
> > >>> 在 2019年10月22日，上午9:10，Kurt Young  写道：
> > >>>
> > >>> +1 (binding)
> > >>>
> > >>> Best,
> > >>> Kurt
> > >>>
> > >>>
> > >>> On Tue, Oct 22, 2019 at 12:56 AM Fabian Hueske <
> > >> fhue...@gmail.com>
> > >> wrote:
> > >>>
> >  +1 (binding)
> > 
> >  Am Mo., 21. Okt. 2019 um 16:18 Uhr schrieb Thomas Weise <
> > > t...@apache.org
> > >>> :
> > 
> > > +1 (binding)
> > >
> > >
> > > On Mon, Oct 21, 2019 at 7:10 AM Timo Walther <
> > >> twal...@apache.org
> > 
> > >> wrote:
> > >
> > >> +1 (binding)
> > >>
> > >> Thanks,
> > >> Timo
> > >>
> > >>
> > >>> On 21.10.19 15:59, Till Rohrmann wrote:
> > >>> +1 (binding)
> > >>>
> > >>> Cheers,
> > >>> Till
> > >>>
> > >>> On Mon, Oct 21, 2019 at 12:13 PM Robert Metzger <
> > > rmetz...@apache.org
> > >
> > >> wrote:
> > >>>
> >  +1 (binding)
> > 
> >  On Mon, Oct 21, 2019 at 12:06 PM Stephan Ewen <
> > >>> se...@apache.org
> > >
> > > wrote:
> > 
> > > This is the official vote whether to accept the
> > >>> Stateful
> > > Functions
> > > code
> > > contribution to Apache Flink.
> > >
> > > The current Stateful Functions code, documentation,
> > >>> and
> > >>> website
> > > can
> > > be
> > > found here:
> > > https://statefun.io/
> > > https://github.com/ververica/stateful-functions
> > >
> > > This vote should capture whether the Apache Flink
> > > community
> > >>> is
> > >> interested
> > > in accepting, maintaining, and evolving Stateful
> > > Functions.
> > >
> > > Reiterating my original motivation, I believe that
> > >>> this
> > >>> project
> > > is
> >  a
> >  

Re: [DISCUSS] Gelly planning for release 1.3 and roadmap

[Vasiliki Kalavri]

Hi Xingcan,

thank you for your input!

On 27 February 2017 at 14:03, Xingcan Cui <xingc...@gmail.com> wrote:

> Hi Vasia and Greg,
>
> thanks for the discussion. I'd like to share my thoughts.
>
> 1) I don't think it's necessary to extend the algorithm list intentionally.
> It's just like a textbook that can not cover all the existing algorithms
> (even if we can). Just representative and commonly used ones will be
> enough. After all, Gelly is mainly designed for providing a framework
> rather than an algorithm library. Besides, it seems that Gelly's API is not
> stable now and thus a huge work of refactoring or even rewriting will rise
> once the API changes.
>

​In fact, Gelly's main APIs haven't changed much throughout the last 3
releases.
Maybe it's time we explicitly mark stable API​s for 1.3.



>
> 2) Unlike other "pure" graph computing framework (e.g. giraph), Gelly is
> built on top of Flink, which means that it can only use operations that
> provided by it. In my own opinion, Flink's batch processing is not so
> outstanding as it's stream. As Grey said, one problem lies on intermediate
> results caching. Though it's not clear for me (I'm still a ignorant new
> comer...) why this feature has not been implemented for such a long time,
> there must be some reasons. What I see is that, to some extent, it's
> already obstructed Gelly's development. From this point of view,
> self-blessing is better than blessing from others and I'm sure some MLers
> may be more anxious than us :) So, I guess "within Gelly" just means a
> Gelly-driven development?
>
> In a nutshell, I will encourage more concentrations on Gelly's API (or even
> related Flink's API if necessary), rather than high-level things (e.g.
> algorithms, performance) on top of it. What if we can change both the
> edges' values and vertices' values during an iteration one day? :)
>

​Changing both edge and vertex values during an iteration has also been
brought up before.
This one could be implemented by providing an alternative representation of
the graph (e.g. adjacency list)
and would (hopefully) leave existing iteration APIs unchanged. I'm onboard
with adding this to the roadmap​.

Best,
-Vasia.


>
> Best,
> Xingcan
>
>
> On Sat, Feb 25, 2017 at 2:43 AM, Vasiliki Kalavri <
> vasilikikala...@gmail.com
> > wrote:
>
> > Hi Greg,
> >
> > On 24 February 2017 at 18:09, Greg Hogan <c...@greghogan.com> wrote:
> >
> > > Thanks, Vasia, for starting the discussion.
> > >
> > > I was expecting more changes from the recent discussion on
> restructuring
> > > the project, in particular regarding the libraries. Gelly has always
> > > collected algorithms and I have personally taken an algorithms-first
> > > approach for contributions. Is that manageable and maintainable? I'd
> > prefer
> > > to see no limit to good contributions, and if necessary split the
> > codebase
> > > or the project.
> > >
> >
> > ​I don't think there should be a limit either. I do think though that
> > development should be community-driven, i.e. not making contributions
> just
> > for the sake of it, but evaluating their benefit first.
> > The library already has a quite long list of algorithms. Shall we keep on
> > extending it? And if yes, how do we choose which algorithms to add? Do we
> > accept any algorithm even if it hasn't been asked by anyone? So far,
> we've
> > added algorithms that we thought were useful and common. But continuing
> to
> > extend the library like this doesn't seem maintainable to me, because we
> > might end up with a lot of code to maintain that nobody uses. On the
> other
> > hand, adding more algorithms might attract more users, so I see a
> trade-off
> > there.
> >
> >
> > >
> > > If so, then a secondary goal is to make the algorithms user-accessible
> > and
> > > easier to review (especially at scale!). FLINK-4949 rewrites
> > > flink-gelly-examples with modular inputs and algorithms, allows users
> to
> > > run all existing algorithms, and makes it trivial to create a driver
> for
> > > new algorithms (and when comparing different implementations).
> > >
> >
> > ​I'm +1 for anything that makes using existing functionality easier.
> > FLINK-4949 sounds like a great addition. Could you maybe extend the JIRA
> > and/or PR description a bit? I understand the rationale but it would be
> > nice to have a high-level description of the changes and the new
> > functionality that the PR adds or the interfaces it modifies. Otherwise,
> it
> > will be difficult to re

Re: [DISCUSS] Gelly planning for release 1.3 and roadmap

[Vasiliki Kalavri]

Hi Greg,

On 24 February 2017 at 18:09, Greg Hogan <c...@greghogan.com> wrote:

> Thanks, Vasia, for starting the discussion.
>
> I was expecting more changes from the recent discussion on restructuring
> the project, in particular regarding the libraries. Gelly has always
> collected algorithms and I have personally taken an algorithms-first
> approach for contributions. Is that manageable and maintainable? I'd prefer
> to see no limit to good contributions, and if necessary split the codebase
> or the project.
>

​I don't think there should be a limit either. I do think though that
development should be community-driven, i.e. not making contributions just
for the sake of it, but evaluating their benefit first.
The library already has a quite long list of algorithms. Shall we keep on
extending it? And if yes, how do we choose which algorithms to add? Do we
accept any algorithm even if it hasn't been asked by anyone? So far, we've
added algorithms that we thought were useful and common. But continuing to
extend the library like this doesn't seem maintainable to me, because we
might end up with a lot of code to maintain that nobody uses. On the other
hand, adding more algorithms might attract more users, so I see a trade-off
there.


>
> If so, then a secondary goal is to make the algorithms user-accessible and
> easier to review (especially at scale!). FLINK-4949 rewrites
> flink-gelly-examples with modular inputs and algorithms, allows users to
> run all existing algorithms, and makes it trivial to create a driver for
> new algorithms (and when comparing different implementations).
>

​I'm +1 for anything that makes using existing functionality easier.
FLINK-4949 sounds like a great addition. Could you maybe extend the JIRA
and/or PR description a bit? I understand the rationale but it would be
nice to have a high-level description of the changes and the new
functionality that the PR adds or the interfaces it modifies. Otherwise, it
will be difficult to review a PR with +5k line changes :)



>
> Regarding BipartiteGraphs, without algorithms or ideas for algorithms it's
> not possible to review the structure of the open pull requests.
>


​I'm not sure I understand this point. There was a design document and an
extensive discussion on this issue. Do you think we should revisit? Some
common algorithms for bipartitite graphs that I am aware of is SALSA for
recommendations and relevance search for anomaly detection.



>
> +1 to evaluating performance and promoting Flink!
>
> Gelly has two shepherds whereas CEP and ML share one committer. New
> algorithms in Gelly require new features in the Batch API (Gelly may also
> start doing streaming, we're cool kids, too)


​^^​


> so we need to find a process
> for snuffing ideas early and for the right balance in dependence on core
> committers' time. For example, reworking the iteration scheduler to allow
> for intermediate outputs and nested iterations. Can this feature be
> developed and reviewed within Gelly?

Does it need the blessing of a Stephan
> or Fabian? I'd like to see contributors and committers less dependent on
> the core team and more autonomous.
>

​What do you mean
​developed and reviewed ​
"within Gelly"?
​This feature would require changes in the batch iterations code and will
probably need to be proposed and reviewed as a FLIP, so it would need the
blessing of the community :)

Having someone who is more familiar with this part of the code help is of
course favorable, but I don't think it's absolutely necessary.

​-V.​


> Greg
>
> On Fri, Feb 24, 2017 at 10:39 AM, Vasiliki Kalavri <
> vasilikikala...@gmail.com> wrote:
>
> > Hello squirrels,
> >
> > this is a discussion thread to organize the Gelly component development
> for
> > release 1.3 and discuss longer-term plans for the library.
> >
> > I am hoping that with time-based releases, we can distribute the load for
> > PR reviewing and make better use of our time, and also point contributors
> > to "useful" tickets when they offer to help.
> >
> > I'm expecting the outcome of this discussion to be:
> >
> > (1) a set of open PRs to review and try merging for 1.3
> > (2) a set of open JIRAs to work-on before feature freeze
> > (3) a set of JIRAs and PRs to reorganize/close
> > (4) ideas on possible FLIPs
> >
> > Here's my initial take on things, i.e. features *I* see as important in
> the
> > short-term. Feel free to add/remove/discuss:
> >
> > Release 1.3
> > ==
> > - Bipartite graph support. Initial support has been added, but there
> > are unreviewed
> > PRs
> > <https://github.com/apache/flink/pulls?utf8=%E2%9C%93=
> > is%3Apr%20is%3Aopen%20bipartite%20>
> >

[DISCUSS] Gelly planning for release 1.3 and roadmap

[Vasiliki Kalavri]

Hello squirrels,

this is a discussion thread to organize the Gelly component development for
release 1.3 and discuss longer-term plans for the library.

I am hoping that with time-based releases, we can distribute the load for
PR reviewing and make better use of our time, and also point contributors
to "useful" tickets when they offer to help.

I'm expecting the outcome of this discussion to be:

(1) a set of open PRs to review and try merging for 1.3
(2) a set of open JIRAs to work-on before feature freeze
(3) a set of JIRAs and PRs to reorganize/close
(4) ideas on possible FLIPs

Here's my initial take on things, i.e. features *I* see as important in the
short-term. Feel free to add/remove/discuss:

Release 1.3
==
- Bipartite graph support. Initial support has been added, but there
are unreviewed
PRs

and there is no Scala API yet. It would be nice to organize this feature,
decide what functionality we need and what functionality is already covered
by the Graph type and have proper bipartite support for 1.3.
- Driver improvements, i.e. #3294

- Algorithm improvements, #2733 
- Affinity Propagation algorithm. This one has been developed using a bulk
iteration plan and needs a review. The PR is #2885
.
- Object reuse issues, FLINK-5890, FLINK-5891
- Vertex-centric iteration improvement, i.e. FLINK-5127


Roadmap

Regarding longer-term plans, I see the following issues as still being
relevant from the existing roadmap [1]:
- Extending the iteration functionality to support algorithms, more complex
than value-propagation, e.g. with nested loops
- Partitioning methods
- Partition-centric iterations
- Performance evaluation

These two lists are by no means complete or final and the goal of this
thread is to see what the community is interested in, whether these
features / additions make sense to be worked on, or what features are
missing.
So, please provide your feedback!

Cheers,
-V.

[1]: https://cwiki.apache.org/confluence/display/FLINK/Flink+Gelly

Re: ecosystem proposal, Temporal graph analytics on Flink with Gelly

[Vasiliki Kalavri]

Hi Wouter,

thank you for sharing your work with the community! When your project is
ready, I would suggest you open a pull request to the flink-web repository
[1] to add it to the list.
It would be nice to add a description and a readme file to your repository.
Among others, you could include the project's goal (what problem is this
library solving?), maybe a link to your thesis and related work, some
information on how this library is using Flink, and usage information for
users, e.g. how to run an example, use a provided algorithm with test data,
etc.

Cheers,
-Vasia.

[1]: https://github.com/apache/flink-web

On 7 February 2017 at 15:44, Wouter Ligtenberg 
wrote:

> I forgot to attach the github page of the project:
>
> https://github.com/otherwise777/Temporal_Graph_library
>
> Best regards,
> Wouter Ligtenberg
> Game designer at Onzichtbaar Developers
>
>
> On Tue, Feb 7, 2017 at 3:41 PM, Wouter Ligtenberg 
> wrote:
>
> > Hello,
> >
> > For my master thesis i have created Tink, abrevated for temporal Flink,
> > It's a library that's supposed to work on top of Flink in relation with
> > Gelly to do temporal graph analytics.
> > I've worked on this project for the past 6 months as part of my master
> > thesis at the university of Eindhoven.
> > I read on [1] that i could propose my project to have it listed on the
> > flink project website under the eco system page.
> > Could my project also be listed on the web page?
> >
> > Please note that it's not completely done yet, at some point I had to
> > start writing my thesis, which is done now.
> >
> > [1] https://flink.apache.org/ecosystem.html
> >
> >
> > Best regards,
> > Wouter Ligtenberg
> > Game designer at Onzichtbaar Developers
> >
> >
>

Re: [DISCUSS] Time-based releases in Flink

[Vasiliki Kalavri]

Hi Robert,

thanks a lot for starting this discussion and for putting together the wiki
pages.
This proposal makes a lot of sense to me.

Big +1 for merging only features which are tested and *documented*.

I believe that having a clear timeline will not only make users happier but
also contributors more engaged. With the current unpredictability, it is
hard to book time aside to help with testing a release candidate. I hope
that knowing exactly when that needs to happen will help us plan our own
time better and help out more.

Cheers,
-Vasia.

On 18 January 2017 at 09:57, Tzu-Li (Gordon) Tai 
wrote:

> Hi Robert,
>
> Thanks for bringing up the discussion. I like the proposal.
>
> Regarding some of the downsides mentioned in the wiki:
>
> 1. Features that don’t make it in time with the feature freeze:
> I think that’s ok, as long as we’re consistent with the schedules for the
> next release. This way users waiting for that particular features will
> still be able to rely on the fact that the feature will be included in 4
> months.
>
> 2. Frequent releases mean bug fix releases for older branches:
> You mentioned in the email that “old releases are supported for 6 months
> by the community”, but not in the wiki. If this is strictly followed, that
> means we’ll at most be supporting 2 previous major release versions (ex. as
> soon as 1.4.0 comes out, we’ll still be supporting bugfixes for 1.3.0, as
> well as 1.2.0 for another 2 months).
> This might seem a bit odd, so perhaps we can stick to something like
> “support bugfixes for the previous 2 major releases”? Ex. Once 1.4.0 comes
> out, we’ll continue to support only 1.4.0 and 1.3.0.
> Supporting bugfixes for 2 major versions seems workable, and this way
> users can also have a “buffer” that they should not fall behind releases
> for more than 2 major versions (8 months) and preplan upgrades.
>
> - Gordon
>
> On January 18, 2017 at 9:19:41 AM, Robert Metzger (rmetz...@apache.org)
> wrote:
>
> Hi all!
>
> Since the 1.2.0 release is about to come out, I would like to propose a
> change in the way we do releases in the Flink community.
>
> In my opinion, the current model leads to dissatisfaction among users and
> contributors, because releases are really not predictable. A recent example
> for the issues with our current model are the FLIP-6 changes we wanted to
> merge to master, a few weeks before the first RC for 1.2.0. Also, there
> were some emails on the mailing lists asking for a release date.
>
> In order to change that, I’m proposing to follow a strictly time-based
> release model. Other open source projects like Ubuntu, Cassandra, Spark or
> Kafka are following that model as well, and I think we should try it out as
> an experiment for the 1.3.0 release.
>
> I’m proposing to:
>
> -
>
> Do a Flink release every 4 months
> -
>
> Cycle:
> -
>
> 3 months development
> -
>
> 1 month before the release: Feature freeze. Create release branch
> with RC0, start testing. Only fixes, tests and minor improvements are
> allowed in.
> -
>
> 2 weeks before the release: Code freeze. Start voting. Only fixes for
> blockers are allowed in.
> -
>
> Forbid blocking a release because a feature is not done yet.
> -
>
> Features are merged to master, when they are tested and documented, to
> have an always stable master
> -
>
> Bugfix releases are done as needed.
> -
>
> Old releases are supported for 6 months by the Flink community with
> critical bug fixes
>
>
> This means, that we would have the following release dates:
>
> (Flink 1.3.0 by end of January 2017)
>
> Flink 1.4.0 by end of May 2017
>
> Flink 1.5.0 by end of September 2017
>
> Flink 1.6.0 by end of January 2018
>
> I’ve put some more details including some pro’s and con’s into our wiki.
> The page is based on Kafka’s time-based release wiki page (Kafka also
> recently started following a strictly time-based model)
>
> https://cwiki.apache.org/confluence/display/FLINK/Time-based+releases
>
>
> Once we’ve agreed on following that model, I’ll update the release plan
> page accordingly:
> https://cwiki.apache.org/confluence/display/FLINK/
> Flink+Release+and+Feature+Plan
>
>
> Please let me know what you think about this idea!
>
> Regards,
>
> Robert
>

Re: Taking time off

[Vasiliki Kalavri]

Hi Max,

thank you for all your work! Enjoy your time off and hope to have you back
with us soon ^^

Cheers,
-Vasia.

On 14 January 2017 at 09:03, Maximilian Michels  wrote:

> Dear Squirrels,
>
> Thank you! It's been very exciting to see the Flink community grow and
> flourish over the past two years.
>
> For the beginning of this year, I decided to take some time off, which
> means I'll be less engaged on the mailing list or on GitHub/JIRA.
>
> In the meantime, if you have any questions I might be able to answer, feel
> free to contact me. Looking forward to see the squirrels rise further!
>
> Best,
> Max
>

Re: Request for edit rights in Jira

[Vasiliki Kalavri]

Hi!

I've given you contributor rights and assigned the issue to you. You can
also assign issues to yourself now :)

-Vasia.

On 16 November 2016 at 17:37, limcheehau lim  wrote:

> Hi folks,
>
> I would like to request for edit rights in Jira so that I could assign the
> issue that I'm working on (https://issues.apache.org/
> jira/browse/FLINK-3551)
> to myself.
>
> My Jira username is ch33hau
>
> Thanks =)
> Regards,
> Chee Hau
>

Re: A master project about implementing Cypher on Apache Flink

[Vasiliki Kalavri]

Hi Mengqi,

thank you for sharing your work with the community!

-Vasia.

On 14 October 2016 at 23:48, Mengqi Yang  wrote:

> Hi all,
>
> Let me introduce me first. My name is Mengqi Yang.  I was a master student
> in Eindhoven University of Technology. Recently I have finished my master
> project there (Vasia is also one of my supervisors, great thanks to her :)
> ).
>
> The name of this project is  "a study of execution strategies for
> openCypher on Apache Flink". In this project, I have implemented some
> operators of Cypher (Cypher is a graph query language), mainly the core
> operators, by using Flink batch API. One can easily build their own graph
> queries by using these operators. Besides, two graph query optimizers are
> also implemented for providing  different optimization strategies to the
> graph queries. The whole project has been done by using Apache Flink, so I
> would like to share my code and my thesis with the Flink community.
>
> Please check the following link of my code. The thesis also has been
> uploaded to the repository:
> https://github.com/jiujieti/CypherImplementation
> More details about this project could be found there.
>
> If anyone is interested in this work or wants to continue this project or
> has some questions, you can always contact me by my email
> melody2014ymq@gmail.
>
> Best regards,
> Mengqi
>

Re: Re: Flink Gelly

[Vasiliki Kalavri]

trySet()) {
> for (String value : entry.getValue()) {
> if (value.contains(oriTemp)) {
> setsWithOrigin.add(value);
> }
> if (value.contains(destTemp)) {
> setsWithDestination.add(value);
> }
> }
> }
> for (String originIter : setsWithOrigin) {
> for (String destinationIter : setsWithDestination) {
> String concat = "";
> if ((originIter.indexOf(oriTemp) == 0 &&
> destinationIter
> .indexOf(destTemp) == 0)) {
> String reverse = destinationIter;
> if (destinationIter.length() > 1) {
> reverse = "";
> int d = destinationIter.length();
> for (int a = 0; a <
> destinationIter.length(); a++) {
> reverse = reverse
> + destinationIter.substring(d
> - 1,
> d);
> d--;
> }
> }
> concat = originIter + ";" + vertex.getId() +
> ";"
> + reverse;
> }
> if (isFormatValid(concat) && concat.length() > 0) {
> if (!tempList.contains(concat)) {
> tempList.add(concat);
> }
> }
>     }
> }
> multiPaths.put(oriDest, tempList);
> }
>
>
> // The combined paths are also saved into a HashMap which is
> additionally set as a Vertex Value
> // Later the paths are filtered for redundance
>
> Tuple2<HashMap<String, List>, HashMap<Integer,
> List>> testTuple3 = new Tuple2<HashMap<String, List>,
> HashMap<Integer, List>>(
> multiPaths, newHashMap);
>
>
> setNewVertexValue(testTuple3);
> }
> }
>
>
> Let me know if you need any further information.
> Thanks in advance.
>
> All the best,
> Dennis
>
>
> Gesendet: Donnerstag, 06. Oktober 2016 um 15:22 Uhr
> Von: "Vasiliki Kalavri" <vasilikikala...@gmail.com>
> An: dev@flink.apache.org
> Betreff: Re: Flink Gelly
> Hi Dennis,
>
> can you give us some details about your setup? e.g. where you are running
> your job, your input size, the configured memory, etc. It would also be
> helpful if you could share your code. Getting an out of memory error with
> just 100 nodes seems weird.
>
> Best,
> -Vasia.
>
> On 6 October 2016 at 13:29, <d...@web.de> wrote:
>
> >
> > Dear ladies and gentlemen,
> >
> > I got a problem using Gelly in Flink. Currently I am loading a Virtuoso
> > Graph into
> > Flink's Gelly and I want to analyze it for the different paths one can
> > take to link
> > the different nodes. Therefore I am using the ScatterGatherIteration.
> > However, my code just works with about ten to twenty nodes. When I try to
> > upload
> > a hundred nodes, the following error occurs:
> >
> > Exception in thread "main" org.apache.flink.runtime.
> > client.JobExecutionException: Job execution failed.
> > at org.apache.flink.runtime.jobmanager.JobManager$$
> > anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply$
> > mcV$sp(JobManager.scala:822)
> > at org.apache.flink.runtime.jobmanager.JobManager$$
> > anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply(
> JobManager.scala:768)
> > at org.apache.flink.runtime.jobmanager.JobManager$$
> > anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply(
> JobManager.scala:768)
> > at scala.concurrent.impl.Future$PromiseCompletingRunnable.
> > liftedTree1$1(Future.scala:24)
> > at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(
> > Future.scala:24)
> > at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
> > at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(
> > AbstractDispatcher.scala:401)
> > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> > at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.
> > runT

Re: Flink Gelly

[Vasiliki Kalavri]

Hi Dennis,

can you give us some details about your setup? e.g. where you are running
your job, your input size, the configured memory, etc. It would also be
helpful if you could share your code. Getting an out of memory error with
just 100 nodes seems weird.

Best,
-Vasia.

On 6 October 2016 at 13:29,  wrote:

>
> Dear ladies and gentlemen,
>
> I got a problem using Gelly in Flink. Currently I am loading a Virtuoso
> Graph into
> Flink's Gelly and I want to  analyze it for the different paths one can
> take to link
> the different nodes. Therefore I am using the ScatterGatherIteration.
> However, my code just works with about ten to twenty nodes. When I try to
> upload
> a hundred nodes, the following error occurs:
>
> Exception in thread "main" org.apache.flink.runtime.
> client.JobExecutionException: Job execution failed.
> at org.apache.flink.runtime.jobmanager.JobManager$$
> anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply$
> mcV$sp(JobManager.scala:822)
> at org.apache.flink.runtime.jobmanager.JobManager$$
> anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply(JobManager.scala:768)
> at org.apache.flink.runtime.jobmanager.JobManager$$
> anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply(JobManager.scala:768)
> at scala.concurrent.impl.Future$PromiseCompletingRunnable.
> liftedTree1$1(Future.scala:24)
> at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(
> Future.scala:24)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(
> AbstractDispatcher.scala:401)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.
> runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(
> ForkJoinPool.java:1979)
> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(
> ForkJoinWorkerThread.java:107)
> Caused by: java.lang.RuntimeException: Memory ran out. Compaction failed.
> numPartitions: 32 minPartition: 1 maxPartition: 431 number of overflow
> segments: 0 bucketSize: 251 Overall memory: 45613056 Partition memory:
> 33685504 Message: null
> at org.apache.flink.runtime.operators.hash.CompactingHashTable.
> insertRecordIntoPartition(CompactingHashTable.java:457)
> at org.apache.flink.runtime.operators.hash.CompactingHashTable.
> insertOrReplaceRecord(CompactingHashTable.java:392)
> at org.apache.flink.runtime.iterative.io.SolutionSetUpdateOutputCollect
> or.collect(SolutionSetUpdateOutputCollector.java:54)
> at org.apache.flink.graph.spargel.GatherFunction.setNewVertexValue(
> GatherFunction.java:123)
> at org.apache.flink.quickstart.PathRank$PathUpdateFunction.
> updateVertex(PathRank.java:357)
> at org.apache.flink.graph.spargel.ScatterGatherIteration$
> GatherUdfSimpleVV.coGroup(ScatterGatherIteration.java:389)
> at org.apache.flink.runtime.operators.CoGroupWithSolutionSetSecondDr
> iver.run(CoGroupWithSolutionSetSecondDriver.java:218)
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:486)
> at org.apache.flink.runtime.iterative.task.AbstractIterativeTask.run(
> AbstractIterativeTask.java:146)
> at org.apache.flink.runtime.iterative.task.IterationTailTask.run(
> IterationTailTask.java:107)
> at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:351)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
> at java.lang.Thread.run(Thread.java:745)
>
>
> I tried to google it a bit, and this problems seems to occur often when
> using Gelly. I hope you have any ideas or approaches how I can handle this
> error.
>
> Thank you in advance!
> All the best,
> Dennis
>

Re: ClassNotFoundException

[Vasiliki Kalavri]

Hi Olga,

have you copied the flink-gelly jar to your TaskManagers? See the
instructions in [1] on how you can make the gelly classes available during
execution and let us know if you have any problems.

Best,
-Vasia,

[1]:
https://ci.apache.org/projects/flink/flink-docs-master/dev/cluster_execution.html#linking-with-modules-not-contained-in-the-binary-distribution

On 28 September 2016 at 20:58, Olga Golovneva  wrote:

> Hi devs,
>
> I'm trying to run Gelly ClusteringCoefficient example on a cluster using
> this command:
>
> flink run -C
> file://home/ubuntu/flink/flink-libraries/flink-gelly/
> target/flink-gelly_2.10-1.2-SNAPSHOT.jar
> -c org.apache.flink.graph.examples.ClusteringCoefficient
> flink-gelly-examples_2.10-1.2-SNAPSHOT.jar
>
> and get the following ClassNotFoundException:
>
> 
>
>  The program finished with the following exception:
>
>
> java.lang.RuntimeException: Could not look up the main(String[]) method
> from the class org.apache.flink.graph.examples.ClusteringCoefficient:
> org/apache/flink/graph/GraphAnalytic
>
> at
> org.apache.flink.client.program.PackagedProgram.
> hasMainMethod(PackagedProgram.java:478)
>
> at
> org.apache.flink.client.program.PackagedProgram.
> (PackagedProgram.java:216)
>
> at org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:829)
>
> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:222)
>
> at
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1002)
>
> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1045)
>
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/flink/graph/GraphAnalytic
>
> at java.lang.Class.getDeclaredMethods0(Native Method)
>
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2625)
>
> at java.lang.Class.getMethod0(Class.java:2866)
>
> at java.lang.Class.getMethod(Class.java:1676)
>
> at
> org.apache.flink.client.program.PackagedProgram.
> hasMainMethod(PackagedProgram.java:473)
>
> ... 5 more
>
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.graph.GraphAnalytic
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>
> ... 10 more
>
> __
>
> I got similar Exceptions for other examples in Gelly library, only the
> missing methods are different. Perhaps someone more knowledgeable could
> kindly help? Where should I look for org.apache.flink.graph.GraphAnalytic?
> I thought it should be in flink-gelly_2.10-1.2-SNAPSHOT.jar, but it does
> not work
>
> Thank you,
> Olga
>

Re: On (FLINK-1526) JIRA issue

[Vasiliki Kalavri]

Exactly :) That's why we haven't added neither the spanning tree nor the
strongly connected components algorithms yet.

On Sep 22, 2016 12:16 PM, "Stephan Ewen" <se...@apache.org> wrote:

> Just as a general comment:
>
> A program with nested loops is most likely not going to be performant on
> any way. It makes sense to re-think the algorithm, come up with a modified
> or different pattern, rather than trying to implement the exact algorithm
> line by line.
>
> It may be worth checking that, because I am not sure if Gelly should have
> algorithms that don't perform well.
>
> On Thu, Sep 22, 2016 at 11:40 AM, Vasiliki Kalavri <
> vasilikikala...@gmail.com> wrote:
>
> > Hi Olga,
> >
> > when you use mapEdges() or mapVertices() with generics, Flink cannot
> > determine the type because of type erasure, like the exception says.
> That's
> > why we also provide methods that take the type information as a
> parameter.
> > You can use those to make the return type explicit. In your example, you
> > should do something like the following (line 41):
> >
> > final TypeInformation longType = BasicTypeInfo.LONG_TYPE_INFO;
> > final TypeInformation doubleType = BasicTypeInfo.DOUBLE_TYPE_
> INFO;
> > Graph<Long, String, Tuple3<Double, Long, Long>> graphOut =
> > graph.mapEdges(new InitializeEdges<Long, Double>(), new
> > TupleTypeInfo(longType, longType,
> > new TupleTypeInfo<Tuple3<Double, Long, Long>>(doubleType,
> > longType,longType)));
> >
> > Regarding the nested loops, I am almost sure that you will face problems
> if
> > you try to experiment with large datasets. I haven't looked into your
> code
> > yet, but according to the JIRA discussion, we've faced this problem
> before
> > and afaik, this is still an issue.
> >
> > Cheers,
> > -Vasia.
> >
> > On 22 September 2016 at 01:12, Olga Golovneva <melcha...@gmail.com>
> wrote:
> >
> > > Hi Vasia,
> > >
> > > I have uploaded these tests on github:
> > > https://github.com/OlgaGolovneva/MST/tree/master/tests
> > >
> > > I have also uploaded source code, but I'm still working on it:
> > > https://github.com/OlgaGolovneva/MST/tree/master/src
> > >
> > > ​>I think you cannot add attachments to the mailing list. Could you
> > upload
> > > >your example somewhere and post a link here? I'm actually surprised
> that
> > > >the while-loop works without problems.
> > >
> > > I have run the program on several simple tests, and I was going to try
> > > large datasets in the next few days. Please, let me know if this
> approach
> > > is wrong.
> > >
> > > Thanks,
> > > Olga
> > >
> > > On Wed, Sep 21, 2016 at 4:55 PM, Vasiliki Kalavri <
> > > vasilikikala...@gmail.com
> > > > wrote:
> > >
> > > > Hi Olga,
> > > >
> > > > On 21 September 2016 at 18:50, Olga Golovneva <melcha...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi devs,
> > > > >
> > > > > I was working on  (FLINK-1526) "Add Minimum Spanning Tree library
> > > method
> > > > > and example" issue. I've developed (Java) code that implements
> > > > distributed
> > > > > Boruvka's algorithm in Gelly library. I've run several tests and it
> > > seems
> > > > > to work fine, although I didn't test it on extremely large input
> > graphs
> > > > > yet, and I'm also trying to optimize my code.
> > > > > Particularly, I have two main issues:
> > > > >
> > > > > 1. Nested loops.
> > > > > I have to use nested loops, and I do not see the way to avoid them.
> > As
> > > > > they are currently not supported, I'm using Bulk Iterations inside
> a
> > > > > "classic" while loop. I've included in attachment simple example
> > > > > MyNestedIterationExample that shows this issue.
> > > > >
> > > >
> > > > ​I think you cannot add attachments to the mailing list. Could you
> > upload
> > > > your example somewhere and post a link here? I'm actually surprised
> > that
> > > > the while-loop works without problems.
> > > >
> > > >
> > > > >
> > > > > 2. For some reason I cannot create class that works with types with
> > > > > generic variab

Re: On (FLINK-1526) JIRA issue

[Vasiliki Kalavri]

Hi Olga,

when you use mapEdges() or mapVertices() with generics, Flink cannot
determine the type because of type erasure, like the exception says. That's
why we also provide methods that take the type information as a parameter.
You can use those to make the return type explicit. In your example, you
should do something like the following (line 41):

final TypeInformation longType = BasicTypeInfo.LONG_TYPE_INFO;
final TypeInformation doubleType = BasicTypeInfo.DOUBLE_TYPE_INFO;
Graph<Long, String, Tuple3<Double, Long, Long>> graphOut =
graph.mapEdges(new InitializeEdges<Long, Double>(), new
TupleTypeInfo(longType, longType,
new TupleTypeInfo<Tuple3<Double, Long, Long>>(doubleType,
longType,longType)));

Regarding the nested loops, I am almost sure that you will face problems if
you try to experiment with large datasets. I haven't looked into your code
yet, but according to the JIRA discussion, we've faced this problem before
and afaik, this is still an issue.

Cheers,
-Vasia.

On 22 September 2016 at 01:12, Olga Golovneva <melcha...@gmail.com> wrote:

> Hi Vasia,
>
> I have uploaded these tests on github:
> https://github.com/OlgaGolovneva/MST/tree/master/tests
>
> I have also uploaded source code, but I'm still working on it:
> https://github.com/OlgaGolovneva/MST/tree/master/src
>
> ​>I think you cannot add attachments to the mailing list. Could you upload
> >your example somewhere and post a link here? I'm actually surprised that
> >the while-loop works without problems.
>
> I have run the program on several simple tests, and I was going to try
> large datasets in the next few days. Please, let me know if this approach
> is wrong.
>
> Thanks,
> Olga
>
> On Wed, Sep 21, 2016 at 4:55 PM, Vasiliki Kalavri <
> vasilikikala...@gmail.com
> > wrote:
>
> > Hi Olga,
> >
> > On 21 September 2016 at 18:50, Olga Golovneva <melcha...@gmail.com>
> wrote:
> >
> > > Hi devs,
> > >
> > > I was working on  (FLINK-1526) "Add Minimum Spanning Tree library
> method
> > > and example" issue. I've developed (Java) code that implements
> > distributed
> > > Boruvka's algorithm in Gelly library. I've run several tests and it
> seems
> > > to work fine, although I didn't test it on extremely large input graphs
> > > yet, and I'm also trying to optimize my code.
> > > Particularly, I have two main issues:
> > >
> > > 1. Nested loops.
> > > I have to use nested loops, and I do not see the way to avoid them. As
> > > they are currently not supported, I'm using Bulk Iterations inside a
> > > "classic" while loop. I've included in attachment simple example
> > > MyNestedIterationExample that shows this issue.
> > >
> >
> > ​I think you cannot add attachments to the mailing list. Could you upload
> > your example somewhere and post a link here? I'm actually surprised that
> > the while-loop works without problems.
> >
> >
> > >
> > > 2. For some reason I cannot create class that works with types with
> > > generic variables in Tuple2(or Tuple3), thus my code does not support
> > > generic types. I also included simple example MyTuple3Example. Here is
> > the
> > > Exception I get:
> > > "Exception in thread "main" org.apache.flink.api.common.functions.
> > InvalidTypesException:
> > > Type of TypeVariable 'EV' in 'class org.apache.flink.graph.
> > > examples.MyTuple3Example$InitializeEdges' could not be determined.
> This
> > > is most likely a type erasure problem. The type extraction currently
> > > supports types with generic variables only in cases where all variables
> > in
> > > the return type can be deduced from the input type(s)."
> > >
> >
> > ​Can you upload this example and link to it too?
> >
> > Thanks,
> > -Vasia.
> >
> >
> > >
> > > I would really appreciate if someone could explain me know how to avoid
> > > this Exception. Otherwise, I could submit my code for testing.
> > >
> > > Best regards,
> > > Olga Golovneva
> > >
> >
>

Re: On (FLINK-1526) JIRA issue

[Vasiliki Kalavri]

Hi Olga,

On 21 September 2016 at 18:50, Olga Golovneva  wrote:

> Hi devs,
>
> I was working on  (FLINK-1526) "Add Minimum Spanning Tree library method
> and example" issue. I've developed (Java) code that implements distributed
> Boruvka's algorithm in Gelly library. I've run several tests and it seems
> to work fine, although I didn't test it on extremely large input graphs
> yet, and I'm also trying to optimize my code.
> Particularly, I have two main issues:
>
> 1. Nested loops.
> I have to use nested loops, and I do not see the way to avoid them. As
> they are currently not supported, I'm using Bulk Iterations inside a
> "classic" while loop. I've included in attachment simple example
> MyNestedIterationExample that shows this issue.
>

​I think you cannot add attachments to the mailing list. Could you upload
your example somewhere and post a link here? I'm actually surprised that
the while-loop works without problems.


>
> 2. For some reason I cannot create class that works with types with
> generic variables in Tuple2(or Tuple3), thus my code does not support
> generic types. I also included simple example MyTuple3Example. Here is the
> Exception I get:
> "Exception in thread "main" 
> org.apache.flink.api.common.functions.InvalidTypesException:
> Type of TypeVariable 'EV' in 'class org.apache.flink.graph.
> examples.MyTuple3Example$InitializeEdges' could not be determined. This
> is most likely a type erasure problem. The type extraction currently
> supports types with generic variables only in cases where all variables in
> the return type can be deduced from the input type(s)."
>

​Can you upload this example and link to it too?

Thanks,
-Vasia.


>
> I would really appreciate if someone could explain me know how to avoid
> this Exception. Otherwise, I could submit my code for testing.
>
> Best regards,
> Olga Golovneva
>

Re: Gelly Library. Need an example

[Vasiliki Kalavri]

Hi,

thanks for looking into this Till! I'm not quite sure what the algorithm
behavior should be when the vertex value is null (probably skip the
record?). Let's wait for Martin's input.

Cheers,
-V.

On 15 September 2016 at 19:19, Olga Golovneva  wrote:

> Hi Till,
>
> Thanks a lot for your help! I'll try to use another variable type in the
> meantime.
>
> Best regards,
> Olga
>
>
> Best regards,
> Olga Golovneva
>
> On Thu, Sep 15, 2016 at 1:03 PM, Till Rohrmann 
> wrote:
>
> > Hi Olga,
> >
> > it’s indeed an error in Flink’s Summarization algorithm. The problem is
> the
> > following: The vertex group value of the VertexGroupItem is null in the
> > VertexGroupReducer. This works in the SummarizationIT case because the
> > vertex value is of type String and the StringSerializer can deal with
> null
> > values.
> >
> > However, in your case where you use longs, it fails, because the
> > LongSerializer cannot handle null values. You can verify this behaviour
> by
> > changing the vertex value type to String. Then everything should work
> > without a problem.
> >
> > I’ve cc’ed Martin who can tell you probably more about the Summarization
> > algorithm. I’ve also opened a JIRA ticket [1] to fix this problem.
> >
> > Thanks for reporting this bug.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-4624
> >
> > Cheers,
> > Till
> > ​
> >
> > On Thu, Sep 15, 2016 at 5:05 PM, Olga Golovneva 
> > wrote:
> >
> > > Hi Till,
> > >
> > > I've created a simple (Java) example to show you what's going on. The
> > code
> > > is in attachment and shown below. This example creates simple graph
> with
> > > Double EV and Long VV. Then it runs Summarization, that should compute
> a
> > > condensed version of the input graph by grouping vertices and edges
> based
> > > on their values. I run this code with IntelliJ IDEA. The code executes
> > fine
> > > until you want to see what is written in resulted edges (just uncomment
> > > line 46, edgesOut.print();). Then it throws the following Exception:
> > >
> > > _EXCEPTION START_
> > > Exception in thread "main" org.apache.flink.runtime.
> > client.JobExecutionException:
> > > Job execution failed.
> > > at org.apache.flink.runtime.jobmanager.JobManager$$
> > > anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply$
> > > mcV$sp(JobManager.scala:830)
> > > at org.apache.flink.runtime.jobmanager.JobManager$$
> > > anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply(
> > JobManager.scala:773)
> > > at org.apache.flink.runtime.jobmanager.JobManager$$
> > > anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply(
> > JobManager.scala:773)
> > > at scala.concurrent.impl.Future$PromiseCompletingRunnable.
> > > liftedTree1$1(Future.scala:24)
> > > at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(
> > > Future.scala:24)
> > > at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
> > > at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(
> > > AbstractDispatcher.scala:401)
> > > at scala.concurrent.forkjoin.ForkJoinTask.doExec(
> ForkJoinTask.java:260)
> > > at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.
> > > pollAndExecAll(ForkJoinPool.java:1253)
> > > at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.
> > > runTask(ForkJoinPool.java:1346)
> > > at scala.concurrent.forkjoin.ForkJoinPool.runWorker(
> > > ForkJoinPool.java:1979)
> > > at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(
> > > ForkJoinWorkerThread.java:107)
> > > Caused by: org.apache.flink.types.NullFieldException: Field 2 is null,
> > > but expected to hold a value.
> > > at org.apache.flink.api.java.typeutils.runtime.
> > TupleSerializer.serialize(
> > > TupleSerializer.java:126)
> > > at org.apache.flink.api.java.typeutils.runtime.
> > TupleSerializer.serialize(
> > > TupleSerializer.java:30)
> > > at org.apache.flink.runtime.plugable.SerializationDelegate.write(
> > > SerializationDelegate.java:56)
> > > at org.apache.flink.runtime.io.network.api.serialization.
> > > SpanningRecordSerializer.addRecord(SpanningRecordSerializer.java:83)
> > > at org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(
> > > RecordWriter.java:85)
> > > at org.apache.flink.runtime.operators.shipping.
> OutputCollector.collect(
> > > OutputCollector.java:65)
> > > at org.apache.flink.runtime.operators.util.metrics.
> > > CountingCollector.collect(CountingCollector.java:35)
> > > at org.apache.flink.api.java.operators.translation.PlanFilterOperator$
> > > FlatMapFilter.flatMap(PlanFilterOperator.java:51)
> > > at org.apache.flink.runtime.operators.chaining.
> > > ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:80)
> > > at org.apache.flink.runtime.operators.util.metrics.
> > > CountingCollector.collect(CountingCollector.java:35)
> > > at org.apache.flink.graph.library.Summarization$
> > VertexGroupReducer.reduce(
> > > Summarization.java:323)
> > > at 

Re: Some thoughts about the lower-level Flink APIs

[Vasiliki Kalavri]

Hi Jamie,

thanks for sharing your thoughts on this! You're raising some interesting
points.

Whether users find the lower-level primitives more intuitive depends on
their background I believe. From what I've seen, if users are coming from
the S4/Storm world and are used to the "compositional" way of streaming,
then indeed it's easier for them to think and operate on that level. These
are usually people who have seen/built streaming things before trying out
Flink.
But if we're talking about analysts and people coming from the "batch" way
of thinking or people used to working with SQL/python, then the
higher-level declarative API is probably easier to understand.

I do think that we should make the lower-level API more visible and
document it properly, but I'm not sure if we should teach Flink on this
level first. I think that presenting it as a set of "advanced" features
makes more sense actually.

Cheers,
-Vasia.

On 16 August 2016 at 04:24, Jamie Grier  wrote:

> You lost me at lattice, Aljoscha ;)
>
> I do think something like the more powerful N-way FlatMap w/ Timers
> Aljoscha is describing here would probably solve most of the problem.
> Often Flink's higher level primitives work well for people and that's
> great.  It's just that I also spend a fair amount of time discussing with
> people how to map what they know they want to do onto operations that
> aren't a perfect fit and it sometimes liberates them when they realize they
> can just implement it the way they want by dropping down a level.  They
> usually don't go there themselves, though.
>
> I mention teaching this "first" and then the higher layers I guess because
> that's just a matter of teaching philosophy.  I think it's good to to see
> the basic operations that are available first and then understand that the
> other abstractions are built on top of that.  That way you're not afraid to
> drop-down to basics when you know what you want to get done.
>
> -Jamie
>
>
> On Mon, Aug 15, 2016 at 2:11 AM, Aljoscha Krettek 
> wrote:
>
> > Hi All,
> > I also thought about this recently. A good think would be to add a good
> > user facing operator that behaves more or less like an enhanced FlatMap
> > with multiple inputs, multiple outputs, state access and keyed timers.
> I'm
> > a bit hesitant, though, since users rarely think about the implications
> > that come with state updating and out-of-order events. If you don't
> > implement a stateful operator correctly you have pretty much arbitrary
> > results.
> >
> > The problem with out-of-order event arrival and state update is that the
> > state basically has to monotonically transition "upwards" through a
> lattice
> > for the computation to make sense. I know this sounds rather theoretical
> so
> > I'll try to explain with an example. Say you have an operator that waits
> > for timestamped elements A, B, C to arrive in timestamp order and then
> does
> > some processing. The naive approach would be to have a small state
> machine
> > that tracks what element you have seen so far. The state machine has
> three
> > states: NOTHING, SEEN_A, SEEN_B and SEEN_C. The state machine is supposed
> > to traverse these states linearly as the elements arrive. This doesn't
> > work, however, when elements arrive in an order that does not match their
> > timestamp order. What the user should do is to have a "Set" state that
> > keeps track of the elements that it has seen. Once it has seen {A, B, C}
> > the operator must check the timestamps and then do the processing, if
> > required. The set of possible combinations of A, B, and C forms a lattice
> > when combined with the "subset" operation. And traversal through these
> sets
> > is monotonically "upwards" so it works regardless of the order that the
> > elements arrive in. (I recently pointed this out on the Beam mailing list
> > and Kenneth Knowles rightly pointed out that what I was describing was in
> > fact a lattice.)
> >
> > I know this is a bit off-topic but I think it's very easy for users to
> > write wrong operations when they are dealing with state. We should still
> > have a good API for it, though. Just wanted to make people aware of this.
> >
> > Cheers,
> > Aljoscha
> >
> > On Mon, 15 Aug 2016 at 08:18 Matthias J. Sax  wrote:
> >
> > > It really depends on the skill level of the developer. Using low-level
> > > API requires to think about many details (eg. state handling etc.) that
> > > could be done wrong.
> > >
> > > As Flink gets a broader community, more people will use it who might
> not
> > > have the required skill level to deal with low-level API. For more
> > > trained uses, it is of course a powerful tool!
> > >
> > > I guess it boils down to the question, what type of developer Flink
> > > targets, if low-level API should be offensive advertised or not. Also
> > > keep in mind, that many people criticized Storm's low-level API as hard
> > > to program etc.
> > >
> > 

Re: [VOTE] Release Apache Flink 1.1.1 (RC1)

[Vasiliki Kalavri]

On 9 August 2016 at 18:27, Ufuk Celebi  wrote:

> Dear Flink community,
>
> Please vote on releasing the following candidate as Apache Flink version
> 1.1.1.
>
> *Important*: I would like to reduce the voting time to 24 hours (with
> a majority of at least three +1 PMC votes as usual). We discovered
> that the Maven artifacts published with version 1.1.0 have dependency
> issues, which will affect users running on Hadoop 2 infrastructure
> (like HDFS). Since Maven artifacts are immutable, we cannot override
> them and we have to publish a new version to fix this.
>
> The release script contained a bug, which resulted in no deployment of
> a Hadoop 1 specific version (1.1.0-hadoop1) and regular 1.1.0
> artifacts having a dependency on Hadoop 1 instead of Hadoop 2. I've
> updated the release announcement accordingly with a warning (see
> http://flink.apache.org/news/2016/08/08/release-1.1.0.html).
>
> Please indicate whether you are OK with the reduced voting time.
>

​+1 fine with me​



>
> The commit to be voted on:
> 61bfb36 (http://git-wip-us.apache.org/repos/asf/flink/commit/61bfb36)
>
> Branch:
> release-1.1.1-rc1
> (https://git1-us-west.apache.org/repos/asf/flink/repo?p=
> flink.git;a=shortlog;h=refs/heads/release-1.1.1-rc1)
>
> The release artifacts to be voted on can be found at:
> http://people.apache.org/~uce/flink-1.1.1-rc1/
>
> The release artifacts are signed with the key with fingerprint 9D403309:
> http://www.apache.org/dist/flink/KEYS
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapacheflink-1101
>
> -
>
> The vote is open for the next 24 hours (see above for explanation) and
> passes if a majority of at least three +1 PMC votes are cast.
>
> The vote ends on Wednesday, August 10th, 2016.
>
> [ ] +1 Release this package as Apache Flink 1.1.1
> [ ] -1 Do not release this package, because ...
>

Re: [ANNOUNCE] Flink 1.1.0 Released

[Vasiliki Kalavri]

yoo-hoo finally announced 
Thanks for managing the release Ufuk!

On 8 August 2016 at 18:36, Ufuk Celebi  wrote:

> The Flink PMC is pleased to announce the availability of Flink 1.1.0.
>
> On behalf of the PMC, I would like to thank everybody who contributed
> to the release.
>
> The release announcement:
> http://flink.apache.org/news/2016/08/08/release-1.1.0.html
>
> Release binaries:
> http://apache.openmirror.de/flink/flink-1.1.0/
>
> Please update your Maven dependencies to the new 1.1.0 version and
> update your binaries.
>
> – Ufuk
>

Re: sampling function

[Vasiliki Kalavri]

Hi Do,

Paris and Martha worked on sampling techniques for data streams on Flink
last year. If you want to implement your own samplers, you might find
Martha's master thesis helpful [1].

-Vasia.

[1]: http://kth.diva-portal.org/smash/get/diva2:910695/FULLTEXT01.pdf

On 11 July 2016 at 11:31, Kostas Kloudas 
wrote:

> Hi Do,
>
> In DataStream you can always implement your own
> sampling function, hopefully without too much effort.
>
> Adding such functionality it to the API could be a good idea.
> But given that in sampling there is no “one-size-fits-all”
> solution (as not every use case needs random sampling and not
> all random samplers fit to all workloads), I am not sure if we
> should start adding different sampling operators.
>
> Thanks,
> Kostas
>
> > On Jul 9, 2016, at 5:43 PM, Greg Hogan  wrote:
> >
> > Hi Do,
> >
> > DataSet provides a stable @Public interface. DataSetUtils is marked
> > @PublicEvolving which is intended for public use, has stable behavior,
> but
> > method signatures may change. It's also good to limit DataSet to common
> > methods whereas the utility methods tend to be used for specific
> > applications.
> >
> > I don't have the pulse of streaming but this sounds like a useful feature
> > that could be added.
> >
> > Greg
> >
> > On Sat, Jul 9, 2016 at 10:47 AM, Le Quoc Do  wrote:
> >
> >> Hi all,
> >>
> >> I'm working on approximate computing using sampling techniques. I
> >> recognized that Flink supports the sample function for Dataset
> >> (org/apache/flink/api/java/utils/DataSetUtils.java). I'm just wondering
> why
> >> you didn't merge the function to org/apache/flink/api/java/DataSet.java
> >> since the sample function works as a transformation operator?
> >>
> >> The second question is that are you planning to support the sample
> >> function for DataStream (within windows) since I did not see it in
> >> DataStream code ?
> >>
> >> Thank you,
> >> Do
> >>
>
>

Re: How to run table api in 1.1-SNAPSHOT

[Vasiliki Kalavri]

Hi Cody,

could it be you're getting this error because you've named a SQL table
column "count"? Can you try renaming it to "myCount" or something else? I
think the parser recognizes the aggregate function instead :)

Cheers,
-V.
On Jun 2, 2016 1:56 PM, "Cody Innowhere"  wrote:

> Hi guys,
> I'm trying to run Table-API in master trunk using the sql/registerDataSet
> APIs in TableEnvironment class.
>
> According to the doc in table.md, after registering a table, I should be
> able to use a sql query on the tabelEnv, so I made a slight change in
> WordCountTable.scala by simply adding two lines:
>
> -
> object WordCountTable {
>
>   case class WC(word: String, count: Int)
>
>   def main(args: Array[String]): Unit = {
>
> // set up execution environment
> val env = ExecutionEnvironment.getExecutionEnvironment
> val tEnv = TableEnvironment.getTableEnvironment(env)
>
> val input = env.fromElements(WC("hello", 1), WC("hello", 2), WC("ciao",
> 3))
> val expr = input.toTable(tEnv)
>
> // *** added lines ***
> tEnv.registerDataSet("WC", input, 'word, 'count)
> val result1 = tEnv.sql("SELECT word FROM WC ")
>
> val result = expr
>   .groupBy('word)
>   .select('word, 'count.sum as 'count)
>   .toDataSet[WC]
>
> result.print()
>   }
> }
>
> As you can see current query sql is "SELECT word FROM WC" and it works.
> But when I change query sql to :
> "SELECT word, count FROM WC" it does not work with the exception:
> "Exception in thread "main"
> org.apache.calcite.sql.parser.SqlParseException: Encountered "count FROM"
> at line 1, column 13.
> Was expecting one of:
>   ...
>   ..."
>
> Do I miss something?
>
> BTW., I read the doc at
>
> https://docs.google.com/document/d/1TLayJNOTBle_-m1rQfgA6Ouj1oYsfqRjPcp1h2TVqdI/
> ,
> I suppose Task2 has been finished already, right? And is somebody working
> on Task3? Do we have a time map for SQL on Flink?
>
> Thanks~
>

Re: [PROPOSAL] Structure the Flink Open Source Development

[Vasiliki Kalavri]

Hi,

we could go for something like "sponsor" or "champion" :)
I'm fine with the proposal. Good to see more than 1 person for both Gelly
and Table API.

cheers,
-V.

On 1 June 2016 at 05:46, Tzu-Li (Gordon) Tai  wrote:

> I'd like to be added to the Streaming Connectors component (already edited
> Wiki) :)
>
> Ah, naming, one of the hardest problems in programming :P Some comments:
> I agree with Robert that the name "maintainers" will be somewhat misleading
> regarding the authoritative difference with committers / PMCs, especially
> for future newcomers to the community who don't come across the original
> discussion on this thread.
>
> Simone's suggestion of Overseer seems good. The name naturally matches its
> role -
> - A group of "Overseers" for components, who keeps an eye on related mail
> threads, known limitations and issues, JIRAs, open PRs, requested features,
> and potential new overseers and committers, etc, for the component
> (original
> maintainer role).
> - A "Shepherd" for individual PRs, assigned from the overseers of the
> component with the aim to guide the submitting contributor. Overseers
> typically pick up new PRs to shepherd themselves, or the leading overseer
> allocates an overseer to shepherd a PR which hasn't been picked up yet
> after
> a certain period of time.
>
> Or perhaps we can also simply go for "Shepherds" for components and
> "Assigned Shepherd" for individual PRs?
>
>
>
> --
> View this message in context:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/PROPOSAL-Structure-the-Flink-Open-Source-Development-tp11598p11932.html
> Sent from the Apache Flink Mailing List archive. mailing list archive at
> Nabble.com.
>

Re: Iteration Intermediate Output

[Vasiliki Kalavri]

Hey,

it would be great to add this feature indeed! Thanks for bringing it up
Greg :)
Would the best way be to extend the iteration operators to support
intermediate outputs or revisit the idea of caching intermediate results
and thus allow efficient for-loop iterations?

-Vasia.

On 26 May 2016 at 22:41, Greg Hogan  wrote:

> Hi y'all,
>
> I think this is an oft-requested feature [0] and there are many graph
> algorithms for which intermediate output is the desired result. I'd like to
> take Stephan up on his offer [1] for pointers.
>
> I have yet to get in deep, but I see that iteration tasks are treated
> specially as IterationIntermediateTask for synchronization between
> supersteps. Also, when OperatorTranslation and GraphCreatingVisitor are
> walking the program DAG an iteration must be first reached through the
> tail.
>
> Greg
>
> [0]
>
> http://stackoverflow.com/questions/37224140/possibility-of-saving-partial-outputs-from-bulk-iteration-in-flink-dataset
> [1]
>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Intermediate-output-during-delta-iterations-td436.html
>

Re: Hotfixes on the master

[Vasiliki Kalavri]

Hi all,

in principle I agree with Max. I personally avoid hotfixes and always open
a PR, even for javadoc improvements.

I believe the main problem is that we don't have a clear definition of what
constitutes a "hotfix". Ideally, even cosmetic changes and documentation
should be reviewed; I've seen documentation added as a hotfix that had
spelling mistakes, which led to another hotfix... Using hotfixes to do
major refactoring or add features is absolutely unacceptable, in my view.
On the other hand, with the current PR load it's not practical to ban
hotfixes all together.

I would suggest to update our contribution guidelines with some definition
of a hotfix. We could add a list of questions to ask before pushing one.
e.g.:
- does the change fix a spelling mistake in the docs? => hotfix
- does the change add a missing javadoc? => hotfix
- does the change improve a comment? => hotfix?
- is the change a small refactoring in a code component you are maintainer
of? => hotfix
- did you change code in a component you are not very familiar with / not
the maintainer of? => open PR
- is this major refactoring? (e.g. more than X lines of code) => open PR
- does it fix a trivial bug? => open JIRA and PR

and so on...

What do you think?

Cheers,
-V.

On 27 May 2016 at 17:40, Greg Hogan  wrote:

> Max,
>
> I certainly agree that hotfixes are not ideal for large refactorings and
> new features. Some thoughts ...
>
> A hotfix should be maven verified, as should a rebased PR. Travis is often
> backed up for half a day or more.
>
> Is our Jira and PR process sufficiently agile to handle these hotfixes?
> Will committers simply include hotfixes with other PRs, and would it be
> better to retain these as smaller, separate commits?
>
> For these cosmetic changes and small updates will the Jira and PR yield
> beneficial documentation addition to what is provided in the commit
> message?
>
> Counting hotfixes by contributor, the top of the list looks as I would
> expect.
>
> Greg
>
> Note: this summary is rather naive and includes non-squashed hotfix commits
> included in a PR
> $ git shortlog --grep 'hotfix' -s -n release-0.9.0..
> 94  Stephan Ewen
> 42  Aljoscha Krettek
> 20  Till Rohrmann
> 16  Robert Metzger
> 13  Ufuk Celebi
>  9  Fabian Hueske
>  9  Maximilian Michels
>  6  Greg Hogan
>  5  Stefano Baghino
>  3  smarthi
>  2  Andrea Sella
>  2  Gyula Fora
>  2  Jun Aoki
>  2  Sachin Goel
>  2  mjsax
>  2  zentol
>  1  Alexander Alexandrov
>  1  Gabor Gevay
>  1  Prez Cannady
>  1  Steve Cosenza
>  1  Suminda Dharmasena
>  1  chengxiang li
>  1  jaoki
>  1  kl0u
>  1  qingmeng.wyh
>  1  sksamuel
>  1  vasia
>
> On Fri, May 27, 2016 at 6:10 AM, Maximilian Michels 
> wrote:
>
> > Hi Flinksters,
> >
> > I'd like to address an issue that has been concerning me for a while.
> > In the Flink community we like to push "hotfixes" to the master.
> > Hotfixes come in various shapes: From very small cosmetic changes
> > (JavaDoc) to major refactoring and even new features.
> >
> > IMHO we should move away from these hotfixes. Here's why:
> >
> > 1) They tend to break the master because they lack test coverage
> > 2) They are usually not communicated with the maintainer or person
> > working on the part being fixed
> > 3) They are not properly documented for future reference or follow-ups
> > (JIRA/Github)
> >
> > That's why I have chosen not to push hotfixes anymore. Even for small
> > fixes, I'll open a JIRA/Github issue. The only exception might be
> > fixing a comment. It improves communication, documentation, and test
> > coverage. All this helps to mature Flink and develop the community in
> > a transparent way.
> >
> > I'm not sure what our contribution guidelines say about this but I
> > would like to update them to explicitly address hotfixes. Let me know
> > what you think.
> >
> > Best,
> > Max
> >
>

Re: Performance and accuracy of Flink iterations

[Vasiliki Kalavri]

Hi Greg,

I think there is confusion between what delta means in the "delta iteration
operator" of Flink and the "delta approximate implementation" of an
algorithm, such as in PageRank.

Assuming that we have a graph with a set of vertices and an iterative
fixpoint algorithm that updates the vertex values in each iteration, the
bulk iteration operator will generate a new set of vertex values at
iteration k, which will be the input of iteration k+1, etc..

Using the delta iteration operator, we can keep state inside the iteration
and thus, exploit the asymmetrical convergence of some algorithms, i.e.
only compute on the "active" vertices. An active vertex is defined as one
that has changed its value since the last iteration.

If the state update function is idempotent and weakly monotonic, e.g. min
(like in connected components, sssp), then the delta iteration result is
equivalent to the bulk iteration.
If the state update function is linear and decomposable, e.g. sum (like in
PageRank), then you can re-write the update function and propagate only
_deltas_, instead of value updates. This way the result will again be
equivalent to the bulk iteration result.

Now, if your application can tolerate some level of inaccuracy, then you
can define a threshold on what it means for a vertex value to "have changed
its value". The higher the threshold, the less active vertices you'll have
and probably faster convergence, but higher error as well. Thus, accuracy
is not a property of the bulk or the delta iteration themselves, but rather
depends on the algorithm and on the vertex activation criterion.

I hope this clears things up!
-Vasia.

On 17 May 2016 at 14:39, Till Rohrmann  wrote:

> Hi Greg,
>
> as far as I know there has not been an exhaustive comparison to what extent
> the delta iterations can achieve the same accuracy as bulk iterations or
> how much accuracy you'll lose. I think it strongly depends on the problem.
> For example, graph algorithms such as connected components shouldn't suffer
> from it. In contrast, the PageRank implementation with the THRESHOLD value
> should not produce the (most) accurate result. Of course this depends on
> the threshold value. Do you want to make such a comparison?
>
> Cheers,
> Till
>
> On Mon, May 16, 2016 at 3:10 PM, Greg Hogan  wrote:
>
> > Hi,
> >
> > This question has arisen with the HITS algorithm (Hubs and Authorities)
> but
> > the question is the same as with PageRank, for which Stephan published an
> > excellent discussion and comparison of bulk and delta iterations [0].
> >
> > Delta iterations are clearly faster. Has there been a comparison as to
> > whether, when, or how delta iterations are more accurate?
> >
> > Greg
> >
> > [0]
> >
> >
> http://data-artisans.com/data-analysis-with-flink-a-case-study-and-tutorial/
> >
>

Updated Gelly Roadmap

[Vasiliki Kalavri]

Hi all,

as promised, I have updated the Gelly roadmap [1].
Below, I am describing and reasoning about the changes I made. Please, let
me know whether you agree and if you have any other ideas for further
improvements and feature additions.

*1. Operators for highly skewed graphs*:
I have removed this item completely. It referred to Andra's master thesis
which has been completed. We had a discussion back then (see [2]), but no
activity after that.

*2. Scala API*:
I removed this item because it's done 

*3. Graph Streaming*:
I removed this item also. We have built an experimental API for graph
streaming with Paris and KTH students. The code is available in [3]. If you
think it would be a valuable addition as a library to Flink, we can start a
separate discussion thread about it.

*4. Library Methods*:
- Affinity Propagation: this is WIP in FLINK-1707
- HITS + Adsorption: Removed. They were started by TUB students and not
finished. I propose to only revisit these only if someone asks for them.
- Strongly CC + DIA: Removed. Nobody has worked on them AFAIK.

*5. Graph partitioning*:
This is still relevant in my opinion and thus I kept it in the updated
roadmap.

*6. Partition-centric iterations*:
We have created a POC implementation with KTH students [4]. In my opinion,
it would be nice to add this to Gelly and most of the work has already been
done, so I kept it.

*7. Generic Iterations*:
This requires caching intermediate results. Anyone has a status update on
that?

*8. Performance evaluation*:
I'm currently working on this, integrating Gelly with the Graphalytics
benchmark. My WIP is in [5] in case you want to get involved :)

*9. Bipartite support*:
This is still relevant in my opinion. Someone had started working on it,
but has been inactive for a while. I pinged the JIRA (FLINK-2254).

>From the wishlist:
- *Neo4j* input/output formats have been implemented as an external project
by Martin Junghanns. Shall we go ahead and them to project-flink? We should
definitely link to this from the third-party packages.

- *TinkerPop*: Discussion started from the TinkerPop community, but there
was not much activity from our side [6]. I wrote my opinion then and I am
very much in favor. Anyone else wants to share their thoughts?

Looking forward to your input,
-Vasia.


[1]: https://cwiki.apache.org/confluence/display/FLINK/Flink+Gelly
[2]:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Proposal-Addition-to-Gelly-td7436.html
[3]: https://github.com/vasia/gelly-streaming
[4]: https://github.com/vasia/gelly-partition-centric
[5]: https://github.com/vasia/graphalytics-platforms-gelly
[6]:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Apache-Tinkerpop-amp-Geode-Integration-td9252.html

Re: [DISCUSS] Graph algorithms for vertex and edge degree

[Vasiliki Kalavri]

Hey Greg,

when we made the initial design for Gelly, we discussed having separate
directed and undirected graph types.
We decided to represent undirected graphs by simply adding the
opposite-direction edges, like Apache Giraph does. I'm not sure whether
that was a good idea, but, back then, it seemed to simplify the design of
the API.
If you think we can improve the API and the usability of Gelly by
introducing separate types, I would love to hear your proposal!

Cheers,
-Vasia.

On 25 April 2016 at 22:49, Greg Hogan <c...@greghogan.com> wrote:

> Hi Fabian,
>
> I don't know if this has been looked at. There is discussion of
> BipartiteGraph in FLINK-2254. If Gelly had DirectedGraph and
> UndirectedGraph then the API could stay lean while methods could be tuned
> for the specific graph type. I do like having simple APIs such as DataSet
> and Graph for for interactive use in small scripts or Scala and Python
> shells.
>
> Greg
>
> On Mon, Apr 25, 2016 at 8:46 AM, Fabian Hueske <fhue...@gmail.com> wrote:
>
> > Hi Greg and Vasia,
> >
> > thanks for starting this discussion.
> > I think it is a good idea to update the Gelly roadmap. As Vasia said,
> many
> > items on the list have been implemented and other have been more or less
> > dropped.
> > Also new persons who want to improve Gelly have joint the community while
> > others have become idle.
> > So from my point of view it makes perfect sense to gather the input of
> the
> > community and update the Gelly roadmap.
> >
> > Regarding the specific changes of FLINK-3772, I am usually always in
> favor
> > of performance improvements.
> > How much would the suggested functionality for degree computation
> overlaps
> > with the current methods?
> > Could the current methods be replaced by the more efficient
> implementations
> > or would there be two methods which look very similar and behave almost
> the
> > same?
> >
> > Best, Fabian
> >
> >
> >
> > 2016-04-22 11:00 GMT+02:00 Vasiliki Kalavri <vasilikikala...@gmail.com>:
> >
> > > Hi all,
> > >
> > > I asked Greg to start a discussion here about FLINK-3771 and
> FLINK-3772,
> > to
> > > make sure we're all on the same page about future Gelly development.
> > >
> > > About a year ago we created the Gelly roadmap [1]. Many of these items
> > have
> > > been implemented and others were researched and either developed
> > externally
> > > or dropped. Graph translators and degree annotations were not in the
> > > roadmap, but I personally think that they are both helpful features and
> > I'm
> > > in favor of adding them to Gelly.
> > >
> > > That said, I find this a perfect timing for revisiting and updating our
> > > Gelly development plans. We should update the roadmap so that it
> reflects
> > > current state and future plans. This way, the community can have a
> clear
> > > picture of what we are working on and what are upcoming features. This
> is
> > > also very important for new contributors. They can always refer to the
> > > roadmap to see what the community is interested in and we can avoid
> > having
> > > people spending their time on obsolete or dropped features. We should
> > also
> > > go through existing JIRAs and clean them up. Several are assigned to
> > people
> > > who have been inactive or might need help.
> > >
> > > In the following days I will go through the roadmap and then start a
> new
> > > thread where we can discuss features and agree on future plans. In the
> > > meantime, it would be great if you give us your view on FLINK-3771 and
> > > FLINK-3772.
> > >
> > > Cheers,
> > > -Vasia.
> > >
> > > [1]: https://cwiki.apache.org/confluence/display/FLINK/Flink+Gelly
> > >
> > > On 21 April 2016 at 18:52, Greg Hogan <c...@greghogan.com> wrote:
> > >
> > > > Vasia and I are looking for additional feedback on FLINK-3772. This
> > > ticket
> > > > [0] and PR [1] provides a set of graph algorithms which compute and
> > store
> > > > the degree for vertices and edges.
> > > >
> > > > Degree annotation is a basic component of many algorithms. For
> example,
> > > > PageRank requires the vertex out-degree to evenly divide the vertex
> > score
> > > > among neighbors. The triangle enumeration algorithm in Gelly requires
> > > both
> > > > source and target degree for each edge as an opt

Re: 答复: 答复: Effort to add SQL / StreamSQL to Flink

[Vasiliki Kalavri]

Great to see people excited about this :)
SQL is indeed coming up next. We should have the SQL on DataSets programs
(see FLINK-3640 [1]) pretty soon.

-Vasia.

[1]: https://issues.apache.org/jira/browse/FLINK-3640

On 29 March 2016 at 14:02, Jiangsong (Hi) <hi.jiangs...@huawei.com> wrote:

> So excited!!   SQL on Flink is ready?
>
> Are there any show case or howto use?
>
>
>
> -邮件原件-
> 发件人: ewenstep...@gmail.com [mailto:ewenstep...@gmail.com] 代表 Stephan Ewen
> 发送时间: 2016年3月29日 20:00
> 收件人: dev@flink.apache.org
> 主题: Re: 答复: Effort to add SQL / StreamSQL to Flink
>
> Cool stuff!
>
> SQL coming up next? ;-)
>
>
> On Tue, Mar 29, 2016 at 1:39 PM, Maximilian Michels <m...@apache.org>
> wrote:
>
> > Yeah! I'm a little late to the party but exciting stuff! :)
> >
> > On Fri, Mar 18, 2016 at 3:15 PM, Vasiliki Kalavri <
> > vasilikikala...@gmail.com
> > > wrote:
> >
> > > Hi all,
> > >
> > > tableOnCalcite has been merged to master :)
> > >
> > > Cheers,
> > > -Vasia.
> > >
> > > On 17 March 2016 at 11:11, Fabian Hueske <fhue...@gmail.com> wrote:
> > >
> > > > Thanks for the initiative Vasia!
> > > > I went over the diff and didn't find anything crucial.
> > > >
> > > > I would like to do another pass over the tests though and improve
> > > > the exceptions for invalid joins before merging.
> > > > Will open a PR later today.
> > > >
> > > > 2016-03-16 21:17 GMT+01:00 Vasiliki Kalavri
> > > > <vasilikikala...@gmail.com
> > >:
> > > >
> > > > > Yes, the current state corresponds to Task 1. PR #1770
> > > > > corresponds to
> > > > Task
> > > > > 5. Task 6 should come right after :)
> > > > >
> > > > > -V.
> > > > >
> > > > > On 16 March 2016 at 20:35, Robert Metzger <rmetz...@apache.org>
> > wrote:
> > > > >
> > > > > > Cool, this is great news!
> > > > > > So "Task 1" from the document [1] is done with the merge? And
> > > > > > PR
> > > #1770
> > > > is
> > > > > > going towards "Task 6".
> > > > > > I think good support for Stream SQL is a very interesting new
> > feature
> > > > for
> > > > > > Flink.
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://docs.google.com/document/d/1TLayJNOTBle_-m1rQfgA6Ouj1oYsfqRjPc
> > p1h2TVqdI/edit#heading=h.28dvisn56su0
> > > > > >
> > > > > > On Wed, Mar 16, 2016 at 6:17 PM, Vasiliki Kalavri <
> > > > > > vasilikikala...@gmail.com
> > > > > > > wrote:
> > > > > >
> > > > > > > Hello everyone,
> > > > > > >
> > > > > > > We are happy to announce that the "tableOnCalcite" branch is
> > > finally
> > > > > > ready
> > > > > > > to be merged.
> > > > > > > It essentially provides the existing functionality of the
> > > > > > > Table
> > > API,
> > > > > but
> > > > > > > now the translation happens through Apache Calcite.
> > > > > > > You can find the changes rebased on top of the current
> > > > > > > master in
> > > [1].
> > > > > > > We have removed the prototype streaming Table API
> > > > > > > functionality,
> > > > which
> > > > > > will
> > > > > > > be added back once PR [2] is merged.
> > > > > > >
> > > > > > > We'll go through the changes once more and, if no
> > > > > > > objections, we
> > > > would
> > > > > > like
> > > > > > > to go ahead and merge this.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > -Vasia.
> > > > > > >
> > > > > > > [1]: https://github.com/vasia/flink/tree/merge-table
> > > > > > > [2]: https://github.com/apache/flink/pull/1770
> > > > > > >
> > > > > > >
> > > > > > > On 15 January 2016 at 10:59, Fabia

Re: Next steps: SQL / StreamSQL support

[Vasiliki Kalavri]

Thanks for the nice summary and for updating the design documents Fabian!

As we proceed with the upcoming tasks, we should also go through existing
JIRAs and update them, too.
There are some old issues referring to SQL and adding external data
sources, but these were created before the decision of using Calcite. It
would be nice to clean up theTable API JIRAs a bit by removing the invalid
issues and updating the ones that are still relevant.

Cheers,
-Vasia.

On 21 March 2016 at 17:56, Fabian Hueske  wrote:

> Hi everybody,
>
> on Friday we merged the working branch to put the Table API on top of
> Calcite back to master.
> This was the first step towards adding SQL support to Flink as outlined in
> the design document [1] (the document was updated to reflect design
> decisions done while implementing task 1).
>
> According to the design doc, the next step is to add support for SQL
> queries on DataSets and Table API Tables.  We created two JIRA issues to
> track this effort:
> - FLINK-3639: Add methods to register DataSets and Tables in
> TableEnvironment
> - FLINK-3640: Add support for SQL queries on registered DataSets and Tables
>
> Subsequent efforts will be to add support for SQL queries on external
> tables (CSV, Parquet, etc files, DBMS, etc.), extending coverage of SQL
> standard (sort, outer joins, etc.), and defining table sinks to emit the
> result.
>
> The following document shows the syntax to register tables (DataSets,
> DataStreams, Tables, external sources), query them, and to define table
> sinks to write a Table to an external storage system [2].
>
> At the same time, we are working on extending the Table API for streaming
> tables (FLINK-3547).
>
> As usual, feedback, comments, and contributions are highly welcome :-)
>
> Best, Fabian
>
> [1]
>
> https://docs.google.com/document/d/1TLayJNOTBle_-m1rQfgA6Ouj1oYsfqRjPcp1h2TVqdI
> [2]
>
> https://docs.google.com/document/d/1sITIShmJMGegzAjGqFuwiN_iw1urwykKsLiacokxSw0
>

Re: 答复: Effort to add SQL / StreamSQL to Flink

[Vasiliki Kalavri]

Hi all,

tableOnCalcite has been merged to master :)

Cheers,
-Vasia.

On 17 March 2016 at 11:11, Fabian Hueske <fhue...@gmail.com> wrote:

> Thanks for the initiative Vasia!
> I went over the diff and didn't find anything crucial.
>
> I would like to do another pass over the tests though and improve the
> exceptions for invalid joins before merging.
> Will open a PR later today.
>
> 2016-03-16 21:17 GMT+01:00 Vasiliki Kalavri <vasilikikala...@gmail.com>:
>
> > Yes, the current state corresponds to Task 1. PR #1770 corresponds to
> Task
> > 5. Task 6 should come right after :)
> >
> > -V.
> >
> > On 16 March 2016 at 20:35, Robert Metzger <rmetz...@apache.org> wrote:
> >
> > > Cool, this is great news!
> > > So "Task 1" from the document [1] is done with the merge? And PR #1770
> is
> > > going towards "Task 6".
> > > I think good support for Stream SQL is a very interesting new feature
> for
> > > Flink.
> > >
> > > [1]
> > >
> > >
> >
> https://docs.google.com/document/d/1TLayJNOTBle_-m1rQfgA6Ouj1oYsfqRjPcp1h2TVqdI/edit#heading=h.28dvisn56su0
> > >
> > > On Wed, Mar 16, 2016 at 6:17 PM, Vasiliki Kalavri <
> > > vasilikikala...@gmail.com
> > > > wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > We are happy to announce that the "tableOnCalcite" branch is finally
> > > ready
> > > > to be merged.
> > > > It essentially provides the existing functionality of the Table API,
> > but
> > > > now the translation happens through Apache Calcite.
> > > > You can find the changes rebased on top of the current master in [1].
> > > > We have removed the prototype streaming Table API functionality,
> which
> > > will
> > > > be added back once PR [2] is merged.
> > > >
> > > > We'll go through the changes once more and, if no objections, we
> would
> > > like
> > > > to go ahead and merge this.
> > > >
> > > > Cheers,
> > > > -Vasia.
> > > >
> > > > [1]: https://github.com/vasia/flink/tree/merge-table
> > > > [2]: https://github.com/apache/flink/pull/1770
> > > >
> > > >
> > > > On 15 January 2016 at 10:59, Fabian Hueske <fhue...@gmail.com>
> wrote:
> > > >
> > > > > Hi everybody,
> > > > >
> > > > > as previously announced, I pushed a feature branch called
> > > > "tableOnCalcite"
> > > > > to the Flink repository.
> > > > > We will use this branch to work on FLINK-3221 and its sub-issues.
> > > > >
> > > > > Cheers, Fabian
> > > > >
> > > > > 2016-01-11 18:29 GMT+01:00 Fabian Hueske <fhue...@gmail.com>:
> > > > >
> > > > > > We haven't defined the StreamSQL syntax yet (and I think it will
> > take
> > > > > some
> > > > > > time until we are at that point).
> > > > > > So we are quite flexible with both featurs.
> > > > > >
> > > > > > Let's keep this opportunity in mind and coordinate when before
> > making
> > > > > > decisions about CEP or StreamSQL.
> > > > > >
> > > > > > Fabian
> > > > > >
> > > > > > 2016-01-11 17:29 GMT+01:00 Till Rohrmann <trohrm...@apache.org>:
> > > > > >
> > > > > >> First of all, it's a great design document. Looking forward
> having
> > > > > stream
> > > > > >> SQL in the foreseeable future :-)
> > > > > >>
> > > > > >> I think it is a good idea to consolidate stream SQL and CEP in
> the
> > > > long
> > > > > >> run. CEP's additional features compared to SQL boil down to
> > pattern
> > > > > >> detection. Once we have this, it should be only a question of
> > > defining
> > > > > the
> > > > > >> SQL syntax for event patterns in order to integrate CEP with
> > stream
> > > > SQL.
> > > > > >> Oracle has already defined an extension [1] to detect patterns
> in
> > a
> > > > set
> > > > > of
> > > > > >> table rows. This or Esper's event processing language (EPL) [2]
> > > could
> > > &g

Re: 答复: Effort to add SQL / StreamSQL to Flink

[Vasiliki Kalavri]

Yes, the current state corresponds to Task 1. PR #1770 corresponds to Task
5. Task 6 should come right after :)

-V.

On 16 March 2016 at 20:35, Robert Metzger <rmetz...@apache.org> wrote:

> Cool, this is great news!
> So "Task 1" from the document [1] is done with the merge? And PR #1770 is
> going towards "Task 6".
> I think good support for Stream SQL is a very interesting new feature for
> Flink.
>
> [1]
>
> https://docs.google.com/document/d/1TLayJNOTBle_-m1rQfgA6Ouj1oYsfqRjPcp1h2TVqdI/edit#heading=h.28dvisn56su0
>
> On Wed, Mar 16, 2016 at 6:17 PM, Vasiliki Kalavri <
> vasilikikala...@gmail.com
> > wrote:
>
> > Hello everyone,
> >
> > We are happy to announce that the "tableOnCalcite" branch is finally
> ready
> > to be merged.
> > It essentially provides the existing functionality of the Table API, but
> > now the translation happens through Apache Calcite.
> > You can find the changes rebased on top of the current master in [1].
> > We have removed the prototype streaming Table API functionality, which
> will
> > be added back once PR [2] is merged.
> >
> > We'll go through the changes once more and, if no objections, we would
> like
> > to go ahead and merge this.
> >
> > Cheers,
> > -Vasia.
> >
> > [1]: https://github.com/vasia/flink/tree/merge-table
> > [2]: https://github.com/apache/flink/pull/1770
> >
> >
> > On 15 January 2016 at 10:59, Fabian Hueske <fhue...@gmail.com> wrote:
> >
> > > Hi everybody,
> > >
> > > as previously announced, I pushed a feature branch called
> > "tableOnCalcite"
> > > to the Flink repository.
> > > We will use this branch to work on FLINK-3221 and its sub-issues.
> > >
> > > Cheers, Fabian
> > >
> > > 2016-01-11 18:29 GMT+01:00 Fabian Hueske <fhue...@gmail.com>:
> > >
> > > > We haven't defined the StreamSQL syntax yet (and I think it will take
> > > some
> > > > time until we are at that point).
> > > > So we are quite flexible with both featurs.
> > > >
> > > > Let's keep this opportunity in mind and coordinate when before making
> > > > decisions about CEP or StreamSQL.
> > > >
> > > > Fabian
> > > >
> > > > 2016-01-11 17:29 GMT+01:00 Till Rohrmann <trohrm...@apache.org>:
> > > >
> > > >> First of all, it's a great design document. Looking forward having
> > > stream
> > > >> SQL in the foreseeable future :-)
> > > >>
> > > >> I think it is a good idea to consolidate stream SQL and CEP in the
> > long
> > > >> run. CEP's additional features compared to SQL boil down to pattern
> > > >> detection. Once we have this, it should be only a question of
> defining
> > > the
> > > >> SQL syntax for event patterns in order to integrate CEP with stream
> > SQL.
> > > >> Oracle has already defined an extension [1] to detect patterns in a
> > set
> > > of
> > > >> table rows. This or Esper's event processing language (EPL) [2]
> could
> > > be a
> > > >> good starting point.
> > > >>
> > > >> [1]
> https://docs.oracle.com/database/121/DWHSG/pattern.htm#DWHSG8959
> > > >> [2]
> > http://www.espertech.com/esper/release-5.2.0/esper-reference/html/
> > > >>
> > > >> Cheers,
> > > >> Till
> > > >>
> > > >> On Mon, Jan 11, 2016 at 10:12 AM, Fabian Hueske <fhue...@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > Thanks for the feedback!
> > > >> >
> > > >> > We will start the SQL effort with putting the existing (batch)
> Table
> > > >> API on
> > > >> > top of Apache Calcite.
> > > >> > From there we continue to add streaming support for the Table API
> > > >> before we
> > > >> > put a StreamSQL interface on top.
> > > >> >
> > > >> > Consolidating the efforts with the CEP library sounds like a good
> > idea
> > > >> to
> > > >> > me.
> > > >> > Maybe it can be nicely integrated with the streaming table API and
> > > >> later as
> > > >> > well with the StreamSQL interface (the StreamSQL dialect is not
> > > defined
> > > >> > yet).
> >

Re: 答复: Effort to add SQL / StreamSQL to Flink

[Vasiliki Kalavri]

Hello everyone,

We are happy to announce that the "tableOnCalcite" branch is finally ready
to be merged.
It essentially provides the existing functionality of the Table API, but
now the translation happens through Apache Calcite.
You can find the changes rebased on top of the current master in [1].
We have removed the prototype streaming Table API functionality, which will
be added back once PR [2] is merged.

We'll go through the changes once more and, if no objections, we would like
to go ahead and merge this.

Cheers,
-Vasia.

[1]: https://github.com/vasia/flink/tree/merge-table
[2]: https://github.com/apache/flink/pull/1770


On 15 January 2016 at 10:59, Fabian Hueske  wrote:

> Hi everybody,
>
> as previously announced, I pushed a feature branch called "tableOnCalcite"
> to the Flink repository.
> We will use this branch to work on FLINK-3221 and its sub-issues.
>
> Cheers, Fabian
>
> 2016-01-11 18:29 GMT+01:00 Fabian Hueske :
>
> > We haven't defined the StreamSQL syntax yet (and I think it will take
> some
> > time until we are at that point).
> > So we are quite flexible with both featurs.
> >
> > Let's keep this opportunity in mind and coordinate when before making
> > decisions about CEP or StreamSQL.
> >
> > Fabian
> >
> > 2016-01-11 17:29 GMT+01:00 Till Rohrmann :
> >
> >> First of all, it's a great design document. Looking forward having
> stream
> >> SQL in the foreseeable future :-)
> >>
> >> I think it is a good idea to consolidate stream SQL and CEP in the long
> >> run. CEP's additional features compared to SQL boil down to pattern
> >> detection. Once we have this, it should be only a question of defining
> the
> >> SQL syntax for event patterns in order to integrate CEP with stream SQL.
> >> Oracle has already defined an extension [1] to detect patterns in a set
> of
> >> table rows. This or Esper's event processing language (EPL) [2] could
> be a
> >> good starting point.
> >>
> >> [1] https://docs.oracle.com/database/121/DWHSG/pattern.htm#DWHSG8959
> >> [2] http://www.espertech.com/esper/release-5.2.0/esper-reference/html/
> >>
> >> Cheers,
> >> Till
> >>
> >> On Mon, Jan 11, 2016 at 10:12 AM, Fabian Hueske 
> >> wrote:
> >>
> >> > Thanks for the feedback!
> >> >
> >> > We will start the SQL effort with putting the existing (batch) Table
> >> API on
> >> > top of Apache Calcite.
> >> > From there we continue to add streaming support for the Table API
> >> before we
> >> > put a StreamSQL interface on top.
> >> >
> >> > Consolidating the efforts with the CEP library sounds like a good idea
> >> to
> >> > me.
> >> > Maybe it can be nicely integrated with the streaming table API and
> >> later as
> >> > well with the StreamSQL interface (the StreamSQL dialect is not
> defined
> >> > yet).
> >> >
> >> > @Till: What do you think about adding CEP features to the Table API.
> >> From
> >> > the CEP design doc, it looks like we need to add a pattern matching
> >> > operator in addition to the window features that we need to add for
> >> > streaming Table API in any case.
> >> >
> >> > Best, Fabian
> >> >
> >> > 2016-01-11 4:03 GMT+01:00 Jiangsong (Hi) :
> >> >
> >> > > I suggest refering to Esper EPL[1], which is a SQL-standard language
> >> > > extend to offering a cluster of window, pattern matching.  EPL can
> >> both
> >> > > support Streaming SQL and CEP with one unified syntax.
> >> > >
> >> > > [1]
> >> > >
> >> >
> >>
> http://www.espertech.com/esper/release-5.2.0/esper-reference/pdf/esper_reference.pdf
> >> > >   (Chapter 5. EPL Reference: Clauses)
> >> > >
> >> > >
> >> > > Regards
> >> > > Song
> >> > >
> >> > >
> >> > > -邮件原件-
> >> > > 发件人: Chiwan Park [mailto:chiwanp...@apache.org]
> >> > > 发送时间: 2016年1月11日 10:31
> >> > > 收件人: dev@flink.apache.org
> >> > > 主题: Re: Effort to add SQL / StreamSQL to Flink
> >> > >
> >> > > We still don’t have a concensus about the streaming SQL and CEP
> >> library
> >> > on
> >> > > Flink. Some people want to merge these two libraries. Maybe we have
> to
> >> > > discuss about this in mailing list.
> >> > >
> >> > > > On Jan 11, 2016, at 10:53 AM, Nick Dimiduk 
> >> wrote:
> >> > > >
> >> > > > What's the relationship between the streaming SQL proposed here
> and
> >> > > > the CEP syntax proposed earlier in the week?
> >> > > >
> >> > > > On Sunday, January 10, 2016, Henry Saputra <
> henry.sapu...@gmail.com
> >> >
> >> > > wrote:
> >> > > >
> >> > > >> Awesome! Thanks for the reply, Fabian.
> >> > > >>
> >> > > >> - Henry
> >> > > >>
> >> > > >> On Sunday, January 10, 2016, Fabian Hueske  >> > > >> > wrote:
> >> > > >>
> >> > > >>> Hi Henry,
> >> > > >>>
> >> > > >>> There is https://issues.apache.org/jira/browse/FLINK-2099 and a
> >> few
> >> > > >>> subissues.
> >> > > >>> I'll reorganize these and add more issues for the tasks
> described
> >> in
> >> > > >>> the design document 

Re: Release 1.0 Migration Guide

[Vasiliki Kalavri]

Thanks Ufuk! I'll add the Gelly-related breaking changes.

-Vasia.

On 3 March 2016 at 10:55, Márton Balassi  wrote:

> Thanks for initiating this Ufuk. Updated the streaming hashing mention -
> whether it is api breaking is questionable, so I would place it last in the
> list. But definitely good to mention it there.
>
> On Thu, Mar 3, 2016 at 10:48 AM, Ufuk Celebi  wrote:
>
> > Hey all,
> >
> > let's make sure that we have a good migration guide for the 1.0
> > release with the announcement.
> >
> > Can you please add relevant changes that you are aware of to this
> document:
> >
> >
> >
> https://docs.google.com/document/d/13RqZ-_7_w1pYCi4cm_94XfaxwyTS1R_9KYUmD7jmGV8
> >
> > I've created two sections, one for API breaking changes and the other
> > for deprecated components.
> >
> > Best,
> >
> > Ufuk
> >
>

Re: [VOTE] Release Apache Flink 1.0.0 (RC3)

[Vasiliki Kalavri]

Thank you Robert! +1 for the google doc.

On 1 March 2016 at 09:10, Aljoscha Krettek  wrote:

> Very good, I’ll test the savepoints again and also try to hammer some of
> the recent API fixes.
>
> Could we also have the usual google doc for keeping track of the basic
> checks?
> > On 01 Mar 2016, at 09:08, Robert Metzger  wrote:
> >
> > Dear Flink community,
> >
> > Please vote on releasing the following candidate as Apache Flink version
> 1.0
> > .0.
> >
> > This is the third RC.
> >
> >
> > The commit to be voted on
> > (*http://git-wip-us.apache.org/repos/asf/flink/commit/41451e22
> > *)
> > 41451e22b0d37f78e39a0f60ab2402658edfd3c4
> >
> > Branch:
> > release-1.0.0-rc3 (see
> >
> https://git1-us-west.apache.org/repos/asf/flink/repo?p=flink.git;a=shortlog;h=refs/heads/release-
> > 1.0.0-rc3)
> >
> > The release artifacts to be voted on can be found at:
> > http://people.apache.org/~rmetzger/flink-1.0.0-rc3/
> >
> > The release artifacts are signed with the key with fingerprint D9839159:
> > http://www.apache.org/dist/flink/KEYS
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapacheflink-1065
> >
> > -
> >
> > The vote is open until Friday and passes if a majority of at least three
> > +1 PMC votes are cast.
> >
> > The vote ends on Friday, March 4, 11:00 CET.
> >
> > [ ] +1 Release this package as Apache Flink 1.0.0
> > [ ] -1 Do not release this package because ...
>
>

Re: Inconvenient (unforeseen?) consequences of PR #1683

[Vasiliki Kalavri]

In my opinion, the fat jar solution is easier than having to copy the Gelly
jar to all task managers.
I would be in favor of including flink-gelly in the flink-examples module,
but we can also simply document the current behavior nicely.

-V.

On 29 February 2016 at 12:15, Till Rohrmann <trohrm...@apache.org> wrote:

> Good catch :-) I mean we could also change the behaviour to include
> flink-gelly in the flink-gelly-examples module.
>
> On Mon, Feb 29, 2016 at 12:13 PM, Vasiliki Kalavri <
> vasilikikala...@gmail.com> wrote:
>
> > Thanks Till! Then, we'd better update the docs, too. Currently both links
> > to examples and cluster execution instructions are broken.
> > I'll create an issue.
> >
> > -V.
> >
> > On 29 February 2016 at 11:54, Till Rohrmann <trohrm...@apache.org>
> wrote:
> >
> > > Hi Vasia,
> > >
> > > that is because you're missing the flink-gelly dependency. If you just
> > > build flink-gelly-examples, then it won't contain flink-gelly because
> it
> > is
> > > not a fat jar. You either have to install flink-gelly on your cluster
> or
> > > package it in the final user jar.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Sat, Feb 27, 2016 at 7:19 PM, Vasiliki Kalavri <
> > > vasilikikala...@gmail.com
> > > > wrote:
> > >
> > > > Hi squirrels,
> > > >
> > > > sorry I've been slow to respond to this, but I'm now testing RC1 and
> > I'm
> > > a
> > > > bit confused with this change.
> > > >
> > > > So far, the easier way to run a Gelly example on a cluster was to
> > package
> > > > and submit the Gelly jar.
> > > > Now, since the flink-gelly project doesn't contain the examples
> > anymore,
> > > I
> > > > tried running a Gelly example by the packaging flink-gelly-examples
> jar
> > > > (mvn package). However, this gives me a ClassNotFoundException.
> > > > e.g. for SingleSourceShortestPaths, the following:
> > > >
> > > > java.lang.RuntimeException: Could not look up the main(String[])
> method
> > > > from the class
> > org.apache.flink.graph.examples.SingleSourceShortestPaths:
> > > > org/apache/flink/graph/spargel/MessagingFunction
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:478)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:216)
> > > > at
> > org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:922)
> > > > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:301)
> > > > at
> > > >
> > >
> >
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189)
> > > > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239)
> > > > Caused by: java.lang.NoClassDefFoundError:
> > > > org/apache/flink/graph/spargel/MessagingFunction
> > > > at java.lang.Class.getDeclaredMethods0(Native Method)
> > > > at java.lang.Class.privateGetDeclaredMethods(Class.java:2531)
> > > > at java.lang.Class.getMethod0(Class.java:2774)
> > > > at java.lang.Class.getMethod(Class.java:1663)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:473)
> > > > ... 5 more
> > > > Caused by: java.lang.ClassNotFoundException:
> > > > org.apache.flink.graph.spargel.MessagingFunction
> > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> > > > ... 10 more
> > > >
> > > >
> > > > What am I missing here?
> > > >
> > > > Thanks!
> > > > -Vasia.
> > > >
> > > > On 26 February 2016 at 14:10, Márton Balassi <
> balassi.mar...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Thanks, +1.
> > > > >
>

Re: Inconvenient (unforeseen?) consequences of PR #1683

[Vasiliki Kalavri]

Thanks Till! Then, we'd better update the docs, too. Currently both links
to examples and cluster execution instructions are broken.
I'll create an issue.

-V.

On 29 February 2016 at 11:54, Till Rohrmann <trohrm...@apache.org> wrote:

> Hi Vasia,
>
> that is because you're missing the flink-gelly dependency. If you just
> build flink-gelly-examples, then it won't contain flink-gelly because it is
> not a fat jar. You either have to install flink-gelly on your cluster or
> package it in the final user jar.
>
> Cheers,
> Till
>
> On Sat, Feb 27, 2016 at 7:19 PM, Vasiliki Kalavri <
> vasilikikala...@gmail.com
> > wrote:
>
> > Hi squirrels,
> >
> > sorry I've been slow to respond to this, but I'm now testing RC1 and I'm
> a
> > bit confused with this change.
> >
> > So far, the easier way to run a Gelly example on a cluster was to package
> > and submit the Gelly jar.
> > Now, since the flink-gelly project doesn't contain the examples anymore,
> I
> > tried running a Gelly example by the packaging flink-gelly-examples jar
> > (mvn package). However, this gives me a ClassNotFoundException.
> > e.g. for SingleSourceShortestPaths, the following:
> >
> > java.lang.RuntimeException: Could not look up the main(String[]) method
> > from the class org.apache.flink.graph.examples.SingleSourceShortestPaths:
> > org/apache/flink/graph/spargel/MessagingFunction
> > at
> >
> >
> org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:478)
> > at
> >
> >
> org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:216)
> > at org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:922)
> > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:301)
> > at
> >
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189)
> > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239)
> > Caused by: java.lang.NoClassDefFoundError:
> > org/apache/flink/graph/spargel/MessagingFunction
> > at java.lang.Class.getDeclaredMethods0(Native Method)
> > at java.lang.Class.privateGetDeclaredMethods(Class.java:2531)
> > at java.lang.Class.getMethod0(Class.java:2774)
> > at java.lang.Class.getMethod(Class.java:1663)
> > at
> >
> >
> org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:473)
> > ... 5 more
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.flink.graph.spargel.MessagingFunction
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> > ... 10 more
> >
> >
> > What am I missing here?
> >
> > Thanks!
> > -Vasia.
> >
> > On 26 February 2016 at 14:10, Márton Balassi <balassi.mar...@gmail.com>
> > wrote:
> >
> > > Thanks, +1.
> > >
> > > On Fri, Feb 26, 2016 at 12:35 PM, Stephan Ewen <se...@apache.org>
> wrote:
> > >
> > > > Hi!
> > > >
> > > > I think it is a release blocker.
> > > >
> > > > That was a change to make it possible to use Flink with SBT (and make
> > it
> > > > easier to use it with Maven).
> > > > Just had an offline chat with Till, and we suggest the following
> > > solution:
> > > >
> > > > All libraries should be split into two projects:
> > > >   - library
> > > >   - library-examples
> > > >
> > > > The "library" project has the core flink dependencies as "provided"
> > > > The "library-examples" project has both the "library" and the core
> > flink
> > > > dependencies with scope "compile"
> > > >
> > > > That way the example should run in the IDE out of the cox, and users
> > that
> > > > reference the libraries will still get the correct packaging (include
> > the
> > > > library in the user jar, but not additionally the core flink jars).
> > > >
> > > > Greetings,
> > > > Stephan
> > > >
> > > >
> > > > On Thu, Feb 25, 2016 at 3:45 PM, Márton Balassi <
> > > balassi.mar...@gmail.com>
> > > > wrote:
> > > >
&g

Re: Inconvenient (unforeseen?) consequences of PR #1683

[Vasiliki Kalavri]

Hi squirrels,

sorry I've been slow to respond to this, but I'm now testing RC1 and I'm a
bit confused with this change.

So far, the easier way to run a Gelly example on a cluster was to package
and submit the Gelly jar.
Now, since the flink-gelly project doesn't contain the examples anymore, I
tried running a Gelly example by the packaging flink-gelly-examples jar
(mvn package). However, this gives me a ClassNotFoundException.
e.g. for SingleSourceShortestPaths, the following:

java.lang.RuntimeException: Could not look up the main(String[]) method
from the class org.apache.flink.graph.examples.SingleSourceShortestPaths:
org/apache/flink/graph/spargel/MessagingFunction
at
org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:478)
at
org.apache.flink.client.program.PackagedProgram.(PackagedProgram.java:216)
at org.apache.flink.client.CliFrontend.buildProgram(CliFrontend.java:922)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:301)
at
org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1189)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1239)
Caused by: java.lang.NoClassDefFoundError:
org/apache/flink/graph/spargel/MessagingFunction
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2531)
at java.lang.Class.getMethod0(Class.java:2774)
at java.lang.Class.getMethod(Class.java:1663)
at
org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:473)
... 5 more
Caused by: java.lang.ClassNotFoundException:
org.apache.flink.graph.spargel.MessagingFunction
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 10 more


What am I missing here?

Thanks!
-Vasia.

On 26 February 2016 at 14:10, Márton Balassi 
wrote:

> Thanks, +1.
>
> On Fri, Feb 26, 2016 at 12:35 PM, Stephan Ewen  wrote:
>
> > Hi!
> >
> > I think it is a release blocker.
> >
> > That was a change to make it possible to use Flink with SBT (and make it
> > easier to use it with Maven).
> > Just had an offline chat with Till, and we suggest the following
> solution:
> >
> > All libraries should be split into two projects:
> >   - library
> >   - library-examples
> >
> > The "library" project has the core flink dependencies as "provided"
> > The "library-examples" project has both the "library" and the core flink
> > dependencies with scope "compile"
> >
> > That way the example should run in the IDE out of the cox, and users that
> > reference the libraries will still get the correct packaging (include the
> > library in the user jar, but not additionally the core flink jars).
> >
> > Greetings,
> > Stephan
> >
> >
> > On Thu, Feb 25, 2016 at 3:45 PM, Márton Balassi <
> balassi.mar...@gmail.com>
> > wrote:
> >
> > > Issued JIRA ticket 3511 to make it referable in other discussions. [1]
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-3511
> > >
> > > On Thu, Feb 25, 2016 at 3:36 PM, Márton Balassi <
> > balassi.mar...@gmail.com>
> > > wrote:
> > >
> > > > Recent changes to the build [1] where many libraries got their core
> > > > dependencies (the ones included in the flink-dist fat jar) moved to
> the
> > > > provided scope.
> > > >
> > > > The reasoning was that when submitting to the Flink cluster the
> > > > application already has these dependencies, while when a user writes
> a
> > > > program against these libraries she will include the core
> dependencies
> > > > explicitly anyway.
> > > >
> > > > There is one other case of usage however, namely when someone is
> trying
> > > to
> > > > run an application defined in these libraries depending on the core
> > jars.
> > > > To give an example if you were to run the Gelly ConnectedComponents
> > > example
> > > > [2] from an IDE after importing Flink (or running with java -jar
> > without
> > > > including the flink fat jar in the classpath) you would receive the
> > > > following class not found exception as per the current master:
> > > >
> > > > Exception in thread "main" java.lang.NoClassDefFoundError:
> > > > org/apache/flink/api/common/ProgramDescription
> > > > at java.lang.ClassLoader.defineClass1(Native Method)
> > > > at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
> > > > at
> > > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> > > > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> > > > at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> > > > at 

Re: [VOTE] Release Apache Flink 1.0.0 (RC1)

[Vasiliki Kalavri]

Hi squirrels,

here's my testing outcome so far:

- Examples: Ran all examples locally and on a cluster, both from CLI and
web submission tool
Issues:
1. PageRank example doesn't run without arguments anymore. I have a fix
together with some doc fixes.

- CLI: tested locally and on cluster
Issues: None.

- WebUI: tested locally and on cluster
Issues:
1. Inside a job view, the "Subtasks" and "TaskManagers" tabs have the same
content. Is this desired?

- HA: tested on a 6-node cluster with 2 masters.
Issues:
1. After new leader election, the job history is cleaned up (at least in
the WebUI). Is this on purpose?
2. After cluster restart, the jobmanager remembers and tries to re-submit
previously failed resubmissions.
This is one is a bit tricky:
I had a batch job running and killed the master. After the new master took
over, job resubmission failed because the HDFS output directory already
existed. After re-starting the whole cluster and removing the HDFS
directory, the new jobmanager re-submitted the previously failed batch job.
3. Upon starting the cluster I get the following warning message "[WARNING]
1 instance(s) of jobmanager are already running", when jps shows no
existing jobmanager process.

Let me know if these are valid issues and I should open corresponding JIRAs
or I'm misunderstanding something :)

Cheers,
-Vasia.


On 25 February 2016 at 17:04, Stephan Ewen  wrote:

> Thanks Marton, the issue is quite serious, I agree.
>
> It is a bit tricky to solve, unfortunately. It seems very hard to make
> experience inside the IDE, with Maven, and with SBT smooth.
> Maven/SBT packaging needs "provided" dependencies, while IntelliJ need
> "compile" dependencies.
>
> On Thu, Feb 25, 2016 at 3:47 PM, Márton Balassi 
> wrote:
>
> > @Stephan on PR #1685. Fair enough, 1.0.1 is fine. We will try to get it
> in
> > soon anyway.
> >
> > Please consider a build inconvenience that I have just reported to both
> the
> > mailing list and JIRA in the meantime. [1]
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-3511
> >
> >
> >
> > On Thu, Feb 25, 2016 at 3:27 PM, Stephan Ewen  wrote:
> >
> > > Concerning the Pull Request mentioned by Marton:
> > >
> > > I think it is a transparent bugfix patch. It is not really affecting
> > > behavior that we want users to make assumptions about (assignment of
> keys
> > > to partitions should not be hardwired into applications).
> > > The only affected program I can think of that is affected is the
> > "queryable
> > > state" program (and similar programs), which is using internal API and
> > > behavior that are always subject to change
> > >
> > > As such, I would think it is also okay for to be included in a bugfix
> > > release.
> > > So if the authors don't have the fix ready in time, it should still be
> > fine
> > > to make it part of 1.0.1 (which will probably come quite soon, as usual
> > > after a full release).
> > >
> > >
> > > On Thu, Feb 25, 2016 at 2:42 PM, Ufuk Celebi  wrote:
> > >
> > > > Maybe you can check whether this is a problem, too:
> > > > https://issues.apache.org/jira/browse/FLINK-3501
> > > >
> > > > Under the assumption that no major functionality changes, I will
> > > > continue with the functional checks.
> > > >
> > > > On Thu, Feb 25, 2016 at 2:32 PM, Robert Metzger  >
> > > > wrote:
> > > > > Damn. I agree that this is a blocker.
> > > > > I use the maven-enforcer-plugin to check for the right maven, but
> the
> > > > build
> > > > > profile that runs the profile is only active during "deploy", not
> > when
> > > > > packaging the binaries.
> > > > > That's why I didn't realize that I build the binaries with the
> wrong
> > > > maven
> > > > > version.
> > > > >
> > > > > I suggest that we keep collecting problems until Friday afternoon
> > > (CET).
> > > > > Then I'll create the next release candidate.
> > > > >
> > > > > I'd also like to address this one:
> > > > > https://issues.apache.org/jira/browse/FLINK-3509
> > > > >
> > > > >
> > > > > On Thu, Feb 25, 2016 at 2:23 PM, Fabian Hueske 
> > > > wrote:
> > > > >
> > > > >> Hi folks,
> > > > >>
> > > > >> I think I found a release blocker.
> > > > >> The flink-dist JAR file contains non-relocated classes of Google
> > Guava
> > > > and
> > > > >> Apache HttpComponents.
> > > > >>
> > > > >> Fabian
> > > > >>
> > > > >> 2016-02-25 13:21 GMT+01:00 Chesnay Schepler :
> > > > >>
> > > > >> > tested the RC on Windows:
> > > > >> >
> > > > >> > - source compiles
> > > > >> > - some tests categorically fail: see FLINK-3491 / FLINK-3496
> > > > >> > - start/stop scripts work in both cygwin and windows CMD
> > > > >> > - ran several examples from batch/streaming/python
> > > > >> > - scripts also work on paths containing spaces
> > > > >> >
> > > > >> >
> > > > >> > On 25.02.2016 12:41, Robert Metzger wrote:
> > > > >> >
> > > > >> >> (I'm removing user@ from 

Re: Affiity Propagation

[Vasiliki Kalavri]

Hi Josep,

welcome to the the Flink dev list!

There exists a JIRA issue for Affinity Propagation that is currently
unassigned (FLINK-1707 [1]). You can see in the comments that someone
started working on it, but they dropped it.
As far as I know, nobody is working on this right now, so let me know if
you'd like to pick it up and I'll assign it to you :)

Before you start coding, make sure you go through the contribution
guidelines [2] and the Gelly guide [3].
Feel free to ask us questions here or in the JIRA issue. We'll be happy to
help!

Cheers
-Vasia.

[1]: https://issues.apache.org/jira/browse/FLINK-1707
[2]: https://flink.apache.org/how-to-contribute.html
[3]:
https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/libs/gelly.html

On 17 February 2016 at 20:33, Josep Rubio  wrote:

> Hi,
>
> I’ve been working on an implementation of the binary approach of Affinity
> Propagation for Giraph (
> https://github.com/joseprupi/okapi/tree/master/src/main/java/ml/grafos/okapi/clustering/ap)
> and willing to help Flink I’d like to use it as a starting point to
> implement it for Gelly.
>
> Just to make you know, and of course comments or thoughts are welcome.
>
> Thanks!
>
>
>

Re: User Feedback

[Vasiliki Kalavri]

Hi Martin,

thank you for the feedback. Let me try to answer some of your concerns.


On 9 February 2016 at 15:35, Martin Neumann  wrote:

> During this year's FOSDEM Martin Junghans and I set together and gathered
> some feedback for the Flink project. It is based on our personal experience
> as well as the feedback and questions from People we taught the system.
> This is going to be a longer email therefore I have split things into
> categories:
>
>
> *Website and Documentation:*
>
>1. *Out-dated Google Search results*: Google searches lead to outdated
>web site versions (e.g. “flink transformations” or “flink iterations”
>return the 0.7 version of the corresponding pages).
>

​I'm not sure we can do much about this. I would suggest searching in the
documentation instead of relying on Google.
There is a search box on the top of all documentation pages.



>2. *Invalid Links on Website: *Links are confusing / broken (e.g. the
>Gelly /ML Links on the start page lead to the top of the feature page
>(which start with streaming) *-> maybe this can be validated
>automatically?*
>
>
​That was bug recently reported and fixed (see FLINK-3316). If you find
​ more of those, please report by opening a JIRA or Pull Request​.



>
> *Batch API:*
>
>1. *.reduceGroup(GroupReduceFunction) and
>.groupCombine(CombineGroupFunction): *In other functions such as
>.flatMap(FlatMapFunction) the function call matches the naming of the
>operator. This structure is quite convenient for new user since they can
>make use of the autocompletion features of the IDE, basically start
> typing
>the function call and you get the correct class. This does not work for
>.reduceGroup() and .groupCombine() since the names are switched around.
> *->
>maybe the function can be renamed*
>

​I agree this might be strange for new users, but I think it will be much
more annoying for existing users if we change this. In my view, it's not an
important case to justify breaking the API.



>2. *.print() and env.execute(): *Often .print() is used for debugging
>and developing programs replacing regular data sinks. Such a project
> will
>not run until the env.execute() is removed. It's very easy to forget to
> add
>it back in, once you change the .print() back to a proper sink. The
> project
>now will compile fine but will not produce any output since .execute()
> is
>missing. This is a very difficult bug to find especially since there is
> no
>warning or error when running the job. It’s common that people use more
>than one .print() statement during debugging and development. This can
> lead
>to confusion since each .print() forces the program to execute so the
>execution behavior is different than without the print. This is
> especially
>important, if the program contains non-deterministic data generation
> (like
>generating IDs). In the stream API .print() would not require to
>remove .execute() as a result the behavior of the two interfaces is
>inconsistent.
>

​This is indeed an issue that many users find hard to get used to. We have
changed the behavior of print() a couple of times before and I'm not sure
it would be wise to do so again. Actually, once a user understands the
difference between eager and lazy sinks, I think it's quite easy​ to avoid
mistakes.



>3. *calling new when applying an operator eg: .reduceGroup(new
>GroupReduceFunction()): *Some of the people I taught the API’s to where
>confused by this. They knew it was a distributed system and they were
>wondering where the constructor would be actually called. They expected
> to
>hand a class to the function that would be initialized on each of the
>worker nodes. *-> maybe have a section about this in the documentation*
>

​I'm not sure I understand the confusion with this one. The goal of
high-level APIs is to relieve the users from having to think about
distribution. The only thing they need to understand is the
DataSet/DataStream abstractions and how to create transformations on them.


>4. *.project() loses type information / does not support .returns(..):
> *The
>project transformation currently loses type information which affects
>chained call with other transformations. One workaround is the
> definition
>of an intermediate dataset. However, to be consistent with other
> operators,
>project should support .returns() to define a type information if
> needed.
>
>
​I'm not sure _why_ this is the case. Maybe someone who knows more can
clarify this one.​



>
> *Stream API:*
>
>1. *.keyBy(): *Currently .keyBy() creates a KeyedDataStream but every
>operator that consumes a KeyedDataStream produces a DataStream. This
> means
>it is not possible to create a program that uses a keyBy() followed by a
>sequence of transformation for each key without having to reapply
> keyBy()
>after each of 

Re: [ANNOUNCE] Chengxiang Li added as committer

[Vasiliki Kalavri]

Congratulations! Welcome Chengxiang Li!

On 19 January 2016 at 11:02, Fabian Hueske  wrote:

> Hi everybody,
>
> I'd like to announce that Chengxiang Li accepted the PMC's offer to become
> a committer of the Apache Flink project.
>
> Please join me in welcoming Chengxiang Li!
>
> Best, Fabian
>

Re: Old Flink Graph Repository

[Vasiliki Kalavri]

I believe I do!

On 15 January 2016 at 16:20, Stephan Ewen <se...@apache.org> wrote:

> Sounds good. Do you have push rights there to do that?
>
> On Fri, Jan 15, 2016 at 3:51 PM, Vasiliki Kalavri <
> vasilikikala...@gmail.com
> > wrote:
>
> > Hey,
> >
> > we can delete the content, but I wouldn't remove the repository.
> > In the early Gelly days we shared the link of this repository in several
> > places (e.g. my FOSDEM'14 talk and few other presentations), so people
> > might still land there. I believe it would be nice if we could keep it
> > empty with a note in the readme.
> >
> > -Vasia.
> >
> > On 15 January 2016 at 14:59, Stephan Ewen <se...@apache.org> wrote:
> >
> > > Hi!
> > >
> > > Just found his: https://github.com/project-flink/flink-graph
> > >
> > > Since Gelly is part of Flink now, can we remove this repository, or
> > delete
> > > everything except a readme with a forward pointer?
> > >
> > > Greetings,
> > > Stephan
> > >
> >
>

Re: [gelly] partition-centric iterations

[Vasiliki Kalavri]

Hi Martin,

thanks for your input!


On 7 January 2016 at 09:36, Martin Junghanns <m.jungha...@mailbox.org>
wrote:

> Hi,
>
> this would be a very nice addition! I had a glimpse look into the PC
> implementation and the two library algorithms and when you get the
> idea, it is easy to follow what's happening. The benchmark results are
> also very promising.
>
> I got some questions about partitions:
>
> 1) I was wondering if the coGroup after the PartitionProcessUDF is aware
> of the initial partitioning. To be more precise, is it guaranteed, that
> the (vertex-)partitions do not change during the whole iteration and the
> RichEdges "stay" on the same worker?
>


​I think you can find the questions to your answers in the attached
generated plan [1].
The edges are cached, meaning that the partitions remain on the same
workers across iterations.
Records from the first coGroup to the mapPartition are forwarded (no
shuffling).
The output of the mapPartition operator is a tuple of <vertexID, message>,
which is sorted and shuffled to reach the destination vertices.



> 2) If 1) is true, would it work to call partition(udf) on the vertex
> data set before the PC iteration starts?
>

Do you mean replacing the default hash partitioning with some custom
partition function? I guess that'd would work, yes.​



> 3) If 1) is false, would it work to call partition(udf) after the
> coGroup during the delta iteration to achieve partition stability?
>
> What I got in mind is something like:
>
> 1 Compute partitions or a set of subgraphs
>   (via CC, LP, Pattern Matching, ...)
> 2 Partition Vertices by computed Partition/Subgraph ID
> 3 Compute Algorithm X (Page Rank, BC, SSSP, FSM...) per
> Partition/Subgraph via PC iteration
>
> Again, nice work!
>
> Best,
> Martin
>



​Cheers,
-Vasia.​

[1]:
https://drive.google.com/file/d/0BzQJrI2eGlyYSHZJTHFObmgxTXM/view?usp=sharing


>
> On 06.01.2016 23:29, Vasiliki Kalavri wrote:
> > Hi squirrels,
> >
> > here's some more Gelly iteration love for you ;)
> >
> > Together with Kien and Nikola (students at KTH, cc-ed), we looked into
> > partition-centric iterations for Gelly. The idea of the partition-centric
> > model is to expose the graph partition structure to the user (not just a
> > single vertex, but all vertices belonging to the same partition at once).
> > It allows for local graph structure exploitation inside a partition and
> > potentially avoids unnecessary communication. The model was first
> > introduced in [1] and we used it as inspiration, but our current
> > implementation is closer to the Spargel model.
> >
> > Kien and Nikola prepared some slides with their design choices and
> > experiments [2]. The connected components results are quite impressive :)
> > There is still room for optimization, but you can find the wip repository
> > in [3] if you're interested.
> >
> > What do you think about exposing this as a Gelly iteration model? The
> task
> > has been in the Gelly roadmap and there are certainly performance
> > advantages for some algorithms. This implementation can be basically
> > considered as a Spargel optimization (existing VertexUpdateFunctions from
> > Spargel can actually be re-used). On the other hand, the model is quite
> > low-level compared to Spargel and vertex-centric, since users have to
> deal
> > with inside-partition algorithms themselves (e.g. compare the connected
> > components implementation [4] with the existing Spargel/GSA ones).
> >
> > Thanks!
> > -Vasia.
> >
> > [1]: http://researcher.ibm.com/researcher/files/us-ytian/giraph++.pdf
> > [2]:
> >
> https://drive.google.com/file/d/0BzQJrI2eGlyYRmlKTjMyVEk1T0U/view?usp=sharing
> > [3]: https://github.com/vasia/gelly-partition-centric
> > [4]:
> >
> https://github.com/vasia/gelly-partition-centric/blob/master/src/main/java/org/apache/flink/graph/partition/centric/library/PCConnectedComponents.java
> >
>

[gelly] partition-centric iterations

[Vasiliki Kalavri]

Hi squirrels,

here's some more Gelly iteration love for you ;)

Together with Kien and Nikola (students at KTH, cc-ed), we looked into
partition-centric iterations for Gelly. The idea of the partition-centric
model is to expose the graph partition structure to the user (not just a
single vertex, but all vertices belonging to the same partition at once).
It allows for local graph structure exploitation inside a partition and
potentially avoids unnecessary communication. The model was first
introduced in [1] and we used it as inspiration, but our current
implementation is closer to the Spargel model.

Kien and Nikola prepared some slides with their design choices and
experiments [2]. The connected components results are quite impressive :)
There is still room for optimization, but you can find the wip repository
in [3] if you're interested.

What do you think about exposing this as a Gelly iteration model? The task
has been in the Gelly roadmap and there are certainly performance
advantages for some algorithms. This implementation can be basically
considered as a Spargel optimization (existing VertexUpdateFunctions from
Spargel can actually be re-used). On the other hand, the model is quite
low-level compared to Spargel and vertex-centric, since users have to deal
with inside-partition algorithms themselves (e.g. compare the connected
components implementation [4] with the existing Spargel/GSA ones).

Thanks!
-Vasia.

[1]: http://researcher.ibm.com/researcher/files/us-ytian/giraph++.pdf
[2]:
https://drive.google.com/file/d/0BzQJrI2eGlyYRmlKTjMyVEk1T0U/view?usp=sharing
[3]: https://github.com/vasia/gelly-partition-centric
[4]:
https://github.com/vasia/gelly-partition-centric/blob/master/src/main/java/org/apache/flink/graph/partition/centric/library/PCConnectedComponents.java

Re: [gelly] Spargel model rework

[Vasiliki Kalavri]

issue created: https://issues.apache.org/jira/browse/FLINK-3207

If anyone has any other suggestion about the renaming, let me know :)

-V.

On 5 January 2016 at 11:52, Aljoscha Krettek <aljos...@apache.org> wrote:

> Nice to hear. :D
>
> I think you can go ahead and add the Jira. About the renaming: I also
> think that it would make sense to do it.
> > On 04 Jan 2016, at 19:48, Vasiliki Kalavri <vasilikikala...@gmail.com>
> wrote:
> >
> > Hello squirrels and happy new year!
> >
> > I'm reviving this thread to share some results and discuss next steps.
> >
> > Using the Either type I was able to get rid of redundant messages and
> > vertex state. During the past few weeks, I have been running experiments,
> > which show that the performance of this "Pregel" model has improved a
> lot :)
> > In [1], you can see the speedup of GSA and Pregel over Spargel, for SSSP
> > and Connected Components (CC), for the Livejournal (68m edges), Orkut
> (117m
> > edges) and Wikipedia (340m edges) datasets.
> >
> > Regarding next steps, if no objections, I will open a Jira for adding a
> > Pregel iteration abstraction to Gelly. The Gelly guide has to be updated
> to
> > reflect the spectrum of iteration abstractions that we have discussed in
> > this thread, i.e. Pregel -> Spargel (Scatter-Gather) -> GSA.
> >
> > I think it might also be a good idea to do some renaming. Currently, we
> > call the Spargel iteration "vertex-centric", which fits better to the
> > Pregel abstraction. I propose we rename the spargel iteration into
> > "scatter-gather" or "signal-collect" (where it was first introduced [2]).
> > Any other ideas?
> >
> > Thanks,
> > -Vasia.
> >
> > [1]:
> >
> https://drive.google.com/file/d/0BzQJrI2eGlyYRTRjMkp1d3R6eVE/view?usp=sharing
> > [2]: http://link.springer.com/chapter/10.1007/978-3-642-17746-0_48
> >
> > On 11 November 2015 at 11:05, Stephan Ewen <se...@apache.org> wrote:
> >
> >> See: https://issues.apache.org/jira/browse/FLINK-3002
> >>
> >> On Wed, Nov 11, 2015 at 10:54 AM, Stephan Ewen <se...@apache.org>
> wrote:
> >>
> >>> "Either" an "Optional" types are quite useful.
> >>>
> >>> Let's add them to the core Java API.
> >>>
> >>> On Wed, Nov 11, 2015 at 10:00 AM, Vasiliki Kalavri <
> >>> vasilikikala...@gmail.com> wrote:
> >>>
> >>>> Thanks Fabian! I'll try that :)
> >>>>
> >>>> On 10 November 2015 at 22:31, Fabian Hueske <fhue...@gmail.com>
> wrote:
> >>>>
> >>>>> You could implement a Java Either type (similar to Scala's Either)
> >> that
> >>>>> either has a Message or the VertexState and a corresponding
> >>>> TypeInformation
> >>>>> and TypeSerializer that serializes a byte flag to indicate which both
> >>>> types
> >>>>> is used.
> >>>>> It might actually make sense, to add a generic Either type to the
> Java
> >>>> API
> >>>>> in general (similar to the Java Tuples with resemble the Scala
> >> Tuples).
> >>>>>
> >>>>> Cheers, Fabian
> >>>>>
> >>>>> 2015-11-10 22:16 GMT+01:00 Vasiliki Kalavri <
> >> vasilikikala...@gmail.com
> >>>>> :
> >>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> after running a few experiments, I can confirm that putting the
> >>>> combiner
> >>>>>> after the flatMap is indeed more efficient.
> >>>>>>
> >>>>>> I ran SSSP and Connected Components with Spargel, GSA, and the
> >> Pregel
> >>>>> model
> >>>>>> and the results are the following:
> >>>>>>
> >>>>>> - for SSSP, Spargel is always the slowest, GSA is a ~1.2x faster and
> >>>>> Pregel
> >>>>>> is ~1.1x faster without combiner, ~1.3x faster with combiner.
> >>>>>> - for Connected Components, Spargel and GSA perform similarly, while
> >>>>> Pregel
> >>>>>> is 1.4-1.6x slower.
> >>>>>>
> >>>>>> To start with, this is much better than I expected :)
> >>>>>> However, there is a main shortcoming in my current implementation
> >

Re: [gelly] Spargel model rework

[Vasiliki Kalavri]

Hello squirrels and happy new year!

I'm reviving this thread to share some results and discuss next steps.

Using the Either type I was able to get rid of redundant messages and
vertex state. During the past few weeks, I have been running experiments,
which show that the performance of this "Pregel" model has improved a lot :)
In [1], you can see the speedup of GSA and Pregel over Spargel, for SSSP
and Connected Components (CC), for the Livejournal (68m edges), Orkut (117m
edges) and Wikipedia (340m edges) datasets.

Regarding next steps, if no objections, I will open a Jira for adding a
Pregel iteration abstraction to Gelly. The Gelly guide has to be updated to
reflect the spectrum of iteration abstractions that we have discussed in
this thread, i.e. Pregel -> Spargel (Scatter-Gather) -> GSA.

I think it might also be a good idea to do some renaming. Currently, we
call the Spargel iteration "vertex-centric", which fits better to the
Pregel abstraction. I propose we rename the spargel iteration into
"scatter-gather" or "signal-collect" (where it was first introduced [2]).
Any other ideas?

Thanks,
-Vasia.

[1]:
https://drive.google.com/file/d/0BzQJrI2eGlyYRTRjMkp1d3R6eVE/view?usp=sharing
[2]: http://link.springer.com/chapter/10.1007/978-3-642-17746-0_48

On 11 November 2015 at 11:05, Stephan Ewen <se...@apache.org> wrote:

> See: https://issues.apache.org/jira/browse/FLINK-3002
>
> On Wed, Nov 11, 2015 at 10:54 AM, Stephan Ewen <se...@apache.org> wrote:
>
> > "Either" an "Optional" types are quite useful.
> >
> > Let's add them to the core Java API.
> >
> > On Wed, Nov 11, 2015 at 10:00 AM, Vasiliki Kalavri <
> > vasilikikala...@gmail.com> wrote:
> >
> >> Thanks Fabian! I'll try that :)
> >>
> >> On 10 November 2015 at 22:31, Fabian Hueske <fhue...@gmail.com> wrote:
> >>
> >> > You could implement a Java Either type (similar to Scala's Either)
> that
> >> > either has a Message or the VertexState and a corresponding
> >> TypeInformation
> >> > and TypeSerializer that serializes a byte flag to indicate which both
> >> types
> >> > is used.
> >> > It might actually make sense, to add a generic Either type to the Java
> >> API
> >> > in general (similar to the Java Tuples with resemble the Scala
> Tuples).
> >> >
> >> > Cheers, Fabian
> >> >
> >> > 2015-11-10 22:16 GMT+01:00 Vasiliki Kalavri <
> vasilikikala...@gmail.com
> >> >:
> >> >
> >> > > Hi,
> >> > >
> >> > > after running a few experiments, I can confirm that putting the
> >> combiner
> >> > > after the flatMap is indeed more efficient.
> >> > >
> >> > > I ran SSSP and Connected Components with Spargel, GSA, and the
> Pregel
> >> > model
> >> > > and the results are the following:
> >> > >
> >> > > - for SSSP, Spargel is always the slowest, GSA is a ~1.2x faster and
> >> > Pregel
> >> > > is ~1.1x faster without combiner, ~1.3x faster with combiner.
> >> > > - for Connected Components, Spargel and GSA perform similarly, while
> >> > Pregel
> >> > > is 1.4-1.6x slower.
> >> > >
> >> > > To start with, this is much better than I expected :)
> >> > > However, there is a main shortcoming in my current implementation
> that
> >> > > negatively impacts performance:
> >> > > Since the compute function coGroup needs to output both new vertex
> >> values
> >> > > and new messages, I emit a wrapping tuple that contains both vertex
> >> state
> >> > > and messages and then filter them out based on a boolean field. The
> >> > problem
> >> > > is that since I cannot emit null fields, I emit a dummy message for
> >> each
> >> > > new vertex state and a dummy vertex state for each new message. That
> >> > > essentially means that the intermediate messages result is double in
> >> > size,
> >> > > if say the vertex values are of the same type as the messages (can
> be
> >> > worse
> >> > > if the vertex values are more complex).
> >> > > So my question is, is there a way to avoid this redundancy, by
> either
> >> > > emitting null fields or by creating an operator that could emit 2
> >> > different
> >> > > types of tuples?
> >> > >
> >> > > Thanks!

Re: 2015: A Year in Review for Apache Flink

[Vasiliki Kalavri]

Happy new year everyone!
Looking forward to all the great things the Apache Flink community will
accomplish in 2016 :))

Greetings from snowy Greece!
-Vasia.

On 31 December 2015 at 04:22, Henry Saputra  wrote:

> Dear All,
>
> It is almost end of 2015 and it has been busy and great year for Apache
> Flink =)
>
> Robert Metzger had posted great blog summarizing Apache Flink grow for
> this year:
>
>   https://flink.apache.org/news/2015/12/18/a-year-in-review.html
>
> Happy New Year everyone and thanks for being part of this great community!
>
>
> Thanks,
>
> - Henry
>

Re: Apache Tinkerpop & Geode Integration?

[Vasiliki Kalavri]

Hey,

I think I might have confused you, so let me try to explain :)

First, Gremlin is a language similar to Cypher, but it is also a traversal
machine, which also supports distributed traversals. For distributed
traversals, Gremlin uses a "graph computer", which runs the Gremlin
traversals using the BSP model. Essentially, vertices receive traversers as
messages and execute the traverser's step as the update function (for more
info see section 5 in [1]).

Thus, Tinkerpop has a GiraphGraphComputer to run on top of Giraph, a
SparkGraphComputer to run on top of Spark, etc.

The Tinkerpop community has offered to work on a FlinkGraphComputer, which,
similarly to the existing graph computers, will use one of the Flink/Gelly
iteration abstractions.

Now, there are 2 questions for the Flink community:
(1): do we think this is interesting/useful and something we can help them
with?
(2): do we think it makes sense to "host" the FlinkGraphComputer on the
Flink codebase?


Neo4j/Cypher on Flink is a separate discussion in my opinion. As far as I
understand, Cypher could run on Gremlin, but there is no compiler for it
yet. I have been discussing with people from Neo4j and we have jointly
written a description for a thesis project regarding OpenCypher on Flink.
The idea is to collaboratively supervise/help the student(s). Of course, if
anyone else is interested in this (not necessarily a student) we can always
use more help, so just let me know!

Thanks,
-Vasia.

​[1]: ​
http://arxiv.org/pdf/1508.03843v1.pdf


On 16 December 2015 at 19:21, Stephan Ewen <se...@apache.org> wrote:

> I am not very familiar with Gremlin, but I remember a brainstorming session
> with Martin Neumann on porting Cypher (the neo4j query language) to Flink.
> We looked at Cypher queries for filtering and traversing the graph.
>
> It looked like it would work well. We remember we could even model
> recursive conditions on traversals pretty well with delta iterations.
>
> If Gremlin's use cases are anything like Cypher, I could ping Martin and
> see if we can collect again some of those ideas.
>
> Stephan
>
>
>
> On Tue, Dec 15, 2015 at 5:35 PM, Vasiliki Kalavri <
> vasilikikala...@gmail.com
> > wrote:
>
> > Hi Dr. Fabian,
> >
> > thanks a lot for your answer!
> >
> >
> > On 15 December 2015 at 15:42, Fabian Hueske <fhue...@gmail.com> wrote:
> >
> > > Hi Vasia,
> > >
> > > I agree, Gremlin definitely looks like an interesting API for Flink.
> > > I'm not sure how it relates to Gelly. I guess Gelly would (initially)
> be
> > > more tightly integrated with the DataSet API whereas Gremlin would be a
> > > connector for other languages. Any ideas on this?
> > >
> >
> > The idea is to provide a FlinkGraphComputer which will use Gelly's
> > iterations to compile the Gremlin query language to Flink.
> > In my previous email, I linked to our discussion over at the Tinkerpop
> > mailing list, where you can find more details on this. By adding the
> > FlinkGraphComputer, we basically get any graph query language that
> compiles
> > to the Gremlin VM for free.
> >
> >
> > >
> > > Another question would be whether the connector should to into Flink or
> > > Tinkerpop. For example, the Spark, Giraph, and Neo4J connectors are all
> > > included in Tinkerpop.
> > > This should be discussed with the Tinkerpop community.
> > >
> > >
> > I'm copying from the Tinkerpop mailing list thread (link for full thread
> in
> > my previous email):​
> >
> >
> > *In the past, TinkerPop use to be a "dumping ground" for all
> > implementations, but we decided for TinkerPop3 that we would only have
> > "reference implementations" so users can play, system providers can
> learn,
> > and ultimately, system providers would provide TinkerPop support in their
> > distribution. As such, we would like to have FlinkGraphComputer
> distributed
> > with Flink. If that sounds like something your project would be
> comfortable
> > with, I think we can provide a JIRA/PR for FlinkGraphComputer (as well as
> > any necessary documentation). We can start with a JIRA ticket to get
> things
> > going. Thoughts?*
> >
> >
> > ​This is why I brought the conversation over here, so I hear the opinions
> > of the Flink community on this :)​
> >
> >
> >
> > > Best, Fabian
> > >
> >
> >
> > -Vasia.​
> >
> >
> >
> > >
> > >
> > > 2015-12-14 18:33 GMT+01:00 Vasiliki Kalavri <vasilikikala...@gmail.com
> >:
> > >
> > > > Ping

Re: Apache Tinkerpop & Geode Integration?

[Vasiliki Kalavri]

Hi Dr. Fabian,

thanks a lot for your answer!


On 15 December 2015 at 15:42, Fabian Hueske <fhue...@gmail.com> wrote:

> Hi Vasia,
>
> I agree, Gremlin definitely looks like an interesting API for Flink.
> I'm not sure how it relates to Gelly. I guess Gelly would (initially) be
> more tightly integrated with the DataSet API whereas Gremlin would be a
> connector for other languages. Any ideas on this?
>

The idea is to provide a FlinkGraphComputer which will use Gelly's
iterations to compile the Gremlin query language to Flink.
In my previous email, I linked to our discussion over at the Tinkerpop
mailing list, where you can find more details on this. By adding the
FlinkGraphComputer, we basically get any graph query language that compiles
to the Gremlin VM for free.


>
> Another question would be whether the connector should to into Flink or
> Tinkerpop. For example, the Spark, Giraph, and Neo4J connectors are all
> included in Tinkerpop.
> This should be discussed with the Tinkerpop community.
>
>
I'm copying from the Tinkerpop mailing list thread (link for full thread in
my previous email):​


*In the past, TinkerPop use to be a "dumping ground" for all
implementations, but we decided for TinkerPop3 that we would only have
"reference implementations" so users can play, system providers can learn,
and ultimately, system providers would provide TinkerPop support in their
distribution. As such, we would like to have FlinkGraphComputer distributed
with Flink. If that sounds like something your project would be comfortable
with, I think we can provide a JIRA/PR for FlinkGraphComputer (as well as
any necessary documentation). We can start with a JIRA ticket to get things
going. Thoughts?*


​This is why I brought the conversation over here, so I hear the opinions
of the Flink community on this :)​



> Best, Fabian
>


-Vasia.​



>
>
> 2015-12-14 18:33 GMT+01:00 Vasiliki Kalavri <vasilikikala...@gmail.com>:
>
> > Ping squirrels! Any thoughts/opinions on this?
> >
> > On 9 December 2015 at 20:40, Vasiliki Kalavri <vasilikikala...@gmail.com
> >
> > wrote:
> >
> > > Hello squirrels,
> > >
> > > I have been discussing with the Apache Tinkerpop [1] community
> regarding
> > > an integration with Flink/Gelly.
> > > You can read our discussion in [2].
> > >
> > > Tinkerpop has a graph traversal machine called Gremlin, which supports
> > > many high-level graph processing languages and runs on top of different
> > > systems (e.g. Giraph, Spark, Graph DBs). You can read more in this
> great
> > > blog post [3].
> > >
> > > The idea is to provide a FlinkGraphComputer implementation, which will
> > add
> > > Gremlin support to Flink.
> > >
> > > I believe Tinkerpop is a great project and I would love to see an
> > > integration with Gelly.
> > > Before we move forward, I would like your input!
> > > To me, it seems that this addition would nicely fit in flink-contrib,
> > > where we also have connectors to other projects.
> > > If you agree, I will go ahead and open a JIRA about it.
> > >
> > > Thank you!
> > > -Vasia.
> > >
> > > [1]: https://tinkerpop.incubator.apache.org/
> > > [2]:
> > >
> >
> https://mail-archives.apache.org/mod_mbox/incubator-tinkerpop-dev/201511.mbox/%3ccanva_a390l7g169r8sn+ej1-yfkbudlnd4td6atwnp0uza-...@mail.gmail.com%3E
> > > [3]:
> > >
> >
> http://www.datastax.com/dev/blog/the-benefits-of-the-gremlin-graph-traversal-machine
> > >
> > > On 25 November 2015 at 16:54, Vasiliki Kalavri <
> > vasilikikala...@gmail.com>
> > > wrote:
> > >
> > >> Hi James,
> > >>
> > >> I've just subscribed to the Tinkerpop dev mailing list. Could you
> please
> > >> send a reply to the thread, so then I can reply to it?
> > >> I'm not sure how I can reply to the thread otherwise...
> > >> I also saw that there is a grafos.ml project thread. I could also
> > >> provide some input there :)
> > >>
> > >> Thanks!
> > >> -Vasia.
> > >>
> > >> On 25 November 2015 at 15:09, James Thornton <
> james.thorn...@gmail.com>
> > >> wrote:
> > >>
> > >>> Hi Vasia -
> > >>>
> > >>> Yes, a FlinkGraphComputer should be a straight-forward first step.
> > Also,
> > >>> on
> > >>> the Apache Tinkerpop dev mailing list, Marko thought it might be cool
> > if
> > >>> there was a "Gra

Re: Apache Tinkerpop & Geode Integration?

[Vasiliki Kalavri]

Ping squirrels! Any thoughts/opinions on this?

On 9 December 2015 at 20:40, Vasiliki Kalavri <vasilikikala...@gmail.com>
wrote:

> Hello squirrels,
>
> I have been discussing with the Apache Tinkerpop [1] community regarding
> an integration with Flink/Gelly.
> You can read our discussion in [2].
>
> Tinkerpop has a graph traversal machine called Gremlin, which supports
> many high-level graph processing languages and runs on top of different
> systems (e.g. Giraph, Spark, Graph DBs). You can read more in this great
> blog post [3].
>
> The idea is to provide a FlinkGraphComputer implementation, which will add
> Gremlin support to Flink.
>
> I believe Tinkerpop is a great project and I would love to see an
> integration with Gelly.
> Before we move forward, I would like your input!
> To me, it seems that this addition would nicely fit in flink-contrib,
> where we also have connectors to other projects.
> If you agree, I will go ahead and open a JIRA about it.
>
> Thank you!
> -Vasia.
>
> [1]: https://tinkerpop.incubator.apache.org/
> [2]:
> https://mail-archives.apache.org/mod_mbox/incubator-tinkerpop-dev/201511.mbox/%3ccanva_a390l7g169r8sn+ej1-yfkbudlnd4td6atwnp0uza-...@mail.gmail.com%3E
> [3]:
> http://www.datastax.com/dev/blog/the-benefits-of-the-gremlin-graph-traversal-machine
>
> On 25 November 2015 at 16:54, Vasiliki Kalavri <vasilikikala...@gmail.com>
> wrote:
>
>> Hi James,
>>
>> I've just subscribed to the Tinkerpop dev mailing list. Could you please
>> send a reply to the thread, so then I can reply to it?
>> I'm not sure how I can reply to the thread otherwise...
>> I also saw that there is a grafos.ml project thread. I could also
>> provide some input there :)
>>
>> Thanks!
>> -Vasia.
>>
>> On 25 November 2015 at 15:09, James Thornton <james.thorn...@gmail.com>
>> wrote:
>>
>>> Hi Vasia -
>>>
>>> Yes, a FlinkGraphComputer should be a straight-forward first step. Also,
>>> on
>>> the Apache Tinkerpop dev mailing list, Marko thought it might be cool if
>>> there was a "Graph API" similar to the "Table API" -- hooking in Gremlin
>>> to
>>> Flink's fluent API would give Flink users a full graph query language.
>>>
>>> Stephen Mallette is a TinkerPop core contributor, and he has already
>>> started working on a FlinkGraphComputer. There is a Flink/Tinkerpop
>>> thread
>>> on the TinkerPop dev list -- it would be great to have you part of the
>>> conversation there too as we work on the integration:
>>>
>>>http://mail-archives.apache.org/mod_mbox/incubator-tinkerpop-dev/
>>>
>>> Thanks, Vasia.
>>>
>>> - James
>>>
>>>
>>> On Mon, Nov 23, 2015 at 10:28 AM, Vasiliki Kalavri <
>>> vasilikikala...@gmail.com> wrote:
>>>
>>> > Hi James,
>>> >
>>> > thank you for your e-mail and your interest in Flink :)
>>> >
>>> > I've recently taken a _quick_ look into Apache TinkerPop and I think
>>> it'd
>>> > be very interesting to integrate with Flink/Gelly.
>>> > Are you thinking about something like a Flink GraphComputer, similar to
>>> > Giraph and Spark GraphComputer's?
>>> > I believe such an integration should be straight-forward to implement.
>>> You
>>> > can start by looking into Flink iteration operators [1] and Gelly
>>> iteration
>>> > abstractions [2].
>>> >
>>> > Regarding Apache Geode, I'm not familiar with project, but I'll try to
>>> take
>>> > a look in the following days!
>>> >
>>> > Cheers,
>>> > -Vasia.
>>> >
>>> >
>>> > [1]:
>>> >
>>> >
>>> https://ci.apache.org/projects/flink/flink-docs-master/apis/programming_guide.html#iteration-operators
>>> > [2]:
>>> >
>>> >
>>> https://ci.apache.org/projects/flink/flink-docs-master/libs/gelly_guide.html#iterative-graph-processing
>>> >
>>> >
>>> > On 20 November 2015 at 08:32, James Thornton <james.thorn...@gmail.com
>>> >
>>> > wrote:
>>> >
>>> > > Hi -
>>> > >
>>> > > This is James Thornton (espeed) from the Apache Tinkerpop project (
>>> > > http://tinkerpop.incubator.apache.org/).
>>> > >
>>> > > The Flink iterators should pair well with Gremlin's Graph Traversal
>>> > Machine
>>> > > (
>>> > >
>>> > >
>>> >
>>> http://www.datastax.com/dev/blog/the-benefits-of-the-gremlin-graph-traversal-machine
>>> > > )
>>> > > -- it would be good to coordinate on creating an integration.
>>> > >
>>> > > Also, Apache Geode made a splash today on HN (
>>> > > https://news.ycombinator.com/item?id=10596859) -- connecting Geode
>>> and
>>> > > Flink would be killer. Here's the Geode/Spark connector for
>>> refefference:
>>> > >
>>> > >
>>> > >
>>> > >
>>> >
>>> https://github.com/apache/incubator-geode/tree/develop/gemfire-spark-connector
>>> > >
>>> > > - James
>>> > >
>>> >
>>>
>>
>>
>

Re: New Blog Post Draft

[Vasiliki Kalavri]

Thanks Matthias! This is a very nice blog post and reads easily.

On 9 December 2015 at 19:21, Ufuk Celebi  wrote:

> Great post! Thanks!
>
> I have also made some comments in the commit.
>
> – Ufuk
>
> > On 09 Dec 2015, at 14:19, Maximilian Michels  wrote:
> >
> > Hi Matthias,
> >
> > Thank you for the blog post. You had already shared a first draft with
> > me. This one looks even better!
> >
> > I've made some minor comments. +1 to merge if these are addressed.
> >
> > Cheers,
> > Max
> >
> > On Wed, Dec 9, 2015 at 1:20 PM, Matthias J. Sax 
> wrote:
> >> Just updated the draft (thanks to Till and Slim for feedback) and opened
> >> a PR.
> >>
> >> https://github.com/apache/flink-web/pull/15
> >>
> >> @Slim: we discussed about benchmark result beforehand and decided to do
> >> a second blog post later on
> >>
> >>
> >> -Matthias
> >>
> >> On 12/09/2015 12:14 PM, Slim Baltagi wrote:
> >>> Matthias,
> >>>
> >>> This is great blog!
> >>>
> >>> I would like to suggest the following:
> >>> Change the title to: How to run your existing Storm applications on
> Apache Flink stream processing engine?
> >>> Fixing the few typos
> >>> For this reasons -> For these reasons
> >>> Storm compatibility package which allows users -> Storm compatibility
> package that allows users
> >>> we need to translated it  -> we need to translate it
> >>> in not available -> is not available
> >>> eg, StormWordCount.jar  -> e.g., StormWordCount.jar
> >>> Provide some benchmarks on running a storm application as it is versus
> running it on Flink.
> >>> Thanks
> >>>
> >>> Slim Baltagi
> >>>
> >>> On Dec 9, 2015, at 5:10 AM, Robert Metzger 
> wrote:
> >>>
>  Great, thank you for writing the article.
> 
>  I like the general idea, but I've found some small typos.
>  Can you open a pull request against the "flink-web" repo to make
> reviewing
>  it easier?
> 
>  On Wed, Dec 9, 2015 at 11:32 AM, Matthias J. Sax 
> wrote:
> 
> > Hi,
> >
> > after talking to several people and getting some feedback already, I
> > would like to suggest a new blog post for the project web site about
> the
> > Storm compatibility layer.
> >
> > You can find the draft here:
> >
> >
> https://github.com/mjsax/flink-web/blob/stormCompatibilityBlogPost/_posts/2015-12-07-storm-compatibility.md
> >
> > The missing (just not rendered) picture is this one:
> >
> >
> https://github.com/mjsax/flink-web/blob/stormCompatibilityBlogPost/img/blog/flink-storm.png
> >
> > Looking forward to your feedback!
> >
> >
> > -Matthias
> >
> >
> >>>
> >>>
> >>
>
>

Re: Apache Tinkerpop & Geode Integration?

[Vasiliki Kalavri]

Hello squirrels,

I have been discussing with the Apache Tinkerpop [1] community regarding an
integration with Flink/Gelly.
You can read our discussion in [2].

Tinkerpop has a graph traversal machine called Gremlin, which supports many
high-level graph processing languages and runs on top of different systems
(e.g. Giraph, Spark, Graph DBs). You can read more in this great blog post
[3].

The idea is to provide a FlinkGraphComputer implementation, which will add
Gremlin support to Flink.

I believe Tinkerpop is a great project and I would love to see an
integration with Gelly.
Before we move forward, I would like your input!
To me, it seems that this addition would nicely fit in flink-contrib, where
we also have connectors to other projects.
If you agree, I will go ahead and open a JIRA about it.

Thank you!
-Vasia.

[1]: https://tinkerpop.incubator.apache.org/
[2]:
https://mail-archives.apache.org/mod_mbox/incubator-tinkerpop-dev/201511.mbox/%3ccanva_a390l7g169r8sn+ej1-yfkbudlnd4td6atwnp0uza-...@mail.gmail.com%3E
[3]:
http://www.datastax.com/dev/blog/the-benefits-of-the-gremlin-graph-traversal-machine

On 25 November 2015 at 16:54, Vasiliki Kalavri <vasilikikala...@gmail.com>
wrote:

> Hi James,
>
> I've just subscribed to the Tinkerpop dev mailing list. Could you please
> send a reply to the thread, so then I can reply to it?
> I'm not sure how I can reply to the thread otherwise...
> I also saw that there is a grafos.ml project thread. I could also provide
> some input there :)
>
> Thanks!
> -Vasia.
>
> On 25 November 2015 at 15:09, James Thornton <james.thorn...@gmail.com>
> wrote:
>
>> Hi Vasia -
>>
>> Yes, a FlinkGraphComputer should be a straight-forward first step. Also,
>> on
>> the Apache Tinkerpop dev mailing list, Marko thought it might be cool if
>> there was a "Graph API" similar to the "Table API" -- hooking in Gremlin
>> to
>> Flink's fluent API would give Flink users a full graph query language.
>>
>> Stephen Mallette is a TinkerPop core contributor, and he has already
>> started working on a FlinkGraphComputer. There is a Flink/Tinkerpop thread
>> on the TinkerPop dev list -- it would be great to have you part of the
>> conversation there too as we work on the integration:
>>
>>http://mail-archives.apache.org/mod_mbox/incubator-tinkerpop-dev/
>>
>> Thanks, Vasia.
>>
>> - James
>>
>>
>> On Mon, Nov 23, 2015 at 10:28 AM, Vasiliki Kalavri <
>> vasilikikala...@gmail.com> wrote:
>>
>> > Hi James,
>> >
>> > thank you for your e-mail and your interest in Flink :)
>> >
>> > I've recently taken a _quick_ look into Apache TinkerPop and I think
>> it'd
>> > be very interesting to integrate with Flink/Gelly.
>> > Are you thinking about something like a Flink GraphComputer, similar to
>> > Giraph and Spark GraphComputer's?
>> > I believe such an integration should be straight-forward to implement.
>> You
>> > can start by looking into Flink iteration operators [1] and Gelly
>> iteration
>> > abstractions [2].
>> >
>> > Regarding Apache Geode, I'm not familiar with project, but I'll try to
>> take
>> > a look in the following days!
>> >
>> > Cheers,
>> > -Vasia.
>> >
>> >
>> > [1]:
>> >
>> >
>> https://ci.apache.org/projects/flink/flink-docs-master/apis/programming_guide.html#iteration-operators
>> > [2]:
>> >
>> >
>> https://ci.apache.org/projects/flink/flink-docs-master/libs/gelly_guide.html#iterative-graph-processing
>> >
>> >
>> > On 20 November 2015 at 08:32, James Thornton <james.thorn...@gmail.com>
>> > wrote:
>> >
>> > > Hi -
>> > >
>> > > This is James Thornton (espeed) from the Apache Tinkerpop project (
>> > > http://tinkerpop.incubator.apache.org/).
>> > >
>> > > The Flink iterators should pair well with Gremlin's Graph Traversal
>> > Machine
>> > > (
>> > >
>> > >
>> >
>> http://www.datastax.com/dev/blog/the-benefits-of-the-gremlin-graph-traversal-machine
>> > > )
>> > > -- it would be good to coordinate on creating an integration.
>> > >
>> > > Also, Apache Geode made a splash today on HN (
>> > > https://news.ycombinator.com/item?id=10596859) -- connecting Geode
>> and
>> > > Flink would be killer. Here's the Geode/Spark connector for
>> refefference:
>> > >
>> > >
>> > >
>> > >
>> >
>> https://github.com/apache/incubator-geode/tree/develop/gemfire-spark-connector
>> > >
>> > > - James
>> > >
>> >
>>
>
>

Re: Either not NotSerializableException and InvalidTypesException

[Vasiliki Kalavri]

Hi,

thanks a lot for your answers :)

@Timo: sure, I can open an issue, but I'm not sure I understand the problem
to describe it properly. The with() method calls the
getFlatJoinReturnTypes() with the allowMissing parameter set to false. Is
this what you're referring to? What does this parameter do and when should
it be true/false? Thanks!

@Till: thanks for the explanation Till. I didn't realize Java serialization
was involved there!

Cheers,
-V.

On 30 November 2015 at 11:09, Till Rohrmann <trohrm...@apache.org> wrote:

> I checked the code and the MapFunction InitializeWorkSet contains indeed
> two instances of type Either. Since we use Java serialization to ship the
> operators to the cluster, the MapFunction has to be serializable. Either
> you make the Either type serializable or you create these instances in the
> open method of the operator.
>
> Cheers,
> Till
> ​
>
> On Mon, Nov 30, 2015 at 11:03 AM, Aljoscha Krettek <aljos...@apache.org>
> wrote:
>
> > It seems there is an Either.Left stored somewhere in the Object. Could
> > that be?
> > > On 28 Nov 2015, at 20:18, Vasiliki Kalavri <vasilikikala...@gmail.com>
> > wrote:
> > >
> > >
> >
> org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:307)
> >
> >
>

Either not NotSerializableException and InvalidTypesException

[Vasiliki Kalavri]

Hi squirrels,

I have 2 problems with the new Either type and I could use your help to
understand them.

1. I have a piece of code that looks like this:

TypeInformation>> workSetTypeInfo = ...
DataSet>> initialWorkSet =
initialVertices.map(...).returns(workSetTypeInfo);

This gives me the following exception:

Exception in thread "main"
org.apache.flink.api.common.InvalidProgramException: Object
org.apache.flink.graph.spargelnew.MessagePassingIteration$InitializeWorkSet@75ba8574
not serializable

at
org.apache.flink.api.java.ClosureCleaner.ensureSerializable(ClosureCleaner.java:97)
at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:59)
at org.apache.flink.api.java.DataSet.clean(DataSet.java:184)
at org.apache.flink.api.java.DataSet.map(DataSet.java:214)
at
org.apache.flink.graph.spargelnew.MessagePassingIteration.createResult(MessagePassingIteration.java:160)
at org.apache.flink.api.java.DataSet.runOperation(DataSet.java:1190)
at org.apache.flink.graph.Graph.runMessagePassingIteration(Graph.java:1650)
at org.apache.flink.graph.spargelnew.example.PregelCC.main(PregelCC.java:53)

Caused by: java.io.NotSerializableException:
org.apache.flink.api.java.typeutils.Either$Left
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
at
org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:307)
at
org.apache.flink.api.java.ClosureCleaner.ensureSerializable(ClosureCleaner.java:95)

​Making​ Either implement
java.io.Serializable
solves this, but I am wondering why this is needed. Since I'm registering
the typeinfo with returns(), shouldn't the EitherTypeSerializer be
registered too? Also, this seem to be the only operation where I get this
error, even though I'm using the Either type in other places as well.


2. The second problem appeared after rebasing to the current master,
containing a fix for FLINK-3046 (Integrate the Either Java type with the
TypeExtractor).

Exception in thread "main"
org.apache.flink.api.common.functions.InvalidTypesException: Type of
TypeVariable 'Message' in 'class
org.apache.flink.graph.spargelnew.MessagePassingIteration$AppendVertexState'
could not be determined. This is most likely a type erasure problem. The
type extraction currently supports types with generic variables only in
cases where all variables in the return type can be deduced from the input
type(s).

at
org.apache.flink.api.java.typeutils.TypeExtractor.createSubTypesInfo(TypeExtractor.java:706)
at
org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:458)
at
org.apache.flink.api.java.typeutils.TypeExtractor.createSubTypesInfo(TypeExtractor.java:713)
at
org.apache.flink.api.java.typeutils.TypeExtractor.createTypeInfoWithTypeHierarchy(TypeExtractor.java:425)
at
org.apache.flink.api.java.typeutils.TypeExtractor.privateCreateTypeInfo(TypeExtractor.java:380)
at
org.apache.flink.api.java.typeutils.TypeExtractor.getBinaryOperatorReturnType(TypeExtractor.java:320)
at
org.apache.flink.api.java.typeutils.TypeExtractor.getFlatJoinReturnTypes(TypeExtractor.java:176)
at
org.apache.flink.api.java.typeutils.TypeExtractor.getFlatJoinReturnTypes(TypeExtractor.java:170)
at
org.apache.flink.api.java.operators.JoinOperator$DefaultJoin.with(JoinOperator.java:562)
at
org.apache.flink.graph.spargelnew.MessagePassingIteration.createResult(MessagePassingIteration.java:171)
at org.apache.flink.api.java.DataSet.runOperation(DataSet.java:1190)
at org.apache.flink.graph.Graph.runMessagePassingIteration(Graph.java:1650)
at org.apache.flink.graph.spargelnew.example.PregelCC.main(PregelCC.java:53)


​The code giving this exception is the following:
​
DataSet, Either>> verticesWithMsgs
=
​
​
iteration.getSolutionSet().join(iteration.getWorkset())
​​
.where(0).equalTo(0)
​​
.with(new AppendVertexState())
​​
.
​​
​​
returns(new TupleTypeInfo, Either>>(vertexType, nullableMsgTypeInfo));

​Do I need to register the Either typeinfo differently ​now that it's
integrated with the TypeExtractor or is this a bug?

If you want to see the complete code, I've pushed it here:
https://github.com/vasia/flink/blob/spargel-2/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/spargelnew/MessagePassingIteration.java#L168
.

Thanks!
-Vasia.

Re: [ANNOUNCE] Flink 0.10.1 released

[Vasiliki Kalavri]

Thank you Robert ^^

On 27 November 2015 at 16:23, Till Rohrmann  wrote:

> Thanks Robert for being the release manager for 0.10.1
>
> On Fri, Nov 27, 2015 at 4:21 PM, Maximilian Michels 
> wrote:
>
> > Great. We released that one fast. Thanks Robert.
> >
> > On Fri, Nov 27, 2015 at 3:27 PM, Robert Metzger 
> > wrote:
> > > The Flink PMC is pleased to announce the availability of Flink 0.10.1.
> > >
> > > The official release announcement:
> > > http://flink.apache.org/news/2015/11/27/release-0.10.1.html
> > > Release binaries: http://apache.openmirror.de/flink/flink-0.10.1/
> > >
> > > Please update your maven dependencies to the new 0.10.1 version and
> > update
> > > your binaries.
> > >
> > >
> > > On behalf of the Flink PMC, I would like to thank everybody who
> > contributed
> > > to the release!
> >
>

Re: Apache Tinkerpop & Geode Integration?

[Vasiliki Kalavri]

Hi James,

I've just subscribed to the Tinkerpop dev mailing list. Could you please
send a reply to the thread, so then I can reply to it?
I'm not sure how I can reply to the thread otherwise...
I also saw that there is a grafos.ml project thread. I could also provide
some input there :)

Thanks!
-Vasia.

On 25 November 2015 at 15:09, James Thornton <james.thorn...@gmail.com>
wrote:

> Hi Vasia -
>
> Yes, a FlinkGraphComputer should be a straight-forward first step. Also, on
> the Apache Tinkerpop dev mailing list, Marko thought it might be cool if
> there was a "Graph API" similar to the "Table API" -- hooking in Gremlin to
> Flink's fluent API would give Flink users a full graph query language.
>
> Stephen Mallette is a TinkerPop core contributor, and he has already
> started working on a FlinkGraphComputer. There is a Flink/Tinkerpop thread
> on the TinkerPop dev list -- it would be great to have you part of the
> conversation there too as we work on the integration:
>
>http://mail-archives.apache.org/mod_mbox/incubator-tinkerpop-dev/
>
> Thanks, Vasia.
>
> - James
>
>
> On Mon, Nov 23, 2015 at 10:28 AM, Vasiliki Kalavri <
> vasilikikala...@gmail.com> wrote:
>
> > Hi James,
> >
> > thank you for your e-mail and your interest in Flink :)
> >
> > I've recently taken a _quick_ look into Apache TinkerPop and I think it'd
> > be very interesting to integrate with Flink/Gelly.
> > Are you thinking about something like a Flink GraphComputer, similar to
> > Giraph and Spark GraphComputer's?
> > I believe such an integration should be straight-forward to implement.
> You
> > can start by looking into Flink iteration operators [1] and Gelly
> iteration
> > abstractions [2].
> >
> > Regarding Apache Geode, I'm not familiar with project, but I'll try to
> take
> > a look in the following days!
> >
> > Cheers,
> > -Vasia.
> >
> >
> > [1]:
> >
> >
> https://ci.apache.org/projects/flink/flink-docs-master/apis/programming_guide.html#iteration-operators
> > [2]:
> >
> >
> https://ci.apache.org/projects/flink/flink-docs-master/libs/gelly_guide.html#iterative-graph-processing
> >
> >
> > On 20 November 2015 at 08:32, James Thornton <james.thorn...@gmail.com>
> > wrote:
> >
> > > Hi -
> > >
> > > This is James Thornton (espeed) from the Apache Tinkerpop project (
> > > http://tinkerpop.incubator.apache.org/).
> > >
> > > The Flink iterators should pair well with Gremlin's Graph Traversal
> > Machine
> > > (
> > >
> > >
> >
> http://www.datastax.com/dev/blog/the-benefits-of-the-gremlin-graph-traversal-machine
> > > )
> > > -- it would be good to coordinate on creating an integration.
> > >
> > > Also, Apache Geode made a splash today on HN (
> > > https://news.ycombinator.com/item?id=10596859) -- connecting Geode and
> > > Flink would be killer. Here's the Geode/Spark connector for
> refefference:
> > >
> > >
> > >
> > >
> >
> https://github.com/apache/incubator-geode/tree/develop/gemfire-spark-connector
> > >
> > > - James
> > >
> >
>

Re: Union a data stream with a product of itself

[Vasiliki Kalavri]

Here's the issue: https://issues.apache.org/jira/browse/FLINK-3080

-V.

On 25 November 2015 at 14:38, Gyula Fóra <gyula.f...@gmail.com> wrote:

> Yes, please
>
> Vasiliki Kalavri <vasilikikala...@gmail.com> ezt írta (időpont: 2015. nov.
> 25., Sze, 14:37):
>
> > So, do we all agree that the current behavior is not correct? Shall I
> open
> > a JIRA about this?
> >
> > On 25 November 2015 at 13:58, Gyula Fóra <gyula.f...@gmail.com> wrote:
> >
> > > Well it kind of depends on what definition of union are we using. If
> this
> > > is a union in a set theoretical way we can argue that the union of a
> > stream
> > > with itself should be the same stream because it contains exactly the
> > same
> > > elements with the same timestamps and lineage.
> > >
> > > On the other hand stream and stream.map(id) are not exactly the same as
> > > they might have elements with different order (the lineage differs).
> > >
> > > So I wouldnt say that any self-union semantics is the only possible
> one.
> > >
> > > Gyula
> > >
> > > Bruecke, Christoph <christoph.brue...@campus.tu-berlin.de> ezt írta
> > > (időpont: 2015. nov. 25., Sze, 13:47):
> > >
> > > > Hi,
> > > >
> > > > the operation “stream.union(stream.map(id))” is equivalent to
> > > > “stream.union(stream)” isn’t it? So it might also duplicate the data.
> > > >
> > > > - Christoph
> > > >
> > > >
> > > > > On 25 Nov 2015, at 11:24, Stephan Ewen <se...@apache.org> wrote:
> > > > >
> > > > > "stream.union(stream.map(..))" should definitely be possible. Not
> > sure
> > > > why
> > > > > this is not permitted.
> > > > >
> > > > > "stream.union(stream)" would contain each element twice, so should
> > > either
> > > > > give an error or actually union (or duplicate) elements...
> > > > >
> > > > > Stephan
> > > > >
> > > > >
> > > > > On Wed, Nov 25, 2015 at 10:42 AM, Gyula Fóra <gyf...@apache.org>
> > > wrote:
> > > > >
> > > > >> Yes, I am not sure if this the intentional behaviour. I think you
> > are
> > > > >> supposed to be able to do the things you described.
> > > > >>
> > > > >> stream.union(stream.map(..)) and things like this are fair
> > operations.
> > > > Also
> > > > >> maybe stream.union(stream) should just give stream instead of an
> > > error.
> > > > >>
> > > > >> Could someone comment on this who knows the reasoning behind the
> > > current
> > > > >> mechanics?
> > > > >>
> > > > >> Gyula
> > > > >>
> > > > >> Vasiliki Kalavri <vasilikikala...@gmail.com> ezt írta (időpont:
> > 2015.
> > > > nov.
> > > > >> 24., K, 16:46):
> > > > >>
> > > > >>> Hi squirrels,
> > > > >>>
> > > > >>> when porting the gelly streaming code from 0.9 to 0.10 today with
> > > > Paris,
> > > > >> we
> > > > >>> hit an exception in union: "*A DataStream cannot be unioned with
> > > > >> itself*".
> > > > >>>
> > > > >>> The code raising this exception looks like this:
> > > > >>> stream.union(stream.map(...)).
> > > > >>>
> > > > >>> Taking a look into the union code, we see that it's now not
> allowed
> > > to
> > > > >>> union a stream, not only with itself, but with any product of
> > itself.
> > > > >>>
> > > > >>> First, we are wondering, why is that? Does it make building the
> > > stream
> > > > >>> graph easier in some way?
> > > > >>> Second, we might want to give a better error message there, e.g.
> > "*A
> > > > >>> DataStream cannot be unioned with itself or a product of
> itself*",
> > > and
> > > > >>> finally, we should update the docs, which currently state that
> > union
> > > a
> > > > >>> stream with itself is allowed and that "*If you union a data
> stream
> > > > with
> > > > >>> itself you will still only get each element once.*"
> > > > >>>
> > > > >>> Cheers,
> > > > >>> -Vasia.
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
>

Re: Union a data stream with a product of itself

[Vasiliki Kalavri]

So, do we all agree that the current behavior is not correct? Shall I open
a JIRA about this?

On 25 November 2015 at 13:58, Gyula Fóra <gyula.f...@gmail.com> wrote:

> Well it kind of depends on what definition of union are we using. If this
> is a union in a set theoretical way we can argue that the union of a stream
> with itself should be the same stream because it contains exactly the same
> elements with the same timestamps and lineage.
>
> On the other hand stream and stream.map(id) are not exactly the same as
> they might have elements with different order (the lineage differs).
>
> So I wouldnt say that any self-union semantics is the only possible one.
>
> Gyula
>
> Bruecke, Christoph <christoph.brue...@campus.tu-berlin.de> ezt írta
> (időpont: 2015. nov. 25., Sze, 13:47):
>
> > Hi,
> >
> > the operation “stream.union(stream.map(id))” is equivalent to
> > “stream.union(stream)” isn’t it? So it might also duplicate the data.
> >
> > - Christoph
> >
> >
> > > On 25 Nov 2015, at 11:24, Stephan Ewen <se...@apache.org> wrote:
> > >
> > > "stream.union(stream.map(..))" should definitely be possible. Not sure
> > why
> > > this is not permitted.
> > >
> > > "stream.union(stream)" would contain each element twice, so should
> either
> > > give an error or actually union (or duplicate) elements...
> > >
> > > Stephan
> > >
> > >
> > > On Wed, Nov 25, 2015 at 10:42 AM, Gyula Fóra <gyf...@apache.org>
> wrote:
> > >
> > >> Yes, I am not sure if this the intentional behaviour. I think you are
> > >> supposed to be able to do the things you described.
> > >>
> > >> stream.union(stream.map(..)) and things like this are fair operations.
> > Also
> > >> maybe stream.union(stream) should just give stream instead of an
> error.
> > >>
> > >> Could someone comment on this who knows the reasoning behind the
> current
> > >> mechanics?
> > >>
> > >> Gyula
> > >>
> > >> Vasiliki Kalavri <vasilikikala...@gmail.com> ezt írta (időpont: 2015.
> > nov.
> > >> 24., K, 16:46):
> > >>
> > >>> Hi squirrels,
> > >>>
> > >>> when porting the gelly streaming code from 0.9 to 0.10 today with
> > Paris,
> > >> we
> > >>> hit an exception in union: "*A DataStream cannot be unioned with
> > >> itself*".
> > >>>
> > >>> The code raising this exception looks like this:
> > >>> stream.union(stream.map(...)).
> > >>>
> > >>> Taking a look into the union code, we see that it's now not allowed
> to
> > >>> union a stream, not only with itself, but with any product of itself.
> > >>>
> > >>> First, we are wondering, why is that? Does it make building the
> stream
> > >>> graph easier in some way?
> > >>> Second, we might want to give a better error message there, e.g. "*A
> > >>> DataStream cannot be unioned with itself or a product of itself*",
> and
> > >>> finally, we should update the docs, which currently state that union
> a
> > >>> stream with itself is allowed and that "*If you union a data stream
> > with
> > >>> itself you will still only get each element once.*"
> > >>>
> > >>> Cheers,
> > >>> -Vasia.
> > >>>
> > >>
> >
> >
>

Union a data stream with a product of itself

[Vasiliki Kalavri]

Hi squirrels,

when porting the gelly streaming code from 0.9 to 0.10 today with Paris, we
hit an exception in union: "*A DataStream cannot be unioned with itself*".

The code raising this exception looks like this:
stream.union(stream.map(...)).

Taking a look into the union code, we see that it's now not allowed to
union a stream, not only with itself, but with any product of itself.

First, we are wondering, why is that? Does it make building the stream
graph easier in some way?
Second, we might want to give a better error message there, e.g. "*A
DataStream cannot be unioned with itself or a product of itself*", and
finally, we should update the docs, which currently state that union a
stream with itself is allowed and that "*If you union a data stream with
itself you will still only get each element once.*"

Cheers,
-Vasia.

Re: Either left() vs left(value)

[Vasiliki Kalavri]

Hey Gyula,

I don't think dropping the method is a good idea. We need a way to retrieve
left and right values, no?
How about renaming to getLeft() / getRight()?

-V.

On 23 November 2015 at 09:55, Gyula Fóra  wrote:

> Hey guys,
>
> I know this should have been part of the PR discussion but it kind of
> slipped through the cracks :)
>
> I think it might be useful to change the method name for Either.left(value)
> to Either.Left(value) (or drop the method completely).
>
> The reason is that it is slightly awkward to use it with java 8 lambdas.
> You cannot use Either::left because of the name clash. Maybe it's not a
> huge issue but a small inconvenience that will come up more often as we are
> gradually moving to java 8 anyways :)
>
> What do you think?
> Gyula
>

Re: Either left() vs left(value)

[Vasiliki Kalavri]

Ah I see. Well, as I also said in the PR, Left and Right make no sense on
their own, they're helper classes for Either. Hence, I believe they should
be private. Maybe we could rename the methods to createLeft() /
createRight() ?

On 23 November 2015 at 20:58, Gyula Fóra <gyula.f...@gmail.com> wrote:

> I was actually not suggesting to drop the e.left() method but instead the
> Either.left(val).
> Renaming the left(), right() methods might be confusing as than it would be
> inconsistent with the scala version.
>
> On the other hand we could change the way the user can create the Left
> Right classes, maybe directly expose them instead of the static method. (or
> rename the static method)
>
> Gyula
>
> Vasiliki Kalavri <vasilikikala...@gmail.com> ezt írta (időpont: 2015. nov.
> 23., H, 20:14):
>
> > Hey Gyula,
> >
> > I don't think dropping the method is a good idea. We need a way to
> retrieve
> > left and right values, no?
> > How about renaming to getLeft() / getRight()?
> >
> > -V.
> >
> > On 23 November 2015 at 09:55, Gyula Fóra <gyula.f...@gmail.com> wrote:
> >
> > > Hey guys,
> > >
> > > I know this should have been part of the PR discussion but it kind of
> > > slipped through the cracks :)
> > >
> > > I think it might be useful to change the method name for
> > Either.left(value)
> > > to Either.Left(value) (or drop the method completely).
> > >
> > > The reason is that it is slightly awkward to use it with java 8
> lambdas.
> > > You cannot use Either::left because of the name clash. Maybe it's not a
> > > huge issue but a small inconvenience that will come up more often as we
> > are
> > > gradually moving to java 8 anyways :)
> > >
> > > What do you think?
> > > Gyula
> > >
> >
>

Re: Apache Tinkerpop & Geode Integration?

[Vasiliki Kalavri]

Hi James,

thank you for your e-mail and your interest in Flink :)

I've recently taken a _quick_ look into Apache TinkerPop and I think it'd
be very interesting to integrate with Flink/Gelly.
Are you thinking about something like a Flink GraphComputer, similar to
Giraph and Spark GraphComputer's?
I believe such an integration should be straight-forward to implement. You
can start by looking into Flink iteration operators [1] and Gelly iteration
abstractions [2].

Regarding Apache Geode, I'm not familiar with project, but I'll try to take
a look in the following days!

Cheers,
-Vasia.


[1]:
https://ci.apache.org/projects/flink/flink-docs-master/apis/programming_guide.html#iteration-operators
[2]:
https://ci.apache.org/projects/flink/flink-docs-master/libs/gelly_guide.html#iterative-graph-processing


On 20 November 2015 at 08:32, James Thornton 
wrote:

> Hi -
>
> This is James Thornton (espeed) from the Apache Tinkerpop project (
> http://tinkerpop.incubator.apache.org/).
>
> The Flink iterators should pair well with Gremlin's Graph Traversal Machine
> (
>
> http://www.datastax.com/dev/blog/the-benefits-of-the-gremlin-graph-traversal-machine
> )
> -- it would be good to coordinate on creating an integration.
>
> Also, Apache Geode made a splash today on HN (
> https://news.ycombinator.com/item?id=10596859) -- connecting Geode and
> Flink would be killer. Here's the Geode/Spark connector for refefference:
>
>
>
> https://github.com/apache/incubator-geode/tree/develop/gemfire-spark-connector
>
> - James
>

Re: Either left() vs left(value)

[Vasiliki Kalavri]

Either is abstract already ;)

On 23 November 2015 at 21:54, Gyula Fóra <gyula.f...@gmail.com> wrote:

> I think it is not too bad to only have the Right/Left classes. You can then
> write it like this:
>
> Either<String, Integer> e1 = new Left<>("");
> Either<String, Integer> e2 = new Right<>(1);
>
> (this would be pretty much like in scala)
>
> or we can add static methods like: Left.of(...), Right.of(...) which would
> work exactly as it does now.
>
> And then we can live without the static methods in Either (Either would
> become Abstract).
>
> Gyula
>
> Vasiliki Kalavri <vasilikikala...@gmail.com> ezt írta (időpont: 2015. nov.
> 23., H, 21:25):
>
> > Ah I see. Well, as I also said in the PR, Left and Right make no sense on
> > their own, they're helper classes for Either. Hence, I believe they
> should
> > be private. Maybe we could rename the methods to createLeft() /
> > createRight() ?
> >
> > On 23 November 2015 at 20:58, Gyula Fóra <gyula.f...@gmail.com> wrote:
> >
> > > I was actually not suggesting to drop the e.left() method but instead
> the
> > > Either.left(val).
> > > Renaming the left(), right() methods might be confusing as than it
> would
> > be
> > > inconsistent with the scala version.
> > >
> > > On the other hand we could change the way the user can create the Left
> > > Right classes, maybe directly expose them instead of the static method.
> > (or
> > > rename the static method)
> > >
> > > Gyula
> > >
> > > Vasiliki Kalavri <vasilikikala...@gmail.com> ezt írta (időpont: 2015.
> > nov.
> > > 23., H, 20:14):
> > >
> > > > Hey Gyula,
> > > >
> > > > I don't think dropping the method is a good idea. We need a way to
> > > retrieve
> > > > left and right values, no?
> > > > How about renaming to getLeft() / getRight()?
> > > >
> > > > -V.
> > > >
> > > > On 23 November 2015 at 09:55, Gyula Fóra <gyula.f...@gmail.com>
> wrote:
> > > >
> > > > > Hey guys,
> > > > >
> > > > > I know this should have been part of the PR discussion but it kind
> of
> > > > > slipped through the cracks :)
> > > > >
> > > > > I think it might be useful to change the method name for
> > > > Either.left(value)
> > > > > to Either.Left(value) (or drop the method completely).
> > > > >
> > > > > The reason is that it is slightly awkward to use it with java 8
> > > lambdas.
> > > > > You cannot use Either::left because of the name clash. Maybe it's
> > not a
> > > > > huge issue but a small inconvenience that will come up more often
> as
> > we
> > > > are
> > > > > gradually moving to java 8 anyways :)
> > > > >
> > > > > What do you think?
> > > > > Gyula
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Release Flink 0.10.1 soon

[Vasiliki Kalavri]

*FLINK-3013 you're right :)

On 18 November 2015 at 13:52, Ufuk Celebi <u...@apache.org> wrote:

> Big +1
>
> I think the mentioned issues cover all important fixes.
>
> – Ufuk
>
> > On 18 Nov 2015, at 13:49, Till Rohrmann <trohrm...@apache.org> wrote:
> >
> > @Vasia, do you mean FLINK-3013 or FLINK-3012?
> >
> > Will merge PRs for FLINK-3036 and FLINK-3013 this afternoon.
> >
> > On Wed, Nov 18, 2015 at 1:47 PM, Stephan Ewen <se...@apache.org> wrote:
> >
> >> +1 for a timely 0.10.1 release
> >>
> >> I would like to add FLINK-2974 - periodic kafka offset committer for
> case
> >> where checkpointing is deactivated
> >>
> >> On Wed, Nov 18, 2015 at 12:34 PM, Vasiliki Kalavri <
> >> vasilikikala...@gmail.com> wrote:
> >>
> >>> Hey,
> >>>
> >>> I would also add FLINK-3012 and FLINK-3036 (both pending PRs).
> >>>
> >>> Thanks!
> >>> -Vasia.
> >>>
> >>> On 18 November 2015 at 12:24, Robert Metzger <rmetz...@apache.org>
> >> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I was wondering whether we should release Flink 0.10.1 soon, as there
> >> are
> >>>> some issues we've identified:
> >>>>
> >>>> (pending PRs)
> >>>> - FLINK-3032: Flink does not start on Hadoop 2.7.1 (HDP), due to class
> >>>> conflict
> >>>> - FLINK-3011, 3019, 3028 Cancel jobs in RESTARTING state
> >>>> - FLINK-3021 Fix class loading issue for streaming sources
> >>>> - FLINK-2989 job cancel button doesn't work on YARN
> >>>>
> >>>> (merged)
> >>>> - FLINK-2977 Using reflection to load HBase Kerberos tokens
> >>>> - FLINK-3024 Fix TimestampExtractor.getCurrentWatermark() Behaviour
> >>>> - FLINK-2967 Increase timeout for LOCAL_HOST address detection stratey
> >>>> - FLINK-3025 [kafka consumer] Bump transitive ZkClient dependency
> >>>>
> >>>>
> >>>> Anything else that you would like to add?
> >>>>
> >>>> Do you think we can manage to merge the four PRs until tomorrow and
> >> then
> >>>> start the RC?
> >>>>
> >>>
> >>
>
>

Re: [DISCUSS] Release Flink 0.10.1 soon

[Vasiliki Kalavri]

Hey,

I would also add FLINK-3012 and FLINK-3036 (both pending PRs).

Thanks!
-Vasia.

On 18 November 2015 at 12:24, Robert Metzger  wrote:

> Hi,
>
> I was wondering whether we should release Flink 0.10.1 soon, as there are
> some issues we've identified:
>
> (pending PRs)
> - FLINK-3032: Flink does not start on Hadoop 2.7.1 (HDP), due to class
> conflict
> - FLINK-3011, 3019, 3028 Cancel jobs in RESTARTING state
> - FLINK-3021 Fix class loading issue for streaming sources
> - FLINK-2989 job cancel button doesn't work on YARN
>
> (merged)
> - FLINK-2977 Using reflection to load HBase Kerberos tokens
> - FLINK-3024 Fix TimestampExtractor.getCurrentWatermark() Behaviour
> - FLINK-2967 Increase timeout for LOCAL_HOST address detection stratey
> - FLINK-3025 [kafka consumer] Bump transitive ZkClient dependency
>
>
> Anything else that you would like to add?
>
> Do you think we can manage to merge the four PRs until tomorrow and then
> start the RC?
>

Re: [VOTE] [RESULT] Release Apache Flink 0.10.0 (release-0.10.0-rc8)

[Vasiliki Kalavri]

\o/ \o/ \o/
Thank you Max!
On Nov 13, 2015 2:23 AM, "Nick Dimiduk"  wrote:

> Woo hoo!
>
> On Thu, Nov 12, 2015 at 3:01 PM, Maximilian Michels 
> wrote:
>
> > Thanks for voting! The vote passes.
> >
> > The following votes have been cast:
> >
> > +1 votes: 7
> >
> > Stephan
> > Aljoscha
> > Robert
> > Max
> > Chiwan*
> > Henry
> > Fabian
> >
> > * non-binding
> >
> > -1 votes: none
> >
> > I'll upload the release artifacts and release the Maven artifacts.
> > Once the changes are effective, the community may announce the
> > release.
> >
>

Re: [gelly] Spargel model rework

[Vasiliki Kalavri]

Thanks Fabian! I'll try that :)

On 10 November 2015 at 22:31, Fabian Hueske <fhue...@gmail.com> wrote:

> You could implement a Java Either type (similar to Scala's Either) that
> either has a Message or the VertexState and a corresponding TypeInformation
> and TypeSerializer that serializes a byte flag to indicate which both types
> is used.
> It might actually make sense, to add a generic Either type to the Java API
> in general (similar to the Java Tuples with resemble the Scala Tuples).
>
> Cheers, Fabian
>
> 2015-11-10 22:16 GMT+01:00 Vasiliki Kalavri <vasilikikala...@gmail.com>:
>
> > Hi,
> >
> > after running a few experiments, I can confirm that putting the combiner
> > after the flatMap is indeed more efficient.
> >
> > I ran SSSP and Connected Components with Spargel, GSA, and the Pregel
> model
> > and the results are the following:
> >
> > - for SSSP, Spargel is always the slowest, GSA is a ~1.2x faster and
> Pregel
> > is ~1.1x faster without combiner, ~1.3x faster with combiner.
> > - for Connected Components, Spargel and GSA perform similarly, while
> Pregel
> > is 1.4-1.6x slower.
> >
> > To start with, this is much better than I expected :)
> > However, there is a main shortcoming in my current implementation that
> > negatively impacts performance:
> > Since the compute function coGroup needs to output both new vertex values
> > and new messages, I emit a wrapping tuple that contains both vertex state
> > and messages and then filter them out based on a boolean field. The
> problem
> > is that since I cannot emit null fields, I emit a dummy message for each
> > new vertex state and a dummy vertex state for each new message. That
> > essentially means that the intermediate messages result is double in
> size,
> > if say the vertex values are of the same type as the messages (can be
> worse
> > if the vertex values are more complex).
> > So my question is, is there a way to avoid this redundancy, by either
> > emitting null fields or by creating an operator that could emit 2
> different
> > types of tuples?
> >
> > Thanks!
> > -Vasia.
> >
> > On 9 November 2015 at 15:20, Fabian Hueske <fhue...@gmail.com> wrote:
> >
> > > Hi Vasia,
> > >
> > > sorry for the late reply.
> > > I don't think there is a big difference. In both cases, the
> partitioning
> > > and sorting happens at the end of the iteration.
> > > If the groupReduce is applied before the workset is returned, the
> sorting
> > > happens on the filtered result (after the flatMap) which might be a
> > little
> > > bit more efficient (depending on the ratio of messages and solution set
> > > updates). Also it does not require that the initial workset is sorted
> for
> > > the first groupReduce.
> > >
> > > I would put it at the end.
> > >
> > > Cheers, Fabian
> > >
> > > 2015-11-05 17:19 GMT+01:00 Vasiliki Kalavri <vasilikikala...@gmail.com
> >:
> > >
> > > > @Fabian
> > > >
> > > > Is there any advantage in putting the reducer-combiner before
> updating
> > > the
> > > > workset vs. after (i.e. right before the join with the solution set)?
> > > >
> > > > If it helps, here are the plans of these 2 alternatives:
> > > >
> > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/0BzQJrI2eGlyYcFV2RFo5dUFNXzg/view?usp=sharing
> > > >
> > > >
> > >
> >
> https://drive.google.com/file/d/0BzQJrI2eGlyYN014NXp6OEZUdGs/view?usp=sharing
> > > >
> > > > Thanks a lot for the help!
> > > >
> > > > -Vasia.
> > > >
> > > > On 30 October 2015 at 21:28, Fabian Hueske <fhue...@gmail.com>
> wrote:
> > > >
> > > > > We can of course inject an optional ReduceFunction (or GroupReduce,
> > or
> > > > > combinable GroupReduce) to reduce the size of the work set.
> > > > > I suggested to remove the GroupReduce function, because it did only
> > > > collect
> > > > > all messages into a single record by emitting the input iterator
> > which
> > > is
> > > > > quite dangerous. Applying a combinable reduce function is could
> > improve
> > > > the
> > > > > performance considerably.
> > > > >
> > > > > The good news is that it would come "for free" because the
> necess

Re: [gelly] Spargel model rework

[Vasiliki Kalavri]

Hi,

after running a few experiments, I can confirm that putting the combiner
after the flatMap is indeed more efficient.

I ran SSSP and Connected Components with Spargel, GSA, and the Pregel model
and the results are the following:

- for SSSP, Spargel is always the slowest, GSA is a ~1.2x faster and Pregel
is ~1.1x faster without combiner, ~1.3x faster with combiner.
- for Connected Components, Spargel and GSA perform similarly, while Pregel
is 1.4-1.6x slower.

To start with, this is much better than I expected :)
However, there is a main shortcoming in my current implementation that
negatively impacts performance:
Since the compute function coGroup needs to output both new vertex values
and new messages, I emit a wrapping tuple that contains both vertex state
and messages and then filter them out based on a boolean field. The problem
is that since I cannot emit null fields, I emit a dummy message for each
new vertex state and a dummy vertex state for each new message. That
essentially means that the intermediate messages result is double in size,
if say the vertex values are of the same type as the messages (can be worse
if the vertex values are more complex).
So my question is, is there a way to avoid this redundancy, by either
emitting null fields or by creating an operator that could emit 2 different
types of tuples?

Thanks!
-Vasia.

On 9 November 2015 at 15:20, Fabian Hueske <fhue...@gmail.com> wrote:

> Hi Vasia,
>
> sorry for the late reply.
> I don't think there is a big difference. In both cases, the partitioning
> and sorting happens at the end of the iteration.
> If the groupReduce is applied before the workset is returned, the sorting
> happens on the filtered result (after the flatMap) which might be a little
> bit more efficient (depending on the ratio of messages and solution set
> updates). Also it does not require that the initial workset is sorted for
> the first groupReduce.
>
> I would put it at the end.
>
> Cheers, Fabian
>
> 2015-11-05 17:19 GMT+01:00 Vasiliki Kalavri <vasilikikala...@gmail.com>:
>
> > @Fabian
> >
> > Is there any advantage in putting the reducer-combiner before updating
> the
> > workset vs. after (i.e. right before the join with the solution set)?
> >
> > If it helps, here are the plans of these 2 alternatives:
> >
> >
> >
> https://drive.google.com/file/d/0BzQJrI2eGlyYcFV2RFo5dUFNXzg/view?usp=sharing
> >
> >
> https://drive.google.com/file/d/0BzQJrI2eGlyYN014NXp6OEZUdGs/view?usp=sharing
> >
> > Thanks a lot for the help!
> >
> > -Vasia.
> >
> > On 30 October 2015 at 21:28, Fabian Hueske <fhue...@gmail.com> wrote:
> >
> > > We can of course inject an optional ReduceFunction (or GroupReduce, or
> > > combinable GroupReduce) to reduce the size of the work set.
> > > I suggested to remove the GroupReduce function, because it did only
> > collect
> > > all messages into a single record by emitting the input iterator which
> is
> > > quite dangerous. Applying a combinable reduce function is could improve
> > the
> > > performance considerably.
> > >
> > > The good news is that it would come "for free" because the necessary
> > > partitioning and sorting can be reused (given the forwardField
> > annotations
> > > are correctly set):
> > > - The partitioning of the reduce can be reused for the join with the
> > > solution set
> > > - The sort of the reduce is preserved by the join with the in-memory
> > > hash-table of the solution set and can be reused for the coGroup.
> > >
> > > Best,
> > > Fabian
> > >
> > > 2015-10-30 18:38 GMT+01:00 Vasiliki Kalavri <vasilikikala...@gmail.com
> >:
> > >
> > > > Hi Fabian,
> > > >
> > > > thanks so much for looking into this so quickly :-)
> > > >
> > > > One update I have to make is that I tried running a few experiments
> > with
> > > > this on a 6-node cluster. The current implementation gets stuck at
> > > > "Rebuilding Workset Properties" and never finishes a single
> iteration.
> > > > Running the plan of one superstep without a delta iteration
> terminates
> > > > fine. I didn't have access to the cluster today, so I couldn't debug
> > this
> > > > further, but I will do as soon as I have access again.
> > > >
> > > > The rest of my comments are inline:
> > > >
> > > > On 30 October 2015 at 17:53, Fabian Hueske <fhue...@gmail.com>
> wrote:
> > > >
> > > > > Hi Vasia,
> &

Re: Tagging Flink classes with InterfaceAudience and InterfaceStability

[Vasiliki Kalavri]

Hi Robert,

thanks for bringing this up!

I generally like the idea, but I wouldn't rush to annotate the Gelly
classes yet. Gelly hasn't had that many users and I'm quite sure we'll find
things to improve as it gets more exposure.
TBH, I think it's quite unfair to force Gelly (also e.g. ML, Table) to a
"1.0" status (from an API stability point of view) since they're really
young compared to the other Flink APIs.

Cheers,
Vasia.
On Nov 10, 2015 8:04 PM, "Robert Metzger"  wrote:

> Hi everyone,
>
> I would like to bring this discussion back to your attention as we seem to
> approach the 1.0 release of Flink.
>
> My suggestion back in January was to annotate all classes, but I think
> it'll be more feasible to just annotate public classes.
> So how about adding an annotation @PublicInterface
>
> For @PublicInterface, I would annotate classes such as: DataSet,
> DataStream, ExecutionEnvironment, InputFormat, MapFunction, FileSystems but
> also Gelly for example.
>
> I would not annotate as public components such as ML, Storm compatibility,
> internals from runtime, yarn, optimizer.
>
>
> From a tooling perspective, I've looked into different maven plugins and
> java libraries and I found https://github.com/siom79/japicmp to be the
> closest to our needs. I actually opened a pull request to the project to
> allow inclusion/exclusion of classes based on annotations. Lets hope it
> gets merged.
>
> Does everybody agree with adding just the @PublicInterface annotation?
>
> Note that I'll add the annotation on a class-level, making the entire class
> either public or private (from a stability point of view). If we need a
> more fine-grained annotation, we have to add a second @PrivateInterface
> annotation which we'll only apply to certain methods.
>
> The next step is that I'm going to open a pull request with all classes
> annotated that I consider public.
>
>
> On Fri, Jan 30, 2015 at 7:10 PM, Henry Saputra 
> wrote:
>
> > I like the idea. But would love to have different name for the
> > "LimitedPrivate" to make it easier to distinguish.
> > How about "Module" or "Package" ?
> >
> > - Henry
> >
> > On Wed, Jan 28, 2015 at 1:29 AM, Robert Metzger 
> > wrote:
> > > I think in Hadoop they use LimitedPrivate for the different components
> of
> > > the project.
> > > For example LimitedPrivate("yarn").
> > > Here is a very good documentation on the topic:
> > >
> >
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/InterfaceClassification.html
> > >
> > > On Tue, Jan 27, 2015 at 3:58 PM, Alexander Alexandrov <
> > > alexander.s.alexand...@gmail.com> wrote:
> > >
> > >> I don't get the difference between Private and LimitedPrivate, but
> > >> otherwise seems like quite a nice idea.
> > >>
> > >> It will be also good if we can agree upon what these tags actually
> mean
> > and
> > >> add this meaning to the documentation.
> > >>
> > >> 2015-01-27 15:46 GMT+01:00 Robert Metzger :
> > >>
> > >> > Hi,
> > >> >
> > >> > Hadoop has annotations for tagging the stability and audience of
> > classes
> > >> > and methods.
> > >> >
> > >> > Through that, you can have @InterfaceAudience.Public, Private,
> > >> > LimitedPrivate
> > >> > and also @InterfaceStability.Evolving, Unstable, and Stable.
> > >> >
> > >> > I guess there are tools which allow to automatically check if
> > interfaces,
> > >> > which are marked as Stable have been changed between releases (or by
> > pull
> > >> > requests).
> > >> >
> > >> > I think such annotations are crucial if we want to guarantee
> interface
> > >> > stability between releases.
> > >> >
> > >> > What do you think? Should we add those annotations? Which one would
> > you
> > >> > like to add?
> > >> >
> > >> >
> > >> > Robert
> > >> >
> > >>
> >
>

Re: Tagging Flink classes with InterfaceAudience and InterfaceStability

[Vasiliki Kalavri]

Yes, my opinion is that we shouldn't declare the Gelly API frozen yet.
We can reconsider when we're closer to the 1.0 release, but if possible, I
would give it some more time.

-V.

On 10 November 2015 at 21:06, Stephan Ewen <se...@apache.org> wrote:

> I think no component should be forced to be stable. It should be an
> individual decision for each component, and in some cases even for
> individual classes.
>
> @Vasia If you think Gelly should not be declared interface-frozen, then
> this is a good point to raise and this should definitely be reflected.
> There is no point in declaring certain APIs as frozen when we are not yet
> confident they have converged.
>
>
>
> On Tue, Nov 10, 2015 at 8:39 PM, Vasiliki Kalavri <
> vasilikikala...@gmail.com
> > wrote:
>
> > Hi Robert,
> >
> > thanks for bringing this up!
> >
> > I generally like the idea, but I wouldn't rush to annotate the Gelly
> > classes yet. Gelly hasn't had that many users and I'm quite sure we'll
> find
> > things to improve as it gets more exposure.
> > TBH, I think it's quite unfair to force Gelly (also e.g. ML, Table) to a
> > "1.0" status (from an API stability point of view) since they're really
> > young compared to the other Flink APIs.
> >
> > Cheers,
> > Vasia.
> > On Nov 10, 2015 8:04 PM, "Robert Metzger" <rmetz...@apache.org> wrote:
> >
> > > Hi everyone,
> > >
> > > I would like to bring this discussion back to your attention as we seem
> > to
> > > approach the 1.0 release of Flink.
> > >
> > > My suggestion back in January was to annotate all classes, but I think
> > > it'll be more feasible to just annotate public classes.
> > > So how about adding an annotation @PublicInterface
> > >
> > > For @PublicInterface, I would annotate classes such as: DataSet,
> > > DataStream, ExecutionEnvironment, InputFormat, MapFunction, FileSystems
> > but
> > > also Gelly for example.
> > >
> > > I would not annotate as public components such as ML, Storm
> > compatibility,
> > > internals from runtime, yarn, optimizer.
> > >
> > >
> > > From a tooling perspective, I've looked into different maven plugins
> and
> > > java libraries and I found https://github.com/siom79/japicmp to be the
> > > closest to our needs. I actually opened a pull request to the project
> to
> > > allow inclusion/exclusion of classes based on annotations. Lets hope it
> > > gets merged.
> > >
> > > Does everybody agree with adding just the @PublicInterface annotation?
> > >
> > > Note that I'll add the annotation on a class-level, making the entire
> > class
> > > either public or private (from a stability point of view). If we need a
> > > more fine-grained annotation, we have to add a second @PrivateInterface
> > > annotation which we'll only apply to certain methods.
> > >
> > > The next step is that I'm going to open a pull request with all classes
> > > annotated that I consider public.
> > >
> > >
> > > On Fri, Jan 30, 2015 at 7:10 PM, Henry Saputra <
> henry.sapu...@gmail.com>
> > > wrote:
> > >
> > > > I like the idea. But would love to have different name for the
> > > > "LimitedPrivate" to make it easier to distinguish.
> > > > How about "Module" or "Package" ?
> > > >
> > > > - Henry
> > > >
> > > > On Wed, Jan 28, 2015 at 1:29 AM, Robert Metzger <rmetz...@apache.org
> >
> > > > wrote:
> > > > > I think in Hadoop they use LimitedPrivate for the different
> > components
> > > of
> > > > > the project.
> > > > > For example LimitedPrivate("yarn").
> > > > > Here is a very good documentation on the topic:
> > > > >
> > > >
> > >
> >
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/InterfaceClassification.html
> > > > >
> > > > > On Tue, Jan 27, 2015 at 3:58 PM, Alexander Alexandrov <
> > > > > alexander.s.alexand...@gmail.com> wrote:
> > > > >
> > > > >> I don't get the difference between Private and LimitedPrivate, but
> > > > >> otherwise seems like quite a nice idea.
> > > > >>
> > > > >> It will be also good if we can agree upon what these tags actually
> > > mean
> > > 

Re: [VOTE] Release Apache Flink 0.10.0 (release-0.10.0-rc6)

[Vasiliki Kalavri]

I also think this is it :-)
Apart from the tests in the doc, I ran a bunch of tests and the dataset
examples on a 5-node cluster with Hadoop 2.7.1.

+1

Cheers,
-Vasia.

On 9 November 2015 at 13:53, Maximilian Michels  wrote:

> +1
>
> - Checked the source files for binaries
> - Ran mvn clean verify for Hadoop 2.3, 2.4, 2.6.
> - Tested Scala packaging for Scala 2.11
> - Checked POM file versions (0.10.0)
> - Local and remote cluster startup and examination of the log/out files
> - YARN cluster deployment and execution of examples
> - Check documentation in source release
> - Verified quickstarts work with included examples
> - Ran Streaming foldl example to verify streaming API fix for rc6
>
> On Mon, Nov 9, 2015 at 11:53 AM, Ufuk Celebi 
> wrote:
> > +1
> >
> > I
> > - ran fault-tolerant job with Kafka and randomly killed workers and
> masters (for Hadoop 2.4, 2.6, 2.7 and standalone)
> > - verified the local split assignment
> > - read the README file
> > - verify that the log is free of errors/exceptions
> > - checked the quickstarts
> >
> > – Ufuk
> >
> >> On 09 Nov 2015, at 10:34, Aljoscha Krettek  wrote:
> >>
> >> +1
> >>
> >> I think this one could be it. I did:
> >> - verify the checksums of some of the release artifacts, I assume that
> the rest is also OK
> >> - test build for custom Hadoop versions 2.4, 2.5, 2.6
> >> - verify that LICENSE/NOTICE are correct
> >> - verify that licenses of dependencies are compatible
> >> - read the README file
> >> - verify that the start/stop scripts work (multi-cluster mode)
> >> - run the bundled examples with built-in and external data
> >> - verify that the log is free of errors/exceptions
> >> - run fault-tolerant job with Kafka with randomly killing TMs and JM
> >> - check that java/scala quickstarts work (also with IntelliJ)
> >> - run an example against a running cluster with RemoteEnvironment
> >> - run the manual tests in flink-tests
> >>> On 05 Nov 2015, at 10:41, Maximilian Michels  wrote:
> >>>
> >>> Here's the current testing document:
> >>>
> https://docs.google.com/document/d/1PP9ar_Astl9TZ7rX2kkGXxxuM0gPr51QFvriNVVGK3s/edit
> >>>
> >>> On Thu, Nov 5, 2015 at 10:28 AM, Maximilian Michels 
> wrote:
> >>>
>  Please vote on releasing the following candidate as Apache Flink
> version
>  0.10.0:
> 
>  The commit to be voted on:
>  b2e9bf1ed1e3b60d8012b217166145b936835ea7
> 
>  Branch:
>  release-0.10.0-rc6 (see
>  https://git1-us-west.apache.org/repos/asf/flink/?p=flink.git)
> 
>  The release artifacts to be voted on can be found at:
>  http://people.apache.org/~mxm/flink-0.10.0-rc6/
> 
>  The release artifacts are signed with the key with fingerprint
> C2909CBF:
>  http://www.apache.org/dist/flink/KEYS
> 
>  The staging repository for this release can be found at:
> 
> https://repository.apache.org/content/repositories/orgapacheflink-1053
> 
>  -
> 
>  The vote is open for the next 72 hours and passes if a majority of at
>  least three +1 PMC votes are cast.
> 
>  The vote ends on Monday November 9, 2015.
> 
>  [ ] +1 Release this package as Apache Flink 0.10.0
>  [ ] -1 Do not release this package because ...
> 
>  ===
> 
>  The following commits have been added on top of release-0.10.0-rc5:
> 
>  af05911 [FLINK-2968] [streaming] Let AbstractUdfStreamOperator forward
>  output type information to WindowFunction
>  81f8c05 [FLINK-2903][web-dashboard] Format numeric values of
> received/sent
>  counters.
>  9bd27ca [FLINK-2943][web-dashboard] Rename bytes/records read/written
> to
>  received/sent. [hotfix][web-dashboard] Fix taskmanager charts legend.
>  f1bf282 [FLINK-2964] [runtime] Fix broken spilling of MutableHashTable
>  e83bf74 [FLINK-2918] Add methods to read Hadoop SequenceFiles.
>  199e7d0 [FLINK-2958] Remove hard coded number of execution retries
>  630aae6 [FLINK-2930] Respect ExecutionConfig execution retry delay
>  bd81abb [release][scripts] deploy Scala 2.11 version to Maven
> 
> >>
> >
>

Re: Long cannot be cast to org.apache.flink.types.CopyableValue

[Vasiliki Kalavri]

Thanks for the explanation Stephan! I solved it :-)

On 9 November 2015 at 10:07, Stephan Ewen <se...@apache.org> wrote:

> The CopyableValue serializer is probably instantiated for the NullValue
> (which extends CopyableValue).
>
> It looks like you are passing a function that puts a Long into that field,
> but the TypeExtraction thinks you return a NullValue. I would guess that
> there are some unsafe generic casts in your code that emit a Long at a
> place that declares a NullValue in its signature.
>
> On Sun, Nov 8, 2015 at 9:20 PM, Vasiliki Kalavri <
> vasilikikala...@gmail.com>
> wrote:
>
> > Hello squirrels,
> >
> > I'm writing a few graph algorithms to test the performance of different
> > iteration models  and I am quite stuck with an error. While my sssp
> example
> > works fine, I get the following in my connected components job (local
> > execution inside eclipse):
> >
> >
> > Exception in thread "main"
> > org.apache.flink.runtime.client.JobExecutionException: Job execution
> > failed.
> > at
> >
> >
> org.apache.flink.runtime.jobmanager.JobManager$anonfun$handleMessage$1$anonfun$applyOrElse$5.apply$mcV$sp(JobManager.scala:563)
> > at
> >
> >
> org.apache.flink.runtime.jobmanager.JobManager$anonfun$handleMessage$1$anonfun$applyOrElse$5.apply(JobManager.scala:509)
> > at
> >
> >
> org.apache.flink.runtime.jobmanager.JobManager$anonfun$handleMessage$1$anonfun$applyOrElse$5.apply(JobManager.scala:509)
> > at
> >
> >
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> > at
> >
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> > at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
> > at
> >
> >
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)
> > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> > at
> >
> >
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> > at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at
> >
> >
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> >
> > Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to
> > org.apache.flink.types.CopyableValue
> > at
> >
> >
> org.apache.flink.api.java.typeutils.runtime.CopyableValueSerializer.serialize(CopyableValueSerializer.java:1)
> > at
> >
> >
> org.apache.flink.api.java.typeutils.runtime.TupleSerializer.serialize(TupleSerializer.java:116)
> > at
> >
> >
> org.apache.flink.api.java.typeutils.runtime.TupleSerializer.serialize(TupleSerializer.java:1)
> > at
> >
> >
> org.apache.flink.runtime.plugable.SerializationDelegate.write(SerializationDelegate.java:56)
> > at
> >
> >
> org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer.addRecord(SpanningRecordSerializer.java:79)
> > at
> >
> >
> org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:84)
> > at
> >
> >
> org.apache.flink.runtime.operators.shipping.OutputCollector.collect(OutputCollector.java:65)
> > at org.apache.flink.runtime.operators.MapDriver.run(MapDriver.java:97)
> > at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:489)
> > at
> org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:354)
> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
> > at java.lang.Thread.run(Thread.java:745)
> >
> >
> > The 2 jobs are almost identical​, expect that in connected components I
> > have Long values instead of Double and edges have no weights (NullValue
> > types).
> >
> > With this exception message, ​I'm not really sure where to look ​for the
> > error. What is the CopyableValueSerializer and when is it used? I tried
> > turning the object reuse mode on/off, but no luck. I would really
> > appreciate any hint or idea on where to look :)
> >
> > Cheers,
> > -Vasia.
> >
>

Long cannot be cast to org.apache.flink.types.CopyableValue

[Vasiliki Kalavri]

Hello squirrels,

I'm writing a few graph algorithms to test the performance of different
iteration models  and I am quite stuck with an error. While my sssp example
works fine, I get the following in my connected components job (local
execution inside eclipse):


Exception in thread "main"
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at
org.apache.flink.runtime.jobmanager.JobManager$anonfun$handleMessage$1$anonfun$applyOrElse$5.apply$mcV$sp(JobManager.scala:563)
at
org.apache.flink.runtime.jobmanager.JobManager$anonfun$handleMessage$1$anonfun$applyOrElse$5.apply(JobManager.scala:509)
at
org.apache.flink.runtime.jobmanager.JobManager$anonfun$handleMessage$1$anonfun$applyOrElse$5.apply(JobManager.scala:509)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to
org.apache.flink.types.CopyableValue
at
org.apache.flink.api.java.typeutils.runtime.CopyableValueSerializer.serialize(CopyableValueSerializer.java:1)
at
org.apache.flink.api.java.typeutils.runtime.TupleSerializer.serialize(TupleSerializer.java:116)
at
org.apache.flink.api.java.typeutils.runtime.TupleSerializer.serialize(TupleSerializer.java:1)
at
org.apache.flink.runtime.plugable.SerializationDelegate.write(SerializationDelegate.java:56)
at
org.apache.flink.runtime.io.network.api.serialization.SpanningRecordSerializer.addRecord(SpanningRecordSerializer.java:79)
at
org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:84)
at
org.apache.flink.runtime.operators.shipping.OutputCollector.collect(OutputCollector.java:65)
at org.apache.flink.runtime.operators.MapDriver.run(MapDriver.java:97)
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:489)
at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:354)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
at java.lang.Thread.run(Thread.java:745)


The 2 jobs are almost identical​, expect that in connected components I
have Long values instead of Double and edges have no weights (NullValue
types).

With this exception message, ​I'm not really sure where to look ​for the
error. What is the CopyableValueSerializer and when is it used? I tried
turning the object reuse mode on/off, but no luck. I would really
appreciate any hint or idea on where to look :)

Cheers,
-Vasia.

Re: [gelly] Spargel model rework

[Vasiliki Kalavri]

@Fabian

Is there any advantage in putting the reducer-combiner before updating the
workset vs. after (i.e. right before the join with the solution set)?

If it helps, here are the plans of these 2 alternatives:

https://drive.google.com/file/d/0BzQJrI2eGlyYcFV2RFo5dUFNXzg/view?usp=sharing
https://drive.google.com/file/d/0BzQJrI2eGlyYN014NXp6OEZUdGs/view?usp=sharing

Thanks a lot for the help!

-Vasia.

On 30 October 2015 at 21:28, Fabian Hueske <fhue...@gmail.com> wrote:

> We can of course inject an optional ReduceFunction (or GroupReduce, or
> combinable GroupReduce) to reduce the size of the work set.
> I suggested to remove the GroupReduce function, because it did only collect
> all messages into a single record by emitting the input iterator which is
> quite dangerous. Applying a combinable reduce function is could improve the
> performance considerably.
>
> The good news is that it would come "for free" because the necessary
> partitioning and sorting can be reused (given the forwardField annotations
> are correctly set):
> - The partitioning of the reduce can be reused for the join with the
> solution set
> - The sort of the reduce is preserved by the join with the in-memory
> hash-table of the solution set and can be reused for the coGroup.
>
> Best,
> Fabian
>
> 2015-10-30 18:38 GMT+01:00 Vasiliki Kalavri <vasilikikala...@gmail.com>:
>
> > Hi Fabian,
> >
> > thanks so much for looking into this so quickly :-)
> >
> > One update I have to make is that I tried running a few experiments with
> > this on a 6-node cluster. The current implementation gets stuck at
> > "Rebuilding Workset Properties" and never finishes a single iteration.
> > Running the plan of one superstep without a delta iteration terminates
> > fine. I didn't have access to the cluster today, so I couldn't debug this
> > further, but I will do as soon as I have access again.
> >
> > The rest of my comments are inline:
> >
> > On 30 October 2015 at 17:53, Fabian Hueske <fhue...@gmail.com> wrote:
> >
> > > Hi Vasia,
> > >
> > > I had a look at your new implementation and have a few ideas for
> > > improvements.
> > > 1) Sending out the input iterator as you do in the last GroupReduce is
> > > quite dangerous and does not give a benefit compared to collecting all
> > > elements. Even though it is an iterator, it needs to be completely
> > > materialized in-memory whenever the record is touched by Flink or user
> > > code.
> > > I would propose to skip the reduce step completely and handle all
> > messages
> > > separates and only collect them in the CoGroup function before giving
> > them
> > > into the VertexComputeFunction. Be careful, to only do that with
> > > objectReuse disabled or take care to properly copy the messages. If you
> > > collect the messages in the CoGroup, you don't need the GroupReduce,
> have
> > > smaller records and you can remove the MessageIterator class
> completely.
> > >
> >
> > ​I see. The idea was to expose to message combiner that user could
> > ​implement if the messages are combinable, e.g. min, sum. This is a
> common
> > case and reduces the message load significantly. Is there a way I could
> do
> > something similar before the coGroup?
> >
> >
> >
> > > 2) Add this annotation to the AppendVertexState function:
> > > @ForwardedFieldsFirst("*->f0"). This indicates that the complete
> element
> > of
> > > the first input becomes the first field of the output. Since the input
> is
> > > partitioned on "f0" (it comes out of the partitioned solution set) the
> > > result of ApplyVertexState will be partitioned on "f0.f0" which is
> > > (accidentially :-D) the join key of the following coGroup function ->
> no
> > > partitioning :-)
> > >
> >
> > ​Great! I totally missed that ;)​
> >
> >
> >
> > > 3) Adding the two flatMap functions behind the CoGroup prevents
> chaining
> > > and causes therefore some serialization overhead but shouldn't be too
> > bad.
> > >
> > > So in total I would make this program as follows:
> > >
> > > iVertices<K,VV>
> > > iMessage<K, Message> = iVertices.map(new InitWorkSet());
> > >
> > > iteration = iVertices.iterateDelta(iMessages, maxIt, 0)
> > > verticesWithMessage<Vertex, Message> = iteration.getSolutionSet()
> > >   .join(iteration.workSet())
> > >   .where(0) // solution set is local and

Re: Question about limitations of iterative algorithms

[Vasiliki Kalavri]

Hi Andre,

I'm happy you were able to solve your problem :)

Improvements to the documentation are always welcome!
To me ST != WT is straight-forward from the javadocs, but I guess it
wouldn't hurt to stress it in the docs.
Do you think you could simplify your implementation a bit to make for a
nice example? It might be a bit too complicated to follow as it is right
now.

In any case, if you would like to improve the delta iteration docs, please
go ahead and open a JIRA. We can discuss the details of what improvements
to make over there.

Thanks!
-Vasia.


On 5 November 2015 at 11:41, André Petermann <
peterm...@informatik.uni-leipzig.de> wrote:

> Hi Vasia!
>
> Many thanks for your reply! Your hints finally enabled me implementing the
> problems first-choice solution using delta iteration:
> https://gist.github.com/p3et/12deb7d6321b48e9efab
>
> Do you think this could be worth to be contributed as an example within
> the Flink documentation? The examples I found so far could not help
> enlightening me how to use delta iteration for this kind of loop
> (ST != WT, start from empty solution set, ...).
>
> Cheers,
> Andre
>
>
> On 04.11.2015 16:32, Vasiliki Kalavri wrote:
>
>> Hi Andre,
>>
>> On 4 November 2015 at 16:04, André Petermann <
>> peterm...@informatik.uni-leipzig.de> wrote:
>>
>> Hi Fabian,
>>>
>>> thanks for your fast reply!
>>>
>>> I created a gist to explain the while-not-empty loop in more detail:
>>> https://gist.github.com/p3et/9f6e56cf0b68213e3e2b
>>>
>>> It is an approach to create a minimal example of the kind of algorithm
>>> corresponding to page 1 of the PDF, in particular frequent substrings.
>>> Please don't care about the algorithm itself, the actual algorithm is
>>> much
>>> more complex. However, even this simple case runs only for 7 iterations
>>> on
>>> my machine before "too few memory segments" is raising.
>>>
>>> If I understood delta iteration right, it's, necessary to provide a 1:1
>>> mapping between working and solution set items. The empty-case you
>>> referred
>>> to is the no-more-updates case, but not no-more-items, right?
>>>
>>>
>>
>> I think it's possible to do what you want with a delta iteration.
>> ​
>> The solution set and workset *don't* need to be of the same type. When you
>> define the delta iteration,
>> e.g. DeltaIteration<ST, WT> iteration = ...,
>> ST is the type of the solution set and WT is the type of the workset.
>>
>> ​
>>
>>
>>> In my example, the working set is completely replaced for each iteration
>>> (line 52), with only a parent-child mapping. The solution set is
>>> initially
>>> empty (line 33) and stores results of all iterations (line 48). I hope
>>> this
>>> shows the difference to the delta iteration and while-not-empty. Further
>>> on, you see the different data types of working and solution sets.
>>>
>>
>>
>> I will provide a further Gist for the "second choice" solution using delta
>>> iteration. However, what we actually would prefer is to replace the
>>> for-loop by something like while(embeddings.isNotEmpty()) including a
>>> truly
>>> iterative execution.
>>>
>>>
>>
>> ​The default convergence criterion for a delta iteration is an empty
>> workset. Thus, if you set embeddings as the workset, you have your
>> "while(embeddings.isNotEmpty())" logic.​
>> ​ ​Also, as far as I know, there is no problem with appending new elements
>> to the solution set. So, using allFrequentPatterns as the solution set
>> should be fine.
>>
>>
>>
>>
>>> But please let me know if I missed some Flink features already enabling
>>> such loops.
>>>
>>> Thanks,
>>> Andre
>>>
>>
>>
>>
>> Does this clear things out a bit? Let me know if I misunderstood what you
>> want to do.
>>
>> Cheers,
>> -Vasia.
>>
>>
>>
>>
>>>
>>> On 03.11.2015 15:47, Fabian Hueske wrote:
>>>
>>> Hi Andre,
>>>>
>>>> Thanks for reaching out to the Flink community!
>>>>
>>>> I am not sure your analysis is based on correct assumptions about
>>>> Flink's
>>>> delta iterations.
>>>> Flink's delta iterations do support
>>>> - working and solution sets of different types
>>>> - worksets that grow and shrink or are completely replaced in each

Re: [gelly] Spargel model rework

[Vasiliki Kalavri]

Hi Fabian,

thanks so much for looking into this so quickly :-)

One update I have to make is that I tried running a few experiments with
this on a 6-node cluster. The current implementation gets stuck at
"Rebuilding Workset Properties" and never finishes a single iteration.
Running the plan of one superstep without a delta iteration terminates
fine. I didn't have access to the cluster today, so I couldn't debug this
further, but I will do as soon as I have access again.

The rest of my comments are inline:

On 30 October 2015 at 17:53, Fabian Hueske <fhue...@gmail.com> wrote:

> Hi Vasia,
>
> I had a look at your new implementation and have a few ideas for
> improvements.
> 1) Sending out the input iterator as you do in the last GroupReduce is
> quite dangerous and does not give a benefit compared to collecting all
> elements. Even though it is an iterator, it needs to be completely
> materialized in-memory whenever the record is touched by Flink or user
> code.
> I would propose to skip the reduce step completely and handle all messages
> separates and only collect them in the CoGroup function before giving them
> into the VertexComputeFunction. Be careful, to only do that with
> objectReuse disabled or take care to properly copy the messages. If you
> collect the messages in the CoGroup, you don't need the GroupReduce, have
> smaller records and you can remove the MessageIterator class completely.
>

​I see. The idea was to expose to message combiner that user could
​implement if the messages are combinable, e.g. min, sum. This is a common
case and reduces the message load significantly. Is there a way I could do
something similar before the coGroup?



> 2) Add this annotation to the AppendVertexState function:
> @ForwardedFieldsFirst("*->f0"). This indicates that the complete element of
> the first input becomes the first field of the output. Since the input is
> partitioned on "f0" (it comes out of the partitioned solution set) the
> result of ApplyVertexState will be partitioned on "f0.f0" which is
> (accidentially :-D) the join key of the following coGroup function -> no
> partitioning :-)
>

​Great! I totally missed that ;)​



> 3) Adding the two flatMap functions behind the CoGroup prevents chaining
> and causes therefore some serialization overhead but shouldn't be too bad.
>
> So in total I would make this program as follows:
>
> iVertices<K,VV>
> iMessage<K, Message> = iVertices.map(new InitWorkSet());
>
> iteration = iVertices.iterateDelta(iMessages, maxIt, 0)
> verticesWithMessage<Vertex, Message> = iteration.getSolutionSet()
>   .join(iteration.workSet())
>   .where(0) // solution set is local and build side
>   .equalTo(0) // workset is shuffled and probe side of hashjoin
> superstepComp<Vertex,Tuple2<K, Message>,Bool> =
> verticesWithMessage.coGroup(edgessWithValue)
>   .where("f0.f0") // vwm is locally forward and sorted
>   .equalTo(0) //  edges are already partitioned and sorted (if cached
> correctly)
>   .with(...) // The coGroup collects all messages in a collection and gives
> it to the ComputeFunction
> delta = superStepComp.flatMap(...) // partitioned when merged into
> solution set
> workSet<K, Message> = superStepComp.flatMap(...) // partitioned for join
> iteration.closeWith(delta, workSet)
>
> So, if I am correct, the program will
> - partition the workset
> - sort the vertices with messages
> - partition the delta
>
> One observation I have is that this program requires that all messages fit
> into memory. Was that also the case before?
>

​I believe not. The plan has one coGroup that produces the messages and a
following coGroup that groups by the messages "target ID" and consumes
them​ in an iterator. That doesn't require them to fit in memory, right?


​I'm also working on a version where the graph is represented as an
adjacency list, instead of two separate datasets of vertices and edges. The
disadvantage is that the graph has to fit in memory, but I think the
advantages are many​. We'll be able to support edge value updates, edge
mutations and different edge access order guarantees. I'll get back to this
thread when I have a working prototype.


>
> Cheers,
> Fabian
>

​Thanks again!

Cheers,
-Vasia.
​


>
>
> 2015-10-27 19:10 GMT+01:00 Vasiliki Kalavri <vasilikikala...@gmail.com>:
>
> > @Martin: thanks for your input! If you ran into any other issues that I
> > didn't mention, please let us know. Obviously, even with my proposal,
> there
> > are still features we cannot support, e.g. updating edge values and graph
> > mutations. We'll need to re-think the underlying iteration and/or graph
> > representation for those.
> >
&g

neo4j - Flink connector

[Vasiliki Kalavri]

Hello everyone,

Martin, Martin, Alex (cc'ed) and myself have started discussing about
implementing a neo4j-Flink connector. I've opened a corresponding JIRA
(FLINK-2941) containing an initial document [1], but we'd also like to
share our ideas here to engage the community and get your feedback.

We've had a skype call today and I will try to summarize some of the key
points here. The main use-cases we see are the following:

- Use Flink for ETL / creating the graph and then insert it to a graph
database, like neo4j, for querying and search.
- Query neo4j on some topic or search the graph for patterns and extract a
subgraph, on which we'd then like to run some iterative graph analysis
task. This is where Flink/Gelly can help, by complementing the querying
(neo4j) with efficient iterative computation.

We all agreed that the main challenge is efficiently getting the data out
of neo4j and into Flink. There have been some attempts to do similar things
with neo4j and Spark, but the performance results are not very promising:

- Mazerunner [2] is using HDFS for communication. We think that's it's not
worth it going towards this direction, as dumping the neo4j database to
HDFS and then reading it back to Flink would probably be terribly slow.
- In [3], you can see Michael Hunger's findings on using neo's HTTP
interface to import data into Spark, run PageRank and then put data back
into neo4j. It seems that this took > 2h for a 125m edge graph. The main
bottlenecks appear to be (1) reading the data as an RDD => this had to be
performed into small batches to avoid OOM errors and (2) PageRank
computation itself, which seems weird to me.

We decided to experiment with neo4j HTTP and Flink and we'll report back
when we have some results.

In the meantime, if you have any ideas on how we could speed up reading
from neo4j or any suggestion on approaches that I haven't mentioned, please
feel free to reply to this e-mail or add your comment in the shared
document.

Cheers,
-Vasia.

[1]:
https://docs.google.com/document/d/13qT_e-y8aTNWQnD43jRBq1074Y1LggPNDsic_Obwc28/edit?usp=sharing
[2]: https://github.com/kbastani/neo4j-mazerunner
[3]: https://gist.github.com/jexp/0dfad34d49a16000e804

Fwd: neo4j - Flink connector

[Vasiliki Kalavri]

Forwarding these here to keep dev@ in the loop :)


-- Forwarded message --
From: Martin Junghanns <m.jungha...@mailbox.org>
Date: 29 October 2015 at 18:37
Subject: Re: neo4j - Flink connector
To: Martin Liesenberg <martin.liesenb...@gmail.com>, Vasia Kalavri <
vasilikikala...@gmail.com>
Cc: Alexander Keller <a...@graphaware.com>, Martin Neumann <mneum...@sics.se
>


My idea was to start with a (non-parallel) Neo4jInputFormat for Flink and
see how this REST endpoint works with streaming. As Cypher query results
are rows in the end, this should work well with Flink Tuples (similar to
our JDBCInputFormat).

I'll keep you updated!

Best,
Martin


On 29.10.2015 17:00, Martin Liesenberg wrote:

Also, if you need any help with the implementation, I'd be glad to chime
in.

Best regards
Martin

Martin Liesenberg <martin.liesenb...@gmail.com> schrieb am Do., 29. Okt.
2015 15:07:

> While using neo4j for a small project at work I came across the feature of
> streaming the results from neo4j with the REST API [1]. we didnt end up
> using it, so I can't comment on performance etc., but intuitively it seems
> like a better chunking.
>
> Nice to see another connector for Flink. :)
>
> Best regards,
>
> Martin
>
> [1] http://neo4j.com/docs/stable/rest-api-streaming.html
>
> Vasiliki Kalavri < <vasilikikala...@gmail.com>vasilikikala...@gmail.com>
> schrieb am Do., 29. Okt. 2015 um 14:51 Uhr:
>
>> Hello everyone,
>>
>> Martin, Martin, Alex (cc'ed) and myself have started discussing about
>> implementing a neo4j-Flink connector. I've opened a corresponding JIRA
>> (FLINK-2941) containing an initial document [1], but we'd also like to
>> share our ideas here to engage the community and get your feedback.
>>
>> We've had a skype call today and I will try to summarize some of the key
>> points here. The main use-cases we see are the following:
>>
>> - Use Flink for ETL / creating the graph and then insert it to a graph
>> database, like neo4j, for querying and search.
>> - Query neo4j on some topic or search the graph for patterns and extract a
>> subgraph, on which we'd then like to run some iterative graph analysis
>> task. This is where Flink/Gelly can help, by complementing the querying
>> (neo4j) with efficient iterative computation.
>>
>> We all agreed that the main challenge is efficiently getting the data out
>> of neo4j and into Flink. There have been some attempts to do similar
>> things
>> with neo4j and Spark, but the performance results are not very promising:
>>
>> - Mazerunner [2] is using HDFS for communication. We think that's it's not
>> worth it going towards this direction, as dumping the neo4j database to
>> HDFS and then reading it back to Flink would probably be terribly slow.
>> - In [3], you can see Michael Hunger's findings on using neo's HTTP
>> interface to import data into Spark, run PageRank and then put data back
>> into neo4j. It seems that this took > 2h for a 125m edge graph. The main
>> bottlenecks appear to be (1) reading the data as an RDD => this had to be
>> performed into small batches to avoid OOM errors and (2) PageRank
>> computation itself, which seems weird to me.
>>
>> We decided to experiment with neo4j HTTP and Flink and we'll report back
>> when we have some results.
>>
>> In the meantime, if you have any ideas on how we could speed up reading
>> from neo4j or any suggestion on approaches that I haven't mentioned,
>> please
>> feel free to reply to this e-mail or add your comment in the shared
>> document.
>>
>> Cheers,
>> -Vasia.
>>
>> [1]:
>>
>> https://docs.google.com/document/d/13qT_e-y8aTNWQnD43jRBq1074Y1LggPNDsic_Obwc28/edit?usp=sharing
>> [2]: https://github.com/kbastani/neo4j-mazerunner
>> [3]: https://gist.github.com/jexp/0dfad34d49a16000e804
>>
>

Re: [VOTE] Release Apache Flink 0.10.0 (release-0.10.0-rc2)

[Vasiliki Kalavri]

I think I found 2 more issues with the web interface.

When inside a running job's view:
- I think the colorful boxes with the number of tasks in each status show
wrong values (or show something else?). I get different values than the
ones I see in "Overview" and "Running Jobs" tabs.
- In the "Plan" tab, it seems that some information is hidden and I cannot
scroll right to see it. I can only see 3 columns for each operator: bytes
read, records read and bytes written. I'm using Firefox.

-Vasia.

On 28 October 2015 at 13:13, Sachin Goel <sachingoel0...@gmail.com> wrote:

> While we're at it, we should also remove the dummy log and stdout tabs for
> task managers. The work on that hasn't been finished yet.
> I'll file a jira for both.
> On Oct 28, 2015 5:39 PM, "Vasiliki Kalavri" <vasilikikala...@gmail.com>
> wrote:
>
> > I see, thank you! +1 for removing before the release :)
> >
> > On 28 October 2015 at 13:06, Sachin Goel <sachingoel0...@gmail.com>
> wrote:
> >
> > > Those are hard coded values.
> > > What exactly should be there, I'm not sure either.
> > > On Oct 28, 2015 5:25 PM, "Vasiliki Kalavri" <vasilikikala...@gmail.com
> >
> > > wrote:
> > >
> > > > I have a question regarding the web interface :)
> > > > What is the "Job Accumulator/Statistics" tab supposed to show? No
> > matter
> > > > what job I run, the values are the same (operator=1, parallelism=2,
> > > > subtasks=3). Are these hard-coded defaults?
> > > >
> > > > Thanks!
> > > > -Vasia.
> > > >
> > > > On 28 October 2015 at 10:50, Maximilian Michels <m...@apache.org>
> > wrote:
> > > >
> > > > > Thanks for testing, Vasia :)
> > > > >
> > > > > Here is the new document:
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1CR3DH4tUJvukxGFQ1ySxfnzO00LjPhSTwkeE7Mf98CY/edit
> > > > >
> > > > > I've transferred results which are unaffected by the changes of the
> > new
> > > > RC.
> > > > >
> > > > > On Wed, Oct 28, 2015 at 10:33 AM, Vasiliki Kalavri <
> > > > > vasilikikala...@gmail.com> wrote:
> > > > >
> > > > > > Is there a new testing doc for rc2 or are we using the previous
> > one?
> > > > > > Thanks!
> > > > > >
> > > > > > On 27 October 2015 at 22:17, Maximilian Michels <m...@apache.org>
> > > > wrote:
> > > > > >
> > > > > > > Please vote on releasing the following candidate as Apache
> Flink
> > > > > version
> > > > > > > 0.10.0:
> > > > > > >
> > > > > > > The commit to be voted on:
> > > > > > > ed75049dfc9748eae81ace9d4d686907dcd7835c
> > > > > > >
> > > > > > > Branch:
> > > > > > > release-0.10.0-rc2 (see
> > > > > > > https://git1-us-west.apache.org/repos/asf/flink/?p=flink.git)
> > > > > > >
> > > > > > > The release artifacts to be voted on can be found at:
> > > > > > > http://people.apache.org/~mxm/flink-0.10.0-rc2/
> > > > > > >
> > > > > > > The release artifacts are signed with the key with fingerprint
> > > > > C2909CBF:
> > > > > > > http://www.apache.org/dist/flink/KEYS
> > > > > > >
> > > > > > > The staging repository for this release can be found at:
> > > > > > >
> > > >
> https://repository.apache.org/content/repositories/orgapacheflink-1049
> > > > > > >
> > > > > > > -
> > > > > > >
> > > > > > > The vote is open for the next 72 hours and passes if a majority
> > of
> > > at
> > > > > > least
> > > > > > > three +1 PMC votes are cast.
> > > > > > >
> > > > > > > The vote ends on Thursday Friday 30, 2015.
> > > > > > >
> > > > > > > [ ] +1 Release this package as Apache Flink 0.10.0
> > > > > > > [ ] -1 Do not release this package because ...
> > > > > > >
> > > > > > > ===
> > > > > > >
> > > > > > > The following commits have been added on top of
> > release-0.10.0-rc1:
> > > > > > >
> > > > > > > ae19d2b [FLINK-2927] [runtime] Provide default required
> > > configuration
> > > > > > keys
> > > > > > > in flink-conf of binary distribution
> > > > > > > 874c500 Add org.apache.httpcomponents:(httpcore, httpclient) to
> > > > > > dependency
> > > > > > > management
> > > > > > > 04e25e1 [docs] remove hard-coded version artifacts
> > > > > > > b240a80 [FLINK-2878] [webmonitor] Fix unexpected leader address
> > > > pattern
> > > > > > > 16a9edc [hotfix] Fix issue with spaces in Path in
> > > > start-*-streaming.sh
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Release Apache Flink 0.10.0 (release-0.10.0-rc2)

[Vasiliki Kalavri]

I see, thank you! +1 for removing before the release :)

On 28 October 2015 at 13:06, Sachin Goel <sachingoel0...@gmail.com> wrote:

> Those are hard coded values.
> What exactly should be there, I'm not sure either.
> On Oct 28, 2015 5:25 PM, "Vasiliki Kalavri" <vasilikikala...@gmail.com>
> wrote:
>
> > I have a question regarding the web interface :)
> > What is the "Job Accumulator/Statistics" tab supposed to show? No matter
> > what job I run, the values are the same (operator=1, parallelism=2,
> > subtasks=3). Are these hard-coded defaults?
> >
> > Thanks!
> > -Vasia.
> >
> > On 28 October 2015 at 10:50, Maximilian Michels <m...@apache.org> wrote:
> >
> > > Thanks for testing, Vasia :)
> > >
> > > Here is the new document:
> > >
> > >
> >
> https://docs.google.com/document/d/1CR3DH4tUJvukxGFQ1ySxfnzO00LjPhSTwkeE7Mf98CY/edit
> > >
> > > I've transferred results which are unaffected by the changes of the new
> > RC.
> > >
> > > On Wed, Oct 28, 2015 at 10:33 AM, Vasiliki Kalavri <
> > > vasilikikala...@gmail.com> wrote:
> > >
> > > > Is there a new testing doc for rc2 or are we using the previous one?
> > > > Thanks!
> > > >
> > > > On 27 October 2015 at 22:17, Maximilian Michels <m...@apache.org>
> > wrote:
> > > >
> > > > > Please vote on releasing the following candidate as Apache Flink
> > > version
> > > > > 0.10.0:
> > > > >
> > > > > The commit to be voted on:
> > > > > ed75049dfc9748eae81ace9d4d686907dcd7835c
> > > > >
> > > > > Branch:
> > > > > release-0.10.0-rc2 (see
> > > > > https://git1-us-west.apache.org/repos/asf/flink/?p=flink.git)
> > > > >
> > > > > The release artifacts to be voted on can be found at:
> > > > > http://people.apache.org/~mxm/flink-0.10.0-rc2/
> > > > >
> > > > > The release artifacts are signed with the key with fingerprint
> > > C2909CBF:
> > > > > http://www.apache.org/dist/flink/KEYS
> > > > >
> > > > > The staging repository for this release can be found at:
> > > > >
> > https://repository.apache.org/content/repositories/orgapacheflink-1049
> > > > >
> > > > > -
> > > > >
> > > > > The vote is open for the next 72 hours and passes if a majority of
> at
> > > > least
> > > > > three +1 PMC votes are cast.
> > > > >
> > > > > The vote ends on Thursday Friday 30, 2015.
> > > > >
> > > > > [ ] +1 Release this package as Apache Flink 0.10.0
> > > > > [ ] -1 Do not release this package because ...
> > > > >
> > > > > ===
> > > > >
> > > > > The following commits have been added on top of release-0.10.0-rc1:
> > > > >
> > > > > ae19d2b [FLINK-2927] [runtime] Provide default required
> configuration
> > > > keys
> > > > > in flink-conf of binary distribution
> > > > > 874c500 Add org.apache.httpcomponents:(httpcore, httpclient) to
> > > > dependency
> > > > > management
> > > > > 04e25e1 [docs] remove hard-coded version artifacts
> > > > > b240a80 [FLINK-2878] [webmonitor] Fix unexpected leader address
> > pattern
> > > > > 16a9edc [hotfix] Fix issue with spaces in Path in
> > start-*-streaming.sh
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Release Apache Flink 0.10.0 (release-0.10.0-rc2)

[Vasiliki Kalavri]

I have a question regarding the web interface :)
What is the "Job Accumulator/Statistics" tab supposed to show? No matter
what job I run, the values are the same (operator=1, parallelism=2,
subtasks=3). Are these hard-coded defaults?

Thanks!
-Vasia.

On 28 October 2015 at 10:50, Maximilian Michels <m...@apache.org> wrote:

> Thanks for testing, Vasia :)
>
> Here is the new document:
>
> https://docs.google.com/document/d/1CR3DH4tUJvukxGFQ1ySxfnzO00LjPhSTwkeE7Mf98CY/edit
>
> I've transferred results which are unaffected by the changes of the new RC.
>
> On Wed, Oct 28, 2015 at 10:33 AM, Vasiliki Kalavri <
> vasilikikala...@gmail.com> wrote:
>
> > Is there a new testing doc for rc2 or are we using the previous one?
> > Thanks!
> >
> > On 27 October 2015 at 22:17, Maximilian Michels <m...@apache.org> wrote:
> >
> > > Please vote on releasing the following candidate as Apache Flink
> version
> > > 0.10.0:
> > >
> > > The commit to be voted on:
> > > ed75049dfc9748eae81ace9d4d686907dcd7835c
> > >
> > > Branch:
> > > release-0.10.0-rc2 (see
> > > https://git1-us-west.apache.org/repos/asf/flink/?p=flink.git)
> > >
> > > The release artifacts to be voted on can be found at:
> > > http://people.apache.org/~mxm/flink-0.10.0-rc2/
> > >
> > > The release artifacts are signed with the key with fingerprint
> C2909CBF:
> > > http://www.apache.org/dist/flink/KEYS
> > >
> > > The staging repository for this release can be found at:
> > > https://repository.apache.org/content/repositories/orgapacheflink-1049
> > >
> > > -
> > >
> > > The vote is open for the next 72 hours and passes if a majority of at
> > least
> > > three +1 PMC votes are cast.
> > >
> > > The vote ends on Thursday Friday 30, 2015.
> > >
> > > [ ] +1 Release this package as Apache Flink 0.10.0
> > > [ ] -1 Do not release this package because ...
> > >
> > > ===
> > >
> > > The following commits have been added on top of release-0.10.0-rc1:
> > >
> > > ae19d2b [FLINK-2927] [runtime] Provide default required configuration
> > keys
> > > in flink-conf of binary distribution
> > > 874c500 Add org.apache.httpcomponents:(httpcore, httpclient) to
> > dependency
> > > management
> > > 04e25e1 [docs] remove hard-coded version artifacts
> > > b240a80 [FLINK-2878] [webmonitor] Fix unexpected leader address pattern
> > > 16a9edc [hotfix] Fix issue with spaces in Path in start-*-streaming.sh
> > >
> >
>

Re: [VOTE] Release Apache Flink 0.10.0 (release-0.10.0-rc2)

[Vasiliki Kalavri]

The numbers I see in the overview are different.

See
https://drive.google.com/file/d/0BzQJrI2eGlyYMHZZUGs2ZFJzaXc/view?usp=sharing
vs.
https://drive.google.com/file/d/0BzQJrI2eGlyYc3kzMlQ4OXN6a3c/view?usp=sharing

-Vasia.

On 28 October 2015 at 14:51, Maximilian Michels <m...@apache.org> wrote:

> @Vasia: This is a CSS problem which manifests because of a long class
> name. The colored boxes show the status of tasks from your job which
> you are viewing. Are the number not correct?
>
> @Sachin: Could you fix the wrapping of the column?
>
> On Wed, Oct 28, 2015 at 2:44 PM, Sachin Goel <sachingoel0...@gmail.com>
> wrote:
> > The name of the vertex is very long and isn't getting wrapped around to
> > accommodate all the columns. There's a TODO at the relevant place in
> > app/scripts/common/filters.coffee which was probably meant to handle
> this.
> >
> >
> > -- Sachin Goel
> > Computer Science, IIT Delhi
> > m. +91-9871457685
> >
> > On Wed, Oct 28, 2015 at 7:04 PM, Vasiliki Kalavri <
> vasilikikala...@gmail.com
> >> wrote:
> >
> >> It's Firefox 41.0.2. Resizing doesn't work :/
> >> See this screenshot for the colored boxes I'm referring to:
> >>
> >>
> https://drive.google.com/file/d/0BzQJrI2eGlyYX1VYQTlrQWNfUkE/view?usp=sharing
> >> .
> >> Shouldn't these numbers show tasks?
> >>
> >> On 28 October 2015 at 14:26, Maximilian Michels <m...@apache.org> wrote:
> >>
> >> > @Vasia:
> >> >
> >> > - There are two types of shapes which are colored :) The circles mark
> >> > the running/finished/cancelled/failed jobs while the squares mark the
> >> > status of a task within a job
> >> > (cancelled/running/failed/restart/pending/finished/total).
> >> >
> >> > - I can see all four columns in the "Plan" tab on Firefox. Which
> >> > version are you using? Does resizing the window make any difference?
> >> >
> >> > @Sachin: Thanks for your pull requests. Will pull them in for the next
> >> RC.
> >> >
> >> > On Wed, Oct 28, 2015 at 2:03 PM, Vasiliki Kalavri
> >> > <vasilikikala...@gmail.com> wrote:
> >> > > I think I found 2 more issues with the web interface.
> >> > >
> >> > > When inside a running job's view:
> >> > > - I think the colorful boxes with the number of tasks in each status
> >> show
> >> > > wrong values (or show something else?). I get different values than
> the
> >> > > ones I see in "Overview" and "Running Jobs" tabs.
> >> > > - In the "Plan" tab, it seems that some information is hidden and I
> >> > cannot
> >> > > scroll right to see it. I can only see 3 columns for each operator:
> >> bytes
> >> > > read, records read and bytes written. I'm using Firefox.
> >> > >
> >> > > -Vasia.
> >> > >
> >> > > On 28 October 2015 at 13:13, Sachin Goel <sachingoel0...@gmail.com>
> >> > wrote:
> >> > >
> >> > >> While we're at it, we should also remove the dummy log and stdout
> tabs
> >> > for
> >> > >> task managers. The work on that hasn't been finished yet.
> >> > >> I'll file a jira for both.
> >> > >> On Oct 28, 2015 5:39 PM, "Vasiliki Kalavri" <
> >> vasilikikala...@gmail.com>
> >> > >> wrote:
> >> > >>
> >> > >> > I see, thank you! +1 for removing before the release :)
> >> > >> >
> >> > >> > On 28 October 2015 at 13:06, Sachin Goel <
> sachingoel0...@gmail.com>
> >> > >> wrote:
> >> > >> >
> >> > >> > > Those are hard coded values.
> >> > >> > > What exactly should be there, I'm not sure either.
> >> > >> > > On Oct 28, 2015 5:25 PM, "Vasiliki Kalavri" <
> >> > vasilikikala...@gmail.com
> >> > >> >
> >> > >> > > wrote:
> >> > >> > >
> >> > >> > > > I have a question regarding the web interface :)
> >> > >> > > > What is the "Job Accumulator/Statistics" tab supposed to
> show?
> >> No
> >> > >> > matter
> >> > >> > > > what job I run, the values are the same (operator=1,
> &g

[gelly] Spargel model rework

[Vasiliki Kalavri]

Hello squirrels,

I want to discuss with you a few concerns I have about our current
vertex-centric model implementation, Spargel, now fully subsumed by Gelly.

Spargel is our implementation of Pregel [1], but it violates some
fundamental properties of the model, as described in the paper and as
implemented in e.g. Giraph, GPS, Hama. I often find myself confused both
when trying to explain it to current Giraph users and when porting my
Giraph algorithms to it.

More specifically:
- in the Pregel model, messages produced in superstep n, are received in
superstep n+1. In Spargel, they are produced and consumed in the same
iteration.
- in Pregel, vertices are active during a superstep, if they have received
a message in the previous superstep. In Spargel, a vertex is active during
a superstep if it has changed its value.

These two differences require a lot of rethinking when porting applications
and can easily cause bugs.

The most important problem however is that we require the user to split the
computation in 2 phases (2 UDFs):
- messaging: has access to the vertex state and can produce messages
- update: has access to incoming messages and can update the vertex value

Pregel/Giraph only expose one UDF to the user:
- compute: has access to both the vertex state and the incoming messages,
can produce messages and update the vertex value.

This might not seem like a big deal, but except from forcing the user to
split their program logic into 2 phases, Spargel also makes some common
computation patterns non-intuitive or impossible to write. A very simple
example is propagating a message based on its value or sender ID. To do
this with Spargel, one has to store all the incoming messages in the vertex
value (might be of different type btw) during the messaging phase, so that
they can be accessed during the update phase.

So, my first question is, when implementing Spargel, were other
alternatives considered and maybe rejected in favor of performance or
because of some other reason? If someone knows, I would love to hear about
them!

Second, I wrote a prototype implementation [2] that only exposes one UDF,
compute(), by keeping the vertex state in the solution set and the messages
in the workset. This way all previously mentioned limitations go away and
the API (see "SSSPComputeFunction" in the example [3]) looks a lot more
like Giraph (see [4]).

I have not run any experiments yet and the prototype has some ugly hacks,
but if you think any of this makes sense, then I'd be willing to follow up
and try to optimize it. If we see that it performs well, we can consider
either replacing Spargel or adding it as an alternative.

Thanks for reading this long e-mail and looking forward to your input!

Cheers,
-Vasia.

[1]: https://kowshik.github.io/JPregel/pregel_paper.pdf
[2]:
https://github.com/vasia/flink/tree/spargel-2/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/spargelnew
[3]:
https://github.com/vasia/flink/blob/spargel-2/flink-libraries/flink-gelly/src/main/java/org/apache/flink/graph/spargelnew/example/SSSPCompute.java
[4]:
https://github.com/grafos-ml/okapi/blob/master/src/main/java/ml/grafos/okapi/graphs/SingleSourceShortestPaths.java

Re: [gelly] Spargel model rework

[Vasiliki Kalavri]

@Martin: thanks for your input! If you ran into any other issues that I
didn't mention, please let us know. Obviously, even with my proposal, there
are still features we cannot support, e.g. updating edge values and graph
mutations. We'll need to re-think the underlying iteration and/or graph
representation for those.

@Fabian: thanks a lot, no rush :)
Let me give you some more information that might make it easier to reason
about performance:

Currently, in Spargel the SolutionSet (SS) keeps the vertex state and the
workset (WS) keeps the active vertices. The iteration is composed of 2
coGroups. The first one takes the WS and the edges and produces messages.
The second one takes the messages and the SS and produced the new WS and
the SS-delta.

In my proposal, the SS has the vertex state and the WS has <vertexId,
MessageIterator> pairs, i.e. the inbox of each vertex. The plan is more
complicated because compute() needs to have two iterators: over the edges
and over the messages.
First, I join SS and WS to get the active vertices (have received a msg)
and their current state. Then I coGroup the result with the edges to access
the neighbors. Now the main problem is that this coGroup needs to have 2
outputs: the new messages and the new vertex value. I couldn't really find
a nice way to do this, so I'm emitting a Tuple that contains both types and
I have a flag to separate them later with 2 flatMaps. From the vertex
flatMap, I crete the SS-delta and from the messaged flatMap I apply a
reduce to group the messages by vertex and send them to the new WS. One
optimization would be to expose a combiner here to reduce message size.

tl;dr:
1. 2 coGroups vs. Join + coGroup + flatMap + reduce
2. how can we efficiently emit 2 different types of records from a coGroup?
3. does it make any difference if we group/combine the messages before
updating the workset or after?

Cheers,
-Vasia.


On 27 October 2015 at 18:39, Fabian Hueske <fhue...@gmail.com> wrote:

> I'll try to have a look at the proposal from a performance point of view in
> the next days.
> Please ping me, if I don't follow up this thread.
>
> Cheers, Fabian
>
> 2015-10-27 18:28 GMT+01:00 Martin Junghanns <m.jungha...@mailbox.org>:
>
> > Hi,
> >
> > At our group, we also moved several algorithms from Giraph to Gelly and
> > ran into some confusing issues (first in understanding, second during
> > implementation) caused by the conceptional differences you described.
> >
> > If there are no concrete advantages (performance mainly) in the Spargel
> > implementation, we would be very happy to see the Gelly API be aligned to
> > Pregel-like systems.
> >
> > Your SSSP example speaks for itself. Straightforward, if the reader is
> > familiar with Pregel/Giraph/...
> >
> > Best,
> > Martin
> >
> >
> > On 27.10.2015 17:40, Vasiliki Kalavri wrote:
> >
> >> Hello squirrels,
> >>
> >> I want to discuss with you a few concerns I have about our current
> >> vertex-centric model implementation, Spargel, now fully subsumed by
> Gelly.
> >>
> >> Spargel is our implementation of Pregel [1], but it violates some
> >> fundamental properties of the model, as described in the paper and as
> >> implemented in e.g. Giraph, GPS, Hama. I often find myself confused both
> >> when trying to explain it to current Giraph users and when porting my
> >> Giraph algorithms to it.
> >>
> >> More specifically:
> >> - in the Pregel model, messages produced in superstep n, are received in
> >> superstep n+1. In Spargel, they are produced and consumed in the same
> >> iteration.
> >> - in Pregel, vertices are active during a superstep, if they have
> received
> >> a message in the previous superstep. In Spargel, a vertex is active
> during
> >> a superstep if it has changed its value.
> >>
> >> These two differences require a lot of rethinking when porting
> >> applications
> >> and can easily cause bugs.
> >>
> >> The most important problem however is that we require the user to split
> >> the
> >> computation in 2 phases (2 UDFs):
> >> - messaging: has access to the vertex state and can produce messages
> >> - update: has access to incoming messages and can update the vertex
> value
> >>
> >> Pregel/Giraph only expose one UDF to the user:
> >> - compute: has access to both the vertex state and the incoming
> messages,
> >> can produce messages and update the vertex value.
> >>
> >> This might not seem like a big deal, but except from forcing the user to
> >> split their program logic into 2 phases, Spargel al

Re: [VOTE] Release Apache Flink 0.10.0 (release-0.10.0-rc1)

[Vasiliki Kalavri]

I tested this for rc0, and I confirm.
Worked fine for Firefox and Chrome, didn't work for Safari (I left a note
in the previous testing doc).

-Vasia.

On 27 October 2015 at 15:18, Maximilian Michels  wrote:

> Good catch, Aljoscha. As far as I know the plan visualizer is only broken
> for Safari. It works for me with Firefox.
>
> On Tue, Oct 27, 2015 at 3:14 PM, Aljoscha Krettek 
> wrote:
>
> > The plan visualizer does not show anything for the output generated by
> > “bin/flink info”
> >
> > > On 27 Oct 2015, at 13:48, Aljoscha Krettek 
> wrote:
> > >
> > > start-cluster-streaming.sh and start-local-streaming.sh don’t work if
> > the flink path has spaces. I’m fixing it on master and on release-0.10.
> > >> On 26 Oct 2015, at 23:06, Maximilian Michels  wrote:
> > >>
> > >> Please vote on releasing the following candidate as Apache Flink
> version
> > >> 0.10.0:
> > >>
> > >> The commit to be voted on:
> > >> d4479404a9a9245ed897189973d8f6dadb9c814b
> > >>
> > >> Branch:
> > >> release-0.10.0-rc1 (see
> > >> https://git1-us-west.apache.org/repos/asf/flink/?p=flink.git)
> > >>
> > >> The release artifacts to be voted on can be found at:
> > >> http://people.apache.org/~mxm/flink-0.10.0-rc1/
> > >>
> > >> The release artifacts are signed with the key with fingerprint
> C2909CBF:
> > >> http://www.apache.org/dist/flink/KEYS
> > >>
> > >> The staging repository for this release can be found at:
> > >>
> https://repository.apache.org/content/repositories/orgapacheflink-1048
> > >>
> > >> -
> > >>
> > >> The vote is open for the next 72 hours and passes if a majority of at
> > least
> > >> three +1 PMC votes are cast.
> > >>
> > >> The vote ends on Thursday October 29, 2015.
> > >>
> > >> [ ] +1 Release this package as Apache Flink 0.10.0
> > >> [ ] -1 Do not release this package because ...
> > >>
> > >> ===
> > >>
> > >> The following commits have been added on top of release-0.10.0-rc0:
> > >>
> > >> 65fcd3a [FLINK-2891] [streaming] Set keys for key/value state in
> window
> > >> evaluation of fast-path windows.
> > >> 8e4cb0a [FLINK-2888] [streaming] State backends return copies of the
> > >> default values
> > >> c2811ce [FLINK-2866] [runtime] Eagerly close FSDataInputStream in file
> > >> state handle
> > >> ec1730b [docs] add information on how to use Kerberos
> > >> 856b278 [hotfix] Improve handling of Window Trigger results
> > >> bc5b852 [hotfix] Add Window Parameter in
> > >> Trigger.onEventTime/onProcessingTime
> > >> 15d3f10 [FLINK-2895] Duplicate immutable object creation
> > >> 8ec828c [FLINK-2893] [runtime] Consistent naming of recovery config
> > >> parameters
> > >> c0d7073 [FLINK-1982] [record-api] Remove dependencies on Record API
> from
> > >> flink-runtime tests
> > >> 712c868 [hotfix] Fix Mutable Object window aggregator/Disable Object
> > Copy
> > >> 45ab0eb [hotfix] Fix broken copy in OperatorChain
> > >> c257abf Add copy() to Tuple base class.
> > >> 85b73e0 [hotfix] Fix processing time triggering on Window Operator
> > >> c72eff4 [FLINK-2874] Fix recognition of Scala default setters
> > >> 42b5ead [FLINK-2874] Fix Avro getter/setter recognition
> > >> 5c3eb8b [FLINK-2668] [DataSet] [api-breaking] Chained Projections are
> no
> > >> longer appended
> > >> dadb1a8 [FLINK-2206] [webui] Fix incorrect counts of finished,
> canceled,
> > >> and failed jobs in new web dashboard
> > >> e340f83 [FLINK-2891] Set KV-State key upon Window Evaluation in
> General
> > >> Windows
> > >> db19973 [FLINK-2887] [gelly] make sendMessageToAllNeighbors respect
> the
> > >> EdgeDirection if set in the configuration
> > >
> >
> >
>

Re: From 0.10 to 1.0

[Vasiliki Kalavri]

+1 ^^

On 23 October 2015 at 13:14, Matthias J. Sax  wrote:

> +1 for 1.0
> it's time to "grow up" :)
>
> On 10/23/2015 12:52 PM, Kostas Tzoumas wrote:
> > +1 for 1.0, it's the right time if not a bit overdue in my opinion
> >
> > On Fri, Oct 23, 2015 at 12:06 PM, Fabian Hueske 
> wrote:
> >
> >> Yes, let's do it
> >> +1
> >>
> >> 2015-10-23 12:00 GMT+02:00 Stephan Ewen :
> >>
> >>> +1 for 1.0 :-)
> >>>
> >>> On Fri, Oct 23, 2015 at 11:59 AM, Maximilian Michels 
> >>> wrote:
> >>>
>  Dear Flink community,
> 
>  We have forked the current release candidate from the master to the
>  release-0.10-rc0 branch. Changes for the next release candidate should
>  be pushed to the release-0.10 branch. The master needed to be updated
>  to deploy only one snapshot version to the Maven repository and to
>  continue development for the next release.
> 
>  Some of us have suggested to go to 1.0 after the 0.10. That's why I
>  updated the version to 1.0-SNAPSHOT in the master. Only after I
>  committed, I realized not everyone might agree with that. I wanted to
>  emphasize that this is just my personal opinion. We may change the
>  version again if we agree on a different version.
> 
>  Let me know how you feel about 1.0 as the next targeted release.
>  Personally, I think it's great for Flink and we can really push for a
>  great 1.0 release after 0.10 is out.
> 
>  Cheers,
>  Max
> 
> >>>
> >>
> >
>
>

Re: [DISCUSS] Java code style

[Vasiliki Kalavri]

Hey,

sorry I haven't replied so far. I was enjoying the thread tough :P

I'm +1 for 120 line length and tabs. I wouldn't voice a -1 for spaces, but
it seems to me like an unnecessary change that would touch every single
Java file and without substantially improving anything.

JavaDocs by-module with JIRAs to track progress seems like the best choice
to me.

-Vasia.

On 23 October 2015 at 11:34, Fabian Hueske  wrote:

> Enforcing JavaDocs (no, by-file, by-module, global) is another open
> question.
>
> Regarding the line length, I am OK with 120 chars.
>
> 2015-10-23 11:29 GMT+02:00 Ufuk Celebi :
>
> > I think that we have two open questions now:
> >
> > 1. Line length
> >
> > From the discussion so far, I think that no one wants 80 characters line
> > length.
> >
> > The final decision will be 100 vs. 120 characters. 120 characters is what
> > we have right now (for most parts), so it is fair to keep it that way,
> but
> > enforce it (get rid of the longer lines).
> >
> > Is everyone OK with this?
> >
> > 2. Tabs vs. Spaces:
> >
> > I hope I’m not misrepresenting someone with the following summary of
> > positions.
> >
> > Tabs:
> > - Robert
> > - Max
> > - Fabian
> > (- Stephan)
> >
> > Spaces:
> > - Matthias
> > - Marton
> > - Till
> > - Gyula
> > - Henry
> > (- Stephan)
> >
> > Let’s wrap the discussion up by focusing on this question.
> >
> > What are the PROs/CONs for the respective approaches? If we went with the
> > opposing approach, would you voice a -1? E.g. would a “tabs person" -1 a
> > "spaces decision" and vice versa?
> >
> > – Ufuk
> >
> > > On 23 Oct 2015, at 10:34, Maximilian Michels  wrote:
> > >
> > > I don't think lazily adding comments will work. However, I'm fine with
> > > adding all the checkstyle rules one module at a time (with a jira
> > > issue to keep track of the modules already converted). It's not going
> > > to happen that we lazily add comments because that's the reason why
> > > comments are missing in the first place...
> > >
> > > On Fri, Oct 23, 2015 at 12:05 AM, Henry Saputra <
> henry.sapu...@gmail.com>
> > wrote:
> > >> Could we make certain rules to give warning instead of error?
> > >>
> > >> This would allow us to cherry-pick certain rules we would like people
> > >> to follow but not strictly enforced.
> > >>
> > >> - Henry
> > >>
> > >> On Thu, Oct 22, 2015 at 9:20 AM, Stephan Ewen 
> wrote:
> > >>> I don't think a "let add comments to everything" effort gives us good
> > >>> comments, actually. It just gives us checkmark comments that make the
> > rules
> > >>> pass.
> > >>>
> > >>> On Thu, Oct 22, 2015 at 3:29 PM, Fabian Hueske 
> > wrote:
> > >>>
> >  Sure, I don't expect it to be free.
> >  But everybody should be aware of the cost of adding this code style,
> > i.e.,
> >  spending a huge amount of time on reformatting and documenting code.
> > 
> >  Alternatively, we could drop the JavaDocs rule and make the
> transition
> >  significantly cheaper.
> > 
> >  2015-10-22 15:24 GMT+02:00 Till Rohrmann :
> > 
> > > There ain’t no such thing as a free lunch and code style.
> > >
> > > On Thu, Oct 22, 2015 at 3:13 PM, Maximilian Michels <
> m...@apache.org>
> > > wrote:
> > >
> > >> I think we have to document all these classes. Code Style doesn't
> > come
> > >> for free :)
> > >>
> > >> On Thu, Oct 22, 2015 at 3:09 PM, Fabian Hueske  >
> > > wrote:
> > >>> Any ideas how to deal with the mandatory JavaDoc rule for
> existing
> > > code?
> > >>> Just adding empty headers to make the checkstyle pass or start a
> > > serious
> > >>> effort to add the missing docs?
> > >>>
> > >>> 2015-10-21 13:31 GMT+02:00 Matthias J. Sax :
> > >>>
> >  Agreed. That's the reason why I am in favor of using vanilla
> > Google
> > > code
> >  style.
> > 
> >  On 10/21/2015 12:31 PM, Stephan Ewen wrote:
> > > We started out originally with mixed tab/spaces, but it ended
> up
> > > with
> > > people mixing spaces and tabs arbitrarily, and there is little
> > way
> > > to
> > > enforce Matthias' specific suggestion via checkstyle.
> > > That's why we dropped spaces alltogether...
> > >
> > > On Wed, Oct 21, 2015 at 12:03 PM, Gyula Fóra <
> >  gyula.f...@gmail.com>
> >  wrote:
> > >
> > >> I think the nice thing about a common codestyle is that
> everyone
> > > can
> > >> set
> > >> the template in the IDE and use the formatting commands.
> > >>
> > >> Matthias's suggestion makes this practically impossible so -1
> > for
> > >> mixed
> > >> tabs/spaces from my side.
> > >>
> > >> Matthias J. Sax  ezt írta (időpont: 2015.
> > okt.
> > 

Re: [VOTE] Release Apache Flink 0.10.0 (release-0.10.0-rc0)

[Vasiliki Kalavri]

Hi,

I found an issue with the web interface. It only shows the last 5 finished
jobs and the "Jobs Finished" counter also goes up to 5.
I found FLINK-2206, but it seems that this was fixed for the old interface
and broken again in the new one?
Shall I open another issue or is there something I need to configure in the
new web interface and I haven't?

Thanks,
-V.

On 22 October 2015 at 11:20, Stephan Ewen <se...@apache.org> wrote:

> I am onto FLINK-2800 and FLINK-2888
>
> I would not disable YARN detached mode, it is used quite a bit by streaming
> users and makes perfect sense for streaming jobs, which are always one-shot
> currently.
>
> On Thu, Oct 22, 2015 at 11:03 AM, Maximilian Michels <m...@apache.org>
> wrote:
>
> > I had a gut feeling this wouldn't be the last RC :)
> >
> > I second Stephan, Ufuk, and Fabian. It's a good idea to start testing the
> > release candidate. We might discover more issues on the way. In the
> > meantime, let's fix FLINK-2763 and FLINK-2800 and push those to the
> > release-0.10 branch.
> >
> > Eager execution calls are not supported by the detached YARN execution
> > mode. The Flink standalone cluster mode doesn't have a detached mode.
> > Sachin was working on throwing a proper exception for programs that try
> to
> > submit a job which contains eager execution calls. In the course of
> > throwing a proper exception, he also added detached execution mode
> support
> > for the standalone cluster mode. The reason why eager execution doesn't
> > work with detached programs is that the client submits the first Flink
> job
> > (call to execute/count/collect/print) to the cluster and then returns an
> > empty ExecutionResult. This leads to error if the user program tries to
> > access the ExecutionResult.
> >
> > Sachin's pull request hasn't been merged in time for the release. I would
> > like Flink to support detached execution mode but I suggest to disable
> > detached execution mode for YARN in this release. We can include a proper
> > support for the next release.
> >
> > On Thu, Oct 22, 2015 at 9:50 AM, Sachin Goel <sachingoel0...@gmail.com>
> > wrote:
> >
> > > Not sure if it's a blocker, but the yarn detached mode is faulty for
> > > interactive programs with eager execution calls. The most basic
> starting
> > > point for yarn, i.e. *bin/flink run -m yarn-cluster -yd -n <>
> > > examples/Wordcount.jar* fails in a bad way.
> > >
> > >
> > > -- Sachin Goel
> > > Computer Science, IIT Delhi
> > > m. +91-9871457685
> > >
> > > On Thu, Oct 22, 2015 at 3:57 AM, fhueske <fhue...@gmail.com> wrote:
> > >
> > > > +1 to that, Stephan.
> > > >
> > > > I can help with FLINK-2763 or FLINK-2800.
> > > >
> > > >
> > > > From: Stephan Ewen
> > > > Sent: Thursday, October 22, 2015 0:02
> > > > To: dev@flink.apache.org
> > > > Subject: Re: [VOTE] Release Apache Flink 0.10.0 (release-0.10.0-rc0)
> > > >
> > > >
> > > > From my side 2888 is a valid blocker. Aljoscha also found another
> > blocker
> > > > bug, so this RC will need a few patches.
> > > >
> > > > I think for 2824 there was no consensus to what would actually be the
> > > > desired behavior, which makes it a bad candidate for a release
> blocker.
> > > >
> > > > I would try and fix FLINK-2763 and FLINK-2800 if possible, but not
> > block
> > > > the release on that. They seem to be very corner case. Good to fix
> > them,
> > > > but not blockers. Too many people are on the 0.10 SNAPSHOT right now
> > and
> > > > too many urgent fixes are in that people wait to be available in a
> > > release.
> > > >
> > > > How about we start testing anyways, because I would expect us to find
> > > more
> > > > issues, and we save time if we do not create a new release candidate
> > for
> > > > each patch.
> > > >
> > > > On Wed, Oct 21, 2015 at 11:44 PM, Flavio Pompermaier <
> > > pomperma...@okkam.it
> > > > >
> > > > wrote:
> > > >
> > > > > I would also point out that Flink-2763 and Flink-2800 could be
> worth
> > of
> > > > > further investigations before this release
> > > > >
> > > > > Best,
> > > > > Flavio
> > > > > On 21 Oct 2015 23:33, "Gyula Fóra" <gyf...@ap

Re: Design document for FLINK-2254

[Vasiliki Kalavri]

Hi Saumitra,

could you maybe add your ideas above in a google doc and share the link, so
we can easily comment and/or edit?

Thank you!
-Vasia.

On 22 October 2015 at 10:53, Andra Lungu  wrote:

> Hi Saumitra,
>
> As you already noticed, the first version (with duplicates) is highly
> inefficient and consumes a lot of memory. So, I suggest we drop it for now.
> The version with the label makes a lot of modifications on the base Graph
> class, and this, in my opinion would make it more difficult to grasp.
> Bipartite Graphs are not as common as regular graphs. And then, when you
> would add an Isomorphic Graph class (stupid example), you'll need to strip
> stuff out again.
>
> I believe we should go with a BipartiteGraph class which extends Graph and
> which has a DataSet of topVertices and a DataSet of bottomVertices. Apart
> from getTopVertices() methods, we would still need functions such as
> getNumberOfTopVertices(). For the time being, just support the funstiones
> mentiones as well as the basics: fromCollection, fromDataSet, etc, get*,
> join*, map*, filter*.
>
> Wait for a few more opinions and dig in. I believe we should discuss this
> even further and propose Jiras for examples of algos for bipartite graphs.
>
> Once we have the design specs well figured out, and if I find some time
> I'll gladly chime in :)
>
> Cheers!
> Andra
>
> On Wed, Oct 21, 2015 at 8:45 PM, Saumitra Shahapure <
> saumitra.offic...@gmail.com> wrote:
>
> > In FLINK-2254  , we
> > want to extend Graph API of Gelly by adding support for bipartite graphs
> > too. In the long term, the support for newer graph types can be added in
> > the same way. Also specialised algorithms for special types of graphs
> > should be implemented.
> >
> > For bipartite graph, a new class BipartiteGraph can be implemented which
> > extends Graph. Other graph algorithms which are written for Graph can be
> > used for BipartiteGraph too, because of inheritance.
> > Initialisation functions like  public static  Graph
> > fromCollection(Collection> vertices,Collection>
> > edges, ExecutionEnvironment context) need to be duplicated as public
> static
> >  BipartiteGraph fromCollection(Collection > VV>> topVertices,Collection> bottomVertices,
> > Collection> edges, ExecutionEnvironment context)
> > Graph validation does not need to happen implicitly. user can call
> > BipartiteGraph.validate() in case he wants to check sanity of data.
> > Vertex modes is primary requirement. BipartiteGraph should have
> functions,
> > getTopVertices() and getBottomVertices(). There are three ways to
> implement
> > it:
> > Simplest and least efficient way is to maintain vertices variable in
> Graph
> > in addition to two more Datasets topVertices, bottomVertices. Benefit of
> > this approach is that Graph class would not need any modification at all.
> > this.vertices variable access inside Graph class would work correctly.
> > Disadvantage is that multiple copies of vertex dataset are created and
> > maintained. So this is inefficient in terms of memory.
> > Another way is to keep topVertices and bottomVertices variables in
> > BipartiteGraph. vertices variable in Graph would stay empty.
> getVertices()
> > function of Graph would be overloaded by BipartiteGraph and reimplemented
> > as union of of topVertices and bottomVertices. Since, vertices is a
> private
> > variable, external view of vertices stays unaffected. All functions of
> > Graph class which use vertices local variable (e.g.
> getVerticesAsTuple2())
> > need to use getVertices() instead of this.vertices. Disadvantage of this
> > method is Graph.vertices variable would have invalid value throughout for
> > BipartiteGraph.
> > Another way is to ‘label’ vertices with an integer. So public class
> > BipartiteGraph extends Graph, EV> would be
> > the signature. Initialisers would tag the vertices according to their
> mode.
> > getVertices() should be overridden to strip-off the tag values so that
> > users would get consistent view of vertex dataset for Graph and
> > BipartiteGraph. getTopVertices/getBottomVertices would be filter
> functions
> > on vertex tags.
> > I personally feel method 2 to be most elegant.
> > Functions like addVertices(vertexList) are not relevant without
> specifying
> > whether they are top or bottom. A possibility could be to add them to
> > topVertices by default. And have overloaded function
> > addVertices(vertexList, mode) to add them to specific partition.
> > Specialised algorithms for Bipartite graphs can implement new
> > BipartiteGraphAlgorithm interface. It would be similar to GraphAlgorithm.
> > In future, newer types of graphs can be similarly derived from Graph and
> > type hierarchy of Graph can be created. Class hierarchy would be better
> > here rather than 

Re: [VOTE] Release Apache Flink 0.10.0 (release-0.10.0-rc0)

[Vasiliki Kalavri]

Thanks!
But shouldn't the "completed jobs" counter be updated regardless of the
configuration value of how much history we store?

On 22 October 2015 at 11:32, Stephan Ewen <se...@apache.org> wrote:

> @Vasia: We cannot keep infinite history currently, because the history is
> kept in memory. Should improve that in the future...
>
> On Thu, Oct 22, 2015 at 11:31 AM, Maximilian Michels <m...@apache.org>
> wrote:
>
> > @Stephan: That's right, the detached mode is very useful for streaming
> > programs. Let's see if we can merge Sachin's pull request to give more
> > meaningful exceptions in case of user programs which are not
> > compatible with the detached execution mode.
> >
> > @Vasia: That's a feature :) You can adjust the number of old jobs to
> > be kept by setting 'jobmanager.web.history' (default is 5).
> >
> >
> > On Thu, Oct 22, 2015 at 11:28 AM, Vasiliki Kalavri
> > <vasilikikala...@gmail.com> wrote:
> > > Hi,
> > >
> > > I found an issue with the web interface. It only shows the last 5
> > finished
> > > jobs and the "Jobs Finished" counter also goes up to 5.
> > > I found FLINK-2206, but it seems that this was fixed for the old
> > interface
> > > and broken again in the new one?
> > > Shall I open another issue or is there something I need to configure in
> > the
> > > new web interface and I haven't?
> > >
> > > Thanks,
> > > -V.
> > >
> > > On 22 October 2015 at 11:20, Stephan Ewen <se...@apache.org> wrote:
> > >
> > >> I am onto FLINK-2800 and FLINK-2888
> > >>
> > >> I would not disable YARN detached mode, it is used quite a bit by
> > streaming
> > >> users and makes perfect sense for streaming jobs, which are always
> > one-shot
> > >> currently.
> > >>
> > >> On Thu, Oct 22, 2015 at 11:03 AM, Maximilian Michels <m...@apache.org>
> > >> wrote:
> > >>
> > >> > I had a gut feeling this wouldn't be the last RC :)
> > >> >
> > >> > I second Stephan, Ufuk, and Fabian. It's a good idea to start
> testing
> > the
> > >> > release candidate. We might discover more issues on the way. In the
> > >> > meantime, let's fix FLINK-2763 and FLINK-2800 and push those to the
> > >> > release-0.10 branch.
> > >> >
> > >> > Eager execution calls are not supported by the detached YARN
> execution
> > >> > mode. The Flink standalone cluster mode doesn't have a detached
> mode.
> > >> > Sachin was working on throwing a proper exception for programs that
> > try
> > >> to
> > >> > submit a job which contains eager execution calls. In the course of
> > >> > throwing a proper exception, he also added detached execution mode
> > >> support
> > >> > for the standalone cluster mode. The reason why eager execution
> > doesn't
> > >> > work with detached programs is that the client submits the first
> Flink
> > >> job
> > >> > (call to execute/count/collect/print) to the cluster and then
> returns
> > an
> > >> > empty ExecutionResult. This leads to error if the user program tries
> > to
> > >> > access the ExecutionResult.
> > >> >
> > >> > Sachin's pull request hasn't been merged in time for the release. I
> > would
> > >> > like Flink to support detached execution mode but I suggest to
> disable
> > >> > detached execution mode for YARN in this release. We can include a
> > proper
> > >> > support for the next release.
> > >> >
> > >> > On Thu, Oct 22, 2015 at 9:50 AM, Sachin Goel <
> > sachingoel0...@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > Not sure if it's a blocker, but the yarn detached mode is faulty
> for
> > >> > > interactive programs with eager execution calls. The most basic
> > >> starting
> > >> > > point for yarn, i.e. *bin/flink run -m yarn-cluster -yd -n <>
> > >> > > examples/Wordcount.jar* fails in a bad way.
> > >> > >
> > >> > >
> > >> > > -- Sachin Goel
> > >> > > Computer Science, IIT Delhi
> > >> > > m. +91-9871457685
> > >> > >
> > >> > > On Thu, Oct 22, 2015 at 3:57 AM, fhueske <fhue...@

Re: [VOTE] Release Apache Flink 0.10.0 (release-0.10.0-rc0)

[Vasiliki Kalavri]

Awesome! Thanks Max :))

I have a couple of questions:
- what about the blocker issue (according to the wiki) FLINK-2747?
- weren't we going to get rid of staging altogether?

Cheers,
-V.

On 21 October 2015 at 19:54, Stephan Ewen  wrote:

> Super, thanks Max!
>
> We should also bump the master to the next version then, to separate what
> goes into release fixes and what goes into the next version...
>
> Is that going to be 1.0-SNAPSHOT? ;-) That is a separate thread, I guess...
>
> On Wed, Oct 21, 2015 at 7:12 PM, Maximilian Michels 
> wrote:
>
> > Release candidates have to be tested thoroughly. Therefore, I would like
> > everybody to take a look at the release page in the wiki:
> > https://cwiki.apache.org/confluence/display/FLINK/0.10+Release
> >
> > I've compiled the checks into a document. I would like everyone to assign
> > one of the checks in the documents to test the release candidate:
> >
> >
> https://docs.google.com/document/d/1TWCFj55xTyJjGYe8x9YEqmICgSvcexDPlbgP4CnLpLY/edit?usp=sharing
> >
> > On Wed, Oct 21, 2015 at 7:10 PM, Maximilian Michels 
> > wrote:
> >
> > > Dear community,
> > >
> > > The past months we have been working very hard to push towards 0.10. I
> > > would like to propose the first release candidate.
> > >
> > > ===
> > > Please vote on releasing the following candidate as Apache Flink
> version
> > > 0.10.0:
> > >
> > > The commit to be voted on:
> > > b697064b71b97e51703caae13660038949d41631
> > >
> > > Branch:
> > > release-0.10.0-rc0 (see
> > > https://git1-us-west.apache.org/repos/asf/flink/?p=flink.git)
> > >
> > > The release artifacts to be voted on can be found at:
> > > http://people.apache.org/~mxm/flink-0.10.0-rc0/
> > >
> > > Release artifacts are signed with the key with fingerprint C2909CBF:
> > > http://www.apache.org/dist/flink/KEYS
> > >
> > > The staging repository for this release can be found at:
> > > https://repository.apache.org/content/repositories/orgapacheflink-1047
> > > -
> > >
> > > Please vote on releasing this package as Apache Flink 0.10.0.
> > >
> > > The vote is open for the next 72 hours and passes if a majority of at
> > > least three +1 PMC votes are cast.
> > >
> > > The vote ends on Monday October 26, 2015.
> > >
> > > [ ] +1 Release this package as Apache Flink 0.10.0
> > > [ ] -1 Do not release this package because ...
> > >
> > > ===
> > >
> >
>

Re: [DISCUSS] flink-external

[Vasiliki Kalavri]

Thank you Matthias!

I'm not sure where the "Downloads" section is the right place for this.
I would actually put it under "Community", with a header "External
Contributions" or something like this, but I'm not feeling strong about
this :)

-Vasia.


On 9 October 2015 at 15:29, Matthias J. Sax <mj...@apache.org> wrote:

> I was not sure what we should add and was hoping for input from the
> community.
>
> I am aware of the following projects we might want to add:
>
>   - Zeppelin
>   - SAMOA
>   - Mahout
>   - Cascading (dataartisan repo)
>   - BigPetStore
>   - Gradoop
>
>
> -Matthias
>
>
>
> On 10/09/2015 03:07 PM, Maximilian Michels wrote:
> > Cool. Right now the list is empty. Do you already have a list you
> > could include in the upcoming pull request? :)
> >
> > On Fri, Oct 9, 2015 at 2:29 PM, Matthias J. Sax <mj...@apache.org>
> wrote:
> >> Hi,
> >>
> >> I just started this. Please see
> >> https://github.com/mjsax/flink-web/tree/flink-external-page
> >>
> >> I think, it is the best way to extend the "Downloads" page. I would also
> >> add a link to this on the main page's "Getting Started" section.
> >>
> >> As a first try, I started like this:
> >>> Third party packages
> >>>
> >>> This is a list of third party packages (ie, libraries, system
> extensions, or examples) build for Flink. The Flink community only collects
> links to those packages but does not maintain them. Thus, they do not
> belong to the Apache Flink project and the community cannot give any
> support for them.
> >>> Package Name
> >>>
> >>> Available for Flink 0.8.x and 0.9.x
> >>>
> >>> Short description
> >>>
> >>> Please let us know, if we missed to list your package. Be aware, that
> we might remove listed packages without notice.
> >>
> >> Can you please give me some input, what projects I should add initially?
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 10/08/2015 04:03 PM, Maximilian Michels wrote:
> >>> IMHO we can do that. There should be a disclaimer that the third party
> >>> software is not officially supported.
> >>>
> >>> On Thu, Oct 8, 2015 at 2:25 PM, Matthias J. Sax <mj...@apache.org>
> wrote:
> >>>> Should we add a new page at Flink project web page?
> >>>>
> >>>> On 10/08/2015 12:56 PM, Maximilian Michels wrote:
> >>>>> +1 for your pragmatic approach, Vasia. A simple collection of third
> >>>>> party software using Flink should be enough for now; of course,
> >>>>> outside the Apache realm.
> >>>>>
> >>>>> On Thu, Oct 8, 2015 at 12:45 PM, Chiwan Park <chiwanp...@apache.org>
> wrote:
> >>>>>> +1 for Vasia’s suggestion. From a long-term perspective, the site
> like Spark Packages [1] would be helpful to manage external contribution.
> >>>>>>
> >>>>>> [1] http://spark-packages.org
> >>>>>>
> >>>>>>> On Oct 8, 2015, at 12:28 PM, Matthias J. Sax <mj...@apache.org>
> wrote:
> >>>>>>>
> >>>>>>> Thanks for the feedback.
> >>>>>>>
> >>>>>>> I think, the repository does not need to build on a single Flink
> >>>>>>> release. From my point of view, there should be a single parent
> module
> >>>>>>> that contains *independent modules* for each extension/library
> (there
> >>>>>>> should be no "cross dependencies" between the modules and each
> module
> >>>>>>> can specify the flink dependencies it needs by itself). This make
> is
> >>>>>>> most flexible. And if a library works on an old release, it might
> just
> >>>>>>> stay there as is. If a library changes (due to Flink changes), it
> might
> >>>>>>> just be contained multiple times for different Flink releases.
> >>>>>>>
> >>>>>>> Each module should provide a short doc (README) that shows how to
> use an
> >>>>>>> integrate it with Flink. Thus, the responsibility goes to the
> >>>>>>> contributor to maintain the library. If it breaks and is not
> maintained
> >>>>>>> any further, we can simple remove it.
> >>&

Re: [DISCUSS] flink-external

[Vasiliki Kalavri]

How about, for now, we simply create a page where we gather links/short
descriptions of all these contributions
and let the maintenance and dependency management to the tool/library
creators?
This way we will at least have these contributions in one place and link to
them somewhere from the website.

-Vasia.

On 8 October 2015 at 12:06, Maximilian Michels  wrote:

> Hi Matthias,
>
> Thanks for bringing up this idea. Actually, it has been discussed a
> couple of times on the mailing list whether we should have a central
> place for third-party extensions/contributions/libraries. This could
> either be something package-based or, like you proposed, another
> repository.
>
> An external place for contributions raises a couple of questions
>
> - Which version should the external contributions be based on?
> - How do we make sure, the extensions are continuously updated?
> (dedicated maintainers or automatic compatibility checks)
> - How do we easily plug-in the external modules into Flink?
>
> In the long term, we really need a solution for these questions. The
> code base of Flink is growing and more and more packages go to
> flink-contrib/flink-staging. I would find something packaged-based
> better than a repository. Quite frankly, momentarily, I think
> developing such a plugin system is out of scope for most Flink
> developers. At the current pace of Flink development, collecting these
> contributions externally without properly maintaining them, doesn't
> make much sense to me.
>
> Cheers,
> Max
>
>
>
> On Wed, Oct 7, 2015 at 11:42 AM, Matthias J. Sax  wrote:
> >
> > Hi,
> >
> > many people are building quite exiting stuff on top of Flink. It is hard
> > to keep an good overview on what stuff is available and what not. What
> > do you think about starting a second git repository "flink-external"
> > that collects all those code?
> >
> > The ideas would be to collect stuff in a central point, such that people
> > can access it easily and get an overview what is already available (this
> > might also avoid duplicate development). It might also be a good point
> > to show common patterns. In order to collect as much as possible, the
> > contributing requirement (with respect to testing etc) could be lower
> > than for Flink itself.
> >
> > For example, I recently started a small flink-clojure module with a
> > simple word-count example to answer a question on SO. Including this in
> > Flink would not be appropriate. However, for a flink-external repro it
> > might be nice to have.
> >
> > What do you think about it?
> >
> >
> > -Matthias
> >
>

Re: [DISCUSS] Introducing a review process for pull requests

[Vasiliki Kalavri]

Hey,

I agree that we need to organize the PR process. A PR management tool would
be great.

However, it seems to me that the shepherding process described is -more or
less- what we've already been doing. There is usually a person that reviews
the PR and kind-of drives the process. Maybe making this explicit will make
things go faster.

There is a problem I see here, quite related to what Theo also mentioned.
For how long do we let a PR lingering around without a shepherd? What do we
do if nobody volunteers?
Could we maybe do a "PR overall status assessment" once per week or so,
where we find those problematic PRs and try to assign them / close them?

Finally, I think shepherding one's own PR is a bad idea (the review part)
unless it's something trivial.

Cheers and see you very soon,
-Vasia.
On Oct 7, 2015 11:24 AM, "Matthias J. Sax"  wrote:

> Ok. That makes sense. So most people will need to change behavior and
> start discussions in JIRA and not over dev list. Furthermore, issues
> list must be monitored more carefully... (I personally, watch dev
> carefully and only skip over issues list right now)
>
> -Matthias
>
> On 10/07/2015 10:22 AM, Fabian Hueske wrote:
> > @Matthias: That's a good point. Each PR should be backed by a JIRA issue.
> > If that's not the case, we have to make the contributor aware of that
> rule.
> > I'll update the process to keep all discussions in JIRA (will be mirrored
> > to issues ML), OK?
> >
> > @Theo: You are right. Adding this process won't be the silver bullet to
> fix
> > all PR related issues.
> > But I hope it will help to improve the overall situation.
> >
> > Are there any other comment? Otherwise I would start to prepare add this
> to
> > our website.
> >
> > Thanks, Fabian
> >
> > 2015-10-06 9:46 GMT+02:00 Theodore Vasiloudis <
> > theodoros.vasilou...@gmail.com>:
> >
> >> One problem that we are seeing with FlinkML PRs is that there are simply
> >> not enough commiters to "shepherd" all of them.
> >>
> >> While I think this process would help generally, I don't think it would
> >> solve this kind of problem.
> >>
> >> Regards,
> >> Theodore
> >>
> >> On Mon, Oct 5, 2015 at 3:28 PM, Matthias J. Sax 
> wrote:
> >>
> >>> One comment:
> >>> We should ensure that contributors follow discussions on the dev
> mailing
> >>> list. Otherwise, they might miss important discussions regarding their
> >>> PR (what happened already). Thus, the contributor was waiting for
> >>> feedback on the PR, while the reviewer(s) waited for the PR to be
> >>> updated according to the discussion consensus, resulting in unnecessary
> >>> delays.
> >>>
> >>> -Matthias
> >>>
> >>>
> >>>
> >>> On 10/05/2015 02:18 PM, Fabian Hueske wrote:
>  Hi everybody,
> 
>  Along with our efforts to improve the “How to contribute” guide, I
> >> would
>  like to start a discussion about a setting up a review process for
> pull
>  requests.
> 
>  Right now, I feel that our PR review efforts are often a bit
> >> unorganized.
>  This leads to situation such as:
> 
>  - PRs which are lingering around without review or feedback
>  - PRs which got a +1 for merging but which did not get merged
>  - PRs which have been rejected after a long time
>  - PRs which became irrelevant because some component was rewritten
>  - PRs which are lingering around and have been abandoned by their
>  contributors
> 
>  To address these issues I propose to define a pull request review
> >> process
>  as follows:
> 
>  1. [Get a Shepherd] Each pull request is taken care of by a shepherd.
>  Shepherds are committers that voluntarily sign up and *feel
> >> responsible*
>  for helping the PR through the process until it is merged (or
> >> discarded).
>  The shepherd is also the person to contact for the author of the PR. A
>  committer becomes the shepherd of a PR by dropping a comment on Github
> >>> like
>  “I would like to shepherd this PR”. A PR can be reassigned with lazy
>  consensus.
> 
>  2. [Accept Decision] The first decision for a PR should be whether it
> >> is
>  accepted or not. This depends on a) whether it is a desired feature or
>  improvement for Flink and b) whether the higher-level solution design
> >> is
>  appropriate. In many cases such as bug fixes or discussed features or
>  improvements, this should be an easy decision. In case of more a
> >> complex
>  feature, the discussion should have been started when the mandatory
> >> JIRA
>  was created. If it is still not clear whether the PR should be
> accepted
> >>> or
>  not, a discussion should be started in JIRA (a JIRA issue needs to be
>  created if none exists). The acceptance decision should be recorded by
> >> a
>  “+1 to accept” message in Github. If the PR is not accepted, it should
> >> be
>  closed in a timely manner.
> 
>  3. [Merge Decision] Once a PR has 

Re: Towards Flink 0.10

[Vasiliki Kalavri]

Yes, FLINK-2785 that's right!
Alright, thanks a lot!

On 5 October 2015 at 18:31, Fabian Hueske <fhue...@gmail.com> wrote:

> Hi Vasia,
>
> I guess you are referring to FLINK-2785. Should be fine, as there is a PR
> already.
> I'll add it to the list.
>
> Would be nice if you could take care of FLINK-2786 (remove Spargel).
>
> Cheers, Fabian
>
> 2015-10-05 18:25 GMT+02:00 Vasiliki Kalavri <vasilikikala...@gmail.com>:
>
> > Thank you Max for putting the list together and to whomever added
> > FLINK-2561 to the list :)
> > I would also add FLINK-2561 (pending PR #1205). It's a sub-task of
> > FLINK-2561, so maybe it's covered as is.
> >
> > If we go for Gelly graduation, I can take care of FLINK-2786 "Remove
> > Spargel from source code and update documentation in favor of Gelly", but
> > maybe it makes sense to wait for the restructuring, since we're getting
> rid
> > of staging altogether?
> >
> > -V.
> >
> > On 5 October 2015 at 17:51, Fabian Hueske <fhue...@gmail.com> wrote:
> >
> > > Thanks Max.
> > > I extended the list of issues to fix for the release.
> > >
> > > 2015-10-05 17:10 GMT+02:00 Maximilian Michels <m...@apache.org>:
> > >
> > > > Thanks Greg, we have added that to the list of API breaking changes.
> > > >
> > > > On Mon, Oct 5, 2015 at 4:36 PM, Greg Hogan <c...@greghogan.com>
> wrote:
> > > >
> > > > > Max,
> > > > >
> > > > > Stephan noted that FLINK-2723 is an API breaking change. The
> > > > CopyableValue
> > > > > interface has a new method "T copy()". Commit
> > > > > e727355e42bd0ad7d403aee703aaf33a68a839d2
> > > > >
> > > > > Greg
> > > > >
> > > > > On Mon, Oct 5, 2015 at 10:20 AM, Maximilian Michels <
> m...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Flinksters,
> > > > > >
> > > > > > After a lot of development effort in the past months, it is about
> > > time
> > > > > > to move towards the next major release. We decided to move
> towards
> > > > > > 0.10 instead of a milestone release. This release will probably
> be
> > > the
> > > > > > last release before 1.0.
> > > > > >
> > > > > > For 0.10 we most noticeably have the new Streaming API which
> comes
> > > > > > with an improved runtime including exactly-once sources and
> sinks.
> > > > > > Additionally, we have a new web interface with support for
> > > > > > live-monitoring. Not to mention the countless fixes and
> > improvements.
> > > > > >
> > > > > > I've been ransacking the JIRA issues to find out what issues we
> > have
> > > > > > to fix before we can release. I've put these issues on the 0.10
> > wiki
> > > > > > page:
> > https://cwiki.apache.org/confluence/display/FLINK/0.10+Release
> > > > > >
> > > > > > It would be great to fix all of these issues. However, I think we
> > > need
> > > > > > to be pragmatic and pick the most pressing ones. Let's do that on
> > the
> > > > > > wiki page on the "To Be Fixed" section.
> > > > > >
> > > > > > Cheers,
> > > > > > Max
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Towards Flink 0.10

[Vasiliki Kalavri]

Thank you Max for putting the list together and to whomever added
FLINK-2561 to the list :)
I would also add FLINK-2561 (pending PR #1205). It's a sub-task of
FLINK-2561, so maybe it's covered as is.

If we go for Gelly graduation, I can take care of FLINK-2786 "Remove
Spargel from source code and update documentation in favor of Gelly", but
maybe it makes sense to wait for the restructuring, since we're getting rid
of staging altogether?

-V.

On 5 October 2015 at 17:51, Fabian Hueske  wrote:

> Thanks Max.
> I extended the list of issues to fix for the release.
>
> 2015-10-05 17:10 GMT+02:00 Maximilian Michels :
>
> > Thanks Greg, we have added that to the list of API breaking changes.
> >
> > On Mon, Oct 5, 2015 at 4:36 PM, Greg Hogan  wrote:
> >
> > > Max,
> > >
> > > Stephan noted that FLINK-2723 is an API breaking change. The
> > CopyableValue
> > > interface has a new method "T copy()". Commit
> > > e727355e42bd0ad7d403aee703aaf33a68a839d2
> > >
> > > Greg
> > >
> > > On Mon, Oct 5, 2015 at 10:20 AM, Maximilian Michels 
> > > wrote:
> > >
> > > > Hi Flinksters,
> > > >
> > > > After a lot of development effort in the past months, it is about
> time
> > > > to move towards the next major release. We decided to move towards
> > > > 0.10 instead of a milestone release. This release will probably be
> the
> > > > last release before 1.0.
> > > >
> > > > For 0.10 we most noticeably have the new Streaming API which comes
> > > > with an improved runtime including exactly-once sources and sinks.
> > > > Additionally, we have a new web interface with support for
> > > > live-monitoring. Not to mention the countless fixes and improvements.
> > > >
> > > > I've been ransacking the JIRA issues to find out what issues we have
> > > > to fix before we can release. I've put these issues on the 0.10 wiki
> > > > page: https://cwiki.apache.org/confluence/display/FLINK/0.10+Release
> > > >
> > > > It would be great to fix all of these issues. However, I think we
> need
> > > > to be pragmatic and pick the most pressing ones. Let's do that on the
> > > > wiki page on the "To Be Fixed" section.
> > > >
> > > > Cheers,
> > > > Max
> > > >
> > >
> >
>

Re: Release Flink 0.10

[Vasiliki Kalavri]

+1 makes sense to me too.

On 29 September 2015 at 12:25, Aljoscha Krettek  wrote:

> +1, seems to be a very sane thing to do
>
> On Tue, 29 Sep 2015 at 12:20 Till Rohrmann  wrote:
>
> > +1 for 0.10
> >
> > On Tue, Sep 29, 2015 at 12:12 PM, Stephan Ewen  wrote:
> >
> > > +1 here as well
> > >
> > > On Tue, Sep 29, 2015 at 12:03 PM, Fabian Hueske 
> > wrote:
> > >
> > > > +1 for moving directly to 0.10.
> > > >
> > > > 2015-09-29 11:40 GMT+02:00 Maximilian Michels :
> > > >
> > > > > Hi Kostas,
> > > > >
> > > > > I think it makes sense to cancel the proposed 0.10-milestone
> release.
> > > > > We are not far away from completing all essential features of the
> > 0.10
> > > > > release. After we manage to complete those, we can test and release
> > > > > 0.10.
> > > > >
> > > > > The 0.10 release will be a major step towards the 1.0 release and,
> > > > > therefore, all new features of 0.10 should get enough exposure
> until
> > > > > we release 1.0.
> > > > >
> > > > > Cheers,
> > > > > Max
> > > > >
> > > > > On Tue, Sep 29, 2015 at 11:26 AM, Kostas Tzoumas <
> > ktzou...@apache.org>
> > > > > wrote:
> > > > > > Hi everyone,
> > > > > >
> > > > > > I would like to propose to cancel the 0.10-milestone release and
> go
> > > > > > directly for a 0.10 release as soon as possible.
> > > > > >
> > > > > > My opinion would be to focus this release on:
> > > > > > - Graduating the streaming API out of staging (depends on some
> open
> > > > pull
> > > > > > requests)
> > > > > > - Master high availability
> > > > > > - New monitoring framework
> > > > > > - Graduating Gelly out of staging
> > > > > >
> > > > > > Flink 1.0 will probably come after 0.10, which gives us time to
> fix
> > > > open
> > > > > > issues and freeze APIs.
> > > > > >
> > > > > > What do you think?
> > > > > >
> > > > > > Best,
> > > > > > Kostas
> > > > >
> > > >
> > >
> >
>

Re: Build get stuck at BarrierBufferMassiveRandomTest

[Vasiliki Kalavri]

@Stephan, have you pushed that fix for SocketClientSinkTest? Local builds
still hang for me :S

On 21 September 2015 at 22:55, Vasiliki Kalavri <vasilikikala...@gmail.com>
wrote:

> Yes, you're right. BarrierBufferMassiveRandomTest has actually finished
> :-)
> Sorry for the confusion! I'll wait for your fix then, thanks!
>
> On 21 September 2015 at 22:51, Stephan Ewen <se...@apache.org> wrote:
>
>> I am actually very happy that it is not the
>> "BarrierBufferMassiveRandomTest", that would be hell to debug...
>>
>> On Mon, Sep 21, 2015 at 10:51 PM, Stephan Ewen <se...@apache.org> wrote:
>>
>> > Ah, actually it is a different test. I think you got confused by the
>> > sysout log, because multiple parallel tests print there (that makes it
>> not
>> > always obvious which one hangs).
>> >
>> > The test is the "SocketClientSinkTest.testSocketSinkRetryAccess()" test.
>> > You can see that by looking in which test case the "main" thread is
>> stuck,
>> >
>> > This test is very unstable, but, fortunately, I made a fix 1h ago and it
>> > is being tested on Travis right now :-)
>> >
>> > Cheers,
>> > Stephan
>> >
>> >
>> >
>> > On Mon, Sep 21, 2015 at 10:23 PM, Vasiliki Kalavri <
>> > vasilikikala...@gmail.com> wrote:
>> >
>> >> Locally yes.
>> >>
>> >> Here's the stack trace:
>> >>
>> >>
>> >> 2015-09-21 22:22:46
>> >> Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.75-b04 mixed
>> mode):
>> >>
>> >> "Attach Listener" daemon prio=5 tid=0x7ff9d104e800 nid=0x4013
>> waiting
>> >> on condition [0x]
>> >>java.lang.Thread.State: RUNNABLE
>> >>
>> >> "Service Thread" daemon prio=5 tid=0x7ff9d3807000 nid=0x4c03
>> runnable
>> >> [0x]
>> >>java.lang.Thread.State: RUNNABLE
>> >>
>> >> "C2 CompilerThread1" daemon prio=5 tid=0x7ff9d2001000 nid=0x4a03
>> >> waiting on condition [0x]
>> >>java.lang.Thread.State: RUNNABLE
>> >>
>> >> "C2 CompilerThread0" daemon prio=5 tid=0x7ff9d201e000 nid=0x4803
>> >> waiting on condition [0x]
>> >>java.lang.Thread.State: RUNNABLE
>> >>
>> >> "Signal Dispatcher" daemon prio=5 tid=0x7ff9d3012800 nid=0x451b
>> >> runnable [0x]
>> >>java.lang.Thread.State: RUNNABLE
>> >>
>> >> "Finalizer" daemon prio=5 tid=0x7ff9d4005800 nid=0x3303 in
>> >> Object.wait() [0x00011430d000]
>> >>java.lang.Thread.State: WAITING (on object monitor)
>> >> at java.lang.Object.wait(Native Method)
>> >> - waiting on <0x0007ef504858> (a java.lang.ref.ReferenceQueue$Lock)
>> >> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
>> >> - locked <0x0007ef504858> (a java.lang.ref.ReferenceQueue$Lock)
>> >> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
>> >> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>> >>
>> >> "Reference Handler" daemon prio=5 tid=0x7ff9d480b000 nid=0x3103 in
>> >> Object.wait() [0x00011420a000]
>> >>java.lang.Thread.State: WAITING (on object monitor)
>> >> at java.lang.Object.wait(Native Method)
>> >> - waiting on <0x0007ef504470> (a java.lang.ref.Reference$Lock)
>> >> at java.lang.Object.wait(Object.java:503)
>> >> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
>> >> - locked <0x0007ef504470> (a java.lang.ref.Reference$Lock)
>> >>
>> >> "main" prio=5 tid=0x7ff9d480 nid=0xd03 runnable
>> >> [0x00010b764000]
>> >>java.lang.Thread.State: RUNNABLE
>> >> at java.net.PlainSocketImpl.socketAccept(Native Method)
>> >> at
>> >>
>> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
>> >> at java.net.ServerSocket.implAccept(ServerSocket.java:530)
>> >> at java.net.ServerSocket.accept(ServerSocket.java:498)
>> >> at
>> >>
>> >>
>> org.apache.flink.streaming.api.functions.sink.SocketClientSinkTest.testSocketSinkRetryAccess

Re: Build get stuck at BarrierBufferMassiveRandomTest

[Vasiliki Kalavri]

x7faebb002000 nid=0x2703
runnable

"GC task thread#5 (ParallelGC)" prio=5 tid=0x7faebb002800 nid=0x2903
runnable

"GC task thread#6 (ParallelGC)" prio=5 tid=0x7faebb003800 nid=0x2b03
runnable

"GC task thread#7 (ParallelGC)" prio=5 tid=0x7faeb9809000 nid=0x2d03
runnable

"VM Periodic Task Thread" prio=5 tid=0x7faeb980e000 nid=0x4f03 waiting
on condition

JNI global references: 195




On 23 September 2015 at 13:35, Stephan Ewen <se...@apache.org> wrote:

> I have pushed it, yes. If you rebase onto the latest master, it should
> work.
>
> If you can verify that it still hangs, can you post a stack trace dump?
>
> Thanks,
> Stephan
>
>
> On Wed, Sep 23, 2015 at 12:37 PM, Vasiliki Kalavri <
> vasilikikala...@gmail.com> wrote:
>
> > @Stephan, have you pushed that fix for SocketClientSinkTest? Local builds
> > still hang for me :S
> >
> > On 21 September 2015 at 22:55, Vasiliki Kalavri <
> vasilikikala...@gmail.com
> > >
> > wrote:
> >
> > > Yes, you're right. BarrierBufferMassiveRandomTest has actually finished
> > > :-)
> > > Sorry for the confusion! I'll wait for your fix then, thanks!
> > >
> > > On 21 September 2015 at 22:51, Stephan Ewen <se...@apache.org> wrote:
> > >
> > >> I am actually very happy that it is not the
> > >> "BarrierBufferMassiveRandomTest", that would be hell to debug...
> > >>
> > >> On Mon, Sep 21, 2015 at 10:51 PM, Stephan Ewen <se...@apache.org>
> > wrote:
> > >>
> > >> > Ah, actually it is a different test. I think you got confused by the
> > >> > sysout log, because multiple parallel tests print there (that makes
> it
> > >> not
> > >> > always obvious which one hangs).
> > >> >
> > >> > The test is the "SocketClientSinkTest.testSocketSinkRetryAccess()"
> > test.
> > >> > You can see that by looking in which test case the "main" thread is
> > >> stuck,
> > >> >
> > >> > This test is very unstable, but, fortunately, I made a fix 1h ago
> and
> > it
> > >> > is being tested on Travis right now :-)
> > >> >
> > >> > Cheers,
> > >> > Stephan
> > >> >
> > >> >
> > >> >
> > >> > On Mon, Sep 21, 2015 at 10:23 PM, Vasiliki Kalavri <
> > >> > vasilikikala...@gmail.com> wrote:
> > >> >
> > >> >> Locally yes.
> > >> >>
> > >> >> Here's the stack trace:
> > >> >>
> > >> >>
> > >> >> 2015-09-21 22:22:46
> > >> >> Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.75-b04 mixed
> > >> mode):
> > >> >>
> > >> >> "Attach Listener" daemon prio=5 tid=0x7ff9d104e800 nid=0x4013
> > >> waiting
> > >> >> on condition [0x]
> > >> >>java.lang.Thread.State: RUNNABLE
> > >> >>
> > >> >> "Service Thread" daemon prio=5 tid=0x7ff9d3807000 nid=0x4c03
> > >> runnable
> > >> >> [0x]
> > >> >>java.lang.Thread.State: RUNNABLE
> > >> >>
> > >> >> "C2 CompilerThread1" daemon prio=5 tid=0x7ff9d2001000
> nid=0x4a03
> > >> >> waiting on condition [0x]
> > >> >>java.lang.Thread.State: RUNNABLE
> > >> >>
> > >> >> "C2 CompilerThread0" daemon prio=5 tid=0x7ff9d201e000
> nid=0x4803
> > >> >> waiting on condition [0x]
> > >> >>java.lang.Thread.State: RUNNABLE
> > >> >>
> > >> >> "Signal Dispatcher" daemon prio=5 tid=0x7ff9d3012800 nid=0x451b
> > >> >> runnable [0x]
> > >> >>java.lang.Thread.State: RUNNABLE
> > >> >>
> > >> >> "Finalizer" daemon prio=5 tid=0x7ff9d4005800 nid=0x3303 in
> > >> >> Object.wait() [0x00011430d000]
> > >> >>java.lang.Thread.State: WAITING (on object monitor)
> > >> >> at java.lang.Object.wait(Native Method)
> > >> >> - waiting on <0x0007ef504858> (a
> > java.lang.ref.ReferenceQueue$Lock)
> > >> >> at java.lang.ref.Refe

Re: Extending and improving our "How to contribute" page

[Vasiliki Kalavri]

Hi,

I agree with you Fabian. Clarifying these issues in the "How to Contribute"
guide will save lots of time both to reviewers and contributors. It is a
really disappointing situation when someone spends time implementing
something and their PR ends up being rejected because either the feature
was not needed or the implementation details were never agreed on.

That said, I think we should also make sure that we don't raise the bar too
high for simple contributions.

Regarding (1) and (2), I think we should clarify what kind of
additions/changes require this process to be followed. e.g. do we need to
discuss additions for which JIRAs already exist? Ideas described in the
roadmaps? Adding a new algorithm to Gelly/Flink-ML?

Regarding (3), maybe we can all suggest some examples/patterns that we've
seen when reviewing PRs and then choose the most common (or all).

(4) sounds good to me.

Cheers,
Vasia.

On 23 September 2015 at 15:08, Kostas Tzoumas  wrote:

> Big +1.
>
> For (1), a discussion in JIRA would also be an option IMO
>
> For (2), let us come up with few examples on what constitutes a feature
> that needs a design doc, and what should be in the doc (IMO
> architecture/general approach, components touched, interfaces changed)
>
>
>
> On Wed, Sep 23, 2015 at 2:24 PM, Fabian Hueske  wrote:
>
> > Hi everybody,
> >
> > I guess we all have noticed that the Flink community is quickly growing
> and
> > more and more contributions are coming in. Recently, a few contributions
> > proposed new features without being discussed on the mailing list. Some
> of
> > these contributions were not accepted in the end. In other cases, pull
> > requests had to be heavily reworked because the approach taken was not
> the
> > best one. These are situations which should be avoided because both the
> > contributor as well as the person who reviewed the contribution invested
> a
> > lot of time for nothing.
> >
> > I had a look at our “How to contribute” and “Coding guideline” pages and
> > think, we can improve them. I see basically two issues:
> >
> >   1. The documents do not explain how to propose and discuss new features
> > and improvements.
> >   2. The documents are quite technical and the structure could be
> improved,
> > IMO.
> >
> > I would like to improve these pages and propose the following additions:
> >
> >   1. Request contributors and committers to start discussions on the
> > mailing list for new features. This discussion should help to figure out
> > whether such a new feature is a good fit for Flink and give first
> pointers
> > for a design to implement it.
> >   2. Require contributors and committers to write design documents for
> all
> > new features and major improvements. These documents should be attached
> to
> > a JIRA issue and follow a template which needs to be defined.
> >   3. Extend the “Coding Style Guides” and add patterns that are commonly
> > remarked in pull requests.
> >   4. Restructure the current pages into three pages: a general guide for
> > contributions and two guides for how to contribute to code and website
> with
> > all technical issues (repository, IDE setup, build system, etc.)
> >
> > Looking forward for your comments,
> > Fabian
> >
>