+1 for Robert's comments. On Thu, Oct 1, 2015 at 3:16 PM, Robert Metzger <rmetz...@apache.org> wrote:
> Big +1 for graduating streaming out of staging. It is widely used, also in > production and we are spending a lot of effort into hardening it. > I also agree with the proposed new maven module structure. > > We have to carefully test the reworked structure for the scripts which are > generating the hadoop1 and the scala 2.11 poms (they are transformed using > a bunch of bash scripts). I can do that once the PR is open. > > @Chesnay: I would be fine with including the language binding into python > > Where would new projects reside in, that previously would have been put > into flink-staging? > > flink-contrib > > > @Kostas: I understand the idea behind your suggested renaming, but thats > just a name. I don't think its going to influence how people are seeing > Flink: It doesn't feel like second class when adding "flink-streaming-core" > to the dependencies to me. > Also, the "flink-datastream-scala" module would depend on > "flink-dataset-scala", which is kind of weird. > > > I'm wondering whether we should remove the "flink-test-utils" module. I > don't think its really necessary, because we can put the test jars into the > flink-tests project and include them using the "test-jar" dependency. > > > On Thu, Oct 1, 2015 at 2:27 PM, Kostas Tzoumas <ktzou...@apache.org> > wrote: > > > +1 > > > > I wanted to suggest that we rename modules to fully accept streaming as > > first class, qualifying also "batch" as "batch" (e.g., flink-java --> > > flink-dataset-java, flink-streaming --> flink-datastream, etc). > > > > This would break maven dependencies (temporary hell :-) so it's not a > > decision to take lightly. I'm not strongly advocating for it. > > > > > > On Thu, Oct 1, 2015 at 12:44 PM, Chesnay Schepler <ches...@apache.org> > > wrote: > > > > > I like it in general. But while we're at it, what is the purpose of the > > > flink-tests project, or rather which tests belong there instead of the > > > individual projects? > > > > > > Where would new projects reside in, that previously would have been put > > > into flink-staging? > > > > > > Lastly, I'd like to merge flink-language-binding into flink-python. I > can > > > go more into detail but the gist of it is that the abstraction just > > doesn't > > > work. > > > > > > > > > On 01.10.2015 12:40, Márton Balassi wrote: > > > > > >> Great to see streaming graduating. :) > > >> > > >> I like the outline, both getting rid of staging, having the examples > > >> together and generally flattening the structure are very reasonable to > > me. > > >> > > >> You have listed flink-streaming-examples under > > flink-streaming-connectors > > >> and left out some less prominent maven modules, but I assume the first > > is > > >> accidental while the second is intentional to make the list a bit > > briefer. > > >> > > >> Best, > > >> > > >> Marton > > >> > > >> > > >> On Thu, Oct 1, 2015 at 12:25 PM, Stephan Ewen <se...@apache.org> > wrote: > > >> > > >> Hi all! > > >>> > > >>> We are making good headway with reworking the last parts of the > Window > > >>> API. > > >>> After that, the streaming API should be good to be pulled out of > > staging. > > >>> > > >>> Since we are reorganizing the projects as part of that, I would > shift a > > >>> bit > > >>> more to bring things a bit more up to date. > > >>> > > >>> In this restructure, I would like to get rid of the "flink-staging" > > >>> project. Anyone who only uses the maven artifacts sees no difference > > >>> whether a project is in "staging" or not, so it does not help much to > > >>> have > > >>> that directory structure. > > >>> On the other hand, projects have a tendency to linger in staging > > forever > > >>> (like avro, spargel, hbase, jdbc, ...) > > >>> > > >>> The new structure could be > > >>> > > >>> flink-core > > >>> flink-java > > >>> flink-scala > > >>> flink-streaming-core > > >>> flink-streaming-scala > > >>> > > >>> flink-runtime > > >>> flink-runtime-web > > >>> flink-optimizer > > >>> flink-clients > > >>> > > >>> flink-shaded > > >>> -> flink-shaded-hadoop > > >>> -> flink-shaded-hadoop2 > > >>> -> flink-shaded-include-yarn-tests > > >>> -> flink-shaded-curator > > >>> > > >>> flink-examples > > >>> -> (have all examples, Scala and Java, Batch and Streaming) > > >>> > > >>> flink-batch-connectors > > >>> -> flink-avro > > >>> -> flink-jdbc > > >>> -> flink-hadoop-compatibility > > >>> -> flink-hbase > > >>> -> flink-hcatalog > > >>> > > >>> flink-streaming-connectors > > >>> -> flink-connector-twitter > > >>> -> flink-streaming-examples > > >>> -> flink-connector-flume > > >>> -> flink-connector-kafka > > >>> -> flink-connector-elasticsearch > > >>> -> flink-connector-rabbitmq > > >>> -> flink-connector-filesystem > > >>> > > >>> flink-libraries > > >>> -> flink-gelly > > >>> -> flink-gelly-scala > > >>> -> flink-ml > > >>> -> flink-table > > >>> -> flink-language-binding > > >>> -> flink-python > > >>> > > >>> > > >>> flink-scala-shell > > >>> > > >>> flink-test-utils > > >>> flink-tests > > >>> flink-fs-tests > > >>> > > >>> flink-contrib > > >>> -> flink-storm-compatibility > > >>> -> flink-storm-compatibility-examples > > >>> -> flink-streaming-utils > > >>> -> flink-tweet-inputformat > > >>> -> flink-operator-stats > > >>> -> flink-tez > > >>> > > >>> flink-quickstart > > >>> -> flink-quickstart-java > > >>> -> flink-quickstart-scala > > >>> -> flink-tez-quickstart > > >>> > > >>> flink-yarn > > >>> flink-yarn-tests > > >>> > > >>> flink-dist > > >>> > > >>> flink-benchmark > > >>> > > >>> > > >>> Let me know if that makes sense! > > >>> > > >>> Greetings, > > >>> Stephan > > >>> > > >>> > > > > > >
