Big +1 for graduating streaming out of staging. It is widely used, also in production and we are spending a lot of effort into hardening it. I also agree with the proposed new maven module structure.
We have to carefully test the reworked structure for the scripts which are generating the hadoop1 and the scala 2.11 poms (they are transformed using a bunch of bash scripts). I can do that once the PR is open. @Chesnay: I would be fine with including the language binding into python > Where would new projects reside in, that previously would have been put into flink-staging? flink-contrib @Kostas: I understand the idea behind your suggested renaming, but thats just a name. I don't think its going to influence how people are seeing Flink: It doesn't feel like second class when adding "flink-streaming-core" to the dependencies to me. Also, the "flink-datastream-scala" module would depend on "flink-dataset-scala", which is kind of weird. I'm wondering whether we should remove the "flink-test-utils" module. I don't think its really necessary, because we can put the test jars into the flink-tests project and include them using the "test-jar" dependency. On Thu, Oct 1, 2015 at 2:27 PM, Kostas Tzoumas <ktzou...@apache.org> wrote: > +1 > > I wanted to suggest that we rename modules to fully accept streaming as > first class, qualifying also "batch" as "batch" (e.g., flink-java --> > flink-dataset-java, flink-streaming --> flink-datastream, etc). > > This would break maven dependencies (temporary hell :-) so it's not a > decision to take lightly. I'm not strongly advocating for it. > > > On Thu, Oct 1, 2015 at 12:44 PM, Chesnay Schepler <ches...@apache.org> > wrote: > > > I like it in general. But while we're at it, what is the purpose of the > > flink-tests project, or rather which tests belong there instead of the > > individual projects? > > > > Where would new projects reside in, that previously would have been put > > into flink-staging? > > > > Lastly, I'd like to merge flink-language-binding into flink-python. I can > > go more into detail but the gist of it is that the abstraction just > doesn't > > work. > > > > > > On 01.10.2015 12:40, Márton Balassi wrote: > > > >> Great to see streaming graduating. :) > >> > >> I like the outline, both getting rid of staging, having the examples > >> together and generally flattening the structure are very reasonable to > me. > >> > >> You have listed flink-streaming-examples under > flink-streaming-connectors > >> and left out some less prominent maven modules, but I assume the first > is > >> accidental while the second is intentional to make the list a bit > briefer. > >> > >> Best, > >> > >> Marton > >> > >> > >> On Thu, Oct 1, 2015 at 12:25 PM, Stephan Ewen <se...@apache.org> wrote: > >> > >> Hi all! > >>> > >>> We are making good headway with reworking the last parts of the Window > >>> API. > >>> After that, the streaming API should be good to be pulled out of > staging. > >>> > >>> Since we are reorganizing the projects as part of that, I would shift a > >>> bit > >>> more to bring things a bit more up to date. > >>> > >>> In this restructure, I would like to get rid of the "flink-staging" > >>> project. Anyone who only uses the maven artifacts sees no difference > >>> whether a project is in "staging" or not, so it does not help much to > >>> have > >>> that directory structure. > >>> On the other hand, projects have a tendency to linger in staging > forever > >>> (like avro, spargel, hbase, jdbc, ...) > >>> > >>> The new structure could be > >>> > >>> flink-core > >>> flink-java > >>> flink-scala > >>> flink-streaming-core > >>> flink-streaming-scala > >>> > >>> flink-runtime > >>> flink-runtime-web > >>> flink-optimizer > >>> flink-clients > >>> > >>> flink-shaded > >>> -> flink-shaded-hadoop > >>> -> flink-shaded-hadoop2 > >>> -> flink-shaded-include-yarn-tests > >>> -> flink-shaded-curator > >>> > >>> flink-examples > >>> -> (have all examples, Scala and Java, Batch and Streaming) > >>> > >>> flink-batch-connectors > >>> -> flink-avro > >>> -> flink-jdbc > >>> -> flink-hadoop-compatibility > >>> -> flink-hbase > >>> -> flink-hcatalog > >>> > >>> flink-streaming-connectors > >>> -> flink-connector-twitter > >>> -> flink-streaming-examples > >>> -> flink-connector-flume > >>> -> flink-connector-kafka > >>> -> flink-connector-elasticsearch > >>> -> flink-connector-rabbitmq > >>> -> flink-connector-filesystem > >>> > >>> flink-libraries > >>> -> flink-gelly > >>> -> flink-gelly-scala > >>> -> flink-ml > >>> -> flink-table > >>> -> flink-language-binding > >>> -> flink-python > >>> > >>> > >>> flink-scala-shell > >>> > >>> flink-test-utils > >>> flink-tests > >>> flink-fs-tests > >>> > >>> flink-contrib > >>> -> flink-storm-compatibility > >>> -> flink-storm-compatibility-examples > >>> -> flink-streaming-utils > >>> -> flink-tweet-inputformat > >>> -> flink-operator-stats > >>> -> flink-tez > >>> > >>> flink-quickstart > >>> -> flink-quickstart-java > >>> -> flink-quickstart-scala > >>> -> flink-tez-quickstart > >>> > >>> flink-yarn > >>> flink-yarn-tests > >>> > >>> flink-dist > >>> > >>> flink-benchmark > >>> > >>> > >>> Let me know if that makes sense! > >>> > >>> Greetings, > >>> Stephan > >>> > >>> > > >
