@Chesnay: Nothing prevents projects from getting stuck there. But at least we don't have two staging repositories (dist, staging). Also I would not say that there has been no graduation out of staging. Yarn was also there once, streaming and gelly are currently leaving it. So I would actually say it worked.
One more thing: Can we actually put the web submission client into a separate module until we have something new and better? It does really not fit into "flink-clients". It makes Flink users pull the jetty dependencies even if they never use them for example when running Flink from an IDE. On Thu, Oct 1, 2015 at 4:45 PM, Kostas Tzoumas <ktzou...@apache.org> wrote: > +1 to Robert and practicality :-) > > As I said before, I do not feel strongly about this, I was torn myself. > > On Thu, Oct 1, 2015 at 3:28 PM, Chesnay Schepler <ches...@apache.org> > wrote: > > > If we remove flink-staging because projects tend to get stuck there, what > > mechanism prevents the same happening with flink-contrib? > > > > > > On 01.10.2015 15:19, Stephan Ewen wrote: > > > >> +1 for Robert's comments. > >> > >> On Thu, Oct 1, 2015 at 3:16 PM, Robert Metzger <rmetz...@apache.org> > >> wrote: > >> > >> Big +1 for graduating streaming out of staging. It is widely used, also > in > >>> production and we are spending a lot of effort into hardening it. > >>> I also agree with the proposed new maven module structure. > >>> > >>> We have to carefully test the reworked structure for the scripts which > >>> are > >>> generating the hadoop1 and the scala 2.11 poms (they are transformed > >>> using > >>> a bunch of bash scripts). I can do that once the PR is open. > >>> > >>> @Chesnay: I would be fine with including the language binding into > python > >>> > >>>> Where would new projects reside in, that previously would have been > put > >>>> > >>> into flink-staging? > >>> > >>> flink-contrib > >>> > >>> > >>> @Kostas: I understand the idea behind your suggested renaming, but > thats > >>> just a name. I don't think its going to influence how people are seeing > >>> Flink: It doesn't feel like second class when adding > >>> "flink-streaming-core" > >>> to the dependencies to me. > >>> Also, the "flink-datastream-scala" module would depend on > >>> "flink-dataset-scala", which is kind of weird. > >>> > >>> > >>> I'm wondering whether we should remove the "flink-test-utils" module. I > >>> don't think its really necessary, because we can put the test jars into > >>> the > >>> flink-tests project and include them using the "test-jar" dependency. > >>> > >>> > >>> On Thu, Oct 1, 2015 at 2:27 PM, Kostas Tzoumas <ktzou...@apache.org> > >>> wrote: > >>> > >>> +1 > >>>> > >>>> I wanted to suggest that we rename modules to fully accept streaming > as > >>>> first class, qualifying also "batch" as "batch" (e.g., flink-java --> > >>>> flink-dataset-java, flink-streaming --> flink-datastream, etc). > >>>> > >>>> This would break maven dependencies (temporary hell :-) so it's not a > >>>> decision to take lightly. I'm not strongly advocating for it. > >>>> > >>>> > >>>> On Thu, Oct 1, 2015 at 12:44 PM, Chesnay Schepler <ches...@apache.org > > > >>>> wrote: > >>>> > >>>> I like it in general. But while we're at it, what is the purpose of > the > >>>>> flink-tests project, or rather which tests belong there instead of > the > >>>>> individual projects? > >>>>> > >>>>> Where would new projects reside in, that previously would have been > put > >>>>> into flink-staging? > >>>>> > >>>>> Lastly, I'd like to merge flink-language-binding into flink-python. I > >>>>> > >>>> can > >>> > >>>> go more into detail but the gist of it is that the abstraction just > >>>>> > >>>> doesn't > >>>> > >>>>> work. > >>>>> > >>>>> > >>>>> On 01.10.2015 12:40, Márton Balassi wrote: > >>>>> > >>>>> Great to see streaming graduating. :) > >>>>>> > >>>>>> I like the outline, both getting rid of staging, having the examples > >>>>>> together and generally flattening the structure are very reasonable > to > >>>>>> > >>>>> me. > >>>> > >>>>> You have listed flink-streaming-examples under > >>>>>> > >>>>> flink-streaming-connectors > >>>> > >>>>> and left out some less prominent maven modules, but I assume the > first > >>>>>> > >>>>> is > >>>> > >>>>> accidental while the second is intentional to make the list a bit > >>>>>> > >>>>> briefer. > >>>> > >>>>> Best, > >>>>>> > >>>>>> Marton > >>>>>> > >>>>>> > >>>>>> On Thu, Oct 1, 2015 at 12:25 PM, Stephan Ewen <se...@apache.org> > >>>>>> > >>>>> wrote: > >>> > >>>> Hi all! > >>>>>> > >>>>>>> We are making good headway with reworking the last parts of the > >>>>>>> > >>>>>> Window > >>> > >>>> API. > >>>>>>> After that, the streaming API should be good to be pulled out of > >>>>>>> > >>>>>> staging. > >>>> > >>>>> Since we are reorganizing the projects as part of that, I would > >>>>>>> > >>>>>> shift a > >>> > >>>> bit > >>>>>>> more to bring things a bit more up to date. > >>>>>>> > >>>>>>> In this restructure, I would like to get rid of the "flink-staging" > >>>>>>> project. Anyone who only uses the maven artifacts sees no > difference > >>>>>>> whether a project is in "staging" or not, so it does not help much > to > >>>>>>> have > >>>>>>> that directory structure. > >>>>>>> On the other hand, projects have a tendency to linger in staging > >>>>>>> > >>>>>> forever > >>>> > >>>>> (like avro, spargel, hbase, jdbc, ...) > >>>>>>> > >>>>>>> The new structure could be > >>>>>>> > >>>>>>> flink-core > >>>>>>> flink-java > >>>>>>> flink-scala > >>>>>>> flink-streaming-core > >>>>>>> flink-streaming-scala > >>>>>>> > >>>>>>> flink-runtime > >>>>>>> flink-runtime-web > >>>>>>> flink-optimizer > >>>>>>> flink-clients > >>>>>>> > >>>>>>> flink-shaded > >>>>>>> -> flink-shaded-hadoop > >>>>>>> -> flink-shaded-hadoop2 > >>>>>>> -> flink-shaded-include-yarn-tests > >>>>>>> -> flink-shaded-curator > >>>>>>> > >>>>>>> flink-examples > >>>>>>> -> (have all examples, Scala and Java, Batch and Streaming) > >>>>>>> > >>>>>>> flink-batch-connectors > >>>>>>> -> flink-avro > >>>>>>> -> flink-jdbc > >>>>>>> -> flink-hadoop-compatibility > >>>>>>> -> flink-hbase > >>>>>>> -> flink-hcatalog > >>>>>>> > >>>>>>> flink-streaming-connectors > >>>>>>> -> flink-connector-twitter > >>>>>>> -> flink-streaming-examples > >>>>>>> -> flink-connector-flume > >>>>>>> -> flink-connector-kafka > >>>>>>> -> flink-connector-elasticsearch > >>>>>>> -> flink-connector-rabbitmq > >>>>>>> -> flink-connector-filesystem > >>>>>>> > >>>>>>> flink-libraries > >>>>>>> -> flink-gelly > >>>>>>> -> flink-gelly-scala > >>>>>>> -> flink-ml > >>>>>>> -> flink-table > >>>>>>> -> flink-language-binding > >>>>>>> -> flink-python > >>>>>>> > >>>>>>> > >>>>>>> flink-scala-shell > >>>>>>> > >>>>>>> flink-test-utils > >>>>>>> flink-tests > >>>>>>> flink-fs-tests > >>>>>>> > >>>>>>> flink-contrib > >>>>>>> -> flink-storm-compatibility > >>>>>>> -> flink-storm-compatibility-examples > >>>>>>> -> flink-streaming-utils > >>>>>>> -> flink-tweet-inputformat > >>>>>>> -> flink-operator-stats > >>>>>>> -> flink-tez > >>>>>>> > >>>>>>> flink-quickstart > >>>>>>> -> flink-quickstart-java > >>>>>>> -> flink-quickstart-scala > >>>>>>> -> flink-tez-quickstart > >>>>>>> > >>>>>>> flink-yarn > >>>>>>> flink-yarn-tests > >>>>>>> > >>>>>>> flink-dist > >>>>>>> > >>>>>>> flink-benchmark > >>>>>>> > >>>>>>> > >>>>>>> Let me know if that makes sense! > >>>>>>> > >>>>>>> Greetings, > >>>>>>> Stephan > >>>>>>> > >>>>>>> > >>>>>>> > > >
