+1 to Robert and practicality :-) As I said before, I do not feel strongly about this, I was torn myself.
On Thu, Oct 1, 2015 at 3:28 PM, Chesnay Schepler <ches...@apache.org> wrote: > If we remove flink-staging because projects tend to get stuck there, what > mechanism prevents the same happening with flink-contrib? > > > On 01.10.2015 15:19, Stephan Ewen wrote: > >> +1 for Robert's comments. >> >> On Thu, Oct 1, 2015 at 3:16 PM, Robert Metzger <rmetz...@apache.org> >> wrote: >> >> Big +1 for graduating streaming out of staging. It is widely used, also in >>> production and we are spending a lot of effort into hardening it. >>> I also agree with the proposed new maven module structure. >>> >>> We have to carefully test the reworked structure for the scripts which >>> are >>> generating the hadoop1 and the scala 2.11 poms (they are transformed >>> using >>> a bunch of bash scripts). I can do that once the PR is open. >>> >>> @Chesnay: I would be fine with including the language binding into python >>> >>>> Where would new projects reside in, that previously would have been put >>>> >>> into flink-staging? >>> >>> flink-contrib >>> >>> >>> @Kostas: I understand the idea behind your suggested renaming, but thats >>> just a name. I don't think its going to influence how people are seeing >>> Flink: It doesn't feel like second class when adding >>> "flink-streaming-core" >>> to the dependencies to me. >>> Also, the "flink-datastream-scala" module would depend on >>> "flink-dataset-scala", which is kind of weird. >>> >>> >>> I'm wondering whether we should remove the "flink-test-utils" module. I >>> don't think its really necessary, because we can put the test jars into >>> the >>> flink-tests project and include them using the "test-jar" dependency. >>> >>> >>> On Thu, Oct 1, 2015 at 2:27 PM, Kostas Tzoumas <ktzou...@apache.org> >>> wrote: >>> >>> +1 >>>> >>>> I wanted to suggest that we rename modules to fully accept streaming as >>>> first class, qualifying also "batch" as "batch" (e.g., flink-java --> >>>> flink-dataset-java, flink-streaming --> flink-datastream, etc). >>>> >>>> This would break maven dependencies (temporary hell :-) so it's not a >>>> decision to take lightly. I'm not strongly advocating for it. >>>> >>>> >>>> On Thu, Oct 1, 2015 at 12:44 PM, Chesnay Schepler <ches...@apache.org> >>>> wrote: >>>> >>>> I like it in general. But while we're at it, what is the purpose of the >>>>> flink-tests project, or rather which tests belong there instead of the >>>>> individual projects? >>>>> >>>>> Where would new projects reside in, that previously would have been put >>>>> into flink-staging? >>>>> >>>>> Lastly, I'd like to merge flink-language-binding into flink-python. I >>>>> >>>> can >>> >>>> go more into detail but the gist of it is that the abstraction just >>>>> >>>> doesn't >>>> >>>>> work. >>>>> >>>>> >>>>> On 01.10.2015 12:40, Márton Balassi wrote: >>>>> >>>>> Great to see streaming graduating. :) >>>>>> >>>>>> I like the outline, both getting rid of staging, having the examples >>>>>> together and generally flattening the structure are very reasonable to >>>>>> >>>>> me. >>>> >>>>> You have listed flink-streaming-examples under >>>>>> >>>>> flink-streaming-connectors >>>> >>>>> and left out some less prominent maven modules, but I assume the first >>>>>> >>>>> is >>>> >>>>> accidental while the second is intentional to make the list a bit >>>>>> >>>>> briefer. >>>> >>>>> Best, >>>>>> >>>>>> Marton >>>>>> >>>>>> >>>>>> On Thu, Oct 1, 2015 at 12:25 PM, Stephan Ewen <se...@apache.org> >>>>>> >>>>> wrote: >>> >>>> Hi all! >>>>>> >>>>>>> We are making good headway with reworking the last parts of the >>>>>>> >>>>>> Window >>> >>>> API. >>>>>>> After that, the streaming API should be good to be pulled out of >>>>>>> >>>>>> staging. >>>> >>>>> Since we are reorganizing the projects as part of that, I would >>>>>>> >>>>>> shift a >>> >>>> bit >>>>>>> more to bring things a bit more up to date. >>>>>>> >>>>>>> In this restructure, I would like to get rid of the "flink-staging" >>>>>>> project. Anyone who only uses the maven artifacts sees no difference >>>>>>> whether a project is in "staging" or not, so it does not help much to >>>>>>> have >>>>>>> that directory structure. >>>>>>> On the other hand, projects have a tendency to linger in staging >>>>>>> >>>>>> forever >>>> >>>>> (like avro, spargel, hbase, jdbc, ...) >>>>>>> >>>>>>> The new structure could be >>>>>>> >>>>>>> flink-core >>>>>>> flink-java >>>>>>> flink-scala >>>>>>> flink-streaming-core >>>>>>> flink-streaming-scala >>>>>>> >>>>>>> flink-runtime >>>>>>> flink-runtime-web >>>>>>> flink-optimizer >>>>>>> flink-clients >>>>>>> >>>>>>> flink-shaded >>>>>>> -> flink-shaded-hadoop >>>>>>> -> flink-shaded-hadoop2 >>>>>>> -> flink-shaded-include-yarn-tests >>>>>>> -> flink-shaded-curator >>>>>>> >>>>>>> flink-examples >>>>>>> -> (have all examples, Scala and Java, Batch and Streaming) >>>>>>> >>>>>>> flink-batch-connectors >>>>>>> -> flink-avro >>>>>>> -> flink-jdbc >>>>>>> -> flink-hadoop-compatibility >>>>>>> -> flink-hbase >>>>>>> -> flink-hcatalog >>>>>>> >>>>>>> flink-streaming-connectors >>>>>>> -> flink-connector-twitter >>>>>>> -> flink-streaming-examples >>>>>>> -> flink-connector-flume >>>>>>> -> flink-connector-kafka >>>>>>> -> flink-connector-elasticsearch >>>>>>> -> flink-connector-rabbitmq >>>>>>> -> flink-connector-filesystem >>>>>>> >>>>>>> flink-libraries >>>>>>> -> flink-gelly >>>>>>> -> flink-gelly-scala >>>>>>> -> flink-ml >>>>>>> -> flink-table >>>>>>> -> flink-language-binding >>>>>>> -> flink-python >>>>>>> >>>>>>> >>>>>>> flink-scala-shell >>>>>>> >>>>>>> flink-test-utils >>>>>>> flink-tests >>>>>>> flink-fs-tests >>>>>>> >>>>>>> flink-contrib >>>>>>> -> flink-storm-compatibility >>>>>>> -> flink-storm-compatibility-examples >>>>>>> -> flink-streaming-utils >>>>>>> -> flink-tweet-inputformat >>>>>>> -> flink-operator-stats >>>>>>> -> flink-tez >>>>>>> >>>>>>> flink-quickstart >>>>>>> -> flink-quickstart-java >>>>>>> -> flink-quickstart-scala >>>>>>> -> flink-tez-quickstart >>>>>>> >>>>>>> flink-yarn >>>>>>> flink-yarn-tests >>>>>>> >>>>>>> flink-dist >>>>>>> >>>>>>> flink-benchmark >>>>>>> >>>>>>> >>>>>>> Let me know if that makes sense! >>>>>>> >>>>>>> Greetings, >>>>>>> Stephan >>>>>>> >>>>>>> >>>>>>> >
