If we remove flink-staging because projects tend to get stuck there,
what mechanism prevents the same happening with flink-contrib?
On 01.10.2015 15:19, Stephan Ewen wrote:
+1 for Robert's comments.
On Thu, Oct 1, 2015 at 3:16 PM, Robert Metzger <rmetz...@apache.org> wrote:
Big +1 for graduating streaming out of staging. It is widely used, also in
production and we are spending a lot of effort into hardening it.
I also agree with the proposed new maven module structure.
We have to carefully test the reworked structure for the scripts which are
generating the hadoop1 and the scala 2.11 poms (they are transformed using
a bunch of bash scripts). I can do that once the PR is open.
@Chesnay: I would be fine with including the language binding into python
Where would new projects reside in, that previously would have been put
into flink-staging?
flink-contrib
@Kostas: I understand the idea behind your suggested renaming, but thats
just a name. I don't think its going to influence how people are seeing
Flink: It doesn't feel like second class when adding "flink-streaming-core"
to the dependencies to me.
Also, the "flink-datastream-scala" module would depend on
"flink-dataset-scala", which is kind of weird.
I'm wondering whether we should remove the "flink-test-utils" module. I
don't think its really necessary, because we can put the test jars into the
flink-tests project and include them using the "test-jar" dependency.
On Thu, Oct 1, 2015 at 2:27 PM, Kostas Tzoumas <ktzou...@apache.org>
wrote:
+1
I wanted to suggest that we rename modules to fully accept streaming as
first class, qualifying also "batch" as "batch" (e.g., flink-java -->
flink-dataset-java, flink-streaming --> flink-datastream, etc).
This would break maven dependencies (temporary hell :-) so it's not a
decision to take lightly. I'm not strongly advocating for it.
On Thu, Oct 1, 2015 at 12:44 PM, Chesnay Schepler <ches...@apache.org>
wrote:
I like it in general. But while we're at it, what is the purpose of the
flink-tests project, or rather which tests belong there instead of the
individual projects?
Where would new projects reside in, that previously would have been put
into flink-staging?
Lastly, I'd like to merge flink-language-binding into flink-python. I
can
go more into detail but the gist of it is that the abstraction just
doesn't
work.
On 01.10.2015 12:40, Márton Balassi wrote:
Great to see streaming graduating. :)
I like the outline, both getting rid of staging, having the examples
together and generally flattening the structure are very reasonable to
me.
You have listed flink-streaming-examples under
flink-streaming-connectors
and left out some less prominent maven modules, but I assume the first
is
accidental while the second is intentional to make the list a bit
briefer.
Best,
Marton
On Thu, Oct 1, 2015 at 12:25 PM, Stephan Ewen <se...@apache.org>
wrote:
Hi all!
We are making good headway with reworking the last parts of the
Window
API.
After that, the streaming API should be good to be pulled out of
staging.
Since we are reorganizing the projects as part of that, I would
shift a
bit
more to bring things a bit more up to date.
In this restructure, I would like to get rid of the "flink-staging"
project. Anyone who only uses the maven artifacts sees no difference
whether a project is in "staging" or not, so it does not help much to
have
that directory structure.
On the other hand, projects have a tendency to linger in staging
forever
(like avro, spargel, hbase, jdbc, ...)
The new structure could be
flink-core
flink-java
flink-scala
flink-streaming-core
flink-streaming-scala
flink-runtime
flink-runtime-web
flink-optimizer
flink-clients
flink-shaded
-> flink-shaded-hadoop
-> flink-shaded-hadoop2
-> flink-shaded-include-yarn-tests
-> flink-shaded-curator
flink-examples
-> (have all examples, Scala and Java, Batch and Streaming)
flink-batch-connectors
-> flink-avro
-> flink-jdbc
-> flink-hadoop-compatibility
-> flink-hbase
-> flink-hcatalog
flink-streaming-connectors
-> flink-connector-twitter
-> flink-streaming-examples
-> flink-connector-flume
-> flink-connector-kafka
-> flink-connector-elasticsearch
-> flink-connector-rabbitmq
-> flink-connector-filesystem
flink-libraries
-> flink-gelly
-> flink-gelly-scala
-> flink-ml
-> flink-table
-> flink-language-binding
-> flink-python
flink-scala-shell
flink-test-utils
flink-tests
flink-fs-tests
flink-contrib
-> flink-storm-compatibility
-> flink-storm-compatibility-examples
-> flink-streaming-utils
-> flink-tweet-inputformat
-> flink-operator-stats
-> flink-tez
flink-quickstart
-> flink-quickstart-java
-> flink-quickstart-scala
-> flink-tez-quickstart
flink-yarn
flink-yarn-tests
flink-dist
flink-benchmark
Let me know if that makes sense!
Greetings,
Stephan
