Hello JB,
can we help in any way to move things forward?
Thanks,
D.
On Mon, Dec 18, 2017 at 4:28 PM, Jean-Baptiste Onofré <j...@nanthrax.net
<mailto:j...@nanthrax.net>> wrote:
Thanks Jan,
It makes sense.
Let me take a look on the code to understand the "interaction".
Regards
JB
On 12/18/2017 04:26 PM, Jan Lukavský wrote:
Hi JB,
basically you are not wrong. The project started about three or four
years ago with a goal to unify batch and streaming processing into
single portable, executor independent API. Because of that, it is
currently "close" to Beam in this sense. But we don't see much added
value keeping this as a separate project, with one of the key
differences to be the API (not the model itself), so we would like to
focus on translation from Euphoria API to Beam's SDK. That's why we
would like to see it as a DSL, so that it would be possible to use
Euphoria API with Beam's runners as much natively as possible.
I hope I didn't make the subject even more unclear, if so, I'll be happy
to explain anything in more detail. :-)
Jan
On 12/18/2017 04:08 PM, Jean-Baptiste Onofré wrote:
Hi Jan,
Thanks for your answers.
However, they confused me ;)
Regarding what you replied, Euphoria seems like a programming
model/SDK "close" to Beam more than a DSL on top of an existing Beam
SDK.
Am I wrong ?
Regards
JB
On 12/18/2017 03:44 PM, Jan Lukavský wrote:
Hi Ismael,
basically we adopted the Beam's design regarding partitioning
(https://github.com/seznam/euphoria/issues/160
<https://github.com/seznam/euphoria/issues/160>) and implemented
the sorting manually
(https://github.com/seznam/euphoria/issues/158
<https://github.com/seznam/euphoria/issues/158>). I'm not aware
of the time model differences (Euphoria supports ingestion and
event time, we don't support processing time by decision).
Regarding other differences (looking into Beam capability
matrix, I'd say that):
- we don't support stateful FlatMap (i.e. ParDo) for now
(https://github.com/seznam/euphoria/issues/192
<https://github.com/seznam/euphoria/issues/192>)
- we don't support side inputs (by decision now, but might be
reconsidered) and outputs
(https://github.com/seznam/euphoria/issues/124
<https://github.com/seznam/euphoria/issues/124>)
- we support complete event-time windows (non-merging,
merging, aligned, unaligned) and time control
- we don't support processing time by decision (might be
reconsidered if a valid use-case is found)
- we support window triggering based on both time and data,
including discarding and accumulating (without accumulating &
retracting)
All our executors (runners) - Flink, Spark and Local - implement
the complete model, which we enforce using "operator test kit"
that all executors must pass. Spark executor supports bounded
sources only (for now). As David said, we currently don't have
serialization abstraction, so there is some work to be done in
that regard.
Our intention is to completely supersede Euphoria, we would like
to consider possibility to use executors that would not rely on
Beam, but that is optional now and should be straightforward.
We'd be happy to answer any more questions you might have and
thanks a lot!
Best,
Jan
On 12/18/2017 03:19 PM, Ismaël Mejía wrote:
Hi,
It is great to see that you guys have achieved a maturity
point to
propose this. Congratulations for your work and the idea to
contribute
it into Beam.
I remember from a previous discussion with Jan about the
model
mismatch between Euphoria and Beam, because of some design
decisions
of both projects. I remember you guys had some issues with
the way
Beam's sources do partitioning, as well as Beam's lack of
sorted data
(on shuffle a la hadoop). Also if I remember well the 'time'
model of
Euphoria was simpler than Beam's. I talk about all of this
because I
am curious about what parts of the Euphoria model you guys
had to
sacrifice to support Beam, and what parts of Beam's model
should still
be integrated into Euphoria (and if there is a
straightforward path to
do it).
If I understand well if this gets merged into Apache this
means that
Euphoria's current implementation would be superseded by
this DSL? I
am curious because I would like to understand your level of
investment
on supporting the future of this DSL.
Thanks and congrats again !
Ismaël
On Mon, Dec 18, 2017 at 10:12 AM, Jean-Baptiste Onofré
<j...@nanthrax.net <mailto:j...@nanthrax.net>> wrote:
Depending of the donation, you would need ICLA for each
contributor, and
CCLA in addition of SGA.
We can sync with Davor and I for the legal stuff.
However, I would wait a little bit just to have feedback
from the whole team
and start a formal vote.
I would be happy to start the formal vote.
Regards
JB
On 12/18/2017 10:03 AM, David Morávek wrote:
Hello,
Thanks for the awesome feedback!
Romain:
We already use Java Stream API in all operators
where it makes sense (eg.:
ReduceByKey). Still not sure if it was a good
choice, but i can be easily
converted to iterator anyway.
Side outputs support is coming soon, we already made
an initial work on
this.
Side inputs are not supported in a way you are used
to from beam, because
it can be replaced by Join operator on the same key
(if annotated with
broadcastHashJoin, it will be turned into map side
join).
Only significant difference from Beam is, that we
decided not to abstract
serialization, so we need to add support for Type
Hints, because of type
erasure.
Fluent API:
API is fluent within one operator. It is designed to
"lead the
programmer", which means, that he we'll be only
offered methods that makes
sense after the last method he used (eg.: in
ReduceByKey, we know that after
keyBy either reduceBy method should come). It is
implemented as a series of
builders.
Davor:
Thanks, I'll contact you, and will start the process
of having all the
necessary paperwork signed on our side, so we can
get things moving.
On Mon, Dec 18, 2017 at 7:46 AM, Romain Manni-Bucau
<rmannibu...@gmail.com
<mailto:rmannibu...@gmail.com>
<mailto:rmannibu...@gmail.com
<mailto:rmannibu...@gmail.com>>> wrote:
Hi guys
A DSL would be very welcomed, in particular if
fluent.
Open question: did you study to implement
Stream API (surely extending
it to
have a BeamStream and a few more features like
sides etc)? Would be
very
natural and integrable easily anywhere and
avoid a new API discovery.
Hazelcast jet did it so I dont see why Beam
couldnt.
Le 18 déc. 2017 07:26, "Davor Bonaci"
<da...@apache.org <mailto:da...@apache.org>
<mailto:da...@apache.org
<mailto:da...@apache.org>>> a écrit :
Hi David,
As JB noted, merging of these two projects
is a great idea. If
fact,
some of us have had those discussions in
the past.
Legally, nothing particular is strictly
necessary as the code seem
to
already be Apache 2.0 licensed. We don't,
however, want to be
perceived
as making hostile forks, so it would be
great to file a Software
Grant
Agreement with the ASF Secretary. I can
help with the process, as
necessary.
Project alignment-wise, there aren't any
particular blockers that
I am
aware of. We welcome DSLs.
Technically, the code would start in a
feature branch. During this
stage, we'd need to validate a few things,
including confirmation
the
code and dependencies match the ASF
policy, automate testing in
Beam's
tooling, etc. At that point, we'd take a
community vote to accept
the
component into master, and consider
author(s) for committership in
the
overall project.
Welcome to the ASF and Beam -- we are
thrilled to have you! Hope
this
helps, and please reach out if anybody on
our end can help,
including JB
or myself.
Davor
On Sun, Dec 17, 2017 at 10:13 AM,
Jean-Baptiste Onofré
<j...@nanthrax.net <mailto:j...@nanthrax.net>
<mailto:j...@nanthrax.net
<mailto:j...@nanthrax.net>>> wrote:
Hi David,
Generally speaking, having different
fluent DSL on top of the
Beam
SDK is great.
I would like to take a look on your
wordcount examples to give
you a
complete feedback. I like the idea and
a fluent Java DSL is
valuable.
Let's wait feedback from others. If we
have a consensus, then
I
would be more than happy to help you
for the donation (I
worked on
the Camel Java DSL while ago, so I
have some experience here).
Thanks !
Regards
JB
On 12/17/2017 07:00 PM, David Morávek
wrote:
Hello,
First of all, thanks for the
amazing work the Apache Beam
community is doing!
In 2014, we've started development
of the runtime
independent
Java 8 API, that helps us to
create unified big-data
processing
flows. It has been used as a core
building block of
Seznam.cz
web crawler data infrastructure
every since. Its design
principles and execution model are
very similar to Apache
Beam.
This API was open sourced in 2016,
under the name Euphoria
API:
https://github.com/seznam/euphoria
<https://github.com/seznam/euphoria>
<https://github.com/seznam/euphoria
<https://github.com/seznam/euphoria>>
As it is very similar to Apache
Beam, we feel, that it is
not
worth of duplicating effort in
terms of development of new
runtimes and fine-tuning of
current ones.
The main blocker for us to switch
to Apache Beam is lack
of the
Java 8 API. *W*e propose the
integration of Euphoria API
into
Apache Beam as a Java 8 DSL, in
order to share our effort
with
the community.
Simple example of the Euphoria API
usage, can be found
here:
https://github.com/seznam/euphoria/tree/master/euphoria-examples/src/main/java/cz/seznam/euphoria/examples/wordcount
<https://github.com/seznam/euphoria/tree/master/euphoria-examples/src/main/java/cz/seznam/euphoria/examples/wordcount>
<https://github.com/seznam/euphoria/tree/master/euphoria-examples/src/main/java/cz/seznam/euphoria/examples/wordcount
<https://github.com/seznam/euphoria/tree/master/euphoria-examples/src/main/java/cz/seznam/euphoria/examples/wordcount>>
If you feel, that Beam community
could leverage from our
work,
we would love to start working on
Euphoria integration
into
Apache Beam (we already have a
working POC, with few basic
operators implemented).
I look forward to hearing from
you,
David
-- Jean-Baptiste Onofré
jbono...@apache.org <mailto:jbono...@apache.org>
<mailto:jbono...@apache.org
<mailto:jbono...@apache.org>>
http://blog.nanthrax.net
Talend - http://www.talend.com
--
s pozdravem
David Morávek
--
Jean-Baptiste Onofré
jbono...@apache.org <mailto:jbono...@apache.org>
http://blog.nanthrax.net
Talend - http://www.talend.com
--
Jean-Baptiste Onofré
jbono...@apache.org <mailto:jbono...@apache.org>
http://blog.nanthrax.net
Talend - http://www.talend.com
--
s pozdravem
David Morávek
