I don't think it's useful to combine them since they are different
projects. But I do think that a lot of work went into Flink's paged
memory system built on byte buffers and if collaboration can take place
to pop that out into like a memory subsystem library that both Spark and
Flink can use then it should raise both ships. If the usage patterns are
too different then sure, don't use their work. But it looks pretty generic:
https://github.com/apache/flink/tree/master/flink-core/src/main/java/org/apache/flink/core/memory
To bring this back into other threads: Flink's memory system uses
java.nio - so it requires Java 1.7 afaik. :)
-Ewan
On 05/01/2015 03:54 PM, Stephen Carman wrote:
I think as long as the two frameworks follow the same paradigm for how their
interfaces work it’s fine to have 2 competing frameworks. This way the
frameworks have some motivation
to be the best at what they do rather than being the only choice whether you
like it or not. They also seem to have some differing opinions about how to do
certain things leaving me to believe
that the 2 projects exist mostly because of disagreements on fundamentals about
how a system such as this should be built and scaled out.
I think spark should definitely take what it can from these projects, but
otherwise they should remain separate projects going their own way.
Steve
On Apr 29, 2015, at 8:01 PM, Sree V <sree_at_ch...@yahoo.com.INVALID> wrote:
I agree, Ewan.
We should also look into combining both Flink and Spark into one.This eases the
industry adaptation instead.
Thanking you.
With Regards
Sree
On Wednesday, April 29, 2015 3:21 AM, Ewan Higgs <ewan.hi...@ugent.be>
wrote:
Hi all,
A quick question about Tungsten. The announcement of the Tungsten
project is on the back of Hadoop Summit in Brussels where some of the
Flink devs were giving talks [1] on how Flink manages memory using byte
arrays and the like to avoid the overhead of all the Java types[2]. Is
there an opportunity for code reuse here? Spark and Flink may have
different needs in some respects, but they work fundamentally towards
the same goal so I imagine there could be come worthwhile collaboration.
-Ewan
[1] http://2015.hadoopsummit.org/brussels/speaker/?speaker=MrtonBalassi
http://2015.hadoopsummit.org/brussels/speaker/?speaker=AljoschaKrettek
[2]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=53741525
https://flink.apache.org/news/2015/03/13/peeking-into-Apache-Flinks-Engine-Room.html
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
This e-mail is intended solely for the above-mentioned recipient and it may
contain confidential or privileged information. If you have received it in
error, please notify us immediately and delete the e-mail. You must not copy,
distribute, disclose or take any action in reliance on it. In addition, the
contents of an attachment to this e-mail may contain software viruses which
could damage your own computer system. While ColdLight Solutions, LLC has taken
every reasonable precaution to minimize this risk, we cannot accept liability
for any damage which you sustain as a result of software viruses. You should
perform your own virus checks before opening the attachment.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
