Hi Georg,
Personally, I didn't have feedback about performance difference between flink
native and beam/flink runner.
For low level runner access, you mean you want to access Spark ML function, etc
from a Beam pipeline ? Even if you can always use a DoFn, you won't have access
to all internal objects/API. That's what you mean ?
Regards
JB
On 04/12/2017 08:33 PM, Georg Heiler wrote:
HI JB,
The overall look and feel of streamline (Kafka, storm, hbase and the simplicity
of the UI) were really compelling. As we are currently starting to build a
streaming platform we are looking forward to such a tight integration, but would
rather use flink or spark (maybe via beam) and not storm integrated into such a
platform.
Regarding streamline performance:
someone told me that e.g. the flink runner for beam seems to be slower than a
native flink job. Is this true? Did you observe such characteristics for several
runners?
Regarding streamline low level runner access:
in case I want to use some low level functionality (specific to a runner) like
ML, graph processing or sql-tables api, is it possible to just drop from the
beam API one level deeper to the actual runner and sort of mesh beam with runner
native code to integrate these features?
Regards,
Georg
Jean-Baptiste Onofré <[email protected] <mailto:[email protected]>> schrieb am
Mi.,
12. Apr. 2017 um 20:04 Uhr:
Hi Georg,
You can use Java API via Scala, or you can use the Scio Scala DSL (this DSL
use
the Beam Java SDK).
For Streamline, can you explain a bit ? Streamline contains different parts:
HBase, Kafka, the web frontend, ...
Using provided IO, it should be possible to store the data and use
streamline on
top of this data for analytics.
Regards
JB
On 04/12/2017 07:57 PM, Georg Heiler wrote:
> Hi,
> new to beam I wonder what API is recommended for using beam from scala.
> Would you recommend simply using the java API from scala or
> https://github.com/spotify/scio?
>
> Are there any plans to support beam in
https://github.com/hortonworks/streamline?
>
> regards,
> Georg
--
Jean-Baptiste Onofré
[email protected] <mailto:[email protected]>
http://blog.nanthrax.net
Talend - http://www.talend.com
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
