By the way, on my side, I will work on a Karaf/OSGi
(http://karaf.apache.org) runner for Beam (with shell commands,
features, etc).
I will start it just after the work on new IOs.
Regards
JB
On 03/09/2016 08:01 PM, Minudika Malshan wrote:
Hi,
Thanks a lot for your quick responses.
I will refer those resources.
Regards,
Minudika
Minudika Malshan
Undergraduate
Department of Computer Science and Engineering
University of Moratuwa
Sri Lanka.
On Thu, Mar 10, 2016 at 12:24 AM, Lukasz Cwik <[email protected]>
wrote:
There are currently two implementations which do not require the cloud:
The DirectPipelineRunner
<
https://github.com/apache/incubator-beam/blob/master/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DirectPipelineRunner.java
which is mainly used for testing and local development. This runner has
several limits (data size, no support for unbounded collections, ...) and
is being expanded to support more use cases, for example adding unbounded
PCollection support <https://issues.apache.org/jira/browse/BEAM-22>.
The FlinkPipelineRunner
<https://github.com/apache/incubator-beam/tree/master/runners/flink> which
can be used to execute locally or on a Flink cluster.
There is also ongoing work to bring Spark
<https://issues.apache.org/jira/browse/BEAM-6> into the mix as a runner
and
suggestions to for other runners such as GearPump
<https://github.com/gearpump/gearpump>.
On Wed, Mar 9, 2016 at 10:37 AM, Minudika Malshan <[email protected]>
wrote:
Hi all,
As per my knowledge about Apache beam and data flow sdk, at the first
data
flow sdk has been developed targeting google cloud platform.
So we have to deploy pipelines in the cloud.
But my question is, can not we use this sdk for standalone
implementations
without cloud. If so, I would love to have a look at some examples of
such
implementations.
Your kind help is much appreciated.
Regards,
Minudika
Minudika Malshan
Undergraduate
Department of Computer Science and Engineering
University of Moratuwa
Sri Lanka.
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
