Different runners have different characteristics, pros, and cons, which is part of the value proposition for Beam. We have some comparisons at https://beam.apache.org/documentation/runners/capability-matrix/ but these were put together a while ago and don't really take into account the state of things with respect to portability (or other features like being managed (pros and cons there) or dynamic work distribution). This would be a welcome addition; I've crated https://issues.apache.org/jira/browse/BEAM-8332 to track.
On Tue, Oct 1, 2019 at 3:33 AM Maximilian Michels <[email protected]> wrote: > > > Probably the most stable is running on Dataflow still. But I’m excited to > > see the progress towards a Spark runner, can’t wait to try TFT on it :) > > That is debatable. It is also hard to compare because Dataflow is a > managed service, whereas you'll have to spin up your own cluster for > other Runners, or use a managed service like Kinesis Data Analytics (for > managed Flink). As Chad pointed out, Dataflow does not fully support the > portability model. One could argue, the most feature-rich and stable > portable Runner is the Flink Runner, but I would not make this comparison ;) > > On 18.09.19 07:51, Holden Karau wrote: > > Probably the most stable is running on Dataflow still. But I’m excited > > to see the progress towards a Spark runner, can’t wait to try TFT on it :) > > > > On Tue, Sep 17, 2019 at 4:37 PM Kyle Weaver <[email protected] > > <mailto:[email protected]>> wrote: > > > > The Flink runner is definitely more stable, as it's been around for > > longer and has more developers and users on it. But a lot of the > > code is shared, so for example some of the issues above would also > > happen on the Flink runner. > > > > Kyle Weaver | Software Engineer | github.com/ibzib > > <http://github.com/ibzib> | [email protected] > > <mailto:[email protected]> > > > > > > On Tue, Sep 17, 2019 at 4:34 PM Benjamin Tan > > <[email protected] <mailto:[email protected]>> > > wrote: > > > > Thanks for all the replies Kyle! You've been super helpful :D. > > > > Would you say that the Flink runner more stable than the Spark > > one? Or which combo is the most stable for now? > > > > On 2019/09/17 19:43:54, Tom Barber <[email protected] > > <mailto:[email protected]>> wrote: > > > Thanks Kyle, > > > > > > From my pov Alpha is fine, I’m just trying to test out some > > of the > > > capabilities currently, but trying to dig around the website > > doesn’t > > > explain a great deal. Luckily Benjamin seems a step ahead of > > me… I hope it > > > stays that way! ;) > > > > > > > > > On 17 September 2019 at 19:33:40, Kyle Weaver > > ([email protected] <mailto:[email protected]>) wrote: > > > > > > > The amount of issues I've encountered as a newbie is indeed > > troubling. > > > Spark portability is very much "alpha" quality software, a > > point we should > > > maybe emphasize on the website more. Anyway, I appreciate > > your patience, > > > and I'll do my best to address all these issues. > > > > > > > > > org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException: > > > UNIMPLEMENTED: Method not found! > > > > AttributeError: module 'apache_beam.coders.coders' has no > > > attribute ‘VarIntCoder' > > > This class of errors occurs when the SDK version does not > > match up with the > > > runner version -- unfortunately, we cannot guarantee > > compatibility between > > > the two, so manual syncing is required for now. We are > > looking for ways to > > > improve this. > > > > > > > ERROR:grpc._server:Exception calling application: u'2-1' > > > It looks like these errors are entirely spurious. Have a fix > > for it here: > > > https://github.com/apache/beam/pull/9600 > > > > > > Note that there may still be other spurious errors like: > > > > > > 19/09/17 11:25:31 ERROR ManagedChannelOrphanWrapper: *~*~*~ > > Channel > > > ManagedChannelImpl{logId=84, target=localhost:36129} was not > > shutdown > > > properly!!! ~*~*~* > > > > > > and > > > > > > 19/09/17 11:25:32 ERROR SerializingExecutor: Exception while > > executing > > > runnable > > > > > > > org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@47db89b1 > > > java.lang.IllegalStateException: call already closed > > > > > > I will try to get rid of them also, but for now you can just > > ignore them. > > > They are annoying but harmless. > > > > > > Kyle Weaver | Software Engineer | github.com/ibzib > > <http://github.com/ibzib> | [email protected] > > <mailto:[email protected]> > > > > > > > > > On Tue, Sep 17, 2019 at 8:05 AM Benjamin Tan > > <[email protected] <mailto:[email protected]>> > > > wrote: > > > > > > > :D. Still, I'm curious as to the error we both are getting. > > Maybe someone > > > > can shed some light on it. > > > > > > > > On Tue, Sep 17, 2019 at 10:54 PM Tom Barber > > <[email protected] <mailto:[email protected]>> wrote: > > > > > > > >> I do see hello written to 1 file and world to another, I > > guess it works! > > > >> Thanks for the pointers Benjamin I was about to give up. > > > >> > > > >> Tom > > > >> > > > >> > > > >> On 17 September 2019 at 15:51:13, Benjamin Tan ( > > > >> [email protected] > > <mailto:[email protected]>) wrote: > > > >> > > > >> Tell me if you see any output. Anyway, here's the link to > > the same issue > > > >> you're facing: > > > >> > > > >> > > > >> > > > > https://lists.apache.org/thread.html/4e8e1455916debe096de32551f9ab05853524cf282bc312cd4620d68@%3Cuser.beam.apache.org%3E > > > >> > > > >> The amount of issues I've encountered as a newbie is > > indeed troubling. > > > >> > > > >> On 2019/09/17 14:43:11, Tom Barber <[email protected] > > <mailto:[email protected]>> wrote: > > > >> > 🤣 okay I’ll look again, I assumed it just crashed in a > > ball of flames! > > > >> > > > > >> > > > > >> > On 17 September 2019 at 15:39:33, Benjamin Tan ( > > > >> [email protected] > > <mailto:[email protected]>) > > > >> > wrote: > > > >> > > > > >> > I got this too! Did you manage to get any output? (I > > did) I reported > > > >> this > > > >> > in another thread. > > > >> > > > > >> > This looks like a key error when StopWorker is called. > > As far as I > > > >> know, it > > > >> > seems like the work has been processed. > > > >> > > > > >> > On Tue, Sep 17, 2019 at 10:31 PM Tom Barber > > <[email protected] <mailto:[email protected]>> wrote: > > > >> > > > > >> > > Well my errors are different but still terminal: > > > >> > > > > > >> > > ERROR:grpc._server:Exception calling application: u'1-1' > > > >> > > Traceback (most recent call last): > > > >> > > File > > "/usr/local/lib/python2.7/dist-packages/grpc/_server.py", line > > > >> 434, > > > >> > > in _call_behavior > > > >> > > response_or_iterator = behavior(argument, context) > > > >> > > File > > > >> > > > > > >> > > > > "/usr/local/lib/python2.7/dist-packages/apache_beam-2.17.0.dev0-py2.7.egg/apache_beam/runners/worker/worker_pool_main.py", > > > >> > > line 126, in StopWorker > > > >> > > worker_process = > > > >> > > self._worker_processes.pop(stop_worker_request.worker_id) > > > >> > > KeyError: u'1-1' > > > >> > > ERROR:grpc._server:Exception calling application: u'2-1' > > > >> > > Traceback (most recent call last): > > > >> > > File > > "/usr/local/lib/python2.7/dist-packages/grpc/_server.py", line > > > >> 434, > > > >> > > in _call_behavior > > > >> > > response_or_iterator = behavior(argument, context) > > > >> > > File > > > >> > > > > > >> > > > > "/usr/local/lib/python2.7/dist-packages/apache_beam-2.17.0.dev0-py2.7.egg/apache_beam/runners/worker/worker_pool_main.py", > > > >> > > line 126, in StopWorker > > > >> > > worker_process = > > > >> > > self._worker_processes.pop(stop_worker_request.worker_id) > > > >> > > KeyError: u'2-1' > > > >> > > > > > >> > > > > > >> > > On 17 September 2019 at 14:46:12, Benjamin Tan ( > > > >> > > [email protected] > > <mailto:[email protected]>) wrote: > > > >> > > > > > >> > > If it helps, I’m using Spark 2.4.4. The Apache Beam > > Python library on > > > >> > > master is 2.17.0-dev. > > > >> > > > > > >> > > > > > >> > > On 17 Sep 2019, at 9:39 PM, Tom Barber > > <[email protected] <mailto:[email protected]>> wrote: > > > >> > > > > > >> > > Cool thanks Benjamin, I’ll give it a shot. > > > >> > > > > > >> > > Tom > > > >> > > > > > >> > > > > > >> > > On 17 September 2019 at 13:56:14, Benjamin Tan ( > > > >> > > [email protected] > > <mailto:[email protected]>) wrote: > > > >> > > > > > >> > > > > > >> > > I encountered the exact same thing today. High five! > > Here’s how I > > > >> managed > > > >> > > to make some progress: > > > >> > > > > > >> > > 1. Used the master branch > > > >> > > 2. Built and installed the Python SDK > > > >> > > > > > >> > > cd into the sdk library and python ./setup install > > > >> > > > > > >> > > I got some other errors but they didn’t seem to be > > show stoppers. > > > >> > > > > > >> > > On 17 Sep 2019, at 8:26 PM, Tom Barber > > <[email protected] <mailto:[email protected]>> wrote: > > > >> > > > > > >> > > Hello folks, > > > >> > > > > > >> > > Day 3 of trying to get the basics going with Python & > > Spark 2.2.3. > > > >> > > > > > >> > > I’ve downgraded the spark version to 2.2.3 in the > > cradle build so > > > >> that I > > > >> > > can run jobs against it. > > > >> > > > > > >> > > I’ve then written this: > > > >> > > > > > >> > > options = PipelineOptions(["--runner=PortableRunner", > > > >> > > "--job_endpoint=localhost:8099", > > "--environment_type=LOOPBACK"]) > > > >> > > > > > >> > > with beam.Pipeline(options=options) as p: > > > >> > > print("hello") > > > >> > > lines = p | 'Create' >> beam.Create(['hello', 'world']) > > > >> > > > > > >> > > Which in turn explodes with: > > > >> > > > > > >> > > Caused by: > > > >> > > > > org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException: > > > >> > > UNIMPLEMENTED: Method not found! > > > >> > > at > > > >> > > > > > >> > > > > org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:235) > > > >> > > at > > > >> > > > > > >> > > > > org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:216) > > > >> > > at > > > >> > > > > > >> > > > > org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:141) > > > >> > > at > > > >> > > > > > >> > > > > org.apache.beam.model.fnexecution.v1.BeamFnExternalWorkerPoolGrpc$BeamFnExternalWorkerPoolBlockingStub.startWorker(BeamFnExternalWorkerPoolGrpc.java:226) > > > >> > > > > > >> > > When run against python 2 and > > > >> > > > > > >> > > AttributeError: module 'apache_beam.coders.coders' has no > > > >> > > attribute ‘VarIntCoder' > > > >> > > > > > >> > > When run with python 3.5. > > > >> > > > > > >> > > Should I just give up trying the portable runner at > > the moment or am I > > > >> > > doing something dumb/incompatible? I’m stuck on Spark > > 2.2.3 so > > > >> upgrading > > > >> > > isn’t currently an option. > > > >> > > > > > >> > > Tom > > > >> > > > > > >> > > Spicule Limited is registered in England & Wales. > > Company Number: > > > >> > > 09954122. Registered office: First Floor, Telecom > > House, 125-135 > > > > <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> > > > > <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> > > >> Preston > > > > <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> > > > > <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> > > >> > > Road, Brighton, England, BN1 6AF > > > > <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>. > > VAT No. 251478891. > > > >> > > > > > >> > > > > > >> > > All engagements are subject to Spicule Terms and > > Conditions of > > > >> Business. > > > >> > > This email and its contents are intended solely for > > the individual to > > > >> whom > > > >> > > it is addressed and may contain information that is > > confidential, > > > >> > > privileged or otherwise protected from disclosure, > > distributing or > > > >> copying. > > > >> > > Any views or opinions presented in this email are > > solely those of the > > > >> > > author and do not necessarily represent those of > > Spicule Limited. The > > > >> > > company accepts no liability for any damage caused by > > any virus > > > >> transmitted > > > >> > > by this email. If you have received this message in > > error, please > > > >> notify us > > > >> > > immediately by reply email before deleting it from > > your system. > > > >> Service of > > > >> > > legal notice cannot be effected on Spicule Limited by > > email. > > > >> > > > > > >> > > > > > >> > > Spicule Limited is registered in England & Wales. > > Company Number: > > > >> > > 09954122. Registered office: First Floor, Telecom > > House, 125-135 > > > > <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> > > > > <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> > > >> Preston > > > > <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> > > > > <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> > > >> > > Road, Brighton, England, BN1 6AF > > > > <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>. > > VAT No. 251478891. > > > >> > > > > > >> > > > > > >> > > All engagements are subject to Spicule Terms and > > Conditions of > > > >> Business. > > > >> > > This email and its contents are intended solely for > > the individual to > > > >> whom > > > >> > > it is addressed and may contain information that is > > confidential, > > > >> > > privileged or otherwise protected from disclosure, > > distributing or > > > >> copying. > > > >> > > Any views or opinions presented in this email are > > solely those of the > > > >> > > author and do not necessarily represent those of > > Spicule Limited. The > > > >> > > company accepts no liability for any damage caused by > > any virus > > > >> transmitted > > > >> > > by this email. If you have received this message in > > error, please > > > >> notify us > > > >> > > immediately by reply email before deleting it from > > your system. > > > >> Service of > > > >> > > legal notice cannot be effected on Spicule Limited by > > email. > > > >> > > > > > >> > > > > > >> > > Spicule Limited is registered in England & Wales. > > Company Number: > > > >> > > 09954122. Registered office: First Floor, Telecom > > House, 125-135 > > > > <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> > > > > <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> > > >> Preston > > > > <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> > > > > <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> > > >> > > Road, Brighton, England, BN1 6AF > > > > <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>. > > VAT No. 251478891. > > > >> > > > > > >> > > > > > >> > > All engagements are subject to Spicule Terms and > > Conditions of > > > >> Business. > > > >> > > This email and its contents are intended solely for > > the individual to > > > >> whom > > > >> > > it is addressed and may contain information that is > > confidential, > > > >> > > privileged or otherwise protected from disclosure, > > distributing or > > > >> copying. > > > >> > > Any views or opinions presented in this email are > > solely those of the > > > >> > > author and do not necessarily represent those of > > Spicule Limited. The > > > >> > > company accepts no liability for any damage caused by > > any virus > > > >> transmitted > > > >> > > by this email. If you have received this message in > > error, please > > > >> notify us > > > >> > > immediately by reply email before deleting it from > > your system. > > > >> Service of > > > >> > > legal notice cannot be effected on Spicule Limited by > > email. > > > >> > > > > > >> > > > > >> > -- > > > >> > > > > >> > > > > >> > Spicule Limited is registered in England & Wales. > > Company Number: > > > >> > 09954122. Registered office: First Floor, Telecom House, > > 125-135 Preston > > > > <https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> > > > > <https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> > > >> > Road, Brighton, England, BN1 6AF > > > > <https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>. > > VAT No. 251478891. > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > All engagements > > > >> > are subject to Spicule Terms and Conditions of Business. > > This email and > > > >> its > > > >> > contents are intended solely for the individual to whom > > it is addressed > > > >> and > > > >> > may contain information that is confidential, privileged > > or otherwise > > > >> > protected from disclosure, distributing or copying. Any > > views or > > > >> opinions > > > >> > presented in this email are solely those of the author > > and do not > > > >> > necessarily represent those of Spicule Limited. The > > company accepts no > > > >> > liability for any damage caused by any virus transmitted > > by this email. > > > >> If > > > >> > you have received this message in error, please notify > > us immediately by > > > >> > reply email before deleting it from your system. Service > > of legal notice > > > >> > cannot be effected on Spicule Limited by email. > > > >> > > > > >> > > > >> > > > >> Spicule Limited is registered in England & Wales. Company > > Number: > > > >> 09954122. Registered office: First Floor, Telecom House, > > 125-135 Preston > > > > <https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> > > > > <https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> > > >> Road, Brighton, England, BN1 6AF > > > > <https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>. > > VAT No. 251478891. > > > >> > > > >> > > > >> All engagements are subject to Spicule Terms and > > Conditions of Business. > > > >> This email and its contents are intended solely for the > > individual to whom > > > >> it is addressed and may contain information that is > > confidential, > > > >> privileged or otherwise protected from disclosure, > > distributing or copying. > > > >> Any views or opinions presented in this email are solely > > those of the > > > >> author and do not necessarily represent those of Spicule > > Limited. The > > > >> company accepts no liability for any damage caused by any > > virus transmitted > > > >> by this email. If you have received this message in error, > > please notify us > > > >> immediately by reply email before deleting it from your > > system. Service of > > > >> legal notice cannot be effected on Spicule Limited by email. > > > >> > > > > > > > > > > -- > > > > > > > > > Spicule Limited is registered in England & Wales. Company > > Number: > > > 09954122. Registered office: First Floor, Telecom House, > > 125-135 Preston > > > > <https://www.google.com/maps/search/125-135+Preston++%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> > > > > <https://www.google.com/maps/search/125-135+Preston++%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> > > Road, Brighton, England, BN1 6AF > > > > <https://www.google.com/maps/search/125-135+Preston++%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>. > > VAT No. 251478891. > > > > > > > > > > > > > > > All engagements > > > are subject to Spicule Terms and Conditions of Business. This > > email and its > > > contents are intended solely for the individual to whom it is > > addressed and > > > may contain information that is confidential, privileged or > > otherwise > > > protected from disclosure, distributing or copying. Any views > > or opinions > > > presented in this email are solely those of the author and do > > not > > > necessarily represent those of Spicule Limited. The company > > accepts no > > > liability for any damage caused by any virus transmitted by > > this email. If > > > you have received this message in error, please notify us > > immediately by > > > reply email before deleting it from your system. Service of > > legal notice > > > cannot be effected on Spicule Limited by email. > > > > > > > -- > > Twitter: https://twitter.com/holdenkarau > > Books (Learning Spark, High Performance Spark, etc.): > > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > > YouTube Live Streams: https://www.youtube.com/user/holdenkarau
