Just note that while Dataflow does have robust python support it does not fully support the portability framework. It’s a bit of a blurry distinction, and honestly I’m not crystal clear on this as I get the impression that Dataflow may be a bit of a Portability hybrid. It does not use the job service or the expansion service but I have heard that it uses the external worker pool, and maybe it uses the protobufs pipeline definitions? It’d be great if a Googler could clarify.
-chad On Wed, Sep 18, 2019 at 7:51 AM Holden Karau <[email protected]> wrote: > Probably the most stable is running on Dataflow still. But I’m excited to > see the progress towards a Spark runner, can’t wait to try TFT on it :) > > On Tue, Sep 17, 2019 at 4:37 PM Kyle Weaver <[email protected]> wrote: > >> The Flink runner is definitely more stable, as it's been around for >> longer and has more developers and users on it. But a lot of the code is >> shared, so for example some of the issues above would also happen on the >> Flink runner. >> >> Kyle Weaver | Software Engineer | github.com/ibzib | [email protected] >> >> >> On Tue, Sep 17, 2019 at 4:34 PM Benjamin Tan <[email protected]> >> wrote: >> >>> Thanks for all the replies Kyle! You've been super helpful :D. >>> >>> Would you say that the Flink runner more stable than the Spark one? Or >>> which combo is the most stable for now? >>> >>> On 2019/09/17 19:43:54, Tom Barber <[email protected]> wrote: >>> > Thanks Kyle, >>> > >>> > From my pov Alpha is fine, I’m just trying to test out some of the >>> > capabilities currently, but trying to dig around the website doesn’t >>> > explain a great deal. Luckily Benjamin seems a step ahead of me… I >>> hope it >>> > stays that way! ;) >>> > >>> > >>> > On 17 September 2019 at 19:33:40, Kyle Weaver ([email protected]) >>> wrote: >>> > >>> > > The amount of issues I've encountered as a newbie is indeed >>> troubling. >>> > Spark portability is very much "alpha" quality software, a point we >>> should >>> > maybe emphasize on the website more. Anyway, I appreciate your >>> patience, >>> > and I'll do my best to address all these issues. >>> > >>> > > org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException: >>> > UNIMPLEMENTED: Method not found! >>> > > AttributeError: module 'apache_beam.coders.coders' has no >>> > attribute ‘VarIntCoder' >>> > This class of errors occurs when the SDK version does not match up >>> with the >>> > runner version -- unfortunately, we cannot guarantee compatibility >>> between >>> > the two, so manual syncing is required for now. We are looking for >>> ways to >>> > improve this. >>> > >>> > > ERROR:grpc._server:Exception calling application: u'2-1' >>> > It looks like these errors are entirely spurious. Have a fix for it >>> here: >>> > https://github.com/apache/beam/pull/9600 >>> > >>> > Note that there may still be other spurious errors like: >>> > >>> > 19/09/17 11:25:31 ERROR ManagedChannelOrphanWrapper: *~*~*~ Channel >>> > ManagedChannelImpl{logId=84, target=localhost:36129} was not shutdown >>> > properly!!! ~*~*~* >>> > >>> > and >>> > >>> > 19/09/17 11:25:32 ERROR SerializingExecutor: Exception while executing >>> > runnable >>> > >>> org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@47db89b1 >>> > java.lang.IllegalStateException: call already closed >>> > >>> > I will try to get rid of them also, but for now you can just ignore >>> them. >>> > They are annoying but harmless. >>> > >>> > Kyle Weaver | Software Engineer | github.com/ibzib | >>> [email protected] >>> > >>> > >>> > On Tue, Sep 17, 2019 at 8:05 AM Benjamin Tan < >>> [email protected]> >>> > wrote: >>> > >>> > > :D. Still, I'm curious as to the error we both are getting. Maybe >>> someone >>> > > can shed some light on it. >>> > > >>> > > On Tue, Sep 17, 2019 at 10:54 PM Tom Barber <[email protected]> >>> wrote: >>> > > >>> > >> I do see hello written to 1 file and world to another, I guess it >>> works! >>> > >> Thanks for the pointers Benjamin I was about to give up. >>> > >> >>> > >> Tom >>> > >> >>> > >> >>> > >> On 17 September 2019 at 15:51:13, Benjamin Tan ( >>> > >> [email protected]) wrote: >>> > >> >>> > >> Tell me if you see any output. Anyway, here's the link to the same >>> issue >>> > >> you're facing: >>> > >> >>> > >> >>> > >> >>> https://lists.apache.org/thread.html/4e8e1455916debe096de32551f9ab05853524cf282bc312cd4620d68@%3Cuser.beam.apache.org%3E >>> > >> >>> > >> The amount of issues I've encountered as a newbie is indeed >>> troubling. >>> > >> >>> > >> On 2019/09/17 14:43:11, Tom Barber <[email protected]> wrote: >>> > >> > 🤣 okay I’ll look again, I assumed it just crashed in a ball of >>> flames! >>> > >> > >>> > >> > >>> > >> > On 17 September 2019 at 15:39:33, Benjamin Tan ( >>> > >> [email protected]) >>> > >> > wrote: >>> > >> > >>> > >> > I got this too! Did you manage to get any output? (I did) I >>> reported >>> > >> this >>> > >> > in another thread. >>> > >> > >>> > >> > This looks like a key error when StopWorker is called. As far as I >>> > >> know, it >>> > >> > seems like the work has been processed. >>> > >> > >>> > >> > On Tue, Sep 17, 2019 at 10:31 PM Tom Barber <[email protected]> >>> wrote: >>> > >> > >>> > >> > > Well my errors are different but still terminal: >>> > >> > > >>> > >> > > ERROR:grpc._server:Exception calling application: u'1-1' >>> > >> > > Traceback (most recent call last): >>> > >> > > File "/usr/local/lib/python2.7/dist-packages/grpc/_server.py", >>> line >>> > >> 434, >>> > >> > > in _call_behavior >>> > >> > > response_or_iterator = behavior(argument, context) >>> > >> > > File >>> > >> > > >>> > >> >>> "/usr/local/lib/python2.7/dist-packages/apache_beam-2.17.0.dev0-py2.7.egg/apache_beam/runners/worker/worker_pool_main.py", >>> > >> > > line 126, in StopWorker >>> > >> > > worker_process = >>> > >> > > self._worker_processes.pop(stop_worker_request.worker_id) >>> > >> > > KeyError: u'1-1' >>> > >> > > ERROR:grpc._server:Exception calling application: u'2-1' >>> > >> > > Traceback (most recent call last): >>> > >> > > File "/usr/local/lib/python2.7/dist-packages/grpc/_server.py", >>> line >>> > >> 434, >>> > >> > > in _call_behavior >>> > >> > > response_or_iterator = behavior(argument, context) >>> > >> > > File >>> > >> > > >>> > >> >>> "/usr/local/lib/python2.7/dist-packages/apache_beam-2.17.0.dev0-py2.7.egg/apache_beam/runners/worker/worker_pool_main.py", >>> > >> > > line 126, in StopWorker >>> > >> > > worker_process = >>> > >> > > self._worker_processes.pop(stop_worker_request.worker_id) >>> > >> > > KeyError: u'2-1' >>> > >> > > >>> > >> > > >>> > >> > > On 17 September 2019 at 14:46:12, Benjamin Tan ( >>> > >> > > [email protected]) wrote: >>> > >> > > >>> > >> > > If it helps, I’m using Spark 2.4.4. The Apache Beam Python >>> library on >>> > >> > > master is 2.17.0-dev. >>> > >> > > >>> > >> > > >>> > >> > > On 17 Sep 2019, at 9:39 PM, Tom Barber <[email protected]> >>> wrote: >>> > >> > > >>> > >> > > Cool thanks Benjamin, I’ll give it a shot. >>> > >> > > >>> > >> > > Tom >>> > >> > > >>> > >> > > >>> > >> > > On 17 September 2019 at 13:56:14, Benjamin Tan ( >>> > >> > > [email protected]) wrote: >>> > >> > > >>> > >> > > >>> > >> > > I encountered the exact same thing today. High five! Here’s how >>> I >>> > >> managed >>> > >> > > to make some progress: >>> > >> > > >>> > >> > > 1. Used the master branch >>> > >> > > 2. Built and installed the Python SDK >>> > >> > > >>> > >> > > cd into the sdk library and python ./setup install >>> > >> > > >>> > >> > > I got some other errors but they didn’t seem to be show >>> stoppers. >>> > >> > > >>> > >> > > On 17 Sep 2019, at 8:26 PM, Tom Barber <[email protected]> >>> wrote: >>> > >> > > >>> > >> > > Hello folks, >>> > >> > > >>> > >> > > Day 3 of trying to get the basics going with Python & Spark >>> 2.2.3. >>> > >> > > >>> > >> > > I’ve downgraded the spark version to 2.2.3 in the cradle build >>> so >>> > >> that I >>> > >> > > can run jobs against it. >>> > >> > > >>> > >> > > I’ve then written this: >>> > >> > > >>> > >> > > options = PipelineOptions(["--runner=PortableRunner", >>> > >> > > "--job_endpoint=localhost:8099", "--environment_type=LOOPBACK"]) >>> > >> > > >>> > >> > > with beam.Pipeline(options=options) as p: >>> > >> > > print("hello") >>> > >> > > lines = p | 'Create' >> beam.Create(['hello', 'world']) >>> > >> > > >>> > >> > > Which in turn explodes with: >>> > >> > > >>> > >> > > Caused by: >>> > >> > > >>> org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException: >>> > >> > > UNIMPLEMENTED: Method not found! >>> > >> > > at >>> > >> > > >>> > >> >>> org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:235) >>> > >> > > at >>> > >> > > >>> > >> >>> org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:216) >>> > >> > > at >>> > >> > > >>> > >> >>> org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:141) >>> > >> > > at >>> > >> > > >>> > >> >>> org.apache.beam.model.fnexecution.v1.BeamFnExternalWorkerPoolGrpc$BeamFnExternalWorkerPoolBlockingStub.startWorker(BeamFnExternalWorkerPoolGrpc.java:226) >>> > >> > > >>> > >> > > When run against python 2 and >>> > >> > > >>> > >> > > AttributeError: module 'apache_beam.coders.coders' has no >>> > >> > > attribute ‘VarIntCoder' >>> > >> > > >>> > >> > > When run with python 3.5. >>> > >> > > >>> > >> > > Should I just give up trying the portable runner at the moment >>> or am I >>> > >> > > doing something dumb/incompatible? I’m stuck on Spark 2.2.3 so >>> > >> upgrading >>> > >> > > isn’t currently an option. >>> > >> > > >>> > >> > > Tom >>> > >> > > >>> > >> > > Spicule Limited is registered in England & Wales. Company >>> Number: >>> > >> > > 09954122. Registered office: First Floor, Telecom House, >>> 125-135 >>> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> >>> >>> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> >>> >> Preston >>> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> >>> >>> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> >>> >> > > Road, Brighton, England, BN1 6AF >>> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>. >>> VAT No. 251478891. >>> > >> > > >>> > >> > > >>> > >> > > All engagements are subject to Spicule Terms and Conditions of >>> > >> Business. >>> > >> > > This email and its contents are intended solely for the >>> individual to >>> > >> whom >>> > >> > > it is addressed and may contain information that is >>> confidential, >>> > >> > > privileged or otherwise protected from disclosure, distributing >>> or >>> > >> copying. >>> > >> > > Any views or opinions presented in this email are solely those >>> of the >>> > >> > > author and do not necessarily represent those of Spicule >>> Limited. The >>> > >> > > company accepts no liability for any damage caused by any virus >>> > >> transmitted >>> > >> > > by this email. If you have received this message in error, >>> please >>> > >> notify us >>> > >> > > immediately by reply email before deleting it from your system. >>> > >> Service of >>> > >> > > legal notice cannot be effected on Spicule Limited by email. >>> > >> > > >>> > >> > > >>> > >> > > Spicule Limited is registered in England & Wales. Company >>> Number: >>> > >> > > 09954122. Registered office: First Floor, Telecom House, >>> 125-135 >>> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> >>> >>> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> >>> >> Preston >>> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> >>> >>> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> >>> >> > > Road, Brighton, England, BN1 6AF >>> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>. >>> VAT No. 251478891. >>> > >> > > >>> > >> > > >>> > >> > > All engagements are subject to Spicule Terms and Conditions of >>> > >> Business. >>> > >> > > This email and its contents are intended solely for the >>> individual to >>> > >> whom >>> > >> > > it is addressed and may contain information that is >>> confidential, >>> > >> > > privileged or otherwise protected from disclosure, distributing >>> or >>> > >> copying. >>> > >> > > Any views or opinions presented in this email are solely those >>> of the >>> > >> > > author and do not necessarily represent those of Spicule >>> Limited. The >>> > >> > > company accepts no liability for any damage caused by any virus >>> > >> transmitted >>> > >> > > by this email. If you have received this message in error, >>> please >>> > >> notify us >>> > >> > > immediately by reply email before deleting it from your system. >>> > >> Service of >>> > >> > > legal notice cannot be effected on Spicule Limited by email. >>> > >> > > >>> > >> > > >>> > >> > > Spicule Limited is registered in England & Wales. Company >>> Number: >>> > >> > > 09954122. Registered office: First Floor, Telecom House, >>> 125-135 >>> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> >>> >>> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> >>> >> Preston >>> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> >>> >>> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> >>> >> > > Road, Brighton, England, BN1 6AF >>> <https://www.google.com/maps/search/125-135+%0D%0A+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>. >>> VAT No. 251478891. >>> > >> > > >>> > >> > > >>> > >> > > All engagements are subject to Spicule Terms and Conditions of >>> > >> Business. >>> > >> > > This email and its contents are intended solely for the >>> individual to >>> > >> whom >>> > >> > > it is addressed and may contain information that is >>> confidential, >>> > >> > > privileged or otherwise protected from disclosure, distributing >>> or >>> > >> copying. >>> > >> > > Any views or opinions presented in this email are solely those >>> of the >>> > >> > > author and do not necessarily represent those of Spicule >>> Limited. The >>> > >> > > company accepts no liability for any damage caused by any virus >>> > >> transmitted >>> > >> > > by this email. If you have received this message in error, >>> please >>> > >> notify us >>> > >> > > immediately by reply email before deleting it from your system. >>> > >> Service of >>> > >> > > legal notice cannot be effected on Spicule Limited by email. >>> > >> > > >>> > >> > >>> > >> > -- >>> > >> > >>> > >> > >>> > >> > Spicule Limited is registered in England & Wales. Company Number: >>> > >> > 09954122. Registered office: First Floor, Telecom House, 125-135 >>> Preston >>> <https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> >>> >>> <https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> >>> >> > Road, Brighton, England, BN1 6AF >>> <https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>. >>> VAT No. 251478891. >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > All engagements >>> > >> > are subject to Spicule Terms and Conditions of Business. This >>> email and >>> > >> its >>> > >> > contents are intended solely for the individual to whom it is >>> addressed >>> > >> and >>> > >> > may contain information that is confidential, privileged or >>> otherwise >>> > >> > protected from disclosure, distributing or copying. Any views or >>> > >> opinions >>> > >> > presented in this email are solely those of the author and do not >>> > >> > necessarily represent those of Spicule Limited. The company >>> accepts no >>> > >> > liability for any damage caused by any virus transmitted by this >>> email. >>> > >> If >>> > >> > you have received this message in error, please notify us >>> immediately by >>> > >> > reply email before deleting it from your system. Service of legal >>> notice >>> > >> > cannot be effected on Spicule Limited by email. >>> > >> > >>> > >> >>> > >> >>> > >> Spicule Limited is registered in England & Wales. Company Number: >>> > >> 09954122. Registered office: First Floor, Telecom House, 125-135 >>> Preston >>> <https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> >>> >>> <https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> >>> >> Road, Brighton, England, BN1 6AF >>> <https://www.google.com/maps/search/125-135+Preston+%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>. >>> VAT No. 251478891. >>> > >> >>> > >> >>> > >> All engagements are subject to Spicule Terms and Conditions of >>> Business. >>> > >> This email and its contents are intended solely for the individual >>> to whom >>> > >> it is addressed and may contain information that is confidential, >>> > >> privileged or otherwise protected from disclosure, distributing or >>> copying. >>> > >> Any views or opinions presented in this email are solely those of >>> the >>> > >> author and do not necessarily represent those of Spicule Limited. >>> The >>> > >> company accepts no liability for any damage caused by any virus >>> transmitted >>> > >> by this email. If you have received this message in error, please >>> notify us >>> > >> immediately by reply email before deleting it from your system. >>> Service of >>> > >> legal notice cannot be effected on Spicule Limited by email. >>> > >> >>> > > >>> > >>> > -- >>> > >>> > >>> > Spicule Limited is registered in England & Wales. Company Number: >>> > 09954122. Registered office: First Floor, Telecom House, 125-135 >>> Preston >>> <https://www.google.com/maps/search/125-135+Preston++%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g> >>> >>> <https://www.google.com/maps/search/125-135+Preston++%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>> >>> Road, Brighton, England, BN1 6AF >>> <https://www.google.com/maps/search/125-135+Preston++%0D%0A+Road,+Brighton,+England,+BN1+6AF?entry=gmail&source=g>. >>> VAT No. 251478891. >>> > >>> > >>> > >>> > >>> > All engagements >>> > are subject to Spicule Terms and Conditions of Business. This email >>> and its >>> > contents are intended solely for the individual to whom it is >>> addressed and >>> > may contain information that is confidential, privileged or otherwise >>> > protected from disclosure, distributing or copying. Any views or >>> opinions >>> > presented in this email are solely those of the author and do not >>> > necessarily represent those of Spicule Limited. The company accepts no >>> > liability for any damage caused by any virus transmitted by this >>> email. If >>> > you have received this message in error, please notify us immediately >>> by >>> > reply email before deleting it from your system. Service of legal >>> notice >>> > cannot be effected on Spicule Limited by email. >>> > >>> >> -- > Twitter: https://twitter.com/holdenkarau > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau >
