[
https://issues.apache.org/jira/browse/BEAM-6249?focusedWorklogId=179232&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-179232
]
ASF GitHub Bot logged work on BEAM-6249:
----------------------------------------
Author: ASF GitHub Bot
Created on: 27/Dec/18 21:29
Start Date: 27/Dec/18 21:29
Worklog Time Spent: 10m
Work Description: chamikaramj commented on issue #7365: [BEAM-6249] Adds
an Streaming Engine based test to release validation
URL: https://github.com/apache/beam/pull/7365#issuecomment-450234587
R: @yifanzou
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 179232)
Time Spent: 20m (was: 10m)
> Vendored gRPC doesn't seem to work with dataflow
> ------------------------------------------------
>
> Key: BEAM-6249
> URL: https://issues.apache.org/jira/browse/BEAM-6249
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
> Affects Versions: 2.9.0
> Reporter: Steve Niemitz
> Assignee: Tyler Akidau
> Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I attempted to migrate an existing pipeline (that worked in 2.8.0) to 2.9.0.
> This pipeline is using the experimental streaming engine
> (–experiments=enable_streaming_engine).
> The pipeline fails to start with these logs:
> {code:java}
> D Unable to load the library
> 'org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_64', trying
> other loading mechanism.
> D org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_64 cannot be
> loaded from java.libary.path, now trying export to -Dio.netty.native.workdir:
> /tmp
> D Unable to load the library
> '/tmp/liborg_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_646918605450681921540.so',
> trying other loading mechanism.
> D Unable to load the library 'netty_tcnative_linux_x86_64', trying next
> name...
> D Unable to load the library
> 'org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_64_fedora',
> trying other loading mechanism.
> D org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_64_fedora
> cannot be loaded from java.libary.path, now trying export to
> -Dio.netty.native.workdir: /tmp
> D Unable to load the library 'netty_tcnative_linux_x86_64_fedora', trying
> next name...
> D Unable to load the library
> 'org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_x86_64', trying other
> loading mechanism.
> D org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_x86_64 cannot be loaded
> from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
> D Unable to load the library 'netty_tcnative_x86_64', trying next name...
> D Unable to load the library
> 'org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative', trying other loading
> mechanism.
> D org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative cannot be loaded from
> java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
> D Unable to load the library 'netty_tcnative', trying next name...
> D Failed to load netty-tcnative; OpenSslEngine will be unavailable, unless
> the application has already loaded the symbols by some other means. See
> http://netty.io/wiki/forked-tomcat-native.html for more information.
> D Failed to initialize netty-tcnative; OpenSslEngine will be unavailable.
> See http://netty.io/wiki/forked-tomcat-native.html for more information.
> I netty-tcnative unavailable (this may be normal)
> I Conscrypt not found (this may be normal)
> I Jetty ALPN unavailable (this may be normal)
> E Uncaught exception in main thread. Exiting with status code 1.
> W Please use a logger instead of System.out or System.err.
> Please switch to using org.slf4j.Logger.
> See: https://cloud.google.com/dataflow/pipelines/logging
> E Uncaught exception in main thread. Exiting with status code 1.
> E java.lang.IllegalStateException: Could not find TLS ALPN provider; no
> working netty-tcnative, Conscrypt, or Jetty NPN/ALPN available
> E at
> org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:256)
>
> E at
> org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171)
>
> E at
> org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:120)
>
> E at
> org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer.remoteChannel(GrpcWindmillServer.java:343)
>
> E at
> org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer.initializeWindmillService(GrpcWindmillServer.java:312)
>
> {code}
>
> The interesting part is in the netty load failure, the stack trace is:
> {code:java}
> exception: "java.lang.UnsatisfiedLinkError at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:276)
> at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:233)
> at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:187)
> at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:85)
> at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:430)
> at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:97)
> at
> org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:242)
> at
> org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171)
> at
> org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:120)
> at
> org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer.remoteChannel(GrpcWindmillServer.java:343)
> at
> org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer.initializeWindmillService(GrpcWindmillServer.java:312)
> at
> org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer.setWindmillServiceEndpoints(GrpcWindmillServer.java:192)
> at
> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.getConfigFromDataflowService(StreamingDataflowWorker.java:1528)
> at
> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.getConfig(StreamingDataflowWorker.java:1583)
> at
> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.getGlobalConfig(StreamingDataflowWorker.java:1568)
> at
> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.schedulePeriodicGlobalConfigRequests(StreamingDataflowWorker.java:1543)
> at
> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.start(StreamingDataflowWorker.java:704)
> at
> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.main(StreamingDataflowWorker.java:228)
> Caused by: java.lang.reflect.InvocationTargetException at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:263)
> at java.security.AccessController.doPrivileged(Native Method) at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:255)
> ... 17 more Caused by: java.lang.NoClassDefFoundError:
> org/apache/beam/vendor/grpc/v1/13/1/io/netty/internal/tcnative/Library at
> java.lang.ClassLoader$NativeLibrary.load(Native Method) at
> java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) at
> java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) at
> java.lang.Runtime.load0(Runtime.java:809) at
> java.lang.System.load(System.java:1086) at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
> ... 24 more Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.vendor.grpc.v1.13.1.io.netty.internal.tcnative.Library at
> java.net.URLClassLoader.findClass(URLClassLoader.java:381) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 30 more{code}
>
> Notice that the class attempting to be loaded is:
> org.apache.beam.vendor.grpc.v1.13.1.io.netty.internal.tcnative.Library, but
> it's actually defined in the jar as
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.internal.tcnative.Library.
> I traced this back to the jni interop code in tcnative:
> [https://github.com/netty/netty-tcnative/blob/master/openssl-dynamic/src/main/c/jnilib.c#L266]
> Here it replaces all _ in the package prefix with /, which won't work here.
> The fix seems like it would be to repackage the vendored gRPC with a
> different prefix that doesn't contain underscores.
> I'm curious how this ever worked though? Maybe the streaming engine is the
> only thing using this vendored gRPC code?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
