[
https://issues.apache.org/jira/browse/BEAM-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16155311#comment-16155311
]
Al Yaros commented on BEAM-2837:
--------------------------------
Hey,
I didn't saw anything else in the error.
I also think that the reason is dependencies issue but I have tried all kind of
combination of versions (Include other and Exclude others) but still it refused
to work.
You can look on the GitHub code (Link added) that this is very simple pipeline
that even if the pom only includes:
{code:java}
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>2.1.0</version>
</dependency>
{code}
It doesn't work.
I verified that the Project\Instance\DB\Table\Columns all exists in Spanner.
I'm running it after I ran "gcloud auth application-default login" from the
Cloud SDK Console so It's not a manner of Authentication. (It starts the
pipeline in DataFlow)
I have tried to Exclude Spanner from the
"beam-runners-google-cloud-dataflow-java" and include the official spanner Java
driver (in different versions) and still it didn't worked.
Anyway, I'll be happy if you can share an example (Simple Code, Pom Etc..) or
Versions I should use to be able to write to Spanner from GCP DataFlow.
Thanks
> Writing To Spanner From Google Cloud DataFlow - Failure
> -------------------------------------------------------
>
> Key: BEAM-2837
> URL: https://issues.apache.org/jira/browse/BEAM-2837
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
> Affects Versions: 2.1.0
> Environment: Google Cloud DataFlow
> Reporter: Al Yaros
> Assignee: Mairbek Khadikov
>
> Simple Java Program That reads from Pub\Sub and Writes to Spanner Fails with
> cryptic error message.
> Simple Program to Demonstrate the Error:
> [https://github.com/alyaros/ExamplePubSubToSpannerViaDataFlow]
> {code:java}
> *Caused by: org.apache.beam.sdk.util.UserCodeException:
> java.lang.NoClassDefFoundError: Could not initialize class
> com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor
>
> org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
> org.apache.beam.sdk.io.
> gcp.spanner.SpannerWriteGroupFn$DoFnInvoker.invokeSetup(Unknown Source)
>
> com.google.cloud.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.deserializeCopy(DoFnInstanceManagers.java:66)
>
> com.google.cloud.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.peek(DoFnInstanceManagers.java:48)
>
> com.google.cloud.dataflow.worker.UserParDoFnFactory.create(UserParDoFnFactory.java:104)
>
> com.google.cloud.dataflow.worker.DefaultParDoFnFactory.create(DefaultParDoFnFactory.java:66)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory.createParDoOperation(MapTaskExecutorFactory.java:360)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory$3.typedApply(MapTaskExecutorFactory.java:271)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory$3.typedApply(MapTaskExecutorFactory.java:253)
>
> com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:55)
>
> com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:43)
>
> com.google.cloud.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:78)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory.create(MapTaskExecutorFactory.java:142)
>
> com.google.cloud.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:925)
>
> com.google.cloud.dataflow.worker.StreamingDataflowWorker.access$800(StreamingDataflowWorker.java:133)
>
> com.google.cloud.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:771)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> java.lang.Thread.run(Thread.java:745)*
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
