I am on 0.6.0
thx,a.
On Friday, 24 March 2017, 8:58, Jean-Baptiste Onofré <[email protected]>
wrote:
Hi Antony,
which Beam version are you using ? We did some improvement about guava shading
recently, wanted to check if it's related.
Regards
JB
On 03/24/2017 08:03 AM, Antony Mayi wrote:
> Hi,
>
> I am using guava's collections (immutables from 21.0) in my beam pipelines and
> when running on spark runner it fails due to kryo unable to serialize those. I
> can see there have been some approaches addressing this using
> de.javakaffee.kryo-serializers
> ->
> https://github.com/apache/beam/commits/master/runners/spark/src/main/java/org/apache/beam/runners/spark/coders/BeamSparkRunnerRegistrator.java
> but
> that's been removed recently.
>
> how should I solve this?
>
> the stacktrace is bellow.
>
> thanks,
> antony.
>
>
> [WARNING]
> java.lang.RuntimeException: org.apache.spark.SparkException: Job aborted due
> to
> stage failure: Exception while getting task result:
> com.esotericsoftware.kryo.KryoException:
> java.lang.UnsupportedOperationException
> Serialization trace:
> fields (my.pkg.types.MyType)
> value (org.apache.beam.sdk.util.WindowedValue$ValueInGlobalWindow)
> at
> org.apache.beam.runners.spark.SparkPipelineResult.runtimeExceptionFrom(SparkPipelineResult.java:60)
> at
> org.apache.beam.runners.spark.SparkPipelineResult.beamExceptionFrom(SparkPipelineResult.java:77)
> at
> org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:113)
> at
> org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:102)
> at my.pkg.Main.main(Main.java:33)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
> at java.lang.Thread.run(Thread.java:745)
>
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
