Seth Fitzsimmons created SEDONA-211:
---------------------------------------
Summary: ShapeSerde.putHeader() doesn't work with Java 8
Key: SEDONA-211
URL: https://issues.apache.org/jira/browse/SEDONA-211
Project: Apache Sedona
Issue Type: Bug
Affects Versions: 1.3.0
Reporter: Seth Fitzsimmons
When deploying an app built with Sedona 1.3.0-incubating to AWS EMR Serverless
(which only supports Java 8), executors that indirectly use the {{ShapeSerde}}
(e.g., by Kryo serializing a geometry column) crash with this error:
{{java.lang.NoSuchMethodError:
java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;}}
{{at
org.apache.sedona.core.formatMapper.shapefileParser.parseUtils.shp.ShapeSerde.putHeader(ShapeSerde.java:163)}}
{{at
org.apache.sedona.core.formatMapper.shapefileParser.parseUtils.shp.ShapeSerde.serialize(ShapeSerde.java:149)}}
{{at
org.apache.sedona.core.formatMapper.shapefileParser.parseUtils.shp.ShapeSerde.serialize(ShapeSerde.java:73)}}
{{at
org.apache.sedona.core.geometryObjects.GeometrySerde.writeGeometry(GeometrySerde.java:104)}}
{{at
org.apache.sedona.core.geometryObjects.GeometrySerde.write(GeometrySerde.java:72)}}
{{at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)}}
{{at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:21)}}
{{at com.twitter.chill.SomeSerializer.write(SomeSerializer.scala:19)}}
{{at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)}}
{{at
com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79)}}
{{at
com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)}}
{{at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)}}
{{at
org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:270)}}
{{at
org.apache.spark.serializer.SerializationStream.writeValue(Serializer.scala:145)}}
{{at
org.apache.spark.serializer.DefaultSerializedRecordAppender.process(SerializedRecordAppender.scala:44)}}
{{at
org.apache.spark.shuffle.sort.UnsafeShuffleWriter.insertRecordIntoSorter(UnsafeShuffleWriter.java:242)}}
{{at
org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:184)}}
{{at
org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)}}
{{at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)}}
{{at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)}}
{{at org.apache.spark.scheduler.Task.run(Task.scala:138)}}
{{at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)}}
{{at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1516)}}
{{at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)}}
{{at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
{{at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
{{at java.lang.Thread.run(Thread.java:750)}}
>From Spark's environment view:
Java Home /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.342.b07-1.amzn2.0.1.x86_64/jre
Java Version 1.8.0_342 (Red Hat, Inc.)
Scala Version version 2.12.15
This appears to be related to {{maven-compiler-plugin}} not doing the right
thing ([{{source}} and {{target}} are both set to
{{8}}|https://repo1.maven.org/maven2/org/apache/sedona/sedona-core-3.0_2.12/1.3.0-incubating/sedona-core-3.0_2.12-1.3.0-incubating.pom]).
Perhaps
[{{release}}|https://maven.apache.org/plugins/maven-compiler-plugin/examples/set-compiler-release.html]
should be used instead?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
