eejbyfeldt commented on PR #41943:
URL: https://github.com/apache/spark/pull/41943#issuecomment-1658336924
> I think the issue is you will target Java 17 bytecode if running on 17,
when we want to target 8 in all cases
If that is the case then the changes currently in this PR are not what we
want. But are we really sure that this is something that is expected or used?
Because as far as I can tell this is not something that actually worked in the
passed. If I take a spark 3.4.1 build that I build using the v3.4.1 tag on Java
11 and then try to run `spark-submit run-example SparkPi` on Java 8 it fails
with
```
2023-07-31 14:59:15,304 INFO scheduler.DAGScheduler: ResultStage 0 (reduce
at SparkPi.scala:38) failed in 0.259 s due to Job aborted due to stage failure:
Task serialization failed: java.lang.NoSuchMethodError:
java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
at
org.apache.spark.util.io.ChunkedByteBufferOutputStream.toChunkedByteBuffer(ChunkedByteBufferOutputStream.scala:115)
at
org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:362)
at
org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:160)
at
org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:99)
at
org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:38)
at
org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:78)
at
org.apache.spark.SparkContext.broadcastInternal(SparkContext.scala:1548)
at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1530)
at
org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1535)
at
org.apache.spark.scheduler.DAGScheduler.submitStage(DAGScheduler.scala:1353)
at
org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:1295)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2931)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2923)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2912)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
```
I think the problem here boils down to that we previously have used the
scalac arg `--target` to attempt to achieve what you describe. But according to
the comment here
(https://github.com/scala/bug/issues/12643#issuecomment-1253761646)
> -target says "emit class file of version N, but I want to use arbitrary
classes from the JDK and take my chances".
so only specifying `-target` is not the proper way to build on a later Java
version and target Java 8. My understanding is that if that is what we actually
want then we would need specify the java version using `-release` and actually
fix the build errors that it causes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
