dossett commented on pull request #29231:
URL: https://github.com/apache/spark/pull/29231#issuecomment-710061815
Running on GCP's dataproc 2.0:
```
aniskodedossett@dossett-delta-w-0:~$ spark-sql --version
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.0.1
/_/
Using Scala version 2.12.12, OpenJDK 64-Bit Server VM, 1.8.0_265
Branch HEAD
Compiled by user on 2020-09-17T10:01:54Z
Revision 0aee93de8ef2a90403093b91843de9777b7ab5ef
Url https://bigdataoss-internal.googlesource.com/third_party/apache/spark
Type --help for more information.
```
I'm playing with the Databricks Delta Lake a simple `vacuum` command fails
with a long stack trace with this at the bottom:
```
Caused by: java.lang.NoSuchFieldError: numNonEmptyBlocks
at
org.apache.spark.scheduler.HighlyCompressedMapStatus.<init>(MapStatus.scala:174)
at
org.apache.spark.scheduler.HighlyCompressedMapStatus$.apply(MapStatus.scala:269)
at org.apache.spark.scheduler.MapStatus$.apply(MapStatus.scala:70)
at
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:71)
at
org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
If this is helpful I'm happy to provide more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
