LuciferYang commented on a change in pull request #34364:
URL: https://github.com/apache/spark/pull/34364#discussion_r734267797
##########
File path:
external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
##########
@@ -1071,7 +1071,8 @@ abstract class AvroSuite
.write.format("avro").option("avroSchema", avroSchema)
.save(s"$tempDir/${UUID.randomUUID()}")
}.getCause.getMessage
- assert(message.contains("Caused by: java.lang.NullPointerException: " +
+ assert(message.contains("Caused by: java.lang.NullPointerException: "))
+ assert(message.contains(
Review comment:
The whole message as follows:
```
"Job aborted due to stage failure: Task 1 in stage 144.0 failed 1 times,
most recent failure: Lost task 1.0 in stage 144.0 (TID 250) (localhost executor
driver): org.apache.spark.SparkException: Task failed while writing rows.
at
org.apache.spark.sql.errors.QueryExecutionErrors$.taskFailedWhileWritingRowsError(QueryExecutionErrors.scala:516)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:345)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$20(FileFormatWriter.scala:252)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:136)
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:507)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1468)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:510)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException:
java.lang.NullPointerException: Cannot invoke "Object.getClass()" because
"datum" is null of string in string in field Name of test_schema in test_schema
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:317)
at
org.apache.spark.sql.avro.SparkAvroKeyRecordWriter.write(SparkAvroKeyOutputFormat.java:84)
at
org.apache.spark.sql.avro.SparkAvroKeyRecordWriter.write(SparkAvroKeyOutputFormat.java:62)
at
org.apache.spark.sql.avro.AvroOutputWriter.write(AvroOutputWriter.scala:84)
at
org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.write(FileFormatDataWriter.scala:175)
at
org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithMetrics(FileFormatDataWriter.scala:85)
at
org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithIterator(FileFormatDataWriter.scala:92)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:328)
at
org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1502)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:335)
... 9 more
Caused by: java.lang.NullPointerException: Cannot invoke "Object.getClass()"
because "datum" is null of string in string in field Name of test_schema in
test_schema
at
org.apache.avro.generic.GenericDatumWriter.npe(GenericDatumWriter.java:184)
at
org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:160)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:314)
... 18 more
Caused by: java.lang.NullPointerException: Cannot invoke "Object.getClass()"
because "datum" is null
at
org.apache.avro.specific.SpecificDatumWriter.writeString(SpecificDatumWriter.java:68)
at
org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:151)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83)
at
org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:158)
at
org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:221)
at
org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:101)
at
org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:184)
at
org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:210)
at
org.apache.avro.specific.SpecificDatumWriter.writeRecord(SpecificDatumWriter.java:83)
at
org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:131)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83)
at
org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:158)
... 20 more
```
so we can't assert with `startWith` and `endWith`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
