pramodbiligiri commented on PR #6665:
URL: https://github.com/apache/hudi/pull/6665#issuecomment-1260496714
Noting down an issue I noticed where the two tests I've added only work on
one of our Spark profiles, and moreover, each of them works on a different
Spark profile :|
1. If I run TestGCSEventsSource on spark-3.2 instead of spark2 profile, I
see below error:
$ mvn -Dspark2 -Dscala-2.12 -Dcheckstyle.skip -Drat.skip
-Dtest=org.apache.hudi.utilities.sources.TestGcsEventsSource -pl
hudi-utilities test
```
[ERROR] org.apache.hudi.utilities.sources.TestGcsEventsSource Time elapsed:
3.433 s <<< ERROR!
java.lang.InstantiationError: org.apache.hadoop.hdfs.protocol.HdfsFileStatus
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.<clinit>(FSDirectory.java:136)
```
2. If I run TestGcsEventsHoodieIncrSource under spark2 instead of spark3.2,
i see the below error:
$ mvn -Dspark3.2 -Dscala-2.12 -Dcheckstyle.skip -Drat.skip
-Dtest=org.apache.hudi.utilities.sources.TestGcsEventsHoodieIncrSource -pl
hudi-utilities test
```
[ERROR] shouldNotFindNewDataIfCommitTimeOfWriteAndReadAreEqual Time
elapsed: 32.22 s <<< ERROR!
org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit
time 1
at
org.apache.hudi.table.action.commit.BaseWriteHelper.write(BaseWriteHelper.java:64)
at
org.apache.hudi.table.action.commit.SparkUpsertCommitActionExecutor.execute(SparkUpsertCommitActionExecutor.java:45)
at
org.apache.hudi.table.HoodieSparkCopyOnWriteTable.upsert(HoodieSparkCopyOnWriteTable.java:113)
at
org.apache.hudi.table.HoodieSparkCopyOnWriteTable.upsert(HoodieSparkCopyOnWriteTable.java:97)
at
org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:153)
at
org.apache.hudi.utilities.sources.TestGcsEventsHoodieIncrSource.writeGcsMetadataRecords(TestGcsEventsHoodieIncrSource.java:227)
...snipped...
Caused by: java.lang.ClassNotFoundException:
org.apache.avro.AvroMissingFieldException
at
[java.net](http://java.net/).URLClassLoader.findClass(URLClassLoader.java:382)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
