geserdugarov commented on issue #12133:
URL: https://github.com/apache/hudi/issues/12133#issuecomment-2431975627
Sorry, I wrote wrong version of Spark first.
When I tried to build Hudi with `-Dspark3.5`, I faced
```Text
An exception or error caused a run to abort:
org.apache.spark.sql.catalyst.plans.logical.InsertIntoStatement.copy$default$7()Z
java.lang.NoSuchMethodError:
org.apache.spark.sql.catalyst.plans.logical.InsertIntoStatement.copy$default$7()Z
at
org.apache.spark.sql.HoodieSpark3CatalystPlanUtils.rebaseInsertIntoStatement(HoodieSpark3CatalystPlanUtils.scala:70)
at
org.apache.spark.sql.HoodieSpark3CatalystPlanUtils.rebaseInsertIntoStatement$(HoodieSpark3CatalystPlanUtils.scala:69)
at
org.apache.spark.sql.HoodieSpark34CatalystPlanUtils$.rebaseInsertIntoStatement(HoodieSpark34CatalystPlanUtils.scala:33)
at
org.apache.spark.sql.hudi.analysis.HoodieAnalysis$AdaptIngestionTargetLogicalRelations$$anonfun$$nestedInanonfun$apply$1$1.applyOrElse(HoodieAnalysis.scala:314)
...
```
As you can see, there calling of `HoodieSpark34CatalystPlanUtils`. So, I
didn't check the reason, and switched to `-Dspark3.4` to skip it.
I'm using Ubuntu 22.04, and run modified test in IntelliJ IDEA. To prevent
removing of data files, I placed breakpoint right after the second insert, and
check filesystem in debug mode.
Rechecked mentioned test again for commit
66597e5080d8ecd071f79473a27117f19f2e793b, and still see:
```Bash
/tmp/spark-bee1984c-74df-4067-8e14-09a61e02e0c8$ tree -a
.
├── dt=2021-01-05
│ ├──
00000001-9e90-410e-bdf3-4ea189ba93ac-0_1-14-12_20241023121839934.parquet
│ ├──
.00000001-9e90-410e-bdf3-4ea189ba93ac-0_1-14-12_20241023121839934.parquet.crc
│ ├──
.00000001-9e90-410e-bdf3-4ea189ba93ac-0_20241023121845855.log.1_0-30-31
│ ├──
..00000001-9e90-410e-bdf3-4ea189ba93ac-0_20241023121845855.log.1_0-30-31.crc
│ ├── .hoodie_partition_metadata
│ └── ..hoodie_partition_metadata.crc
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
