[
https://issues.apache.org/jira/browse/HUDI-8519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sagar Sumit updated HUDI-8519:
------------------------------
Status: In Progress (was: Open)
> Update failed with mismatched partition path if mutliple secondary indexes
> present
> ----------------------------------------------------------------------------------
>
> Key: HUDI-8519
> URL: https://issues.apache.org/jira/browse/HUDI-8519
> Project: Apache Hudi
> Issue Type: Task
> Reporter: Sagar Sumit
> Assignee: Sagar Sumit
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.0.0
>
>
> After building an index on {{rider}} and {{driver}} columns, an update with
> fails with {{mismatched partition path, record partition:}}
> {code:java}
> 24/11/13 21:12:27 ERROR SparkSQLDriver: Failed in [
> update hudi_table set rider = 'rider-E' where rider = 'rider-F']
> org.apache.hudi.exception.HoodieException: Failed to update metadata
> at
> org.apache.hudi.client.BaseHoodieClient.writeTableMetadata(BaseHoodieClient.java:295)
> at
> org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:294)
> at
> org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:244)
> at
> org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:93)
> at
> org.apache.hudi.HoodieSparkSqlWriterInternal.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:991)
> at
> org.apache.hudi.HoodieSparkSqlWriterInternal.writeInternal(HoodieSparkSqlWriter.scala:533)
> at
> org.apache.hudi.HoodieSparkSqlWriterInternal.$anonfun$write$1(HoodieSparkSqlWriter.scala:190)
> at
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:125)
> ...
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:248)
> at
> org.apache.spark.sql.hudi.command.UpdateHoodieTableCommand.run(UpdateHoodieTableCommand.scala:79)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
> ...
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: org.apache.spark.SparkException: Job aborted due to stage failure:
> Task 5 in stage 154.0 failed 1 times, most recent failure: Lost task 5.0 in
> stage 154.0 (TID 495) (192.168.68.107 executor driver):
> org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType
> UPDATE for partition :5
> at
> org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:317)
> at
> org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.lambda$mapPartitionsAsRDD$a3ab3c4$1(BaseSparkCommitActionExecutor.java:250)
> at
> org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1(JavaRDDLike.scala:102)
> at
> org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1$adapted(JavaRDDLike.scala:102)
> ...
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> Caused by: org.apache.hudi.exception.HoodieException: Error writing record
> HoodieRecord{key=HoodieKey { recordKey=driver-P$trip4
> partitionPath=secondary_index_idx_driver},
> currentLocation='HoodieRecordLocation {instantTime=20241113211226184,
> fileId=secondary-index-0004-0, position=-1}', newLocation='null'}
> at
> org.apache.hudi.client.FailOnFirstErrorWriteStatus.markFailure(FailOnFirstErrorWriteStatus.java:45)
> at
> org.apache.hudi.io.HoodieAppendHandle.writeToBuffer(HoodieAppendHandle.java:601)
> at
> org.apache.hudi.io.HoodieAppendHandle.doAppend(HoodieAppendHandle.java:469)
> at
> org.apache.hudi.table.action.deltacommit.BaseSparkDeltaCommitActionExecutor.handleUpdate(BaseSparkDeltaCommitActionExecutor.java:83)
> at
> org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:310)
> ... 33 more
> Caused by: org.apache.hudi.exception.HoodieUpsertException: mismatched
> partition path, record partition: secondary_index_idx_driver but trying to
> insert into partition: secondary_index_idx_rider
> at
> org.apache.hudi.io.HoodieAppendHandle.writeToBuffer(HoodieAppendHandle.java:600)
> ... 36 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
