[
https://issues.apache.org/jira/browse/HUDI-3726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516181#comment-17516181
]
Rajesh commented on HUDI-3726:
------------------------------
Some of the tests around spark-sql are failing
{code:java}
2022-04-01T22:52:11.8369699Z 1360949 [ScalaTest-main-running-TestCreateTable]
WARN org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was
not found at path
/tmp/spark-12cc7375-e24a-4f40-9648-94c2ba6d29cf/h12/.hoodie/metadata
2022-04-01T22:52:15.1508736Z - Test Create Table From Existing Hoodie Table For
Multi-Level Partitioned Table *** FAILED ***
2022-04-01T22:52:15.1509783Z org.apache.hudi.exception.HoodieException:
Config conflict(key current value existing value):
2022-04-01T22:52:15.1510463Z KeyGenerator:
org.apache.hudi.keygen.SimpleKeyGenerator
org.apache.hudi.keygen.ComplexKeyGenerator
2022-04-01T22:52:15.1511169Z at
org.apache.hudi.HoodieWriterUtils$.validateTableConfig(HoodieWriterUtils.scala:166)
2022-04-01T22:52:15.1511918Z at
org.apache.spark.sql.catalyst.catalog.HoodieCatalogTable.parseSchemaAndConfigs(HoodieCatalogTable.scala:211)
2022-04-01T22:52:15.1512711Z at
org.apache.spark.sql.catalyst.catalog.HoodieCatalogTable.initHoodieTable(HoodieCatalogTable.scala:169)
2022-04-01T22:52:15.1513469Z at
org.apache.spark.sql.hudi.command.CreateHoodieTableCommand.run(CreateHoodieTableCommand.scala:65)
2022-04-01T22:52:15.1514247Z at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
2022-04-01T22:52:15.1515004Z at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
2022-04-01T22:52:15.1515737Z at
org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
{code}
{code:java}
val datasourceKeyGen = getOriginKeyGenerator(params)
val tableConfigKeyGen =
tableConfig.getString(HoodieTableConfig.KEY_GENERATOR_CLASS_NAME)
if (null != datasourceKeyGen && null != tableConfigKeyGen
&& datasourceKeyGen != tableConfigKeyGen) {
diffConfigs.append(s"KeyGenerator:\t$datasourceKeyGen\t$tableConfigKeyGen\n")
} {code}
The code is expecting datasourceKeyGen to be null, which is what we changed to
default for HUDI-3726
> Switching from non-partitioned to partitioned key gen does not throw any
> exception
> ----------------------------------------------------------------------------------
>
> Key: HUDI-3726
> URL: https://issues.apache.org/jira/browse/HUDI-3726
> Project: Apache Hudi
> Issue Type: Bug
> Components: spark
> Reporter: sivabalan narayanan
> Assignee: Rajesh
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.11.0
>
> Attachments: Screen Shot 2022-04-01 at 12.28.12 PM.png
>
>
> in commit C1, if non-partitioned key gen is used and for commit C2, if key
> gen option is missed, we fallback to simple key gen and the write succeeds. I
> was expecting to throw exception if key gen class is changed.
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
