rangareddy commented on issue #17353:
URL: https://github.com/apache/hudi/issues/17353#issuecomment-5351237454
This issue was reviewed as part of the JIRA-migrated backlog triage
(HUDI-8813).
**Findings: confirmed, still blocked on `master`.**
The block is a single condition in `CreateIndexCommand.run`,
`hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/IndexCommands.scala`:
```scala
if (indexType.equals(HoodieTableMetadataUtil.PARTITION_NAME_COLUMN_STATS) &&
options.asJava.getOrDefault(EXPRESSION_OPTION,
ExpressionIndexSparkFunctions.IDENTITY_FUNCTION).equals(ExpressionIndexSparkFunctions.IDENTITY_FUNCTION))
{
throw new HoodieIndexException("Column stats index without expression on
any column can be created using datasource configs. " +
"Please refer https://hudi.apache.org/docs/metadata for more info")
}
```
One detail worth flagging for whoever implements this: because the default
passed to `getOrDefault` is `IDENTITY_FUNCTION`, "the user supplied no
expression" and "the user explicitly supplied identity" are indistinguishable
at this point. If identity becomes supported, those two cases probably want to
stay distinguishable, which means the option lookup needs restructuring rather
than just deleting the `throw`.
The transform itself is already a first-class concept in common code -
`hudi-common/src/main/java/org/apache/hudi/core/index/expression/HoodieExpressionIndex.java:54`
declares `String IDENTITY_TRANSFORM = "identity"` - so this is a SQL-layer
restriction rather than a missing capability underneath.
This is the same code block that blocks #17352 (HUDI-8812). Recommend fixing
them as one change.
Keeping this open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
