rangareddy commented on issue #17352:
URL: https://github.com/apache/hudi/issues/17352#issuecomment-5351237219
This issue was reviewed as part of the JIRA-migrated backlog triage
(HUDI-8812).
**Findings: confirmed, still not supported on `master`.**
In
`hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/IndexCommands.scala`,
`CreateIndexCommand.run` handles exactly four cases, and the three index types
in this ticket are not among the supported ones:
```scala
if (indexType.equals(HoodieTableMetadataUtil.PARTITION_NAME_COLUMN_STATS)
|| indexType.equals(HoodieTableMetadataUtil.PARTITION_NAME_BLOOM_FILTERS))
{
if (indexType.equals(HoodieTableMetadataUtil.PARTITION_NAME_COLUMN_STATS)
&&
options.asJava.getOrDefault(EXPRESSION_OPTION,
ExpressionIndexSparkFunctions.IDENTITY_FUNCTION).equals(ExpressionIndexSparkFunctions.IDENTITY_FUNCTION))
{
throw new HoodieIndexException("Column stats index without expression on
any column can be created using datasource configs. " +
"Please refer https://hudi.apache.org/docs/metadata for more info")
}
...
} else if
(indexName.equals(HoodieTableMetadataUtil.PARTITION_NAME_RECORD_INDEX)) { ...
} else if (StringUtils.isNullOrEmpty(indexType)) { ...
} else {
throw new HoodieIndexException(String.format("%s is not supported",
indexType))
}
```
Concretely:
- `column_stats` without an expression - explicitly rejected with a message
pointing users back at datasource configs.
- `bloom_filters` - only reachable via the same expression-index branch, not
as a plain index.
- `partition_stats` - not handled at all; falls through to the final `else`
and fails with "`partition_stats` is not supported".
Only `record_index` and `secondary_index` have real SQL paths today.
Worth noting the workaround your description mentions is itself blocked by
the same code: creating an expression index with the identity function over
`column_stats` is what that first `throw` rejects. That is tracked as #17353
(HUDI-8813), and the two are pinned by the same `if` condition - fixing them
together is probably one change rather than two.
Keeping this open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
