wombatu-kun commented on code in PR #19687:
URL: https://github.com/apache/hudi/pull/19687#discussion_r3877980379
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/MultipleSparkJobExecutionStrategy.java:
##########
@@ -194,11 +196,16 @@ protected BulkInsertPartitioner<JavaRDD<HoodieRecord<T>>>
getRDDPartitioner(Map<
private <I> BulkInsertPartitioner<I> getPartitioner(Map<String, String>
strategyParams,
HoodieSchema schema,
boolean
isRowPartitioner) {
+ // Trim: the config and inline paths pass the list through verbatim (`id,
ts`), while the
+ // partitioners look up the column names as given.
Option<String[]> orderByColumnsOpt =
Option.ofNullable(strategyParams.get(PLAN_STRATEGY_SORT_COLUMNS.key()))
- .map(listStr -> listStr.split(","));
+ .map(listStr ->
Arrays.stream(listStr.split(",")).map(String::trim).toArray(String[]::new));
return orderByColumnsOpt.map(orderByColumns -> {
+ // The custom-columns partitioners re-validate in their constructors;
this earlier check
+ // additionally covers the spatial-curve (ZORDER/HILBERT) partitioners
below.
+ SortUtils.validateSortableColumns(orderByColumns, schema);
Review Comment:
`SpaceCurveSortingHelper.orderDataFrameByMappingValues` matches order
columns against the dataframe's top-level fields only and returns the frame
unordered when one misses, so a dotted column that `resolveSortColumn` now
accepts is silently dropped by the ZORDER/HILBERT partitioners this check
covers. Should the spatial-curve arm reject a dotted path outright, or is an
unordered result intended there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
