rangareddy commented on issue #18161:
URL: https://github.com/apache/hudi/issues/18161#issuecomment-5176164510
Closing this out, since all four questions are now either answered or
tracked elsewhere. Summary for anyone
arriving from a search.
Q1 — numBuckets does not change in .hashing_meta after clustering. Expected.
In
HoodieConsistentHashingMetadata the field is private final int numBuckets,
assigned only in the
constructors with no setter and no reassignment anywhere in the codebase,
and it is used solely by
constructDefaultHashingNodes to lay out the initial ring. The live bucket
count is the nodes list, which
is what split and merge rewrite. So new node entries plus new file groups,
with numBuckets unchanged, is
the correct picture — that is the signal to check, not the counter.
Q2 — consistent hashing with non-partitioned tables. This was the real bug
here, and it is fixed on
master. The failure you hit was:
java.lang.IllegalArgumentException: Partition should not be null or empty
at
SingleSparkJobConsistentHashingExecutionStrategy.performBucketMergeForGroup(...:117)
A non-partitioned table yields an empty partition string, which that
assertion rejected outright.
245f47f50352 ("fix(spark): support consistent hashing clustering on
non-partitioned tables", #18968,
merged 26 Jun 2026) narrowed it to a null check:
// Note: partition can be an empty string for non-partitioned tables, so
only check for null here.
ValidationUtils.checkArgument(partition != null, "Partition should not be
null");
It also added nonPartitioned as a parameter to testResizing in
TestSparkConsistentBucketClustering, so
split and merge on a non-partitioned table are covered by a regression test
rather than only by inspection.
One caveat on availability: as of writing this commit is on master and is
not in any release tag
(git tag --contains 245f47f50352 returns nothing), so it ships in the next
release rather than in 1.1.x. If
you are on a released version you will still see the exception; there is no
config workaround, since the
assertion fires before any clustering work happens.
Q3 — custom merger not recognised during clustering. Tracked separately in
#18980, which is still open.
Nothing here supersedes it.
Q4 — confirmed earlier in the thread.
Since the only defect this issue reported is fixed with test coverage, and
the remaining thread item has its
own issue, I am closing this. Please reopen if the non-partitioned path
still misbehaves for you on master —
ideally with the Hudi version and the clustering config, since the assertion
that failed is gone and a new
failure there would be a different bug.
Thanks for the detailed report and for splitting out #18980.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
