github-actions[bot] commented on code in PR #66913:
URL: https://github.com/apache/doris/pull/66913#discussion_r3820857944
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/HudiExternalMetaCache.java:
##########
@@ -88,8 +88,9 @@ public HudiExternalMetaCache(ExecutorService refreshExecutor)
{
TablePartitionValues.class,
this::loadPartitionValuesCacheValue, defaultEntryCacheSpec(),
MetaCacheEntryInvalidation.forNameMapping(HudiPartitionCacheKey::getNameMapping)));
fsViewEntry = registerEntry(MetaCacheEntryDef.of(ENTRY_FS_VIEW,
HudiFsViewCacheKey.class,
- HoodieTableFileSystemView.class, this::createFsView,
defaultEntryCacheSpec(),
-
MetaCacheEntryInvalidation.forNameMapping(HudiFsViewCacheKey::getNameMapping)));
+ HudiFsViewCacheValue.class, this::createFsView,
defaultEntryCacheSpec(),
+ false,
MetaCacheEntryInvalidation.forNameMapping(HudiFsViewCacheKey::getNameMapping),
Review Comment:
[P1] Preserve filesystem-view freshness
Passing `false` here removes this entry's previous refresh-after-write path.
The remaining 24-hour policy is expire-after-access, so a continuously queried
key may never be recreated. Hudi's in-memory view captures the completed
timeline and loaded partition groups at construction; `HudiScanNode` reloads
the separate meta client and can resolve query instant T2, but it never calls
`fsView.sync()`, so asking the still-T1 view for files before/on T2 omits T2's
base file or MOR log slice indefinitely. Preserve an explicit
sync/recreate/invalidate freshness transition while keeping the retired
generation leased, and add a hot-cache T1-load/T2-commit test. Simply restoring
background refresh also requires removing the assumption that every newly
loaded wrapper has a foreground caller to consume its initial reference.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/source/HudiScanNode.java:
##########
@@ -251,6 +251,11 @@ protected void doInitialize() throws UserException {
// and `the file column name`.
// Split planning and FE-BE schema transport must describe the same
pinned Hudi instant.
ExternalUtil.initSchemaInfo(params, -1L,
table.getFullSchema(relationSnapshot));
+ fsViewLease = Env.getCurrentEnv()
Review Comment:
[P1] Add a planner-failure owner for this lease
This lease is acquired during `scanNode.init()`, before Nereids has
published the node to a planner/coordinator cleanup owner. Later translation
and finalization are still fallible, so a pre-split failure runs none of this
class's release paths. In batch mode, `SplitAssignment.init()` can also start
workers before remaining finalization; if planning then aborts with no source
consumer, the queue can fill and block the producer before it attaches the
`allOf` release callback. These planner-handoff failures are outside the
existing line-560/617 task-internal threads. Install structured failure cleanup
that stops/cancels and joins started work before releasing, or delay both
acquisition and work startup until an owner is published.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/source/HudiScanNode.java:
##########
@@ -494,22 +505,30 @@ private List<HudiSplit> planPartitionSplits(HivePartition
partition) throws IOEx
private void getPartitionsSplits(List<HivePartition> partitions,
List<Split> splits) {
Executor executor =
Env.getCurrentEnv().getExtMetaCacheMgr().getFileListingExecutor();
- CountDownLatch countDownLatch = new CountDownLatch(partitions.size());
+ Phaser tasks = new Phaser(1);
AtomicReference<Throwable> throwable = new AtomicReference<>();
long startTime = System.currentTimeMillis();
- partitions.forEach(partition -> executor.execute(() -> {
- try {
- getPartitionSplits(partition, splits);
- } catch (Throwable t) {
- throwable.set(t);
- } finally {
- countDownLatch.countDown();
- }
- }));
try {
- countDownLatch.await();
- } catch (InterruptedException e) {
- throw new RuntimeException(e.getMessage(), e);
+ for (HivePartition partition : partitions) {
+ tasks.register();
Review Comment:
[P2] Avoid Phaser's 65,535-party ceiling
This keeps one parent party and registers one more for every outstanding
partition, but Java `Phaser` rejects registrations above 65,535. With
`max_external_cache_loader_thread_pool_size >= 66`, the file-listing executor
can accept more than 65,534 running/queued tasks; if batch mode is disabled or
its threshold is raised for such a table, `tasks.register()` now throws even
though the executor would accept the work. The previous latch had no equivalent
ceiling. Use bounded/tiered task groups or another unbounded join mechanism,
and cover the boundary in a deterministic test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
