github-actions[bot] commented on code in PR #66913:
URL: https://github.com/apache/doris/pull/66913#discussion_r3910034450
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalMetaCache.java:
##########
@@ -56,6 +60,9 @@
Review Comment:
[P1] Keep partition-column projection on the snapshot generation
This no-context path closes the table-generation lease in
`withoutRetainedTable()` before `IcebergUtils.getIcebergPartitionColumns()`
resolves the schema. That helper then performs a second borrow, so a concurrent
catalog reset/table replacement can combine the first generation's
snapshot/partition info with a later generation's schema and partition columns
(including spec changes). Keep the schema/partition projection inside the same
operation lease, or carry and validate the pinned generation; please add a
reset-barrier regression for the no-context caller.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/source/HudiScanNode.java:
##########
@@ -494,22 +544,52 @@ private List<HudiSplit> planPartitionSplits(HivePartition
partition) throws IOEx
private void getPartitionsSplits(List<HivePartition> partitions,
List<Split> splits) {
Executor executor =
Env.getCurrentEnv().getExtMetaCacheMgr().getFileListingExecutor();
- CountDownLatch countDownLatch = new CountDownLatch(partitions.size());
+ ListingFsViewOwner createdOwner = new ListingFsViewOwner(fsViewLease,
executor);
+ ListingFsViewOwner owner = createdOwner;
+ ConnectContext connectContext = ConnectContext.get();
+ StatementContext statementContext = connectContext == null ? null :
connectContext.getStatementContext();
+ if (statementContext != null) {
+ try {
+ owner =
statementContext.getOrRegisterStatementResource(listingFsViewResourceKey, () ->
createdOwner);
+ if (owner != createdOwner) {
+ throw new IllegalStateException("Hudi listing owner was
registered twice");
+ }
+ } catch (RuntimeException e) {
+ createdOwner.discardBeforeSubmission();
+ throw e;
+ }
+ }
+ // The owner now releases the exact fs-view generation after every
accepted task terminates.
+ if (!fsViewReleased.compareAndSet(false, true)) {
+ owner.discardBeforeSubmission();
+ throw new IllegalStateException("Hudi filesystem-view lease has
already been released");
+ }
AtomicReference<Throwable> throwable = new AtomicReference<>();
+ RuntimeException submissionFailure = null;
long startTime = System.currentTimeMillis();
- partitions.forEach(partition -> executor.execute(() -> {
+ for (HivePartition partition : partitions) {
+ TerminalTask task = terminalTask(() -> {
+ try {
+ ensureHmsRuntimeGeneration();
+ getPartitionSplits(partition, splits);
+ ensureHmsRuntimeGeneration();
+ } catch (Throwable t) {
+ throwable.compareAndSet(null, t);
+ }
+ }, () -> { });
try {
- getPartitionSplits(partition, splits);
- } catch (Throwable t) {
- throwable.set(t);
Review Comment:
[P2] Remove canceled batch tasks from executor queues
`BatchFsViewOwner.close()` requests stop on each tracked task, but unlike
`ListingFsViewOwner` it does not retain the backing executors or remove queued
tasks. `cancelBeforeStart()` runs `done()` yet leaves an already-enqueued
`FutureTask` in the bounded shared `scheduleExecutor` (and the producer can
remain in the file-listing queue) until a worker dequeues it. With blocked/slow
storage calls, repeated canceled batches can fill these queues and stall or
reject unrelated scans. Give the batch owner the executor handles and remove
not-started tasks race-safely, with a regression that checks queue size after
cancellation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
