JackieTien97 commented on code in PR #18645:
URL: https://github.com/apache/iotdb/pull/18645#discussion_r4033824988


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/exchange/source/SourceHandle.java:
##########
@@ -767,6 +779,115 @@ private void fail(Throwable t) {
         sourceHandleListener.onFailure(SourceHandle.this, t);
       }
     }
+
+    private TGetDataBlockResponse getDataBlockWithFragments(
+        SyncDataNodeMPPDataExchangeServiceClient client,
+        TGetDataBlockRequest request,
+        DataBlockFetchProgress fetchProgress)
+        throws TException {
+      while (!fetchProgress.isFinished()) {
+        TGetDataBlockRequest fragmentRequest = request.deepCopy();
+        fragmentRequest.setStartSequenceId(fetchProgress.nextSequenceId);
+        fragmentRequest.setOffset(fetchProgress.offset);
+        TGetDataBlockResponse response = client.getDataBlock(fragmentRequest);

Review Comment:
   [P2] Stop fetching fragments when the source is closed or aborted
   
   If `close()` or `abort()` runs while a fragment RPC is in flight, this loop 
still consumes its response and downloads the remaining fragments because it 
only checks `fetchProgress.isFinished()`. The source has already returned its 
reservation to the query pool, so it can allocate the full 
`PublicBAOS(totalLength)` and retain accumulated data after cancellation 
without an accounted reservation. A focused test that closes or aborts during 
the first RPC of a three-fragment transfer still executes all three RPCs with 
`getBufferRetainedSizeInBytes() == 0`. For large blocks this also keeps the 
exchange worker and connection occupied until the entire range is fetched. 
Please check the terminal state with appropriate synchronization before each 
new RPC and again before consuming its response, and discard the pending fetch 
state when stopping.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java:
##########
@@ -3083,6 +3092,41 @@ public void loadShuffleProps(TrimProperties properties) {
                 
Integer.toString(commonConfig.getDriverTaskExecutionTimeSliceInMs()))));
   }
 
+  private void loadMppDataExchangeMaxPayloadSize(TrimProperties properties) {
+    int configuredSize =
+        Integer.parseInt(
+            properties.getProperty(
+                "mpp_data_exchange_max_payload_size_in_bytes",
+                
Integer.toString(conf.getMppDataExchangeMaxPayloadSizeInBytes())));
+    if (configuredSize <= 0) {
+      LOGGER.warn(
+          DataNodeMiscMessages
+              
.LOG_MPP_DATA_EXCHANGE_MAX_PAYLOAD_SIZE_ARG_IS_NOT_POSITIVE_USING_DEFAULT_VALUE_ARG_1AA821B2,
+          configuredSize,
+          DEFAULT_MPP_DATA_EXCHANGE_MAX_PAYLOAD_SIZE_IN_BYTES);

Review Comment:
   [P3] Use the correct placeholders for the configuration warnings
   
   The new message constants contain `%d`, but `LOGGER.warn(template, args...)` 
performs SLF4J `{}` substitution. Consequently this warning is emitted as 
`mpp_data_exchange_max_payload_size_in_bytes (%d) is not positive, using 
default value %d`, without the configured or effective values. The 
below-minimum and above-maximum warnings have the same problem in both locales. 
Please use `{}` in the English and Chinese templates, or explicitly apply 
`String.format()` before logging, so operators can see which values were 
rejected and applied.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/exchange/source/SourceHandle.java:
##########
@@ -709,7 +714,14 @@ public void run() {
               return;
             }
             break;
-          } catch (Throwable e) {
+          } catch (IllegalArgumentException | IllegalStateException e) {
+            if (!transferAttemptRecorded) {
+              recordTransferAttempt(
+                  false, 
UserDataTransferErrorCode.UNEXPECTED_RESPONSE_SIZE.name(), e);
+            }
+            fail(e);
+            return;
+          } catch (Exception e) {

Review Comment:
   [P2] Preserve failure cleanup for errors during fragment assembly
   
   Narrowing the previous `catch (Throwable)` to `catch (Exception)` leaves an 
`OutOfMemoryError` from the new `PublicBAOS(totalLength)` allocation uncaught. 
This allocation happens after `client.getDataBlock()` returns, so the Thrift 
client wrapper cannot convert it into a `TException`. The submitted task then 
terminates without calling `fail()`: the executor only logs the error, leaving 
the source blocked and its reservation held until an outer timeout or 
cancellation. Using the JVM array-size limit to safely inject an allocation 
error reproduced zero failure callbacks, a blocked source, and a retained 
1,024-byte reservation. Please also notify the fragment of failure and release 
the reservation on the `Error` path; failing immediately is sufficient and does 
not require retrying an OOM.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to