github-actions[bot] commented on code in PR #65145:
URL: https://github.com/apache/doris/pull/65145#discussion_r3557226669
##########
regression-test/suites/external_table_p0/hive/test_workload_policy_remote_scan_bytes.groovy:
##########
@@ -103,38 +195,23 @@ suite("test_workload_policy_remote_scan_bytes",
"p0,external") {
assertEquals(policyName, policy[0][0])
assertTrue(policy[0][1].toString().contains("be_scan_bytes_from_remote_storage
> 1"))
- // Wait for FE to publish the new BE-side policy before issuing the
query.
- Thread.sleep(15000)
+ waitForBePolicyPublish()
- Throwable queryException = null
- sql """SET workload_group = '${workloadGroupName}'"""
- sql """SET enable_file_cache = false"""
- sql """SET enable_sql_cache = false"""
- try {
- sql """
- SELECT SUM(SLEEP(1) + l_quantity)
- FROM (
- SELECT l_quantity
- FROM ${catalogName}.${lineitemDb}.lineitem
- LIMIT 10
- ) s
- """
- } catch (Throwable t) {
- queryException = t
- }
- assertTrue(queryException != null, "query should be cancelled by
remote scan bytes workload policy")
- String msg = queryException.getMessage()
- logger.info("Remote scan bytes workload policy cancel message: " + msg)
- assertTrue(msg != null && msg.contains("cancelled by workload policy:
${policyName}"),
- "unexpected cancel policy: " + msg)
- assertTrue(msg.contains("scan_bytes_from_remote_storage"),
- "remote scan bytes counter is missing from cancel message: " +
msg)
+ // Test item 1: the legacy FileScanner already publishes remote scan
bytes to the
+ // query-level IO context, so cancellation is required and fully
asserted.
+ runRemoteScanPolicyCase(lineitemDb, false, true)
+
+ // Test item 2: FileScannerV2 currently does not publish file scan
bytes to the
+ // query-level IO context. Keep this path as non-blocking coverage
until the BE-side
+ // FileScannerV2 accounting fix is merged, then change
requireCancellation to true.
Review Comment:
This keeps the FileScannerV2 half of the regression non-blocking, so the
suite would still pass if the V2 path stops enforcing the
`be_scan_bytes_from_remote_storage` policy. In the current BE path,
`enable_file_scanner_v2=true` can select `FileScannerV2` for supported Hive
Parquet scans, and `FileScannerV2::update_realtime_counters()` already updates
the query IO context via
`update_scan_bytes_from_remote_storage(deltas.scan_bytes_from_remote_storage)`.
`WorkloadSchedPolicy::is_match()` reads that same
`scan_bytes_from_remote_storage()` counter for
`SCAN_BYTES_FROM_REMOTE_STORAGE`, with HDFS/S3/HTTP-style uncached reads
classified as remote. This test should therefore require cancellation for the
V2 invocation too, or otherwise assert the expected V2 counter/policy behavior
instead of accepting a successful query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
