venkata91 opened a new pull request, #19022:
URL: https://github.com/apache/iceberg/pull/19022

   ## Intent
   
   Wait for https://github.com/apache/iceberg/pull/18726 to merge, then 
backport to 3.5 and 4.2. This task is the Spark 3.5 backport. PR #18726 ("Spark 
4.1: Keep read metrics from earlier readers in the same task", merged as commit 
d414bdc12) fixed Spark 4.1 so read metrics from earlier readers in the same 
Spark task are kept rather than lost when a later reader is created; it 
includes a numDeletes test. A reviewer (manuzhang) asked for the 3.5 backport, 
and the 4.2 module also exists.
   
   ## What Changed
   
   - Backports #18726 to Spark 3.5: `RowDataReader` and `BatchDataReader` now 
implement `initMetricsValues` and add the metrics passed in from the previous 
reader in the same Spark task to their own `numSplits` and `numDeletes`. Before 
this, a task that opened several readers (for example, with storage-partitioned 
joins grouping splits) reported only the last reader's values.
   - Adds a package-private `InheritedTaskMetrics` helper that stores the 
inherited values and sums them, by metric name, into the current reader's 
metrics.
   - Adds `TestSparkReadMetrics.metricsWhenTaskReadsSeveralPartitions`. It runs 
a storage-partitioned join where each task reads several splits and checks that 
`numSplits` and `numDeletes` both equal 5, with vectorized reads on and off.
   
   ## Risk Assessment
   
   ✅ Low: Faithful backport of the merged Spark 4.1 fix (InheritedTaskMetrics 
is byte-identical); Spark 3.5.9's PartitionReader exposes initMetricsValues and 
DataSourceRDD calls it, so the fix is effective, and the test was adapted 
correctly because 3.5 lacks the subset-partition-key SPJ conf.
   
   ## Testing
   
   I ran TestSparkReadMetrics on a real local Spark 3.5 session with a Hadoop 
catalog. The scenario uses a storage-partitioned join where each task opens 
several readers, over 5 data files with one deleted row each. With the fix, all 
7 tests pass, and the new test sees numSplits=5 and numDeletes=5 for both 
vectorized and non-vectorized reads. With the reader changes reverted, the new 
test fails (numSplits 2 instead of 5), so it reproduces the lost-metrics bug.
   
   - Live validation: ✅ go - 4 of 4 scenarios driven live against the product
   
   | Scenario | Result | Live | Evidence |
   | --- | --- | --- | --- |
   | Vectorized SPJ join over 5 merge-on-read files with one task reading 
several partitions: Spark scan reports numSplits=5 and numDeletes=5 | ✅ pass | 
live | with-fix.log / with-fix-TEST-TestSparkReadMetrics.xml 
(metricsWhenTaskReadsSeveralPartitions, vectorization=true) |
   | Non-vectorized (RowDataReader) SPJ join over the same table: numSplits=5 
and numDeletes=5 | ✅ pass | live | with-fix.log (same test, vectorization=false 
iteration) |
   | Adversarial: without the reader changes, metrics from earlier readers in 
the task are lost | ✅ pass | live | without-fix.log: numSplits expected 5L but 
was 2L |
   | Existing read-metrics behavior (single reader per task, skipped files, 
delete counts) unchanged | ✅ pass | live | 
with-fix-TEST-TestSparkReadMetrics.xml: 7 tests, 0 failures |
   
   
   <details>
   <summary>Evidence: Gradle log with fix (TestSparkReadMetrics 
passes)</summary>
   
   ```text
     % Total    % Received % Xferd  Average Speed   Time    Time     Time  
Current
                                    Dload  Upload   Total   Spent    Left  Speed
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--    
 0 100 43764  100 43764    0     0   288k      0 --:--:-- --:--:-- --:--:--  
290k
   Starting a Gradle Daemon, 1 busy and 1 stopped Daemons could not be reused, 
use --status for details
   Configuration on demand is an incubating feature.
   > Task :iceberg-gcp-bundle:compileJava NO-SOURCE
   > Task :iceberg-common:processResources NO-SOURCE
   > Task :iceberg-core:processResources NO-SOURCE
   > Task :iceberg-hive-metastore:processResources NO-SOURCE
   > Task :iceberg-orc:processResources NO-SOURCE
   > Task :iceberg-azure:processResources NO-SOURCE
   > Task :iceberg-data:processResources NO-SOURCE
   > Task :iceberg-aws-bundle:compileJava NO-SOURCE
   > Task :iceberg-gcp:processResources NO-SOURCE
   > Task :iceberg-azure-bundle:compileJava NO-SOURCE
   > Task :iceberg-api:processResources NO-SOURCE
   > Task :iceberg-arrow:processResources NO-SOURCE
   > Task :iceberg-orc:processTestResources NO-SOURCE
   > Task :iceberg-data:processTestResources NO-SOURCE
   > Task :iceberg-parquet:processResources NO-SOURCE
   > Task :iceberg-azure-bundle:processResources NO-SOURCE
   > Task :iceberg-aws-bundle:processResources NO-SOURCE
   > Task :iceberg-gcp-bundle:processResources NO-SOURCE
   > Task :iceberg-gcp-bundle:classes UP-TO-DATE
   > Task :iceberg-azure-bundle:classes UP-TO-DATE
   > Task :iceberg-aws-bundle:classes UP-TO-DATE
   > Task :iceberg-hive-metastore:processTestResources
   > Task :iceberg-core:processTestResources
   > Task :iceberg-spark:iceberg-spark-3.5_2.12:processResources
   > Task :iceberg-open-api:processTestFixturesResources
   > Task :iceberg-aws:processResources
   > Task :iceberg-spark:iceberg-spark-3.5_2.12:processTestResources
   > Task :iceberg-parquet:processTestFixturesResources
   > Task :iceberg-bundled-guava:compileJava FROM-CACHE
   > Task :iceberg-bundled-guava:processResources NO-SOURCE
   > Task :iceberg-bundled-guava:classes UP-TO-DATE
   > Task :iceberg-bundled-guava:compileTestJava NO-SOURCE
   > Task :iceberg-bundled-guava:shadowJar FROM-CACHE
   > Task :iceberg-azure-bundle:shadowJar FROM-CACHE
   > Task :iceberg-aws-bundle:shadowJar FROM-CACHE
   > Task :iceberg-gcp-bundle:shadowJar FROM-CACHE
   > Task :generateGitProperties
   > Task :buildInfo
   > Task :iceberg-aws-bundle:jar SKIPPED
   > Task :iceberg-azure-bundle:jar SKIPPED
   > Task :iceberg-gcp-bundle:jar SKIPPED
   > Task :iceberg-common:compileJava FROM-CACHE
   > Task :iceberg-common:classes UP-TO-DATE
   > Task :iceberg-common:jar
   > Task :iceberg-api:compileJava FROM-CACHE
   > Task :iceberg-api:classes UP-TO-DATE
   > Task :iceberg-api:jar
   > Task :iceberg-api:compileTestJava FROM-CACHE
   > Task :iceberg-api:processTestResources
   > Task :iceberg-api:testClasses
   
   > Task :iceberg-api:testJar
   file '{run-worktree}/build/iceberg-build.properties' will be copied to 
'iceberg-build.properties', overwriting file 
'{run-worktree}/api/build/resources/test/iceberg-build.properties', which has 
already been copied there.
   
   > Task :iceberg-core:compileJava FROM-CACHE
   > Task :iceberg-core:classes UP-TO-DATE
   > Task :iceberg-core:jar
   > Task :iceberg-gcp:compileJava FROM-CACHE
   > Task :iceberg-gcp:classes UP-TO-DATE
   > Task :iceberg-azure:compileJava FROM-CACHE
   > Task :iceberg-azure:classes UP-TO-DATE
   > Task :iceberg-hive-metastore:compileJava FROM-CACHE
   > Task :iceberg-hive-metastore:classes UP-TO-DATE
   > Task :iceberg-gcp:jar
   > Task :iceberg-orc:compileJava FROM-CACHE
   > Task :iceberg-orc:classes UP-TO-DATE
   > Task :iceberg-azure:jar
   > Task :iceberg-aws:compileJava FROM-CACHE
   > Task :iceberg-aws:classes
   > Task :iceberg-parquet:compileJava FROM-CACHE
   > Task :iceberg-hive-metastore:jar
   > Task :iceberg-aws:jar
   > Task :iceberg-orc:jar
   > Task :iceberg-parquet:compileTestFixturesJava FROM-CACHE
   > Task :iceberg-parquet:classes UP-TO-DATE
   > Task :iceberg-arrow:compileJava FROM-CACHE
   > Task :iceberg-arrow:classes UP-TO-DATE
   > Task :iceberg-data:compileJava FROM-CACHE
   > Task :iceberg-data:classes UP-TO-DATE
   > Task :iceberg-spark:iceberg-spark-3.5_2.12:compileJava NO-SOURCE
   > Task :iceberg-data:jar
   > Task :iceberg-arrow:jar
   > Task :iceberg-parquet:jar
   > Task :iceberg-parquet:testFixturesClasses
   > Task :iceberg-parquet:testFixturesJar
   > Task :iceberg-core:compileTestJava FROM-CACHE
   > Task :iceberg-core:testClasses
   > Task :iceberg-spark:iceberg-spark-3.5_2.12:compileScala FROM-CACHE
   > Task :iceberg-spark:iceberg-spark-3.5_2.12:classes
   > Task :iceberg-core:testJar
   > Task :iceberg-open-api:compileTestFixturesJava FROM-CACHE
   > Task :iceberg-open-api:testFixturesClasses
   > Task :iceberg-open-api:testFixturesJar
   > Task :iceberg-orc:compileTestJava FROM-CACHE
   > Task :iceberg-orc:testClasses UP-TO-DATE
   > Task :iceberg-hive-metastore:compileTestJava FROM-CACHE
   > Task :iceberg-hive-metastore:testClasses
   > Task :iceberg-orc:testJar
   > Task :iceberg-hive-metastore:testJar
   > Task :iceberg-data:compileTestJava FROM-CACHE
   > Task :iceberg-data:testClasses UP-TO-DATE
   > Task :iceberg-data:testJar
   > Task :iceberg-spark:iceberg-spark-3.5_2.12:compileTestJava FROM-CACHE
   > Task :iceberg-spark:iceberg-spark-3.5_2.12:compileTestScala NO-SOURCE
   > Task :iceberg-spark:iceberg-spark-3.5_2.12:testClasses
   [shutdown-hook-0] INFO org.apache.spark.util.ShutdownHookManager - Shutdown 
hook called
   [shutdown-hook-0] INFO org.apache.spark.util.ShutdownHookManager - Deleting 
directory /tmp/spark-4e78ad9c-07fe-46eb-89a6-40bde5aab1fb
   > Task :iceberg-spark:iceberg-spark-3.5_2.12:test
   
   [Incubating] Problems report is available at: 
file://{run-worktree}/build/reports/problems/problems-report.html
   
   Deprecated Gradle features were used in this build, making it incompatible 
with Gradle 9.0.
   
   You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.
   
   For more on this, please refer to 
https://docs.gradle.org/8.14.6/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.
   
   BUILD SUCCESSFUL in 43s
   53 actionable tasks: 28 executed, 25 from cache
   ```
   </details>
   <details>
   <summary>Evidence: JUnit XML with fix (7 tests, 0 failures)</summary>
   
   ```text
   <?xml version="1.0" encoding="UTF-8"?>
   <testsuite name="org.apache.iceberg.spark.source.TestSparkReadMetrics" 
tests="7" skipped="0" failures="0" errors="0" 
timestamp="2026-11-18T04:06:08.532Z" 
hostname="vsowrirajan-microsoft-devbox-vsowrirajan-dev-6b698f4977-zvfhg" 
time="11.688">
     <properties/>
     <testcase name="catalogName = testhadoop, implementation = 
org.apache.iceberg.spark.SparkCatalog, config = {type=hadoop, 
cache-enabled=false}" 
classname="org.apache.iceberg.spark.source.TestSparkReadMetrics" time="5.517"/>
     <testcase name="catalogName = testhadoop, implementation = 
org.apache.iceberg.spark.SparkCatalog, config = {type=hadoop, 
cache-enabled=false}" 
classname="org.apache.iceberg.spark.source.TestSparkReadMetrics" time="2.532"/>
     <testcase name="catalogName = testhadoop, implementation = 
org.apache.iceberg.spark.SparkCatalog, config = {type=hadoop, 
cache-enabled=false}" 
classname="org.apache.iceberg.spark.source.TestSparkReadMetrics" time="1.714"/>
     <testcase name="catalogName = testhadoop, implementation = 
org.apache.iceberg.spark.SparkCatalog, config = {type=hadoop, 
cache-enabled=false}" 
classname="org.apache.iceberg.spark.source.TestSparkReadMetrics" time="0.479"/>
     <testcase name="catalogName = testhadoop, implementation = 
org.apache.iceberg.spark.SparkCatalog, config = {type=hadoop, 
cache-enabled=false}" 
classname="org.apache.iceberg.spark.source.TestSparkReadMetrics" time="0.553"/>
     <testcase name="catalogName = testhadoop, implementation = 
org.apache.iceberg.spark.SparkCatalog, config = {type=hadoop, 
cache-enabled=false}" 
classname="org.apache.iceberg.spark.source.TestSparkReadMetrics" time="0.444"/>
     <testcase name="catalogName = testhadoop, implementation = 
org.apache.iceberg.spark.SparkCatalog, config = {type=hadoop, 
cache-enabled=false}" 
classname="org.apache.iceberg.spark.source.TestSparkReadMetrics" time="0.437"/>
     <system-out><![CDATA[--
   -- Licensed to the Apache Software Foundation (ASF) under one or more
   -- contributor license agreements.  See the NOTICE file distributed with
   -- this work for additional information regarding copyright ownership.
   -- The ASF licenses this file to You under the Apache License, Version 2.0
   -- (the "License"); you may not use this file except in compliance with
   -- the License.  You may obtain a copy of the License at
   --
   --   http://www.apache.org/licenses/LICENSE-2.0
   --
   -- Unless required by applicable law or agreed to in writing, software
   -- distributed under the License is distributed on an "AS IS" BASIS,
   -- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   -- See the License for the specific language governing permissions and
   -- limitations under the License.
   --
   -- This file was copied from Apache Hive, at:
   -- 
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-3.1.0.derby.sql
   --
   -- This has been modified slightly for compatibility with older Hive 
versions.
   --
   -- Timestamp: 2011-09-22 15:32:02.024
   -- Source database is: ~/Work/repos/hive1/metastore/scripts/upgrade/derby/mdb
   -- Connection URL is: 
jdbc:derby:~/Work/repos/hive1/metastore/scripts/upgrade/derby/mdb
   -- Specified schema is: APP
   -- appendLogs: false
   -- ----------------------------------------------
   -- DDL Statements for functions
   -- ----------------------------------------------
   CREATE FUNCTION "APP"."NUCLEUS_ASCII" (C CHAR(1)) RETURNS INTEGER LANGUAGE 
JAVA PARAMETER STYLE JAVA READS SQL DATA CALLED ON NULL INPUT EXTERNAL NAME 
'org.datanucleus.store.rdbms.adapter.DerbySQLFunction.ascii'  
   CREATE FUNCTION "APP"."NUCLEUS_MATCHES" (TEXT VARCHAR(8000),PATTERN 
VARCHAR(8000)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE JAVA READS SQL 
DATA CALLED ON NULL INPUT EXTERNAL NAME 
'org.datanucleus.store.rdbms.adapter.DerbySQLFunction.matches'  
   -- ----------------------------------------------
   -- DDL Statements for tables
   -- ----------------------------------------------
   CREATE TABLE "APP"."DBS" (   "DB_ID" BIGINT NOT NULL,   "DESC" 
VARCHAR(4000),   "DB_LOCATION_URI" VARCHAR(4000) NOT NULL,   "NAME" 
VARCHAR(128),   "OWNER_NAME" VARCHAR(128),   "OWNER_TYPE" VARCHAR(10),   
"CTLG_NAME" VARCHAR(256) ) 
   CREATE TABLE "APP"."TBL_PRIVS" ("TBL_GRANT_ID" BIGINT NOT NULL, 
"CREATE_TIME" INTEGER NOT NULL, "GRANT_OPTION" SMALLINT NOT NULL, "GRANTOR" 
VARCHAR(128), "GRANTOR_TYPE" VARCHAR(128), "PRINCIPAL_NAME" VARCHAR(128), 
"PRINCIPAL_TYPE" VARCHAR(128), "TBL_PRIV" VARCHAR(128), "TBL_ID" BIGINT, 
"AUTHORIZER" VARCHAR(128)) 
   CREATE TABLE "APP"."DATABASE_PARAMS" ("DB_ID" BIGINT NOT NULL, "PARAM_KEY" 
VARCHAR(180) NOT NULL, "PARAM_VALUE" VARCHAR(4000)) 
   CREATE TABLE "APP"."TBL_COL_PRIVS" ("TBL_COLUMN_GRANT_ID" BIGINT NOT NULL, 
"COLUMN_NAME" VARCHAR(767), "CREATE_TIME" INTEGER NOT NULL, "GRANT_OPTION" 
SMALLINT NOT NULL, "GRANTOR" VARCHAR(128), "GRANTOR_TYPE" VARCHAR(128), 
"PRINCIPAL_NAME" VARCHAR(128), "PRINCIPAL_TYPE" VARCHAR(128), "TBL_COL_PRIV" 
VARCHAR(128), "TBL_ID" BIGINT, "AUTHORIZER" VARCHAR(128)) 
   CREATE TABLE "APP"."SERDE_PARAMS" ("SERDE_ID" BIGINT NOT NULL, "PARAM_KEY" 
VARCHAR(256) NOT NULL, "PARAM_VALUE" VARCHAR(32672)) 
   CREATE TABLE "APP"."COLUMNS_V2" ("CD_ID" BIGINT NOT NULL, "COMMENT" 
VARCHAR(4000), "COLUMN_NAME" VARCHAR(767) NOT NULL, "TYPE_NAME" VARCHAR(32672), 
"INTEGER_IDX" INTEGER NOT NULL) 
   CREATE TABLE "APP"."SORT_COLS" ("SD_ID" BIGINT NOT NULL, "COLUMN_NAME" 
VARCHAR(767), "ORDER" INTEGER NOT NULL, "INTEGER_IDX" INTEGER NOT NULL) 
   CREATE TABLE "APP"."CDS" ("CD_ID" BIGINT NOT NULL) 
   CREATE TABLE "APP"."PARTITION_KEY_VALS" ("PART_ID" BIGINT NOT NULL, 
"PART_KEY_VAL" VARCHAR(256), "INTEGER_IDX" INTEGER NOT NULL) 
   CREATE TABLE "APP"."DB_PRIVS" ("DB_GRANT_ID" BIGINT NOT NULL, "CREATE_TIME" 
INTEGER NOT NULL, "DB_ID" BIGINT, "GRANT_OPTION" SMALLINT NOT NULL, "GRANTOR" 
VARCHAR(128), "GRANTOR_TYPE" VARCHAR(128), "PRINCIPAL_NAME" VARCHAR(128), 
"PRINCIPAL_TYPE" VARCHAR(128), "DB_PRIV" VARCHAR(128), "AUTHORIZER" 
VARCHAR(128)) 
   CREATE TABLE "APP"."IDXS" ("INDEX_ID" BIGINT NOT NULL, "CREATE_TIME" INTEGER 
NOT NULL, "DEFERRED_REBUILD" CHAR(1) NOT NULL, "INDEX_HANDLER_CLASS" 
VARCHAR(4000), "INDEX_NAME" VARCHAR(128), "INDEX_TBL_ID" BIGINT, 
"LAST_ACCESS_TIME" INTEGER NOT NULL, "ORIG_TBL_ID" BIGINT, "SD_ID" BIGINT) 
   CREATE TABLE "APP"."INDEX_PARAMS" ("INDEX_ID" BIGINT NOT NULL, "PARAM_KEY" 
VARCHAR(256) NOT NULL, "PARAM_VALUE" VARCHAR(4000)) 
   CREATE TABLE "APP"."PARTITIONS" ("PART_ID" BIGINT NOT NULL, "CREATE_TIME" 
INTEGER NOT NULL, "LAST_ACCESS_TIME" INTEGER NOT NULL, "PART_NAME" 
VARCHAR(767), "SD_ID" BIGINT, "TBL_ID" BIGINT) 
   CREATE TABLE "APP"."SERDES" ("SERDE_ID" BIGINT NOT NULL, "NAME" 
VARCHAR(128), "SLIB" VARCHAR(4000), "DESCRIPTION" VARCHAR(4000), 
"SERIALIZER_CLASS" VARCHAR(4000), "DESERIALIZER_CLASS" VARCHAR(4000), 
SERDE_TYPE INTEGER) 
   CREATE TABLE "APP"."PART_PRIVS" ("PART_GRANT_ID" BIGINT NOT NULL, 
"CREATE_TIME" INTEGER NOT NULL, "GRANT_OPTION" SMALLINT NOT NULL, "GRANTOR" 
VARCHAR(128), "GRANTOR_TYPE" VARCHAR(128), "PART_ID" BIGINT, "PRINCIPAL_NAME" 
VARCHAR(128), "PRINCIPAL_TYPE" VARCHAR(128), "PART_PRIV" VARCHAR(128), 
"AUTHORIZER" VARCHAR(128)) 
   CREATE TABLE "APP"."ROLE_MAP" ("ROLE_GRANT_ID" BIGINT NOT NULL, "ADD_TIME" 
INTEGER NOT NULL, "GRANT_OPTION" SMALLINT NOT NULL, "GRANTOR" VARCHAR(128), 
"GRANTOR_TYPE" VARCHAR(128), "PRINCIPAL_NAME" VARCHAR(128), "PRINCIPAL_TYPE" 
VARCHAR(128), "ROLE_ID" BIGINT) 
   CREATE TABLE "APP"."TYPES" ("TYPES_ID" BIGINT NOT NULL, "TYPE_NAME" 
VARCHAR(128), "TYPE1" VARCHAR(767), "TYPE2" VARCHAR(767)) 
   CREATE TABLE "APP"."GLOBAL_PRIVS" ("USER_GRANT_ID" BIGINT NOT NULL, 
"CREATE_TIME" INTEGER NOT NULL, "GRANT_OPTION" SMALLINT NOT NULL, "GRANTOR" 
VARCHAR(128), "GRANTOR_TYPE" VARCHAR(128), "PRINCIPAL_NAME" VARCHAR(128), 
"PRINCIPAL_TYPE" VARCHAR(128), "USER_PRIV" VARCHAR(128), "AUTHORIZER" 
VARCHAR(128)) 
   CREATE TABLE "APP"."PARTITION_PARAMS" ("PART_ID" BIGINT NOT NULL, 
"PARAM_KEY" VARCHAR(256) NOT NULL, "PARAM_VALUE" VARCHAR(4000)) 
   CREATE TABLE "APP"."PARTITION_EVENTS" (     "PART_NAME_ID" BIGINT NOT NULL,  
   "CAT_NAME" VARCHAR(256),     "DB_NAME" VARCHAR(128),     "EVENT_TIME" BIGINT 
NOT NULL,     "EVENT_TYPE" INTEGER NOT NULL,     "PARTITION_NAME" VARCHAR(767), 
    "TBL_NAME" VARCHAR(256) ) 
   CREATE TABLE "APP"."COLUMNS"
   
   ... [354904 bytes truncated] ...
   
   ated with testhadoop.default.table
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current 
cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, 
loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0}
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - 
Invalidating 0 keys associated with testhadoop.default.table
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current 
cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, 
loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0}
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - 
Invalidating 0 keys associated with testhadoop.default.table
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current 
cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, 
loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0}
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - 
Invalidating 0 keys associated with testhadoop.default.table
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current 
cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, 
loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0}
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - 
Invalidating 0 keys associated with testhadoop.default.table
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current 
cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, 
loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0}
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - 
Invalidating 0 keys associated with testhadoop.default.table
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current 
cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, 
loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0}
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - 
Invalidating 0 keys associated with testhadoop.default.table
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current 
cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, 
loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0}
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - 
Invalidating 0 keys associated with testhadoop.default.table
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current 
cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, 
loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0}
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - 
Invalidating 0 keys associated with testhadoop.default.table
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current 
cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, 
loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0}
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - 
Invalidating 0 keys associated with testhadoop.default.table
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current 
cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, 
loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0}
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - 
Invalidating 0 keys associated with testhadoop.default.table
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current 
cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, 
loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0}
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - 
Invalidating 0 keys associated with testhadoop.default.table
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current 
cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, 
loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0}
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - 
Invalidating 0 keys associated with testhadoop.default.table
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current 
cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, 
loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0}
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - 
Invalidating 0 keys associated with testhadoop.default.table
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current 
cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, 
loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0}
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - 
Invalidating 0 keys associated with testhadoop.default.table
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current 
cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, 
loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0}
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - 
Invalidating 0 keys associated with testhadoop.default.table
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current 
cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, 
loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0}
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - 
Invalidating 0 keys associated with testhadoop.default.table
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current 
cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, 
loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0}
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - 
Invalidating 0 keys associated with testhadoop.default.table
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current 
cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, 
loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0}
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - 
Invalidating 0 keys associated with testhadoop.default.table
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current 
cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, 
loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0}
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - 
Invalidating 0 keys associated with testhadoop.default.table
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current 
cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, 
loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0}
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - 
Invalidating 0 keys associated with testhadoop.default.table
   [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current 
cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, 
loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0}
   [Test worker] INFO org.apache.spark.storage.memory.MemoryStore - MemoryStore 
cleared
   [Test worker] INFO org.apache.spark.storage.BlockManager - BlockManager 
stopped
   [Test worker] INFO org.apache.spark.storage.BlockManagerMaster - 
BlockManagerMaster stopped
   [dispatcher-event-loop-1] INFO 
org.apache.spark.scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint
 - OutputCommitCoordinator stopped!
   [Test worker] INFO org.apache.spark.SparkContext - Successfully stopped 
SparkContext
   [Test worker] INFO org.eclipse.jetty.server.Server - Stopped 
oejs.Server@2d746ce4{STOPPING}[12.1.14,sto=0]
   [Test worker] INFO org.eclipse.jetty.server.AbstractConnector - Stopped 
oejs.ServerConnector@4390f46e{HTTP/1.1, (http/1.1)}{0.0.0.0:39849}
   [Test worker] INFO org.eclipse.jetty.ee10.servlet.ServletContextHandler - 
Stopped 
oeje10s.ServletContextHandler@63ec445c{ROOT,/,b=null,a=AVAILABLE,h=oejcs.CompressionHandler@3104351d{STOPPED,supported=gzip}}
   ]]></system-err>
   </testsuite>
   ```
   </details>
   <details>
   <summary>Evidence: Gradle log without fix (regression reproduced)</summary>
   
   <code>metricsWhenTaskReadsSeveralPartitions FAILED&#10;AssertionFailedError: 
[numSplits with vectorization true]&#10;expected: 5L&#10;but was: 2L</code>
   
   ```text
   Configuration on demand is an incubating feature.
   > Task :iceberg-aws-bundle:compileJava NO-SOURCE
   > Task :iceberg-core:processResources NO-SOURCE
   > Task :iceberg-api:processResources NO-SOURCE
   > Task :iceberg-common:processResources NO-SOURCE
   > Task :iceberg-arrow:processResources NO-SOURCE
   > Task :iceberg-data:processResources NO-SOURCE
   > Task :iceberg-gcp:processResources NO-SOURCE
   > Task :iceberg-azure-bundle:compileJava NO-SOURCE
   > Task :iceberg-azure:processResources NO-SOURCE
   > Task :iceberg-gcp-bundle:compileJava NO-SOURCE
   > Task :iceberg-orc:processResources NO-SOURCE
   > Task :iceberg-hive-metastore:processResources NO-SOURCE
   > Task :iceberg-parquet:processResources NO-SOURCE
   > Task :iceberg-orc:processTestResources NO-SOURCE
   > Task :iceberg-gcp-bundle:processResources NO-SOURCE
   > Task :iceberg-aws-bundle:processResources NO-SOURCE
   > Task :iceberg-data:processTestResources NO-SOURCE
   > Task :iceberg-aws-bundle:classes UP-TO-DATE
   > Task :iceberg-gcp-bundle:classes UP-TO-DATE
   > Task :iceberg-azure-bundle:processResources NO-SOURCE
   > Task :iceberg-azure-bundle:classes UP-TO-DATE
   > Task :iceberg-hive-metastore:processTestResources UP-TO-DATE
   > Task :iceberg-parquet:processTestFixturesResources UP-TO-DATE
   > Task :iceberg-open-api:processTestFixturesResources UP-TO-DATE
   > Task :iceberg-core:processTestResources UP-TO-DATE
   > Task :iceberg-spark:iceberg-spark-3.5_2.12:processResources UP-TO-DATE
   > Task :iceberg-aws:processResources UP-TO-DATE
   > Task :iceberg-spark:iceberg-spark-3.5_2.12:processTestResources UP-TO-DATE
   > Task :generateGitProperties
   > Task :buildInfo
   > Task :iceberg-api:processTestResources
   > Task :iceberg-bundled-guava:compileJava UP-TO-DATE
   > Task :iceberg-bundled-guava:processResources NO-SOURCE
   > Task :iceberg-bundled-guava:classes UP-TO-DATE
   > Task :iceberg-bundled-guava:compileTestJava NO-SOURCE
   > Task :iceberg-bundled-guava:shadowJar FROM-CACHE
   > Task :iceberg-common:compileJava UP-TO-DATE
   > Task :iceberg-common:classes UP-TO-DATE
   > Task :iceberg-api:compileJava UP-TO-DATE
   > Task :iceberg-api:classes UP-TO-DATE
   > Task :iceberg-azure-bundle:shadowJar FROM-CACHE
   > Task :iceberg-azure-bundle:jar SKIPPED
   > Task :iceberg-common:jar
   > Task :iceberg-api:jar
   > Task :iceberg-api:compileTestJava UP-TO-DATE
   > Task :iceberg-api:testClasses
   > Task :iceberg-core:compileJava UP-TO-DATE
   > Task :iceberg-core:classes UP-TO-DATE
   
   > Task :iceberg-api:testJar
   file '{run-worktree}/build/iceberg-build.properties' will be copied to 
'iceberg-build.properties', overwriting file 
'{run-worktree}/api/build/resources/test/iceberg-build.properties', which has 
already been copied there.
   
   > Task :iceberg-azure:compileJava UP-TO-DATE
   > Task :iceberg-azure:classes UP-TO-DATE
   > Task :iceberg-gcp:compileJava UP-TO-DATE
   > Task :iceberg-gcp:classes UP-TO-DATE
   > Task :iceberg-azure:jar
   > Task :iceberg-gcp:jar
   > Task :iceberg-aws:compileJava UP-TO-DATE
   > Task :iceberg-orc:compileJava UP-TO-DATE
   > Task :iceberg-orc:classes UP-TO-DATE
   > Task :iceberg-aws:classes UP-TO-DATE
   > Task :iceberg-parquet:compileJava UP-TO-DATE
   > Task :iceberg-hive-metastore:compileJava UP-TO-DATE
   > Task :iceberg-hive-metastore:classes UP-TO-DATE
   > Task :iceberg-parquet:compileTestFixturesJava UP-TO-DATE
   > Task :iceberg-parquet:classes UP-TO-DATE
   > Task :iceberg-data:compileJava UP-TO-DATE
   > Task :iceberg-data:classes UP-TO-DATE
   > Task :iceberg-hive-metastore:jar
   > Task :iceberg-orc:jar
   > Task :iceberg-aws:jar
   > Task :iceberg-arrow:compileJava UP-TO-DATE
   > Task :iceberg-arrow:classes UP-TO-DATE
   > Task :iceberg-spark:iceberg-spark-3.5_2.12:compileJava NO-SOURCE
   > Task :iceberg-data:jar
   > Task :iceberg-arrow:jar
   > Task :iceberg-aws-bundle:shadowJar FROM-CACHE
   > Task :iceberg-aws-bundle:jar SKIPPED
   > Task :iceberg-parquet:jar
   > Task :iceberg-parquet:testFixturesClasses UP-TO-DATE
   > Task :iceberg-parquet:testFixturesJar
   > Task :iceberg-gcp-bundle:shadowJar FROM-CACHE
   > Task :iceberg-gcp-bundle:jar SKIPPED
   > Task :iceberg-core:jar
   > Task :iceberg-core:compileTestJava UP-TO-DATE
   > Task :iceberg-core:testClasses UP-TO-DATE
   > Task :iceberg-core:testJar
   > Task :iceberg-open-api:compileTestFixturesJava UP-TO-DATE
   > Task :iceberg-open-api:testFixturesClasses UP-TO-DATE
   > Task :iceberg-open-api:testFixturesJar
   > Task :iceberg-orc:compileTestJava UP-TO-DATE
   > Task :iceberg-orc:testClasses UP-TO-DATE
   > Task :iceberg-hive-metastore:compileTestJava UP-TO-DATE
   > Task :iceberg-hive-metastore:testClasses UP-TO-DATE
   > Task :iceberg-orc:testJar
   > Task :iceberg-hive-metastore:testJar
   > Task :iceberg-data:compileTestJava UP-TO-DATE
   > Task :iceberg-data:testClasses UP-TO-DATE
   > Task :iceberg-data:testJar
   
   > Task :iceberg-spark:iceberg-spark-3.5_2.12:compileScala
   Could not determine source for class 
org.apache.iceberg.spark.ImmutableParquetBatchReadConf
   Could not determine source for class 
org.apache.iceberg.spark.ImmutableParquetBatchReadConf$Builder
   Could not determine source for class 
org.apache.iceberg.spark.ImmutableOrcBatchReadConf$Builder
   Could not determine source for class 
org.apache.iceberg.spark.ImmutableOrcBatchReadConf
   
   > Task :iceberg-spark:iceberg-spark-3.5_2.12:classes
   
   > Task :iceberg-spark:iceberg-spark-3.5_2.12:compileTestJava
   Note: 
{run-worktree}/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java
 uses or overrides a deprecated API.
   Note: Recompile with -Xlint:deprecation for details.
   Note: 
{run-worktree}/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java
 uses unchecked or unsafe operations.
   Note: Recompile with -Xlint:unchecked for details.
   
   > Task :iceberg-spark:iceberg-spark-3.5_2.12:compileTestScala NO-SOURCE
   > Task :iceberg-spark:iceberg-spark-3.5_2.12:testClasses
   
   > Task :iceberg-spark:iceberg-spark-3.5_2.12:test
   
   TestSparkReadMetrics > metricsWhenTaskReadsSeveralPartitions() > catalogName 
= testhadoop, implementation = org.apache.iceberg.spark.SparkCatalog, config = 
{type=hadoop, cache-enabled=false} FAILED
       org.opentest4j.AssertionFailedError: [numSplits with vectorization true] 
       expected: 5L
        but was: 2L
           at 
app//org.apache.iceberg.spark.source.TestSparkReadMetrics.lambda$metricsWhenTaskReadsSeveralPartitions$53(TestSparkReadMetrics.java:548)
           at 
app//org.apache.iceberg.spark.TestBase.withSQLConf(TestBase.java:216)
           at 
app//org.apache.iceberg.spark.source.TestSparkReadMetrics.metricsWhenTaskReadsSeveralPartitions(TestSparkReadMetrics.java:533)
   
   [shutdown-hook-0] INFO org.apache.spark.util.ShutdownHookManager - Shutdown 
hook called
   [shutdown-hook-0] INFO org.apache.spark.util.ShutdownHookManager - Deleting 
directory /tmp/spark-ad67c245-e597-435b-a2ba-23c5a7649159
   
   > Task :iceberg-spark:iceberg-spark-3.5_2.12:test FAILED
   
   Failures:
     
org.apache.iceberg.spark.source.TestSparkReadMetrics.metricsWhenTaskReadsSeveralPartitions()[1]
   
   1 test completed, 1 failed
   
   [Incubating] Problems report is available at: 
file://{run-worktree}/build/reports/problems/problems-report.html
   
   FAILURE: Build failed with an exception.
   
   * What went wrong:
   Execution failed for task ':iceberg-spark:iceberg-spark-3.5_2.12:test'.
   > There were failing tests. See the report at: 
file://{run-worktree}/spark/v3.5/spark/build/reports/tests/test/index.html
   
   * Try:
   > Run with --scan to get full insights.
   
   Deprecated Gradle features were used in this build, making it incompatible 
with Gradle 9.0.
   
   You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.
   
   For more on this, please refer to 
https://docs.gradle.org/8.14.6/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.
   
   BUILD FAILED in 38s
   53 actionable tasks: 23 executed, 4 from cache, 26 up-to-date
   ```
   </details>
   
   ## Pipeline
   
   Updates from [git push 
no-mistakes](https://github.com/kunchenguid/no-mistakes)
   
   <!-- no-mistakes-pipeline-attestation:v1 
{"head_sha":"0e16952ddabfaff09b59947ed528f485a26e8922","steps":[{"step":"intent","status":"completed"},{"step":"rebase","status":"completed"},{"step":"review","status":"completed"},{"step":"test","status":"completed"},{"step":"document","status":"completed"},{"step":"lint","status":"completed"},{"step":"push","status":"completed"},{"step":"pr","status":"running"},{"step":"ci","status":"pending"}],"live_validation":{"verdict":"go","live":4,"total":4}}
 -->
   
   <details>
   <summary>✅ **Intent** - passed</summary>
   
   ✅ No issues found.
   </details>
   
   <details>
   <summary>✅ **Rebase** - passed</summary>
   
   ✅ No issues found.
   </details>
   
   <details>
   <summary>✅ **Review** - passed</summary>
   
   ✅ No issues found.
   </details>
   
   <details>
   <summary>✅ **Test** - passed</summary>
   
   ✅ No issues found.
   - Live validation: ✅ go - 4 of 4 scenarios driven live against the product
   
   | Scenario | Result | Live | Evidence |
   | --- | --- | --- | --- |
   | Vectorized SPJ join over 5 merge-on-read files with one task reading 
several partitions: Spark scan reports numSplits=5 and numDeletes=5 | ✅ pass | 
live | with-fix.log / with-fix-TEST-TestSparkReadMetrics.xml 
(metricsWhenTaskReadsSeveralPartitions, vectorization=true) |
   | Non-vectorized (RowDataReader) SPJ join over the same table: numSplits=5 
and numDeletes=5 | ✅ pass | live | with-fix.log (same test, vectorization=false 
iteration) |
   | Adversarial: without the reader changes, metrics from earlier readers in 
the task are lost | ✅ pass | live | without-fix.log: numSplits expected 5L but 
was 2L |
   | Existing read-metrics behavior (single reader per task, skipped files, 
delete counts) unchanged | ✅ pass | live | 
with-fix-TEST-TestSparkReadMetrics.xml: 7 tests, 0 failures |
   
   - <code>`./gradlew -DsparkVersions=3.5 -DscalaVersion=2.12 
:iceberg-spark:iceberg-spark-3.5_2.12:test --tests 
org.apache.iceberg.spark.source.TestSparkReadMetrics` (with fix: 7/7 pass, 
including metricsWhenTaskReadsSeveralPartitions)</code>
   - <code>Regression check: reverted BatchDataReader/RowDataReader to 
d414bdc12, removed InheritedTaskMetrics, ran `--tests 
&#34;org.apache.iceberg.spark.source.TestSparkReadMetrics.metricsWhenTaskReadsSeveralPartitions&#34;`:
 fails with numSplits expected 5 but was 2; sources restored afterward (clean 
worktree)</code>
   </details>
   
   <details>
   <summary>✅ **Document** - passed</summary>
   
   ✅ No issues found.
   </details>
   
   <details>
   <summary>✅ **Lint** - passed</summary>
   
   ✅ No issues found.
   </details>
   
   <details>
   <summary>✅ **Push** - passed</summary>
   
   ✅ No issues found.
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to