venkata91 opened a new pull request, #19022: URL: https://github.com/apache/iceberg/pull/19022
## Intent Wait for https://github.com/apache/iceberg/pull/18726 to merge, then backport to 3.5 and 4.2. This task is the Spark 3.5 backport. PR #18726 ("Spark 4.1: Keep read metrics from earlier readers in the same task", merged as commit d414bdc12) fixed Spark 4.1 so read metrics from earlier readers in the same Spark task are kept rather than lost when a later reader is created; it includes a numDeletes test. A reviewer (manuzhang) asked for the 3.5 backport, and the 4.2 module also exists. ## What Changed - Backports #18726 to Spark 3.5: `RowDataReader` and `BatchDataReader` now implement `initMetricsValues` and add the metrics passed in from the previous reader in the same Spark task to their own `numSplits` and `numDeletes`. Before this, a task that opened several readers (for example, with storage-partitioned joins grouping splits) reported only the last reader's values. - Adds a package-private `InheritedTaskMetrics` helper that stores the inherited values and sums them, by metric name, into the current reader's metrics. - Adds `TestSparkReadMetrics.metricsWhenTaskReadsSeveralPartitions`. It runs a storage-partitioned join where each task reads several splits and checks that `numSplits` and `numDeletes` both equal 5, with vectorized reads on and off. ## Risk Assessment ✅ Low: Faithful backport of the merged Spark 4.1 fix (InheritedTaskMetrics is byte-identical); Spark 3.5.9's PartitionReader exposes initMetricsValues and DataSourceRDD calls it, so the fix is effective, and the test was adapted correctly because 3.5 lacks the subset-partition-key SPJ conf. ## Testing I ran TestSparkReadMetrics on a real local Spark 3.5 session with a Hadoop catalog. The scenario uses a storage-partitioned join where each task opens several readers, over 5 data files with one deleted row each. With the fix, all 7 tests pass, and the new test sees numSplits=5 and numDeletes=5 for both vectorized and non-vectorized reads. With the reader changes reverted, the new test fails (numSplits 2 instead of 5), so it reproduces the lost-metrics bug. - Live validation: ✅ go - 4 of 4 scenarios driven live against the product | Scenario | Result | Live | Evidence | | --- | --- | --- | --- | | Vectorized SPJ join over 5 merge-on-read files with one task reading several partitions: Spark scan reports numSplits=5 and numDeletes=5 | ✅ pass | live | with-fix.log / with-fix-TEST-TestSparkReadMetrics.xml (metricsWhenTaskReadsSeveralPartitions, vectorization=true) | | Non-vectorized (RowDataReader) SPJ join over the same table: numSplits=5 and numDeletes=5 | ✅ pass | live | with-fix.log (same test, vectorization=false iteration) | | Adversarial: without the reader changes, metrics from earlier readers in the task are lost | ✅ pass | live | without-fix.log: numSplits expected 5L but was 2L | | Existing read-metrics behavior (single reader per task, skipped files, delete counts) unchanged | ✅ pass | live | with-fix-TEST-TestSparkReadMetrics.xml: 7 tests, 0 failures | <details> <summary>Evidence: Gradle log with fix (TestSparkReadMetrics passes)</summary> ```text % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 43764 100 43764 0 0 288k 0 --:--:-- --:--:-- --:--:-- 290k Starting a Gradle Daemon, 1 busy and 1 stopped Daemons could not be reused, use --status for details Configuration on demand is an incubating feature. > Task :iceberg-gcp-bundle:compileJava NO-SOURCE > Task :iceberg-common:processResources NO-SOURCE > Task :iceberg-core:processResources NO-SOURCE > Task :iceberg-hive-metastore:processResources NO-SOURCE > Task :iceberg-orc:processResources NO-SOURCE > Task :iceberg-azure:processResources NO-SOURCE > Task :iceberg-data:processResources NO-SOURCE > Task :iceberg-aws-bundle:compileJava NO-SOURCE > Task :iceberg-gcp:processResources NO-SOURCE > Task :iceberg-azure-bundle:compileJava NO-SOURCE > Task :iceberg-api:processResources NO-SOURCE > Task :iceberg-arrow:processResources NO-SOURCE > Task :iceberg-orc:processTestResources NO-SOURCE > Task :iceberg-data:processTestResources NO-SOURCE > Task :iceberg-parquet:processResources NO-SOURCE > Task :iceberg-azure-bundle:processResources NO-SOURCE > Task :iceberg-aws-bundle:processResources NO-SOURCE > Task :iceberg-gcp-bundle:processResources NO-SOURCE > Task :iceberg-gcp-bundle:classes UP-TO-DATE > Task :iceberg-azure-bundle:classes UP-TO-DATE > Task :iceberg-aws-bundle:classes UP-TO-DATE > Task :iceberg-hive-metastore:processTestResources > Task :iceberg-core:processTestResources > Task :iceberg-spark:iceberg-spark-3.5_2.12:processResources > Task :iceberg-open-api:processTestFixturesResources > Task :iceberg-aws:processResources > Task :iceberg-spark:iceberg-spark-3.5_2.12:processTestResources > Task :iceberg-parquet:processTestFixturesResources > Task :iceberg-bundled-guava:compileJava FROM-CACHE > Task :iceberg-bundled-guava:processResources NO-SOURCE > Task :iceberg-bundled-guava:classes UP-TO-DATE > Task :iceberg-bundled-guava:compileTestJava NO-SOURCE > Task :iceberg-bundled-guava:shadowJar FROM-CACHE > Task :iceberg-azure-bundle:shadowJar FROM-CACHE > Task :iceberg-aws-bundle:shadowJar FROM-CACHE > Task :iceberg-gcp-bundle:shadowJar FROM-CACHE > Task :generateGitProperties > Task :buildInfo > Task :iceberg-aws-bundle:jar SKIPPED > Task :iceberg-azure-bundle:jar SKIPPED > Task :iceberg-gcp-bundle:jar SKIPPED > Task :iceberg-common:compileJava FROM-CACHE > Task :iceberg-common:classes UP-TO-DATE > Task :iceberg-common:jar > Task :iceberg-api:compileJava FROM-CACHE > Task :iceberg-api:classes UP-TO-DATE > Task :iceberg-api:jar > Task :iceberg-api:compileTestJava FROM-CACHE > Task :iceberg-api:processTestResources > Task :iceberg-api:testClasses > Task :iceberg-api:testJar file '{run-worktree}/build/iceberg-build.properties' will be copied to 'iceberg-build.properties', overwriting file '{run-worktree}/api/build/resources/test/iceberg-build.properties', which has already been copied there. > Task :iceberg-core:compileJava FROM-CACHE > Task :iceberg-core:classes UP-TO-DATE > Task :iceberg-core:jar > Task :iceberg-gcp:compileJava FROM-CACHE > Task :iceberg-gcp:classes UP-TO-DATE > Task :iceberg-azure:compileJava FROM-CACHE > Task :iceberg-azure:classes UP-TO-DATE > Task :iceberg-hive-metastore:compileJava FROM-CACHE > Task :iceberg-hive-metastore:classes UP-TO-DATE > Task :iceberg-gcp:jar > Task :iceberg-orc:compileJava FROM-CACHE > Task :iceberg-orc:classes UP-TO-DATE > Task :iceberg-azure:jar > Task :iceberg-aws:compileJava FROM-CACHE > Task :iceberg-aws:classes > Task :iceberg-parquet:compileJava FROM-CACHE > Task :iceberg-hive-metastore:jar > Task :iceberg-aws:jar > Task :iceberg-orc:jar > Task :iceberg-parquet:compileTestFixturesJava FROM-CACHE > Task :iceberg-parquet:classes UP-TO-DATE > Task :iceberg-arrow:compileJava FROM-CACHE > Task :iceberg-arrow:classes UP-TO-DATE > Task :iceberg-data:compileJava FROM-CACHE > Task :iceberg-data:classes UP-TO-DATE > Task :iceberg-spark:iceberg-spark-3.5_2.12:compileJava NO-SOURCE > Task :iceberg-data:jar > Task :iceberg-arrow:jar > Task :iceberg-parquet:jar > Task :iceberg-parquet:testFixturesClasses > Task :iceberg-parquet:testFixturesJar > Task :iceberg-core:compileTestJava FROM-CACHE > Task :iceberg-core:testClasses > Task :iceberg-spark:iceberg-spark-3.5_2.12:compileScala FROM-CACHE > Task :iceberg-spark:iceberg-spark-3.5_2.12:classes > Task :iceberg-core:testJar > Task :iceberg-open-api:compileTestFixturesJava FROM-CACHE > Task :iceberg-open-api:testFixturesClasses > Task :iceberg-open-api:testFixturesJar > Task :iceberg-orc:compileTestJava FROM-CACHE > Task :iceberg-orc:testClasses UP-TO-DATE > Task :iceberg-hive-metastore:compileTestJava FROM-CACHE > Task :iceberg-hive-metastore:testClasses > Task :iceberg-orc:testJar > Task :iceberg-hive-metastore:testJar > Task :iceberg-data:compileTestJava FROM-CACHE > Task :iceberg-data:testClasses UP-TO-DATE > Task :iceberg-data:testJar > Task :iceberg-spark:iceberg-spark-3.5_2.12:compileTestJava FROM-CACHE > Task :iceberg-spark:iceberg-spark-3.5_2.12:compileTestScala NO-SOURCE > Task :iceberg-spark:iceberg-spark-3.5_2.12:testClasses [shutdown-hook-0] INFO org.apache.spark.util.ShutdownHookManager - Shutdown hook called [shutdown-hook-0] INFO org.apache.spark.util.ShutdownHookManager - Deleting directory /tmp/spark-4e78ad9c-07fe-46eb-89a6-40bde5aab1fb > Task :iceberg-spark:iceberg-spark-3.5_2.12:test [Incubating] Problems report is available at: file://{run-worktree}/build/reports/problems/problems-report.html Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. For more on this, please refer to https://docs.gradle.org/8.14.6/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation. BUILD SUCCESSFUL in 43s 53 actionable tasks: 28 executed, 25 from cache ``` </details> <details> <summary>Evidence: JUnit XML with fix (7 tests, 0 failures)</summary> ```text <?xml version="1.0" encoding="UTF-8"?> <testsuite name="org.apache.iceberg.spark.source.TestSparkReadMetrics" tests="7" skipped="0" failures="0" errors="0" timestamp="2026-11-18T04:06:08.532Z" hostname="vsowrirajan-microsoft-devbox-vsowrirajan-dev-6b698f4977-zvfhg" time="11.688"> <properties/> <testcase name="catalogName = testhadoop, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hadoop, cache-enabled=false}" classname="org.apache.iceberg.spark.source.TestSparkReadMetrics" time="5.517"/> <testcase name="catalogName = testhadoop, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hadoop, cache-enabled=false}" classname="org.apache.iceberg.spark.source.TestSparkReadMetrics" time="2.532"/> <testcase name="catalogName = testhadoop, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hadoop, cache-enabled=false}" classname="org.apache.iceberg.spark.source.TestSparkReadMetrics" time="1.714"/> <testcase name="catalogName = testhadoop, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hadoop, cache-enabled=false}" classname="org.apache.iceberg.spark.source.TestSparkReadMetrics" time="0.479"/> <testcase name="catalogName = testhadoop, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hadoop, cache-enabled=false}" classname="org.apache.iceberg.spark.source.TestSparkReadMetrics" time="0.553"/> <testcase name="catalogName = testhadoop, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hadoop, cache-enabled=false}" classname="org.apache.iceberg.spark.source.TestSparkReadMetrics" time="0.444"/> <testcase name="catalogName = testhadoop, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hadoop, cache-enabled=false}" classname="org.apache.iceberg.spark.source.TestSparkReadMetrics" time="0.437"/> <system-out><![CDATA[-- -- Licensed to the Apache Software Foundation (ASF) under one or more -- contributor license agreements. See the NOTICE file distributed with -- this work for additional information regarding copyright ownership. -- The ASF licenses this file to You under the Apache License, Version 2.0 -- (the "License"); you may not use this file except in compliance with -- the License. You may obtain a copy of the License at -- -- http://www.apache.org/licenses/LICENSE-2.0 -- -- Unless required by applicable law or agreed to in writing, software -- distributed under the License is distributed on an "AS IS" BASIS, -- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -- See the License for the specific language governing permissions and -- limitations under the License. -- -- This file was copied from Apache Hive, at: -- https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-3.1.0.derby.sql -- -- This has been modified slightly for compatibility with older Hive versions. -- -- Timestamp: 2011-09-22 15:32:02.024 -- Source database is: ~/Work/repos/hive1/metastore/scripts/upgrade/derby/mdb -- Connection URL is: jdbc:derby:~/Work/repos/hive1/metastore/scripts/upgrade/derby/mdb -- Specified schema is: APP -- appendLogs: false -- ---------------------------------------------- -- DDL Statements for functions -- ---------------------------------------------- CREATE FUNCTION "APP"."NUCLEUS_ASCII" (C CHAR(1)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE JAVA READS SQL DATA CALLED ON NULL INPUT EXTERNAL NAME 'org.datanucleus.store.rdbms.adapter.DerbySQLFunction.ascii' CREATE FUNCTION "APP"."NUCLEUS_MATCHES" (TEXT VARCHAR(8000),PATTERN VARCHAR(8000)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE JAVA READS SQL DATA CALLED ON NULL INPUT EXTERNAL NAME 'org.datanucleus.store.rdbms.adapter.DerbySQLFunction.matches' -- ---------------------------------------------- -- DDL Statements for tables -- ---------------------------------------------- CREATE TABLE "APP"."DBS" ( "DB_ID" BIGINT NOT NULL, "DESC" VARCHAR(4000), "DB_LOCATION_URI" VARCHAR(4000) NOT NULL, "NAME" VARCHAR(128), "OWNER_NAME" VARCHAR(128), "OWNER_TYPE" VARCHAR(10), "CTLG_NAME" VARCHAR(256) ) CREATE TABLE "APP"."TBL_PRIVS" ("TBL_GRANT_ID" BIGINT NOT NULL, "CREATE_TIME" INTEGER NOT NULL, "GRANT_OPTION" SMALLINT NOT NULL, "GRANTOR" VARCHAR(128), "GRANTOR_TYPE" VARCHAR(128), "PRINCIPAL_NAME" VARCHAR(128), "PRINCIPAL_TYPE" VARCHAR(128), "TBL_PRIV" VARCHAR(128), "TBL_ID" BIGINT, "AUTHORIZER" VARCHAR(128)) CREATE TABLE "APP"."DATABASE_PARAMS" ("DB_ID" BIGINT NOT NULL, "PARAM_KEY" VARCHAR(180) NOT NULL, "PARAM_VALUE" VARCHAR(4000)) CREATE TABLE "APP"."TBL_COL_PRIVS" ("TBL_COLUMN_GRANT_ID" BIGINT NOT NULL, "COLUMN_NAME" VARCHAR(767), "CREATE_TIME" INTEGER NOT NULL, "GRANT_OPTION" SMALLINT NOT NULL, "GRANTOR" VARCHAR(128), "GRANTOR_TYPE" VARCHAR(128), "PRINCIPAL_NAME" VARCHAR(128), "PRINCIPAL_TYPE" VARCHAR(128), "TBL_COL_PRIV" VARCHAR(128), "TBL_ID" BIGINT, "AUTHORIZER" VARCHAR(128)) CREATE TABLE "APP"."SERDE_PARAMS" ("SERDE_ID" BIGINT NOT NULL, "PARAM_KEY" VARCHAR(256) NOT NULL, "PARAM_VALUE" VARCHAR(32672)) CREATE TABLE "APP"."COLUMNS_V2" ("CD_ID" BIGINT NOT NULL, "COMMENT" VARCHAR(4000), "COLUMN_NAME" VARCHAR(767) NOT NULL, "TYPE_NAME" VARCHAR(32672), "INTEGER_IDX" INTEGER NOT NULL) CREATE TABLE "APP"."SORT_COLS" ("SD_ID" BIGINT NOT NULL, "COLUMN_NAME" VARCHAR(767), "ORDER" INTEGER NOT NULL, "INTEGER_IDX" INTEGER NOT NULL) CREATE TABLE "APP"."CDS" ("CD_ID" BIGINT NOT NULL) CREATE TABLE "APP"."PARTITION_KEY_VALS" ("PART_ID" BIGINT NOT NULL, "PART_KEY_VAL" VARCHAR(256), "INTEGER_IDX" INTEGER NOT NULL) CREATE TABLE "APP"."DB_PRIVS" ("DB_GRANT_ID" BIGINT NOT NULL, "CREATE_TIME" INTEGER NOT NULL, "DB_ID" BIGINT, "GRANT_OPTION" SMALLINT NOT NULL, "GRANTOR" VARCHAR(128), "GRANTOR_TYPE" VARCHAR(128), "PRINCIPAL_NAME" VARCHAR(128), "PRINCIPAL_TYPE" VARCHAR(128), "DB_PRIV" VARCHAR(128), "AUTHORIZER" VARCHAR(128)) CREATE TABLE "APP"."IDXS" ("INDEX_ID" BIGINT NOT NULL, "CREATE_TIME" INTEGER NOT NULL, "DEFERRED_REBUILD" CHAR(1) NOT NULL, "INDEX_HANDLER_CLASS" VARCHAR(4000), "INDEX_NAME" VARCHAR(128), "INDEX_TBL_ID" BIGINT, "LAST_ACCESS_TIME" INTEGER NOT NULL, "ORIG_TBL_ID" BIGINT, "SD_ID" BIGINT) CREATE TABLE "APP"."INDEX_PARAMS" ("INDEX_ID" BIGINT NOT NULL, "PARAM_KEY" VARCHAR(256) NOT NULL, "PARAM_VALUE" VARCHAR(4000)) CREATE TABLE "APP"."PARTITIONS" ("PART_ID" BIGINT NOT NULL, "CREATE_TIME" INTEGER NOT NULL, "LAST_ACCESS_TIME" INTEGER NOT NULL, "PART_NAME" VARCHAR(767), "SD_ID" BIGINT, "TBL_ID" BIGINT) CREATE TABLE "APP"."SERDES" ("SERDE_ID" BIGINT NOT NULL, "NAME" VARCHAR(128), "SLIB" VARCHAR(4000), "DESCRIPTION" VARCHAR(4000), "SERIALIZER_CLASS" VARCHAR(4000), "DESERIALIZER_CLASS" VARCHAR(4000), SERDE_TYPE INTEGER) CREATE TABLE "APP"."PART_PRIVS" ("PART_GRANT_ID" BIGINT NOT NULL, "CREATE_TIME" INTEGER NOT NULL, "GRANT_OPTION" SMALLINT NOT NULL, "GRANTOR" VARCHAR(128), "GRANTOR_TYPE" VARCHAR(128), "PART_ID" BIGINT, "PRINCIPAL_NAME" VARCHAR(128), "PRINCIPAL_TYPE" VARCHAR(128), "PART_PRIV" VARCHAR(128), "AUTHORIZER" VARCHAR(128)) CREATE TABLE "APP"."ROLE_MAP" ("ROLE_GRANT_ID" BIGINT NOT NULL, "ADD_TIME" INTEGER NOT NULL, "GRANT_OPTION" SMALLINT NOT NULL, "GRANTOR" VARCHAR(128), "GRANTOR_TYPE" VARCHAR(128), "PRINCIPAL_NAME" VARCHAR(128), "PRINCIPAL_TYPE" VARCHAR(128), "ROLE_ID" BIGINT) CREATE TABLE "APP"."TYPES" ("TYPES_ID" BIGINT NOT NULL, "TYPE_NAME" VARCHAR(128), "TYPE1" VARCHAR(767), "TYPE2" VARCHAR(767)) CREATE TABLE "APP"."GLOBAL_PRIVS" ("USER_GRANT_ID" BIGINT NOT NULL, "CREATE_TIME" INTEGER NOT NULL, "GRANT_OPTION" SMALLINT NOT NULL, "GRANTOR" VARCHAR(128), "GRANTOR_TYPE" VARCHAR(128), "PRINCIPAL_NAME" VARCHAR(128), "PRINCIPAL_TYPE" VARCHAR(128), "USER_PRIV" VARCHAR(128), "AUTHORIZER" VARCHAR(128)) CREATE TABLE "APP"."PARTITION_PARAMS" ("PART_ID" BIGINT NOT NULL, "PARAM_KEY" VARCHAR(256) NOT NULL, "PARAM_VALUE" VARCHAR(4000)) CREATE TABLE "APP"."PARTITION_EVENTS" ( "PART_NAME_ID" BIGINT NOT NULL, "CAT_NAME" VARCHAR(256), "DB_NAME" VARCHAR(128), "EVENT_TIME" BIGINT NOT NULL, "EVENT_TYPE" INTEGER NOT NULL, "PARTITION_NAME" VARCHAR(767), "TBL_NAME" VARCHAR(256) ) CREATE TABLE "APP"."COLUMNS" ... [354904 bytes truncated] ... ated with testhadoop.default.table [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0} [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Invalidating 0 keys associated with testhadoop.default.table [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0} [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Invalidating 0 keys associated with testhadoop.default.table [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0} [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Invalidating 0 keys associated with testhadoop.default.table [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0} [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Invalidating 0 keys associated with testhadoop.default.table [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0} [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Invalidating 0 keys associated with testhadoop.default.table [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0} [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Invalidating 0 keys associated with testhadoop.default.table [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0} [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Invalidating 0 keys associated with testhadoop.default.table [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0} [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Invalidating 0 keys associated with testhadoop.default.table [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0} [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Invalidating 0 keys associated with testhadoop.default.table [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0} [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Invalidating 0 keys associated with testhadoop.default.table [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0} [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Invalidating 0 keys associated with testhadoop.default.table [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0} [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Invalidating 0 keys associated with testhadoop.default.table [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0} [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Invalidating 0 keys associated with testhadoop.default.table [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0} [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Invalidating 0 keys associated with testhadoop.default.table [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0} [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Invalidating 0 keys associated with testhadoop.default.table [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0} [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Invalidating 0 keys associated with testhadoop.default.table [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0} [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Invalidating 0 keys associated with testhadoop.default.table [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0} [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Invalidating 0 keys associated with testhadoop.default.table [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0} [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Invalidating 0 keys associated with testhadoop.default.table [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0} [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Invalidating 0 keys associated with testhadoop.default.table [Test worker] INFO org.apache.iceberg.spark.SparkExecutorCache - Current cache stats CacheStats{hitCount=5, missCount=16, loadSuccessCount=16, loadFailureCount=0, totalLoadTime=69138754, evictionCount=0, evictionWeight=0} [Test worker] INFO org.apache.spark.storage.memory.MemoryStore - MemoryStore cleared [Test worker] INFO org.apache.spark.storage.BlockManager - BlockManager stopped [Test worker] INFO org.apache.spark.storage.BlockManagerMaster - BlockManagerMaster stopped [dispatcher-event-loop-1] INFO org.apache.spark.scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint - OutputCommitCoordinator stopped! [Test worker] INFO org.apache.spark.SparkContext - Successfully stopped SparkContext [Test worker] INFO org.eclipse.jetty.server.Server - Stopped oejs.Server@2d746ce4{STOPPING}[12.1.14,sto=0] [Test worker] INFO org.eclipse.jetty.server.AbstractConnector - Stopped oejs.ServerConnector@4390f46e{HTTP/1.1, (http/1.1)}{0.0.0.0:39849} [Test worker] INFO org.eclipse.jetty.ee10.servlet.ServletContextHandler - Stopped oeje10s.ServletContextHandler@63ec445c{ROOT,/,b=null,a=AVAILABLE,h=oejcs.CompressionHandler@3104351d{STOPPED,supported=gzip}} ]]></system-err> </testsuite> ``` </details> <details> <summary>Evidence: Gradle log without fix (regression reproduced)</summary> <code>metricsWhenTaskReadsSeveralPartitions FAILED AssertionFailedError: [numSplits with vectorization true] expected: 5L but was: 2L</code> ```text Configuration on demand is an incubating feature. > Task :iceberg-aws-bundle:compileJava NO-SOURCE > Task :iceberg-core:processResources NO-SOURCE > Task :iceberg-api:processResources NO-SOURCE > Task :iceberg-common:processResources NO-SOURCE > Task :iceberg-arrow:processResources NO-SOURCE > Task :iceberg-data:processResources NO-SOURCE > Task :iceberg-gcp:processResources NO-SOURCE > Task :iceberg-azure-bundle:compileJava NO-SOURCE > Task :iceberg-azure:processResources NO-SOURCE > Task :iceberg-gcp-bundle:compileJava NO-SOURCE > Task :iceberg-orc:processResources NO-SOURCE > Task :iceberg-hive-metastore:processResources NO-SOURCE > Task :iceberg-parquet:processResources NO-SOURCE > Task :iceberg-orc:processTestResources NO-SOURCE > Task :iceberg-gcp-bundle:processResources NO-SOURCE > Task :iceberg-aws-bundle:processResources NO-SOURCE > Task :iceberg-data:processTestResources NO-SOURCE > Task :iceberg-aws-bundle:classes UP-TO-DATE > Task :iceberg-gcp-bundle:classes UP-TO-DATE > Task :iceberg-azure-bundle:processResources NO-SOURCE > Task :iceberg-azure-bundle:classes UP-TO-DATE > Task :iceberg-hive-metastore:processTestResources UP-TO-DATE > Task :iceberg-parquet:processTestFixturesResources UP-TO-DATE > Task :iceberg-open-api:processTestFixturesResources UP-TO-DATE > Task :iceberg-core:processTestResources UP-TO-DATE > Task :iceberg-spark:iceberg-spark-3.5_2.12:processResources UP-TO-DATE > Task :iceberg-aws:processResources UP-TO-DATE > Task :iceberg-spark:iceberg-spark-3.5_2.12:processTestResources UP-TO-DATE > Task :generateGitProperties > Task :buildInfo > Task :iceberg-api:processTestResources > Task :iceberg-bundled-guava:compileJava UP-TO-DATE > Task :iceberg-bundled-guava:processResources NO-SOURCE > Task :iceberg-bundled-guava:classes UP-TO-DATE > Task :iceberg-bundled-guava:compileTestJava NO-SOURCE > Task :iceberg-bundled-guava:shadowJar FROM-CACHE > Task :iceberg-common:compileJava UP-TO-DATE > Task :iceberg-common:classes UP-TO-DATE > Task :iceberg-api:compileJava UP-TO-DATE > Task :iceberg-api:classes UP-TO-DATE > Task :iceberg-azure-bundle:shadowJar FROM-CACHE > Task :iceberg-azure-bundle:jar SKIPPED > Task :iceberg-common:jar > Task :iceberg-api:jar > Task :iceberg-api:compileTestJava UP-TO-DATE > Task :iceberg-api:testClasses > Task :iceberg-core:compileJava UP-TO-DATE > Task :iceberg-core:classes UP-TO-DATE > Task :iceberg-api:testJar file '{run-worktree}/build/iceberg-build.properties' will be copied to 'iceberg-build.properties', overwriting file '{run-worktree}/api/build/resources/test/iceberg-build.properties', which has already been copied there. > Task :iceberg-azure:compileJava UP-TO-DATE > Task :iceberg-azure:classes UP-TO-DATE > Task :iceberg-gcp:compileJava UP-TO-DATE > Task :iceberg-gcp:classes UP-TO-DATE > Task :iceberg-azure:jar > Task :iceberg-gcp:jar > Task :iceberg-aws:compileJava UP-TO-DATE > Task :iceberg-orc:compileJava UP-TO-DATE > Task :iceberg-orc:classes UP-TO-DATE > Task :iceberg-aws:classes UP-TO-DATE > Task :iceberg-parquet:compileJava UP-TO-DATE > Task :iceberg-hive-metastore:compileJava UP-TO-DATE > Task :iceberg-hive-metastore:classes UP-TO-DATE > Task :iceberg-parquet:compileTestFixturesJava UP-TO-DATE > Task :iceberg-parquet:classes UP-TO-DATE > Task :iceberg-data:compileJava UP-TO-DATE > Task :iceberg-data:classes UP-TO-DATE > Task :iceberg-hive-metastore:jar > Task :iceberg-orc:jar > Task :iceberg-aws:jar > Task :iceberg-arrow:compileJava UP-TO-DATE > Task :iceberg-arrow:classes UP-TO-DATE > Task :iceberg-spark:iceberg-spark-3.5_2.12:compileJava NO-SOURCE > Task :iceberg-data:jar > Task :iceberg-arrow:jar > Task :iceberg-aws-bundle:shadowJar FROM-CACHE > Task :iceberg-aws-bundle:jar SKIPPED > Task :iceberg-parquet:jar > Task :iceberg-parquet:testFixturesClasses UP-TO-DATE > Task :iceberg-parquet:testFixturesJar > Task :iceberg-gcp-bundle:shadowJar FROM-CACHE > Task :iceberg-gcp-bundle:jar SKIPPED > Task :iceberg-core:jar > Task :iceberg-core:compileTestJava UP-TO-DATE > Task :iceberg-core:testClasses UP-TO-DATE > Task :iceberg-core:testJar > Task :iceberg-open-api:compileTestFixturesJava UP-TO-DATE > Task :iceberg-open-api:testFixturesClasses UP-TO-DATE > Task :iceberg-open-api:testFixturesJar > Task :iceberg-orc:compileTestJava UP-TO-DATE > Task :iceberg-orc:testClasses UP-TO-DATE > Task :iceberg-hive-metastore:compileTestJava UP-TO-DATE > Task :iceberg-hive-metastore:testClasses UP-TO-DATE > Task :iceberg-orc:testJar > Task :iceberg-hive-metastore:testJar > Task :iceberg-data:compileTestJava UP-TO-DATE > Task :iceberg-data:testClasses UP-TO-DATE > Task :iceberg-data:testJar > Task :iceberg-spark:iceberg-spark-3.5_2.12:compileScala Could not determine source for class org.apache.iceberg.spark.ImmutableParquetBatchReadConf Could not determine source for class org.apache.iceberg.spark.ImmutableParquetBatchReadConf$Builder Could not determine source for class org.apache.iceberg.spark.ImmutableOrcBatchReadConf$Builder Could not determine source for class org.apache.iceberg.spark.ImmutableOrcBatchReadConf > Task :iceberg-spark:iceberg-spark-3.5_2.12:classes > Task :iceberg-spark:iceberg-spark-3.5_2.12:compileTestJava Note: {run-worktree}/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: {run-worktree}/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReaderDeletes.java uses unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. > Task :iceberg-spark:iceberg-spark-3.5_2.12:compileTestScala NO-SOURCE > Task :iceberg-spark:iceberg-spark-3.5_2.12:testClasses > Task :iceberg-spark:iceberg-spark-3.5_2.12:test TestSparkReadMetrics > metricsWhenTaskReadsSeveralPartitions() > catalogName = testhadoop, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hadoop, cache-enabled=false} FAILED org.opentest4j.AssertionFailedError: [numSplits with vectorization true] expected: 5L but was: 2L at app//org.apache.iceberg.spark.source.TestSparkReadMetrics.lambda$metricsWhenTaskReadsSeveralPartitions$53(TestSparkReadMetrics.java:548) at app//org.apache.iceberg.spark.TestBase.withSQLConf(TestBase.java:216) at app//org.apache.iceberg.spark.source.TestSparkReadMetrics.metricsWhenTaskReadsSeveralPartitions(TestSparkReadMetrics.java:533) [shutdown-hook-0] INFO org.apache.spark.util.ShutdownHookManager - Shutdown hook called [shutdown-hook-0] INFO org.apache.spark.util.ShutdownHookManager - Deleting directory /tmp/spark-ad67c245-e597-435b-a2ba-23c5a7649159 > Task :iceberg-spark:iceberg-spark-3.5_2.12:test FAILED Failures: org.apache.iceberg.spark.source.TestSparkReadMetrics.metricsWhenTaskReadsSeveralPartitions()[1] 1 test completed, 1 failed [Incubating] Problems report is available at: file://{run-worktree}/build/reports/problems/problems-report.html FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':iceberg-spark:iceberg-spark-3.5_2.12:test'. > There were failing tests. See the report at: file://{run-worktree}/spark/v3.5/spark/build/reports/tests/test/index.html * Try: > Run with --scan to get full insights. Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. For more on this, please refer to https://docs.gradle.org/8.14.6/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation. BUILD FAILED in 38s 53 actionable tasks: 23 executed, 4 from cache, 26 up-to-date ``` </details> ## Pipeline Updates from [git push no-mistakes](https://github.com/kunchenguid/no-mistakes) <!-- no-mistakes-pipeline-attestation:v1 {"head_sha":"0e16952ddabfaff09b59947ed528f485a26e8922","steps":[{"step":"intent","status":"completed"},{"step":"rebase","status":"completed"},{"step":"review","status":"completed"},{"step":"test","status":"completed"},{"step":"document","status":"completed"},{"step":"lint","status":"completed"},{"step":"push","status":"completed"},{"step":"pr","status":"running"},{"step":"ci","status":"pending"}],"live_validation":{"verdict":"go","live":4,"total":4}} --> <details> <summary>✅ **Intent** - passed</summary> ✅ No issues found. </details> <details> <summary>✅ **Rebase** - passed</summary> ✅ No issues found. </details> <details> <summary>✅ **Review** - passed</summary> ✅ No issues found. </details> <details> <summary>✅ **Test** - passed</summary> ✅ No issues found. - Live validation: ✅ go - 4 of 4 scenarios driven live against the product | Scenario | Result | Live | Evidence | | --- | --- | --- | --- | | Vectorized SPJ join over 5 merge-on-read files with one task reading several partitions: Spark scan reports numSplits=5 and numDeletes=5 | ✅ pass | live | with-fix.log / with-fix-TEST-TestSparkReadMetrics.xml (metricsWhenTaskReadsSeveralPartitions, vectorization=true) | | Non-vectorized (RowDataReader) SPJ join over the same table: numSplits=5 and numDeletes=5 | ✅ pass | live | with-fix.log (same test, vectorization=false iteration) | | Adversarial: without the reader changes, metrics from earlier readers in the task are lost | ✅ pass | live | without-fix.log: numSplits expected 5L but was 2L | | Existing read-metrics behavior (single reader per task, skipped files, delete counts) unchanged | ✅ pass | live | with-fix-TEST-TestSparkReadMetrics.xml: 7 tests, 0 failures | - <code>`./gradlew -DsparkVersions=3.5 -DscalaVersion=2.12 :iceberg-spark:iceberg-spark-3.5_2.12:test --tests org.apache.iceberg.spark.source.TestSparkReadMetrics` (with fix: 7/7 pass, including metricsWhenTaskReadsSeveralPartitions)</code> - <code>Regression check: reverted BatchDataReader/RowDataReader to d414bdc12, removed InheritedTaskMetrics, ran `--tests "org.apache.iceberg.spark.source.TestSparkReadMetrics.metricsWhenTaskReadsSeveralPartitions"`: fails with numSplits expected 5 but was 2; sources restored afterward (clean worktree)</code> </details> <details> <summary>✅ **Document** - passed</summary> ✅ No issues found. </details> <details> <summary>✅ **Lint** - passed</summary> ✅ No issues found. </details> <details> <summary>✅ **Push** - passed</summary> ✅ No issues found. </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
