venkata91 commented on code in PR #18726:
URL: https://github.com/apache/iceberg/pull/18726#discussion_r4553686257


##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/source/TestSparkReadMetrics.java:
##########
@@ -341,4 +344,58 @@ public void testDeleteMetrics() throws 
NoSuchTableException {
         .hasEntrySatisfying(
             "skippedDeleteFiles", sqlMetric -> 
assertThat(sqlMetric.value()).isEqualTo(0));
   }
+
+  @TestTemplate
+  public void numSplitsWhenTaskReadsSeveralPartitions() {
+    sql(
+        "CREATE TABLE %s (id BIGINT, dep STRING, region STRING) USING iceberg "
+            + "PARTITIONED BY (dep, region)",
+        tableName);
+    sql(
+        "INSERT INTO %s VALUES (1, 'hr', 'us'), (2, 'hr', 'eu'), (3, 'hr', 
'apac'), "
+            + "(4, 'ops', 'us'), (5, 'ops', 'eu')",
+        tableName);
+
+    // joining on a subset of the partition keys makes Spark group the 3 hr 
and the 2 ops
+    // partitions into one task each, so every task opens several readers
+    Map<String, String> spjConf =
+        ImmutableMap.of(
+            SQLConf.V2_BUCKETING_ENABLED().key(),
+            "true",
+            
SQLConf.V2_BUCKETING_ALLOW_JOIN_KEYS_SUBSET_OF_PARTITION_KEYS().key(),
+            "true",
+            SQLConf.REQUIRE_ALL_CLUSTER_KEYS_FOR_CO_PARTITION().key(),
+            "false",
+            SQLConf.ADAPTIVE_EXECUTION_ENABLED().key(),
+            "false",
+            SQLConf.AUTO_BROADCASTJOIN_THRESHOLD().key(),
+            "-1",
+            SparkSQLProperties.PRESERVE_DATA_GROUPING,
+            "true");
+
+    for (String vectorized : new String[] {"true", "false"}) {
+      sql(
+          "ALTER TABLE %s SET TBLPROPERTIES ('%s'='%s')",
+          tableName, TableProperties.PARQUET_VECTORIZATION_ENABLED, 
vectorized);
+
+      withSQLConf(
+          spjConf,
+          () -> {
+            Dataset<Row> df =
+                spark.sql(
+                    String.format(
+                        "SELECT * FROM %s a JOIN %s b ON a.dep = b.dep", 
tableName, tableName));
+            df.collect();
+
+            List<SparkPlan> scans =
+                
seqAsJavaListConverter(df.queryExecution().executedPlan().collectLeaves()).asJava();
+            assertThat(scans).hasSize(2);
+            for (SparkPlan scan : scans) {
+              assertThat(scan.metrics().get("numSplits").get().value())
+                  .as("numSplits with vectorization %s", vectorized)
+                  .isEqualTo(5L);

Review Comment:
   Good call, updated in 9bec07e6e. The table is now v2 merge-on-read with two 
rows per partition, and I delete one row in each of the 5 partitions before the 
join. The test now checks `numDeletes` is 5 next to `numSplits`, on both the 
vectorized and row paths. Without the `initMetricsValues` change it reports 2 
(one delete per task). Thanks!
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to