gudladona commented on code in PR #20533:
URL: https://github.com/apache/hudi/pull/20533#discussion_r4543839188


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/MultipleSparkJobExecutionStrategy.java:
##########
@@ -137,10 +184,67 @@ public HoodieWriteMetadata<HoodieData<WriteStatus>> 
performClustering(final Hood
       writeMetadata.setWriteStatuses(HoodieJavaRDD.of(writeStatusRDD));
       return writeMetadata;
     } finally {
+      // Every group has been submitted and joined: the groups' RDDs hold what 
they need from the run invariants.
+      runInvariants = null;
       clusteringExecutorService.shutdown();
     }
   }
 
+  /**
+   * What every group of one {@link #performClustering} run shares (see the 
class comment), fixed for the run: built on
+   * the driver before the groups are submitted, dropped when they have all 
been submitted. The caches only memoize:
+   * each derives its value from the key and the run's write config, so a 
key's value never changes within the run.
+   * Never serialized: groups capture its values, not the holder.
+   */
+  private static final class ClusteringRunInvariants {
+    /** The schema the groups' records have, with meta fields, as handed to 
performClusteringWithRecords*. */
+    private final HoodieSchema readerSchema;
+    private final ReaderContextFactory<?> readerContextFactory;
+    /** Row writer only: {@link #readerSchema} as Spark types. */
+    private final StructType sparkReaderSchema;
+    /** Row writer only: the file group readers' properties. */
+    private final TypedProperties readerProperties;
+    /** Groups run concurrently, on the clustering thread pool. */
+    private final ConcurrentMap<Option<Long>, HoodieWriteConfig> 
groupWriteConfigs = new ConcurrentHashMap<>();
+    /** Keyed by dataset schema and table schema: groups may read different 
schemas. */
+    private final ConcurrentMap<Pair<StructType, String>, StructType> 
groupWriterSchemas = new ConcurrentHashMap<>();
+
+    private ClusteringRunInvariants(HoodieSchema readerSchema, 
ReaderContextFactory<?> readerContextFactory,
+                               StructType sparkReaderSchema, TypedProperties 
readerProperties) {
+      this.readerSchema = readerSchema;
+      this.readerContextFactory = readerContextFactory;
+      this.sparkReaderSchema = sparkReaderSchema;
+      this.readerProperties = readerProperties;
+    }
+
+    @SuppressWarnings("unchecked")
+    private <R> ReaderContextFactory<R> readerContextFactory() {
+      return (ReaderContextFactory<R>) readerContextFactory;
+    }
+  }
+
+  /** The invariants of the {@link #performClustering} run that is submitting 
groups. */
+  private ClusteringRunInvariants currentRunInvariants() {
+    ClusteringRunInvariants run = runInvariants;
+    ValidationUtils.checkState(run != null, "Clustering groups are submitted 
by performClustering");

Review Comment:
   Thanks, done in 75b10814. With no run in progress, each hook now builds the 
invariants of its own write path for that one group, so such callers keep the 
old per-group cost but no longer break.
   
   `groupSubmittedOutsideARunIsWrittenInFull` calls each hook directly, with 
the row writer on and off, and checks that the group writes every record of its 
input files. Against the previous `checkState`, both cases fail with 
`IllegalStateException`.
   



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/clustering/run/strategy/MultipleSparkJobExecutionStrategy.java:
##########
@@ -137,10 +184,67 @@ public HoodieWriteMetadata<HoodieData<WriteStatus>> 
performClustering(final Hood
       writeMetadata.setWriteStatuses(HoodieJavaRDD.of(writeStatusRDD));
       return writeMetadata;
     } finally {
+      // Every group has been submitted and joined: the groups' RDDs hold what 
they need from the run invariants.

Review Comment:
   Good catch, done in 75b10814. The comment now says that on failure `join()` 
throws first, so a group may still be running: it keeps the reader state it 
captured when submitted, and builds its own write config and writer schema from 
there. The run fails either way.
   



##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/execution/SQLConfInjectingRDD.scala:
##########
@@ -58,4 +58,30 @@ class SQLConfInjectingRDD[T: ClassTag](var sqlRDD: RDD[T], 
@transient conf: SQLC
       firstParent[T].iterator(split, context)
     }
   }
-}
\ No newline at end of file
+}
+
+object SQLConfInjectingRDD {
+
+  /** The last config map captured; driver-side only, guarded by this object. 
See [[sharedConfigs]]. */
+  private var lastConfigs: Map[String, String] = Map.empty

Review Comment:
   Done in 75b10814, as a comment. I kept the strong reference: a weak one 
could be collected between two groups' captures, and the next group would then 
ship a map of its own, which is the per-group cost this change removes. The map 
holds what every `SQLConfInjectingRDD` already carries to its tasks, so keeping 
it adds no new place the configs go, only time past the session that set them. 
The comment says so.
   



##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/client/clustering/run/strategy/TestClusteringTaskBinaryPayloads.java:
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.client.clustering.run.strategy;
+
+import org.apache.hudi.avro.model.HoodieClusteringPlan;
+import org.apache.hudi.client.SparkRDDWriteClient;
+import org.apache.hudi.client.WriteClientTestUtils;
+import org.apache.hudi.client.WriteStatus;
+import 
org.apache.hudi.client.clustering.plan.strategy.SparkSingleFileSortPlanStrategy;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.ReaderContextFactory;
+import org.apache.hudi.common.schema.HoodieSchema;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.TableSchemaResolver;
+import org.apache.hudi.common.table.marker.MarkerType;
+import org.apache.hudi.common.util.ClusteringUtils;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.config.HoodieClusteringConfig;
+import org.apache.hudi.config.HoodieCompactionConfig;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.data.HoodieJavaRDD;
+import org.apache.hudi.table.HoodieSparkTable;
+import org.apache.hudi.testutils.HoodieClientTestBase;
+
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.rdd.RDD;
+import org.apache.spark.sql.types.StructType;
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.api.Test;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.io.OutputStream;
+import java.util.Collections;
+import java.util.IdentityHashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Predicate;
+import java.util.stream.Collectors;
+
+import static 
org.apache.hudi.common.testutils.HoodieTestUtils.INSTANT_GENERATOR;
+import static org.apache.hudi.testutils.Assertions.assertNoWriteErrors;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Measures what the write stage of {@link 
MultipleSparkJobExecutionStrategy#performClustering} ships to every task.
+ *
+ * <p>Clustering builds one RDD per group and unions them into a single write 
stage. Spark serializes the stage
+ * lineage into one task binary that every task deserializes, so an object a 
group's lineage captures is shipped
+ * to, and deserialized by, every task of the stage, for every group. Objects 
that do not depend on the group
+ * (table schema, write config, reader properties, reader context factory, SQL 
configs) should therefore be shared
+ * instances: Java serialization writes a shared instance once per stream.
+ *
+ * <p>The test runs the real strategy on a small and a large subset of one 
plan, serializes the resulting lineage
+ * the way Spark's closure serializer does (one {@link ObjectOutputStream}), 
and counts the distinct instances of
+ * each such payload. The counts must not grow with the number of groups.
+ */
+@Tag("functional")
+public class TestClusteringTaskBinaryPayloads extends HoodieClientTestBase {
+
+  private static final int SMALL_PLAN_GROUPS = 2;
+  private static final int LARGE_PLAN_GROUPS = 8;
+  // dataGen spreads inserts over 3 partitions; with small-file handling off, 
each batch adds one file group
+  // per partition, and the single-file plan strategy emits one clustering 
group per file group.
+  private static final int PARTITIONS = 3;
+  private static final int INSERT_BATCHES = (LARGE_PLAN_GROUPS + PARTITIONS - 
1) / PARTITIONS;
+  private static final int RECORDS_PER_BATCH = 30;
+  private static final String SQL_CONF_MAPS = "SQL config maps";
+
+  /** Payloads that do not depend on the group, so their count must not depend 
on the number of groups. */
+  private static final Map<String, Predicate<Object>> SHARED_PAYLOADS = new 
LinkedHashMap<>();
+
+  static {
+    SHARED_PAYLOADS.put("HoodieSchema", HoodieSchema.class::isInstance);
+    SHARED_PAYLOADS.put("HoodieWriteConfig", 
HoodieWriteConfig.class::isInstance);
+    SHARED_PAYLOADS.put("TypedProperties", TypedProperties.class::isInstance);
+    SHARED_PAYLOADS.put("ReaderContextFactory", 
ReaderContextFactory.class::isInstance);
+    SHARED_PAYLOADS.put("StructType", StructType.class::isInstance);
+  }
+
+  @Test
+  public void taskBinaryPayloadsDoNotGrowWithClusteringGroups() throws 
Exception {

Review Comment:
   Done in 75b10814. The test now runs `SparkSortAndSizeExecutionStrategy` and 
`SparkSingleFileSortExecutionStrategy`, each with the row writer on and off, 
and every shared payload stays flat in all four.
   
   One difference showed up: on the records path no group builds a DataFrame, 
so no SQL config map is shipped at all (0 at 2 and at 8 groups). The test 
expects the per-group `SQLExecutionRDD` copy only with the row writer.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to