rahil-c commented on code in PR #19575: URL: https://github.com/apache/hudi/pull/19575#discussion_r3859134744
########## hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRliLookupMetricsOnSparkSql.scala: ########## @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hudi.functional + +import org.apache.hudi.DataSourceWriteOptions._ +import org.apache.hudi.common.config.HoodieMetadataConfig +import org.apache.hudi.config.HoodieIndexConfig +import org.apache.hudi.metrics.RecordIndexMetricNames + +import org.apache.spark.sql.SaveMode +import org.junit.jupiter.api.{Tag, Test} +import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} + +/** Record level index lookup counters on the Spark SQL write path. */ +@Tag("functional") +class TestRliLookupMetricsOnSparkSql extends RliLookupMetricsTestBase { + + private val sqlTable = "rli_lookup_metrics_tbl" + private val numSeedRecords = 60 + + /** + * Seeds a table through the DataSource so the record index exists, then exposes it to SQL and applies + * the index settings as session configs -- index type is a write config, not a table property. + */ + private def seedTableAndRegisterForSql(): Unit = { + doWriteAndValidateDataAndRecordIndex(rliOpts, INSERT_OPERATION_OPT_VAL, SaveMode.Overwrite, + validate = false, numInserts = numSeedRecords) + + spark.sql(s"drop table if exists $sqlTable") + spark.sql(s"create table $sqlTable using hudi location '$basePath'") + + spark.sql("set hoodie.write.lock.provider = org.apache.hudi.client.transaction.lock.InProcessLockProvider") + spark.sql(s"set ${HoodieMetadataConfig.ENABLE.key} = true") + spark.sql(s"set ${HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key} = ${!isPartitionedRli}") + spark.sql(s"set ${HoodieMetadataConfig.RECORD_LEVEL_INDEX_ENABLE_PROP.key} = $isPartitionedRli") + spark.sql(s"set ${HoodieIndexConfig.INDEX_TYPE.key} = " + + (if (isPartitionedRli) "RECORD_LEVEL_INDEX" else "GLOBAL_RECORD_LEVEL_INDEX")) + + clearRliRegistry() + } + + /** + * The default path. Optimized writes make UPDATE a prepped write, so no index lookup happens and no + * counters are produced. Documented behaviour, asserted so it cannot change unnoticed. + */ + @Test + def testUpdateWithOptimizedWritesPerformsNoLookup(): Unit = { + seedTableAndRegisterForSql() + spark.sql(s"set ${SPARK_SQL_OPTIMIZED_WRITES.key} = true") + + spark.sql(s"update $sqlTable set rider = 'rider-optimized'") + + val counters = rliCountersFromLatestCommit() + report(s"Spark SQL UPDATE, optimized writes ON ($indexLabel) -- expected empty", counters) + assertTrue(counters.isEmpty, Review Comment: Reversed -- the class is back in the PR, and your point stands in it. What looked like a property of the Spark SQL write path was a bug in the registry key: it digests the raw base path, and SQL builds its config from the catalog location (scheme-qualified) while the DataSource passes the bare path, so one table got two keys and the drain looked under the wrong one. Fixed by digesting authority + path. The class now passes 6/6 including the non-prepped UPDATE and MERGE INTO cases. Your original comment here is addressed too: the prepped-UPDATE test is no longer a vacuous pass, because the sibling cases in the same class now prove the feature works on this path. ########## hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java: ########## @@ -278,11 +279,43 @@ public String getApplicationId() { return javaSparkContext.sc().applicationId(); } + /** + * Drops a registry from both process-wide maps. Only for tests that create their own SparkContexts: + * without it they leave accumulators bound to stopped contexts behind for whatever runs next in the + * same JVM. + */ + @VisibleForTesting + public static void removeMetricRegistryForTesting(String tableName, String registryName) { + DISTRIBUTED_REGISTRY_MAP.remove(tableName.isEmpty() ? registryName : tableName + "." + registryName); + Registry.REGISTRY_MAP.remove(Registry.makeKey(tableName, registryName)); + } + @Override public Registry getMetricRegistry(String tableName, String registryName) { Review Comment: One more piece of evidence for this, found after the fact. The base-path digest was not just awkward, it was silently wrong: it digests the raw string, so a table reached through a scheme-qualified path (Spark SQL, via the catalog location) hashed differently from the same table reached through a bare path (DataSource), and the drain looked under a key nothing had registered. Spark SQL reported no counters at all as a result. Fixed in place by digesting authority + path. But the digest only exists to disambiguate inside the process-wide map, so an instance-scoped registry removes the failure mode rather than correcting it -- which strengthens the case for the follow-up you are describing here. ########## hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetricRegistry.java: ########## @@ -0,0 +1,107 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hudi.metrics; + +import org.apache.hudi.common.metrics.Registry; +import org.apache.hudi.config.HoodieWriteConfig; + +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.util.function.Predicate; + +/** + * Every class of executor-collected metric, and the only thing a new one is added to. The driver must + * declare it up front because an {@code AccumulatorV2} must be registered with the {@code SparkContext} + * before a task can contribute; the bundle sent to executors and the commit drain both iterate this. + */ +public enum ExecutorMetricRegistry implements ExecutorMetricGroup { + + RECORD_INDEX_LOOKUP( + "HoodieRecordIndexLookup", + "hoodie.rli.lookup.", + "rli", + "lookup", + HoodieWriteConfig::isRecordIndexLookupMetricsEnabled); + + private final String registryName; + private final String commitMetadataPrefix; + private final String metricAction; + private final String metricQualifier; + private final Predicate<HoodieWriteConfig> enabled; + + ExecutorMetricRegistry(String registryName, String commitMetadataPrefix, String metricAction, + String metricQualifier, Predicate<HoodieWriteConfig> enabled) { + this.registryName = registryName; + this.commitMetadataPrefix = commitMetadataPrefix; + this.metricAction = metricAction; + this.metricQualifier = metricQualifier; + this.enabled = enabled; + } + + /** The bare name emitting code passes to {@link Registry#getRegistry(String)}. */ + @Override + public String registryName() { + return registryName; + } + + @Override + public String commitMetadataPrefix() { + return commitMetadataPrefix; + } + + @Override + public String metricAction() { + return metricAction; + } + + @Override + public String metricQualifier() { + return metricQualifier; + } + + /** Gating here, rather than in the drain, is what lets a new class of metric need no new config. */ + @Override + public boolean isEnabled(HoodieWriteConfig config) { + return enabled.test(config); + } + + /** + * Driver-side {@code REGISTRY_MAP} key. Table name alone is not an identity: two tables can share one + * and would then share a registry. Executors use the bare {@link #registryName()}. + */ + @Override + public String scopedName(String basePath) { Review Comment: Related to this, and worth recording: the digest in this name was not only ugly, it was wrong. Digesting the raw base path meant one table hashed two ways -- scheme-qualified from the Spark SQL catalog location, bare from the DataSource -- so that write path silently reported nothing at all. Now fixed by digesting authority + path. That does not change your point. The digest exists only because the map is process-wide, and both it and the common-scrape republish go away together when the registry becomes instance-scoped. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
