iemejia commented on code in PR #12400: URL: https://github.com/apache/gluten/pull/12400#discussion_r3549059957
########## backends-velox/src/test/scala/org/apache/spark/sql/execution/benchmark/FileHandleCacheBenchmark.scala: ########## @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.spark.sql.execution.benchmark + +import org.apache.gluten.config.VeloxConfig + +import org.apache.spark.benchmark.Benchmark + +/** + * Benchmark to measure the effect of Velox file handle caching on repeated scans of many small + * Parquet files. File handle caching avoids repeated open/close overhead per file, which is + * especially beneficial for remote filesystems (S3, HDFS, ABFS) where handle creation involves + * network round-trips (DNS, TCP, auth). + * + * Even on local filesystems the overhead is measurable when scanning hundreds of small files + * multiple times (e.g., repeated queries on a heavily-partitioned table). + * + * NOTE: `fileHandleCacheEnabled` is a static config (read at backend init). To compare on vs off, + * run this benchmark twice with different Spark configurations: + * {{{ + * # With file handle cache enabled (default): + * bin/spark-submit --class <this class> \ + * --conf spark.gluten.sql.columnar.backend.velox.fileHandleCacheEnabled=true \ + * --jars <spark core test jar> <sql core test jar> + * + * # With file handle cache disabled: + * bin/spark-submit --class <this class> \ + * --conf spark.gluten.sql.columnar.backend.velox.fileHandleCacheEnabled=false \ + * --jars <spark core test jar> <sql core test jar> Review Comment: Fixed. Changed `--jars` to use comma-separated values and added the application jar as the final positional argument, matching `spark-submit` syntax. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
