[ https://issues.apache.org/jira/browse/TINKERPOP-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17927024#comment-17927024 ]
ASF GitHub Bot commented on TINKERPOP-3133: ------------------------------------------- ministat commented on code in PR #3026: URL: https://github.com/apache/tinkerpop/pull/3026#discussion_r1955477627 ########## spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/PersistedOutputRDD.java: ########## @@ -73,15 +74,29 @@ public <K, V> Iterator<KeyValue<K, V>> writeMemoryRDD(final Configuration config throw new IllegalArgumentException("There is no provided " + Constants.GREMLIN_HADOOP_OUTPUT_LOCATION + " to write the persisted RDD to"); final String memoryRDDName = Constants.getMemoryLocation(configuration.getString(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION), memoryKey); Spark.removeRDD(memoryRDDName); - memoryRDD.setName(memoryRDDName).persist(StorageLevel.fromString(configuration.getString(Constants.GREMLIN_SPARK_PERSIST_STORAGE_LEVEL, "MEMORY_ONLY"))) + final JavaPairRDD<K, V> javaPairRDD = repartitionJavaPairRDD(configuration, memoryRDD); + javaPairRDD.setName(memoryRDDName).persist(StorageLevel.fromString(configuration.getString(Constants.GREMLIN_SPARK_PERSIST_STORAGE_LEVEL, "MEMORY_ONLY"))) // call action to eager store rdd .count(); Spark.refresh(); // necessary to do really fast so the Spark GC doesn't clear out the RDD - return IteratorUtils.map(memoryRDD.collect().iterator(), tuple -> new KeyValue<>(tuple._1(), tuple._2())); + return IteratorUtils.map(javaPairRDD.collect().iterator(), tuple -> new KeyValue<>(tuple._1(), tuple._2())); } @Override public boolean supportsResultGraphPersistCombination(final GraphComputer.ResultGraph resultGraph, final GraphComputer.Persist persist) { return persist.equals(GraphComputer.Persist.NOTHING) || resultGraph.equals(GraphComputer.ResultGraph.NEW); } + + /** + * Allow users to customize the RDD partitions to reduce HDFS small files + */ + private static <K, V> JavaPairRDD<K, V> repartitionJavaPairRDD(final Configuration configuration, JavaPairRDD<K, V> graphRDD) { Review Comment: ok > Customize the file count by repartition the OutputRDD in Spark to reduce HDFS > small files > ----------------------------------------------------------------------------------------- > > Key: TINKERPOP-3133 > URL: https://issues.apache.org/jira/browse/TINKERPOP-3133 > Project: TinkerPop > Issue Type: Improvement > Components: hadoop > Affects Versions: 3.7.3 > Reporter: Redriver > Priority: Major > > The Graph export to HDFS through OutputRDD, but we often saw there are many > small files in production environment. For example, there are more than > 50,000 files and each is about 17 MB, which will trigger HDFS small files > alerts. So, it is better allow customize the output file numbers by > repartition the OutputRDD. -- This message was sent by Atlassian Jira (v8.20.10#820010)