[
https://issues.apache.org/jira/browse/PARQUET-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17275608#comment-17275608
]
Yuming Wang commented on PARQUET-1805:
--------------------------------------
It seems that the previous configuration is better, enabling bloom filter
seriously affects the writing performance:
{code:scala}
val numRows = 1024 * 1024 * 15
val df = spark.range(numRows).selectExpr(
"id",
"cast(id as string) as s",
"cast(id as timestamp) as ts",
"cast(cast(id as timestamp) as date) as td",
"cast(id as decimal) as dec")
val benchmark = new org.apache.spark.benchmark.Benchmark(
"Benchmark bloom filter write",
numRows,
minNumIters = 5)
Seq(false, true).foreach { pushDownEnabled =>
val name = s"Write parquet ${if (pushDownEnabled) s"(bloom filter)" else ""}"
benchmark.addCase(name) { _ =>
withSQLConf(org.apache.parquet.hadoop.ParquetOutputFormat.BLOOM_FILTER_ENABLED
-> s"$pushDownEnabled") {
df.write.mode("overwrite").parquet("/tmp/spark/parquet")
}
}
}
benchmark.run()
{code}
{noformat}
Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.15.7
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark bloom filter write: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Write parquet 5531 6001
503 2.8 351.6 1.0X
Write parquet (bloom filter) 10529 11633
1113 1.5 669.4 0.5X
{noformat}
> Refactor the configuration for bloom filters
> --------------------------------------------
>
> Key: PARQUET-1805
> URL: https://issues.apache.org/jira/browse/PARQUET-1805
> Project: Parquet
> Issue Type: Improvement
> Reporter: Gabor Szadovszky
> Assignee: Gabor Szadovszky
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> Refactor the hadoop configuration for bloom filters according to PARQUET-1784.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)