[
https://issues.apache.org/jira/browse/PARQUET-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276312#comment-17276312
]
Yuming Wang commented on PARQUET-1805:
--------------------------------------
Thank you [~gszadovszky] [~junjie] This is what I want:
{code:sql}
set parquet.bloom.filter.enabled=false;
set parquet.bloom.filter.enabled#ts=true;
set parquet.bloom.filter.enabled#dec=true;
{code}
Benchmark and benchmark result:
{code:scala}
val numRows = 1024 * 1024 * 15
val df = spark.range(numRows).selectExpr(
"id",
"cast(id as string) as s",
"cast(id as timestamp) as ts",
"cast(cast(id as timestamp) as date) as td",
"cast(id as decimal) as dec")
val benchmark = new org.apache.spark.benchmark.Benchmark(
"Benchmark bloom filter write",
numRows,
minNumIters = 5)
benchmark.addCase("default") { _ =>
withSQLConf() {
df.write.mode("overwrite").parquet("/tmp/spark/parquet")
}
}
benchmark.addCase("Build bloom filter for ts column") { _ =>
withSQLConf(
org.apache.parquet.hadoop.ParquetOutputFormat.BLOOM_FILTER_ENABLED ->
"false",
org.apache.parquet.hadoop.ParquetOutputFormat.BLOOM_FILTER_ENABLED + "#ts"
-> "true") {
df.write.mode("overwrite").parquet("/tmp/spark/parquet")
}
}
benchmark.addCase("Build bloom filter for ts and dec column") { _ =>
withSQLConf(
org.apache.parquet.hadoop.ParquetOutputFormat.BLOOM_FILTER_ENABLED ->
"false",
org.apache.parquet.hadoop.ParquetOutputFormat.BLOOM_FILTER_ENABLED + "#ts"
-> "true",
org.apache.parquet.hadoop.ParquetOutputFormat.BLOOM_FILTER_ENABLED + "#dec"
-> "true") {
df.write.mode("overwrite").parquet("/tmp/spark/parquet")
}
}
benchmark.addCase("Build bloom filter for all column") { _ =>
withSQLConf(
org.apache.parquet.hadoop.ParquetOutputFormat.BLOOM_FILTER_ENABLED ->
"true") {
df.write.mode("overwrite").parquet("/tmp/spark/parquet")
}
}
benchmark.run()
{code}
{noformat}
Java HotSpot(TM) 64-Bit Server VM 1.8.0_251-b08 on Mac OS X 10.15.7
Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Benchmark bloom filter write: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
default 5207 5314
72 3.0 331.1 1.0X
Build bloom filter for ts column 5808 6065
245 2.7 369.2 0.9X
Build bloom filter for ts and dec column 6685 6776
79 2.4 425.0 0.8X
Build bloom filter for all column 9077 9889
629 1.7 577.1 0.6X
{noformat}
cc [~dongjoon]
> Refactor the configuration for bloom filters
> --------------------------------------------
>
> Key: PARQUET-1805
> URL: https://issues.apache.org/jira/browse/PARQUET-1805
> Project: Parquet
> Issue Type: Improvement
> Reporter: Gabor Szadovszky
> Assignee: Gabor Szadovszky
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> Refactor the hadoop configuration for bloom filters according to PARQUET-1784.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)