[
https://issues.apache.org/jira/browse/PARQUET-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276854#comment-17276854
]
Yuming Wang commented on PARQUET-41:
------------------------------------
[~nchammas] You can check the related configuration parameters:
[https://github.com/apache/parquet-mr/tree/master/parquet-hadoopÂ
|https://github.com/apache/parquet-mr/tree/master/parquet-hadoop]This is an
example:
{code:scala}
val numRows = 1024 * 1024 * 15
val df = spark.range(numRows).selectExpr(
"id",
"cast(id as string) as s",
"cast(id as timestamp) as ts",
"cast(cast(id as timestamp) as date) as td",
"cast(id as decimal) as dec")
val benchmark = new org.apache.spark.benchmark.Benchmark(
"Benchmark bloom filter write",
numRows,
minNumIters = 5)
benchmark.addCase("default") { _ =>
withSQLConf() {
df.write.mode("overwrite").parquet("/tmp/spark/parquet")
}
}
benchmark.addCase("Build bloom filter for ts column") { _ =>
withSQLConf(
org.apache.parquet.hadoop.ParquetOutputFormat.BLOOM_FILTER_ENABLED ->
"false",
org.apache.parquet.hadoop.ParquetOutputFormat.BLOOM_FILTER_ENABLED + "#ts"
-> "true") {
df.write.mode("overwrite").parquet("/tmp/spark/parquet")
}
}
benchmark.addCase("Build bloom filter for ts and dec column") { _ =>
withSQLConf(
org.apache.parquet.hadoop.ParquetOutputFormat.BLOOM_FILTER_ENABLED ->
"false",
org.apache.parquet.hadoop.ParquetOutputFormat.BLOOM_FILTER_ENABLED + "#ts"
-> "true",
org.apache.parquet.hadoop.ParquetOutputFormat.BLOOM_FILTER_ENABLED + "#dec"
-> "true") {
df.write.mode("overwrite").parquet("/tmp/spark/parquet")
}
}
benchmark.addCase("Build bloom filter for all column") { _ =>
withSQLConf(
org.apache.parquet.hadoop.ParquetOutputFormat.BLOOM_FILTER_ENABLED ->
"true") {
df.write.mode("overwrite").parquet("/tmp/spark/parquet")
}
}
benchmark.run()
{code}
> Add bloom filters to parquet statistics
> ---------------------------------------
>
> Key: PARQUET-41
> URL: https://issues.apache.org/jira/browse/PARQUET-41
> Project: Parquet
> Issue Type: New Feature
> Components: parquet-format, parquet-mr
> Reporter: Alex Levenson
> Assignee: Junjie Chen
> Priority: Major
> Labels: filter2, pull-request-available
> Fix For: format-2.7.0, 1.12.0
>
>
> For row groups with no dictionary, we could still produce a bloom filter.
> This could be very useful in filtering entire row groups.
> Pull request:
> https://github.com/apache/parquet-mr/pull/215
--
This message was sent by Atlassian Jira
(v8.3.4#803005)