[
https://issues.apache.org/jira/browse/HIVE-18340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356426#comment-16356426
]
Ke Jia commented on HIVE-18340:
-------------------------------
[~stakiar]：
{quote}Hive-on-Tez's has an implementation of DynamicValueRegistry that uses
some special Tez APIs such as ProcessorContext#waitForAllInputsReady, how are
we simulating this in HoS?
{quote}
[~kellyzly],Yes, For HoS, I flush the runtime filter info (min/max and bloom
filter) to hdfs in SparkRuntimeFilterPruningSinkOperator operator and get the
info from hdfs in SparkRuntimeFilterPruner , which is similar as
SparkPartitionPruningSinkOperator and SparkDynamicPartitionPruner class in
Spark DPP.
{quote}It would be nice to have some qtests to help visualize what the explain
plan with RF would look like
{quote}
I upload the HIVE-18340.2.patch to add qtest "spark_runtime_filter_pruning.q"
and "spark_runtime_filter_pruning.q.out".
Thanks [~stakiar], [~kellyzly] for your review!
> Dynamic Min-Max/BloomFilter runtime-filtering in HoS
> ----------------------------------------------------
>
> Key: HIVE-18340
> URL: https://issues.apache.org/jira/browse/HIVE-18340
> Project: Hive
> Issue Type: New Feature
> Components: Spark
> Affects Versions: 3.0.0
> Reporter: Ke Jia
> Assignee: Ke Jia
> Priority: Major
> Attachments: HIVE-18340.1.patch, HIVE-18340.2.patch
>
>
> Tez implemented Dynamic Min-Max/BloomFilter runtime-filtering in HIVE-15269
> and we should implement the same in HOS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
