[
https://issues.apache.org/jira/browse/FLINK-10474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644347#comment-16644347
]
Hequn Cheng commented on FLINK-10474:
-------------------------------------
Hi all,
Thanks for your comments. I think there are two things here:
1. Don't translate IN with Literals to JOIN
2. Convert a cascade of predicates to a HashSet way.
For the first problem, I increase the threshold to avoid translating IN to
JOIN. For the second problem, I added a rule to convert the predicates into an
IN. The first problem can also benefit from the solution of the second problem.
I think it is a good choice to use rule to optimize our query during
optimization. It is quite similar to the Calc rules. There are tons of Calc
rules, but i think it would be fine as long as there are orthogonal to each
other.
> Don't translate IN with Literals to JOIN with VALUES for streaming queries
> --------------------------------------------------------------------------
>
> Key: FLINK-10474
> URL: https://issues.apache.org/jira/browse/FLINK-10474
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
> Affects Versions: 1.6.1, 1.7.0
> Reporter: Fabian Hueske
> Assignee: Hequn Cheng
> Priority: Major
> Labels: pull-request-available
>
> IN predicates with literals are translated to JOIN with VALUES if the number
> of elements in the IN clause exceeds a certain threshold. This should not be
> done, because a streaming join is very heavy and materializes both inputs
> (which is fine for the VALUES) input but not for the other.
> There are two ways to solve this:
> # don't translate IN to a JOIN at all
> # translate it to a JOIN but have a special join strategy if one input is
> bound and final (non-updating)
> Option 1. should be easy to do, option 2. requires much more effort.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
