> it looks like if we want to get these problems fixed quickly we can just > remove > EnumerableValues's collation emitting.
I am afraid even removing Values collation enumeration won't actually give it a quick fix, because Multi-sorted table, if there is, might still encounter the same issue with Values. Thanks ~ Haisheng Yuan ------------------------------------------------------------------ 发件人：Hongze Zhang<[email protected]> 日 期：2019年04月16日 18:10:17 收件人：<[email protected]> 主 题：Re: [DISCUSS] RelCompositeTrait If we minimize the issue scope to Calcite itself, I think the 3 JIRA tickets: CALCITE-2010, CALCITE-2593, CALCITE-2764 that Haisheng has listed (thanks, Haisheng!) are all related to the multi-sorted EnumerableValue more or less. An it looks like if we want to get these problems fixed quickly we can just remove EnumerableValues's collation emitting. I recall that (correct me if I am wrong) the rel is even not able to emit descending collations, so I suppose it is not perfect at first. And another discussion is about enumerating traits. IMHO it's hard to tell Calcite didn't really try avoiding enumerating them already. The methods RelCollationImpl#satisfies[1] and RelDistributions#satisfies[2] already did a job of testing the relationship between traits without checking equality. So the whole thing is looking like we already tried to not to enumerate them but failed at last. Regarding the composite traits, one embarrassing thing I can see so far is about the method RelTraitSet#simplify[3]. The JavaDoc says the method is to "return a trait set similar to this one but with all composite traits flattened". But when we look into the related implementations RelCollationTraitDef#getDefault[4]/RelDistributionTraitDef#getDefault[5], they seem not to flatten anything, the traits just simply get wiped. This causes me to worry about if it is really correct that RelCollation/RelDistribution extends RelMultipleTrait, because we can't leverage the trait simplification but are actually hurt by it. If a rel loses it's physical property, we can never prevent from adding unnecessary sorts/exchanges. Besides, even if we decide to add some extra sorts/exchanges, that is somehow not easy so far. See CALCITE-2592/CALCITE-2970, the planner is not that smooth to automatically add them. Overall, regarding these "small" problems, I think none of them is really impossible to be solved (yes coming up right solutions may be not that straightforward). But of course in future if a brand new design can be proposed to improve the entire trait system (such as avoid enumerating traits), I think that would be totally a great thing. Best, Hongze [1]https://github.com/apache/calcite/blob/9538374e8fae5cec7d6f7b270850f5dfb4c1fc06/core/src/main/java/org/apache/calcite/rel/RelCollationImpl.java#L118 [2]https://github.com/apache/calcite/blob/9538374e8fae5cec7d6f7b270850f5dfb4c1fc06/core/src/main/java/org/apache/calcite/rel/RelDistributions.java#L143 [3]https://github.com/apache/calcite/blob/9538374e8fae5cec7d6f7b270850f5dfb4c1fc06/core/src/main/java/org/apache/calcite/plan/RelTraitSet.java#L526-L538 [4]https://github.com/apache/calcite/blob/9538374e8fae5cec7d6f7b270850f5dfb4c1fc06/core/src/main/java/org/apache/calcite/rel/RelCollationTraitDef.java#L58-L60 [5]https://github.com/apache/calcite/blob/9538374e8fae5cec7d6f7b270850f5dfb4c1fc06/core/src/main/java/org/apache/calcite/rel/RelDistributionTraitDef.java#L47-L49 ------ Original Message ------ From: "Haisheng Yuan" <[email protected]> To: "Jacques Nadeau" <[email protected]>; "Apache Calcite dev list" <[email protected]> Sent: 2019/4/15 12:27:04 Subject: Re: Re: [DISCUSS] RelCompositeTrait >> There are major challenges with asking for particular traits as well. >Imagine a desired aggregate on 7 columns. What does the requestor request >with regards to distribution? All seven columns? One column? Some >combination in between? > >The same challenges exist for enumerating all the traits as well. Imagine >there is an order by the 7 grouping keys on top of the aggregate on 7 columns, >but with different sort direction: >select * from foo group by a,b,c... order by c desc, a asc, b desc... >What sort order, direction should the sort-based stream aggregate provide? >All ascending, all descending, order (a,b,c...), order(..c,b,a), or all the >combination? >All of those enumerated traits are useless except one; for others, additional >sort operator will be needed. > >Another example is aggregate on top of join, where join on 7 keys, and >aggregate >on 2 of the join keys. In distributed system, what distribution trait would >the join >operator provide? The 2 grouping keys? All the join keys? All the combination? > >Enumerating some/all the deliverable traits, is not prupose driven. All the >traits >may be just useless for parent operator. On the other hand, asking the child >operator particular traits, is purpose driven, at least the traits asked by >parent >operator are worth consideration, not as wasteful as the former. > >If I understand RelCompositeTrait's intent correctly, the enumerated traits, no >matter some combination or all combination, should be saved here. But in fact, >it seems not. And as Jacques mentioned, many people rely on RelMetadata >operations to pull up the traitsets through operators. > >This makes me curious and wonder if there are any true use cases or systems >who rely on RelCompositeTrait. If someone has the story, we would love to hear. > >Put that aside, even RelCompositeTrait is indispensible, why do we bother >optimizing >Values node? For values with several tuples, it is not worth optimization, with >many tuples, it may take more time to enumerate the RelCollation than just >sorting it. >Specifically for Values with 0 or 1 tuple, but with many columns, it is >definitely not >worth the optimization, because sort removal rule and empty rel removal rule >should >do the work. > > >Thanks ~ >Haisheng Yuan >------------------------------------------------------------------ >发件人：Jacques Nadeau<[email protected]> >日 期：2019年04月15日 07:36:51 >收件人：<[email protected]> >主 题：Re: [DISCUSS] RelCompositeTrait > >There are major challenges with asking for particular traits as well. >Imagine a desired aggregate on 7 columns. What does the requestor request >with regards to distribution? All seven columns? One column? Some >combination in between? The trait system in Calcite is very challenging to >work with because it is up to downstream users to try to figure out trait >propagation outside the core. So challenging, that I believe that many >people move to relying on RelMetadata operations since those can be pulled >across several operators at once. > >It would be great if someone could spend the time to come up with a more >global design for these items and we avoid solving one-off problems. >Rationalizing when something should be trait, how to avoid trait planning >cost explosion, how to propagate, when something should be handled via >RelMetadataQuery, when something should be managed via traits versus >materialized view alternatives, etc. > >An example of overlapping functionality I'd start with is: should >multitraits for collation really exist or would exposing these as >materialized view alternatives be more appropriate? Why is it necessary to >have a 'shortcut' for this situation while other alternatives don't have >one? > > > >On Mon, Apr 8, 2019 at 4:38 PM Julian Hyde <[email protected]> wrote: > >> It seemed reasonable when I introduced it, and seems very reasonable, that >> a relational expression (even in the relational model) can have multiple >> physical properties. Consider these questions that the planner might ask: >> >> Example 1: >> >> “Are you sorted on hiredate?” >> “Yes” >> “Are you sorted on empno?” >> “Yes” >> “Are you sorted on deptno?” >> “No” >> >> Example 2: >> >> “Can you fit into less than 100MB of memory?” >> “Yes” >> “Can you fit into less than 10MB of memory?” >> “Yes” >> “Can you fit into less than 1MB of memory?” >> “No” >> >> We manage traits like those in example 1 using RelCompositeTrait. We can’t >> handle traits like this in example 2, and so we have trained ourselves to >> not think of “can fit into memory X” as a trait at all. >> >> Perhaps our mistake is to have an API “tell me all of your traits” rather >> than an API “do you have trait X?”. Asking a RelNode to enumerate its >> traits can be painful: the extreme case is an empty Values with 100 >> columns; it satisfies any sort order, and there are 100! of these. >> >> Julian >> >> >> >> > On Apr 8, 2019, at 3:51 PM, Stamatis Zampetakis <[email protected]> >> wrote: >> > >> > Hi Haisheng, >> > >> > Thanks for raising awareness around this topic. I also think we should >> try >> > to find a solution. >> > >> > Initially, the Volcano planner was designed to be able to cover multiple >> > models (and not only the relational). For non-relational models composite >> > traits may be indispensable. I don't know if there are people in this >> list >> > that are using the planner for other models but if there are it would be >> > nice to hear from them. >> > >> > Focusing exclusively on the relational model, I think composite traits >> are >> > useful. One use-case that comes to my mind is data replication. It >> > perfectly makes sense to partition (distribute) your table on two (or >> more) >> > columns to be able execute efficiently queries using special partition >> > joins. A concrete use-case is RDF data where many distributed systems >> store >> > the triples table partitioned by subject and object. I guess such >> use-cases >> > could possibly be modelled in other ways but composite traits is what >> comes >> > naturally to my mind. >> > >> > Regarding multi-sorted tables it is not that rare if you import sorted >> data >> > into a table with an auto-increment primary key for example. >> > >> > I think all the trait-related issues can be solved if we prioritize them >> > correctly. Apart from Vladimir and Hongze, who already spend quite some >> > time on these, the rest of us should also jump in and try to help. >> > >> > Best, >> > Stamatis >> > >> > >> > >> > >> > On Sun, Apr 7, 2019 at 9:48 AM Haisheng Yuan <[email protected]> >> wrote: >> > >> >> Hi, >> >> >> >> I found there are some RelCompositeTrait related issues: >> >> https://issues.apache.org/jira/browse/CALCITE-2010 >> >> https://issues.apache.org/jira/browse/CALCITE-2593 >> >> https://issues.apache.org/jira/browse/CALCITE-2764 >> >> >> >> Multi-sorted table are rare in pratice, mutil-distributed table doesn't >> >> exist either. Values node with several tuples is not worth optimization, >> >> with many tuples is not worth optimization either, because the time it >> >> takes optimizer to figure out the ordering may be longer than just sort >> it >> >> in runtime. >> >> >> >> In issue https://issues.apache.org/jira/browse/CALCITE-1990, >> >> Leo extended RelDistribution to inherit RelMultipleTrait, just like >> >> RelCollation does, to solve his problem in the example. But I don't >> think >> >> this is an appropriate way to represent the equivalence classes (in >> >> PostgreSQL's term). >> >> >> >> So why did we introduce RelCompisteTrait and RelMultipleTrait in the >> >> beginning? Seems like it gives us more pain than gain. >> >> >> >> Thanks ~ >> >> Haisheng Yuan >> >> >> >> >
