> There are major challenges with asking for particular traits as well. Imagine a desired aggregate on 7 columns. What does the requestor request with regards to distribution? All seven columns? One column? Some combination in between?
The same challenges exist for enumerating all the traits as well. Imagine there is an order by the 7 grouping keys on top of the aggregate on 7 columns, but with different sort direction: select * from foo group by a,b,c... order by c desc, a asc, b desc... What sort order, direction should the sort-based stream aggregate provide? All ascending, all descending, order (a,b,c...), order(..c,b,a), or all the combination? All of those enumerated traits are useless except one; for others, additional sort operator will be needed. Another example is aggregate on top of join, where join on 7 keys, and aggregate on 2 of the join keys. In distributed system, what distribution trait would the join operator provide? The 2 grouping keys? All the join keys? All the combination? Enumerating some/all the deliverable traits, is not prupose driven. All the traits may be just useless for parent operator. On the other hand, asking the child operator particular traits, is purpose driven, at least the traits asked by parent operator are worth consideration, not as wasteful as the former. If I understand RelCompositeTrait's intent correctly, the enumerated traits, no matter some combination or all combination, should be saved here. But in fact, it seems not. And as Jacques mentioned, many people rely on RelMetadata operations to pull up the traitsets through operators. This makes me curious and wonder if there are any true use cases or systems who rely on RelCompositeTrait. If someone has the story, we would love to hear. Put that aside, even RelCompositeTrait is indispensible, why do we bother optimizing Values node? For values with several tuples, it is not worth optimization, with many tuples, it may take more time to enumerate the RelCollation than just sorting it. Specifically for Values with 0 or 1 tuple, but with many columns, it is definitely not worth the optimization, because sort removal rule and empty rel removal rule should do the work. Thanks ~ Haisheng Yuan ------------------------------------------------------------------ 发件人：Jacques Nadeau<[email protected]> 日 期：2019年04月15日 07:36:51 收件人：<[email protected]> 主 题：Re: [DISCUSS] RelCompositeTrait There are major challenges with asking for particular traits as well. Imagine a desired aggregate on 7 columns. What does the requestor request with regards to distribution? All seven columns? One column? Some combination in between? The trait system in Calcite is very challenging to work with because it is up to downstream users to try to figure out trait propagation outside the core. So challenging, that I believe that many people move to relying on RelMetadata operations since those can be pulled across several operators at once. It would be great if someone could spend the time to come up with a more global design for these items and we avoid solving one-off problems. Rationalizing when something should be trait, how to avoid trait planning cost explosion, how to propagate, when something should be handled via RelMetadataQuery, when something should be managed via traits versus materialized view alternatives, etc. An example of overlapping functionality I'd start with is: should multitraits for collation really exist or would exposing these as materialized view alternatives be more appropriate? Why is it necessary to have a 'shortcut' for this situation while other alternatives don't have one? On Mon, Apr 8, 2019 at 4:38 PM Julian Hyde <[email protected]> wrote: > It seemed reasonable when I introduced it, and seems very reasonable, that > a relational expression (even in the relational model) can have multiple > physical properties. Consider these questions that the planner might ask: > > Example 1: > > “Are you sorted on hiredate?” > “Yes” > “Are you sorted on empno?” > “Yes” > “Are you sorted on deptno?” > “No” > > Example 2: > > “Can you fit into less than 100MB of memory?” > “Yes” > “Can you fit into less than 10MB of memory?” > “Yes” > “Can you fit into less than 1MB of memory?” > “No” > > We manage traits like those in example 1 using RelCompositeTrait. We can’t > handle traits like this in example 2, and so we have trained ourselves to > not think of “can fit into memory X” as a trait at all. > > Perhaps our mistake is to have an API “tell me all of your traits” rather > than an API “do you have trait X?”. Asking a RelNode to enumerate its > traits can be painful: the extreme case is an empty Values with 100 > columns; it satisfies any sort order, and there are 100! of these. > > Julian > > > > > On Apr 8, 2019, at 3:51 PM, Stamatis Zampetakis <[email protected]> > wrote: > > > > Hi Haisheng, > > > > Thanks for raising awareness around this topic. I also think we should > try > > to find a solution. > > > > Initially, the Volcano planner was designed to be able to cover multiple > > models (and not only the relational). For non-relational models composite > > traits may be indispensable. I don't know if there are people in this > list > > that are using the planner for other models but if there are it would be > > nice to hear from them. > > > > Focusing exclusively on the relational model, I think composite traits > are > > useful. One use-case that comes to my mind is data replication. It > > perfectly makes sense to partition (distribute) your table on two (or > more) > > columns to be able execute efficiently queries using special partition > > joins. A concrete use-case is RDF data where many distributed systems > store > > the triples table partitioned by subject and object. I guess such > use-cases > > could possibly be modelled in other ways but composite traits is what > comes > > naturally to my mind. > > > > Regarding multi-sorted tables it is not that rare if you import sorted > data > > into a table with an auto-increment primary key for example. > > > > I think all the trait-related issues can be solved if we prioritize them > > correctly. Apart from Vladimir and Hongze, who already spend quite some > > time on these, the rest of us should also jump in and try to help. > > > > Best, > > Stamatis > > > > > > > > > > On Sun, Apr 7, 2019 at 9:48 AM Haisheng Yuan <[email protected]> > wrote: > > > >> Hi, > >> > >> I found there are some RelCompositeTrait related issues: > >> https://issues.apache.org/jira/browse/CALCITE-2010 > >> https://issues.apache.org/jira/browse/CALCITE-2593 > >> https://issues.apache.org/jira/browse/CALCITE-2764 > >> > >> Multi-sorted table are rare in pratice, mutil-distributed table doesn't > >> exist either. Values node with several tuples is not worth optimization, > >> with many tuples is not worth optimization either, because the time it > >> takes optimizer to figure out the ordering may be longer than just sort > it > >> in runtime. > >> > >> In issue https://issues.apache.org/jira/browse/CALCITE-1990, > >> Leo extended RelDistribution to inherit RelMultipleTrait, just like > >> RelCollation does, to solve his problem in the example. But I don't > think > >> this is an appropriate way to represent the equivalence classes (in > >> PostgreSQL's term). > >> > >> So why did we introduce RelCompisteTrait and RelMultipleTrait in the > >> beginning? Seems like it gives us more pain than gain. > >> > >> Thanks ~ > >> Haisheng Yuan > >> > >
