Thx, Julian I think the hint path is a good way for searching RelNode’s parents, broadly, there may be these modules/things need to be modified:
1. Supports hints grammar for parser.jj 2. Cache the hints in the RelNode instance, and add method like RelNode#getHints() to fetch all the hints inherited for this node. 3. Modify #copy method for every kind of RelNode so that the hints can be copied when creating new equivalent nodes. 4. Add a visitor in after sql-to-rel phrase, to set up full hints list for every children RelNode if there exists any. 5. Add hints metadata handler and handles the hints fetching and overriding for specific kind of RelNode The 2 and 3 are the modifications that i really want to confirm, that is, shall we store the hints in the RelNode instance ? These are initial thoughts and if we make agreement, I would output a detail design doc which contains: 1. The hints grammar supported for the major sql engines 2. The hints grammar supported for Apache Calcite 3. The interface and design ideas of the proposed modifications Best, Danny Chan 在 2019年4月24日 +0800 AM3:04，Julian Hyde <[email protected]>，写道： > I see that if you have a hint on, say, the root node then it would be nice > for its child or grand-child to be able to see that hint. > > How about giving each hint an inherit path? Thus given > > Filter Hint1 > +- Join > +- Scan > +- Project Hint2 > +- Scan > > > Filter would have hints {Hint1[]} > Join would have hints {Hint1[0]} > Scan would have hints {Hint1[0, 0]} > Project would have hints {Hint1[0,1], Hint2} > Scan2 would have hints {[Hint1[0, 0, 1, 0], Hint2[0]} > > You could populate the hints and inherit paths with a single visitor pass > after sql-to-rel conversion. > > By the way, I still like the idea of having kinds as a kind of RelMetadata, > but I realize that a given RelNode might have more than one hint. So I think > that the getHints(RelNode) method would return a List<Hint>, with Hint as > follows: > > class Hint { > public final List<Integer> inheritPath; // immutable, not null > public final String type; // not null > public final Object operand; // immutable, may be null, must be JSON data > } > > operand must be JSON-style data (null, boolean, number, String, immutable > List of JSON data, or immutable order-preserving Map from String to JSON > data). > > > On Apr 23, 2019, at 1:25 AM, Yuzhao Chen <[email protected]> wrote: > > > > Thx, Andrew > > > > I don’t want to have a custom RelNode class, I hope all the work about > > hints would be contributed to the community. I want to find an acceptable > > way to keep and propagate the hints if we use the MetadataHandler to cache > > and query the hints. > > > > I don’t think the hints should be mixed into the cost model, that would > > make the cost computation very complex and hard to maintain, we only need > > the hints in our planning phrase to give suggestions, hints is more like > > another guideline for me and transparent to the planner. > > > > Best, > > Danny Chan > > 在 2019年4月23日 +0800 PM2:24，Андрей Цвелодуб <[email protected]>，写道： > > > Hi Danny, > > > > > > I would also agree with Julian on his position. I've tried to get around > > > this limitation in several different ways, but none of it ended well :) > > > > > > For your idea with hints, if you have custom RelNode classes, you can add > > > hint as an additional field of the class and you can write a simple rule > > > that propagates the hint downwards, step by step. And also include the > > > hint > > > in your cost estimation, so that nodes with hints would be more attractive > > > to the planner. I'm not sure this would be the most correct way to use the > > > cost mechanism, but at least it is straightforward and it works. > > > > > > Best Regards, > > > Andrew Tsvelodub > > > > > > On Tue, 23 Apr 2019 at 08:44, Yuzhao Chen <[email protected]> wrote: > > > > > > > Julian, > > > > > > > > I want to add hint support for Calcite, the initial idea was to tag a > > > > RelNode(transformed from a SqlNode with hint) with a hit attribute(or > > > > trait), then I hope that the children (inputs) of it can see this hint, > > > > so > > > > to make some decisions if it should consume or propagate the hint. > > > > > > > > The problem I got here is the trait propagate from inputs from, which is > > > > the opposite as what I need, can you give some suggestions ? If I use > > > > MetadataHandler to cache and propagate the hints, how to propagate from > > > > parents to children ? > > > > > > > > Best, > > > > Danny Chan > > > > 在 2019年4月23日 +0800 AM3:14，Julian Hyde <[email protected]>，写道： > > > > > TL;DR: RelNodes don’t really have parents. Be careful if you are > > > > > relying > > > > on the parent concept too much. Rely on rules instead. > > > > > > > > > > In the Volcano model, a RelNode doesn’t really have a parent. It might > > > > be used in several places. (RelSet has a field ‘List<RelNode> parents’ > > > > that > > > > is kept up to date as planing progresses. But it’s really for Volcano’s > > > > internal use.) > > > > > > > > > > Even if you are not using Volcano, there are reasons to want the > > > > > RelNode > > > > graph to be a dag, so again, a RelNode doesn’t have a unique parent. > > > > > > > > > > RelShuttleImpl has a stack. You can use that to find the parent. But > > > > > the > > > > “parent” is just “where we came from as we traversed the RelNode graph”. > > > > There may be other “parents” that you do not know about. > > > > > > > > > > If you have a Project and want to find all parents that are Filters, > > > > don’t even think about “iterating over the parents” of the Project. Just > > > > write a rule that matches a Filter on a Project, and trust Volcano to do > > > > its job. > > > > > > > > > > Julian > > > > > > > > > > > > > > > > > > > > > > > > > > On Apr 22, 2019, at 6:15 AM, Yuzhao Chen <[email protected]> > > > > > > wrote: > > > > > > > > > > > > Thx, Stamatis, that somehow make sense, if i pass around the parent > > > > node every time I visit a RelNode and keep the parents in the cache, > > > > but it > > > > is still not that intuitive. Actually I what a to add a new RelTrait > > > > which > > > > bind to a specific scope, for example: > > > > > > > > > > > > join-rel(trait1) > > > > > > / \ > > > > > > join2 join3 > > > > > > > > > > > > Join-rel has a trait trait1, and I want all the children of join-rel > > > > can see this trait, with Calcite’s default metadata handler, I can only > > > > see > > > > the trait from children nodes(traits propagate from the inputs), and I > > > > have > > > > no idea how to propagate a trait reversely? > > > > > > > > > > > > > > > > > > Best, > > > > > > Danny Chan > > > > > > 在 2019年4月22日 +0800 PM8:44，Stamatis Zampetakis > > > > > > <[email protected]>，写道： > > > > > > > Hi Danny, > > > > > > > > > > > > > > Apart from RelShuttle there is also RelVisitor which has a visit > > > > method > > > > > > > that provides the parent [1]. Not sure, if it suits your needs. > > > > > > > > > > > > > > Best, > > > > > > > Stamatis > > > > > > > > > > > > > > [1] > > > > > > > > > > > https://github.com/apache/calcite/blob/ee83efd360793ef4201f4cdfc2af8d837b76ca69/core/src/main/java/org/apache/calcite/rel/RelVisitor.java#L43 > > > > > > > > > > > > > > > > > > > > > On Mon, Apr 22, 2019 at 2:14 PM Yuzhao Chen <[email protected]> > > > > wrote: > > > > > > > > > > > > > > > Now for RelNode, we have method getInput()[1] to fetch the input > > > > > > > > RelNodes, but how we fetch the parent ? > > > > > > > > > > > > > > > > For example, we have plan: > > > > > > > > > > > > > > > > join-rel > > > > > > > > / \ > > > > > > > > scan1 scan2 > > > > > > > > > > > > > > > > > > > > > > > > We can get scan1 and scan2 in join-rel directly with method > > > > getInput, but > > > > > > > > how can we get the join rel in scan1 and scan 2 ? > > > > > > > > > > > > > > > > I know that there is a RelShuttle that can visit every RelNode > > > > > > > > and > > > > if I > > > > > > > > make a cache for the inputs mapping, finally I can get the > > > > ‘parents’ from > > > > > > > > the cache, but this is boring code and not that intuitive. > > > > > > > > > > > > > > > > Do you guys have any good ideas ? > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > https://github.com/apache/calcite/blob/ee83efd360793ef4201f4cdfc2af8d837b76ca69/core/src/main/java/org/apache/calcite/rel/RelNode.java#L132 > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > Danny Chan > > > > > > > > > > > > > > > > > >
