Hi Vladmir, Thanks for your feedback.
I see your point. The example I gave above was not a good one. I agree with you that making the decision locally solves most problems. However, it may be sub-optimal for some scenarios: The project operator may contain unnecessary columns (even if the columns appear in the query). For example, the optimizer may determine that the column is always a constant, and should only be included in the final write. Best, Liya Fan On Fri, Dec 18, 2020 at 9:32 PM Vladimir Ozerov <[email protected]> wrote: > Hi Liya, > > I do not think, this is a global decision. If you go bottom-up in the plan, > and find the Project operator, than none parent operators will use any > columns from the lower scans, other than those present in the Project. > > It is a common practice for Calcite integrations to restrict the number of > columns returned from scans. It is achieved as follows: > 1) There should be rules to extract projections from some operators (e.g. > Joins) > 2) There should be rules to transpose projects with child operators > 3) Finally, you should create a rule to move project into scan for the > pattern [Project<-Scan] > > When all three prerequisites are ready, the optimizer will produce missing > projects (if any), then push them down the operator tree, and finally merge > into the scan. > > Regards, > Vladimir > > Пт, 18 дек. 2020 г. в 13:34, Fan Liya <[email protected]>: > > > Hi Bhavya, > > > > Thanks for your explanation. > > > > I do not think a rule like ProjectFilterTableScan solves the problem. > > To see this, please note that the parent operator of the LogicalProject > may > > need other columns from the table > > (e.g. the parent operator may be a join, which uses some other columns as > > the join key). > > > > So the column pruning of table scan should be a global decision, > depending > > on the whole plan. > > > > Calcite has RelMdColumnOrigins, which can help in this scenario. > > In particular, given an output column in the output operator, it gives > the > > dependent columns in the underlying source tables. > > So columns not referenced from the final output can be pruned from the > > source table. > > > > Best, > > Liya Fan > > > > > > On Fri, Dec 18, 2020 at 4:43 PM Bhavya Aggarwal <[email protected]> > > wrote: > > > > > Hi Liya, > > > > > > Please see below , I have a query as given below > > > > > > SELECT t_1.ID FROM USERS t_1 where NAME = 'HELLO'; > > > > > > Now I know that I have to fetch two columns from my columnar database > > i.e. > > > ID, Name, so need to pass these parameters in my custom Table Scan. > > > But when I convert the LogicalTableScan to my custom Table scan I do > not > > > have that information, please see the tree below, in the Logical Table > > Scan > > > the Input is null, I was hoping that I will have the columns in the > > > LogicalTableScan, The question is what do I need to do in Calcite to > pass > > > that information to the LogicalTableScan or in my physical operator. > The > > > way I have found is that I create a Rule like ProjectFilterTableScan > rule > > > and then extract what information I need. Is there a better way to do > > this? > > > > > > -----------------Logical JSON Plan -------------------- > > > > { > > > > "rels": [ > > > > { > > > > "id": "0", > > > > "relOp": "LogicalTableScan", > > > > "table": [ > > > > "USERS" > > > > ], > > > > "inputs": [] > > > > }, > > > > { > > > > "id": "1", > > > > "relOp": "LogicalFilter", > > > > "condition": { > > > > "op": { > > > > "name": "=", > > > > "kind": "EQUALS", > > > > "syntax": "BINARY" > > > > }, > > > > "operands": [ > > > > { > > > > "op": { > > > > "name": "CAST", > > > > "kind": "CAST", > > > > "syntax": "SPECIAL" > > > > }, > > > > "operands": [ > > > > { > > > > "input": 1, > > > > "name": "$1" > > > > } > > > > ], > > > > "type": { > > > > "type": "CHAR", > > > > "nullable": true, > > > > "precision": 5 > > > > } > > > > }, > > > > { > > > > "literal": "HELLO", > > > > "type": { > > > > "type": "CHAR", > > > > "nullable": false, > > > > "precision": 5 > > > > } > > > > } > > > > ] > > > > } > > > > }, > > > > { > > > > "id": "2", > > > > "relOp": "LogicalProject", > > > > "fields": [ > > > > "ID" > > > > ], > > > > "exprs": [ > > > > { > > > > "input": 0, > > > > "name": "$0" > > > > } > > > > ] > > > > } > > > > ] > > > > } > > > > > > > >> ----------------------------------------------------------- > > > > > > > > > > > Thanks and regards > > > Bhavya > > > > > > On Fri, Dec 18, 2020 at 11:54 AM Fan Liya <[email protected]> > wrote: > > > > > > > Hi Bhavya, > > > > > > > > Sorry I do not understand your question. Why is it difficult to pass > > sth. > > > > to child operators? > > > > > > > > Best, > > > > Liya Fan > > > > > > > > > > > > On Fri, Dec 18, 2020 at 12:38 PM Bhavya Aggarwal <[email protected] > > > > > > wrote: > > > > > > > > > Hi Liya, > > > > > > > > > > Actually the question is how to pass the projection columns to > Table > > > Scan > > > > > as right now in the LogicalTableScan there are no projection > columns > > > > being > > > > > passed so when I am trying to create my custom JSON , I do not have > > > those > > > > > projected columns or columns that are being used in the query. I > want > > > to > > > > > understand what is the calcite mechanism for passing it to child > > > > operators > > > > > without using Rules as it will be a lot of rule that we need to > > > > implement. > > > > > > > > > > Thanks and Regards > > > > > Bhavya > > > > > > > > > > On Fri, Dec 18, 2020 at 9:47 AM Fan Liya <[email protected]> > > wrote: > > > > > > > > > > > Hi Bhavya, > > > > > > > > > > > > It seems the problem is that the json format provided by Calcite > > does > > > > not > > > > > > include properties specific to your operator? > > > > > > If so, I think you can override RelNode#explainTerms in your > > operator > > > > to > > > > > > attach the properties. > > > > > > > > > > > > Best, > > > > > > Liya Fan > > > > > > > > > > > > On Thu, Dec 17, 2020 at 10:45 PM Bhavya Aggarwal < > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > Please let me know if we have an example for writing a custom > > > visitor > > > > > > that > > > > > > > I can use. I see a RelVisitor class that I can extend but how > to > > > make > > > > > it > > > > > > to > > > > > > > visit the specific Physical Operator. > > > > > > > > > > > > > > Thanks and Regards > > > > > > > Bhavya > > > > > > > > > > > > > > On Thu, Dec 17, 2020 at 7:43 PM Bhavya Aggarwal < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > I tried using the RelJsonWriter and tried to add my custom > > > > > > implementation > > > > > > > > in the my custom Rel Nodes but I am not getting all the data > in > > > my > > > > > > child > > > > > > > > operators. We are trying to use Calcite for a columnar data > and > > > we > > > > > need > > > > > > > the > > > > > > > > information of the projections in the TableScan as well > because > > > we > > > > > need > > > > > > > to > > > > > > > > pass those projections to read specific columns but I am not > > able > > > > to > > > > > > find > > > > > > > > the projections in my Table scan. As per my understanding > > Calcite > > > > > > should > > > > > > > be > > > > > > > > passing this information as it is required for me reading the > > > > table. > > > > > > > Please > > > > > > > > let me know if I need to do something special to get that > > > > information > > > > > > in > > > > > > > > the child operator. > > > > > > > > > > > > > > > > Regards > > > > > > > > Bhavya > > > > > > > > > > > > > > > > On Sun, Dec 13, 2020 at 11:32 PM Michael Mior < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > >> That would likely be the best approach if you have some > > specific > > > > > JSON > > > > > > > >> format you're trying to generate. If you're happy with a > JSON > > > > format > > > > > > > >> defined by Calcite, you can look at RelJsonWriter. > > > > > > > >> > > > > > > > >> -- > > > > > > > >> Michael Mior > > > > > > > >> [email protected] > > > > > > > >> > > > > > > > >> Le dim. 13 déc. 2020 à 05:36, Muhammad Gelbana < > > > > [email protected] > > > > > > > > > > > > a > > > > > > > >> écrit : > > > > > > > >> > > > > > > > > >> > I would use a visitor to traverse the optimized/physical > > plan. > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > On Sun, Dec 13, 2020 at 6:42 AM Bhavya Aggarwal < > > > > > [email protected] > > > > > > > > > > > > > > >> wrote: > > > > > > > >> > > > > > > > > >> > > Hi All, > > > > > > > >> > > > > > > > > > >> > > We need to generate a JSON object for the physical > > execution > > > > > tree > > > > > > > >> that has > > > > > > > >> > > been created. Is there an option in Calcite that we can > > use > > > to > > > > > do > > > > > > > >> this. I > > > > > > > >> > > am not sure what is the right approach to do it. Please > > let > > > me > > > > > > know > > > > > > > if > > > > > > > >> > > there are different ways to achieve this. > > > > > > > >> > > > > > > > > > >> > > Regards > > > > > > > >> > > Bhavya > > > > > > > >> > > > > > > > > > >> > > -- > > > > > > > >> > > Your feedback matters - At Knoldus we aim to be very > > > > > professional > > > > > > in > > > > > > > >> our > > > > > > > >> > > quality of work, commitment to results, and proactive > > > > > > communication. > > > > > > > >> If > > > > > > > >> > > you > > > > > > > >> > > feel otherwise please share your feedback > > > > > > > >> > > <https://forms.gle/Ax1Te1DDpirAQuQ8A> and we would work > > on > > > > it. > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > *Bhavya Aggarwal* > > > > > > > > CTO & Partner > > > > > > > > Knoldus Inc. <http://www.knoldus.com/> > > > > > > > > +91-9910483067 > > > > > > > > Canada - USA - India - Singapore > > > > > > > > <https://in.linkedin.com/company/knoldus> < > > > > > > https://twitter.com/Knolspeak > > > > > > > > > > > > > > > > <https://www.facebook.com/KnoldusSoftware/> < > > > > > https://blog.knoldus.com/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > *Bhavya Aggarwal* > > > > > > > CTO & Partner > > > > > > > Knoldus Inc. <http://www.knoldus.com/> > > > > > > > +91-9910483067 > > > > > > > Canada - USA - India - Singapore > > > > > > > <https://in.linkedin.com/company/knoldus> < > > > > > https://twitter.com/Knolspeak > > > > > > > > > > > > > > <https://www.facebook.com/KnoldusSoftware/> < > > > > https://blog.knoldus.com/ > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Your feedback matters - At Knoldus we aim to be very > professional > > > in > > > > > our > > > > > > > quality of work, commitment to results, and proactive > > > communication. > > > > If > > > > > > > you > > > > > > > feel otherwise please share your feedback > > > > > > > <https://forms.gle/Ax1Te1DDpirAQuQ8A> and we would work on it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > *Bhavya Aggarwal* > > > > > CTO & Partner > > > > > Knoldus Inc. <http://www.knoldus.com/> > > > > > +91-9910483067 > > > > > Canada - USA - India - Singapore > > > > > <https://in.linkedin.com/company/knoldus> < > > > https://twitter.com/Knolspeak > > > > > > > > > > <https://www.facebook.com/KnoldusSoftware/> < > > https://blog.knoldus.com/ > > > > > > > > > > > > > > -- > > > > > Your feedback matters - At Knoldus we aim to be very professional > in > > > our > > > > > quality of work, commitment to results, and proactive > communication. > > If > > > > > you > > > > > feel otherwise please share your feedback > > > > > <https://forms.gle/Ax1Te1DDpirAQuQ8A> and we would work on it. > > > > > > > > > > > > > > > > > > -- > > > *Bhavya Aggarwal* > > > CTO & Partner > > > Knoldus Inc. <http://www.knoldus.com/> > > > +91-9910483067 > > > Canada - USA - India - Singapore > > > <https://in.linkedin.com/company/knoldus> < > https://twitter.com/Knolspeak > > > > > > <https://www.facebook.com/KnoldusSoftware/> <https://blog.knoldus.com/ > > > > > > > > -- > > > Your feedback matters - At Knoldus we aim to be very professional in > our > > > quality of work, commitment to results, and proactive communication. If > > > you > > > feel otherwise please share your feedback > > > <https://forms.gle/Ax1Te1DDpirAQuQ8A> and we would work on it. > > > > > >
