Hi Liya, I do not think, this is a global decision. If you go bottom-up in the plan, and find the Project operator, than none parent operators will use any columns from the lower scans, other than those present in the Project.
It is a common practice for Calcite integrations to restrict the number of columns returned from scans. It is achieved as follows: 1) There should be rules to extract projections from some operators (e.g. Joins) 2) There should be rules to transpose projects with child operators 3) Finally, you should create a rule to move project into scan for the pattern [Project<-Scan] When all three prerequisites are ready, the optimizer will produce missing projects (if any), then push them down the operator tree, and finally merge into the scan. Regards, Vladimir Пт, 18 дек. 2020 г. в 13:34, Fan Liya <[email protected]>: > Hi Bhavya, > > Thanks for your explanation. > > I do not think a rule like ProjectFilterTableScan solves the problem. > To see this, please note that the parent operator of the LogicalProject may > need other columns from the table > (e.g. the parent operator may be a join, which uses some other columns as > the join key). > > So the column pruning of table scan should be a global decision, depending > on the whole plan. > > Calcite has RelMdColumnOrigins, which can help in this scenario. > In particular, given an output column in the output operator, it gives the > dependent columns in the underlying source tables. > So columns not referenced from the final output can be pruned from the > source table. > > Best, > Liya Fan > > > On Fri, Dec 18, 2020 at 4:43 PM Bhavya Aggarwal <[email protected]> > wrote: > > > Hi Liya, > > > > Please see below , I have a query as given below > > > > SELECT t_1.ID FROM USERS t_1 where NAME = 'HELLO'; > > > > Now I know that I have to fetch two columns from my columnar database > i.e. > > ID, Name, so need to pass these parameters in my custom Table Scan. > > But when I convert the LogicalTableScan to my custom Table scan I do not > > have that information, please see the tree below, in the Logical Table > Scan > > the Input is null, I was hoping that I will have the columns in the > > LogicalTableScan, The question is what do I need to do in Calcite to pass > > that information to the LogicalTableScan or in my physical operator. The > > way I have found is that I create a Rule like ProjectFilterTableScan rule > > and then extract what information I need. Is there a better way to do > this? > > > > -----------------Logical JSON Plan -------------------- > > > { > > > "rels": [ > > > { > > > "id": "0", > > > "relOp": "LogicalTableScan", > > > "table": [ > > > "USERS" > > > ], > > > "inputs": [] > > > }, > > > { > > > "id": "1", > > > "relOp": "LogicalFilter", > > > "condition": { > > > "op": { > > > "name": "=", > > > "kind": "EQUALS", > > > "syntax": "BINARY" > > > }, > > > "operands": [ > > > { > > > "op": { > > > "name": "CAST", > > > "kind": "CAST", > > > "syntax": "SPECIAL" > > > }, > > > "operands": [ > > > { > > > "input": 1, > > > "name": "$1" > > > } > > > ], > > > "type": { > > > "type": "CHAR", > > > "nullable": true, > > > "precision": 5 > > > } > > > }, > > > { > > > "literal": "HELLO", > > > "type": { > > > "type": "CHAR", > > > "nullable": false, > > > "precision": 5 > > > } > > > } > > > ] > > > } > > > }, > > > { > > > "id": "2", > > > "relOp": "LogicalProject", > > > "fields": [ > > > "ID" > > > ], > > > "exprs": [ > > > { > > > "input": 0, > > > "name": "$0" > > > } > > > ] > > > } > > > ] > > > } > > > > > >> ----------------------------------------------------------- > > > > > > > > Thanks and regards > > Bhavya > > > > On Fri, Dec 18, 2020 at 11:54 AM Fan Liya <[email protected]> wrote: > > > > > Hi Bhavya, > > > > > > Sorry I do not understand your question. Why is it difficult to pass > sth. > > > to child operators? > > > > > > Best, > > > Liya Fan > > > > > > > > > On Fri, Dec 18, 2020 at 12:38 PM Bhavya Aggarwal <[email protected]> > > > wrote: > > > > > > > Hi Liya, > > > > > > > > Actually the question is how to pass the projection columns to Table > > Scan > > > > as right now in the LogicalTableScan there are no projection columns > > > being > > > > passed so when I am trying to create my custom JSON , I do not have > > those > > > > projected columns or columns that are being used in the query. I want > > to > > > > understand what is the calcite mechanism for passing it to child > > > operators > > > > without using Rules as it will be a lot of rule that we need to > > > implement. > > > > > > > > Thanks and Regards > > > > Bhavya > > > > > > > > On Fri, Dec 18, 2020 at 9:47 AM Fan Liya <[email protected]> > wrote: > > > > > > > > > Hi Bhavya, > > > > > > > > > > It seems the problem is that the json format provided by Calcite > does > > > not > > > > > include properties specific to your operator? > > > > > If so, I think you can override RelNode#explainTerms in your > operator > > > to > > > > > attach the properties. > > > > > > > > > > Best, > > > > > Liya Fan > > > > > > > > > > On Thu, Dec 17, 2020 at 10:45 PM Bhavya Aggarwal < > [email protected] > > > > > > > > wrote: > > > > > > > > > > > Please let me know if we have an example for writing a custom > > visitor > > > > > that > > > > > > I can use. I see a RelVisitor class that I can extend but how to > > make > > > > it > > > > > to > > > > > > visit the specific Physical Operator. > > > > > > > > > > > > Thanks and Regards > > > > > > Bhavya > > > > > > > > > > > > On Thu, Dec 17, 2020 at 7:43 PM Bhavya Aggarwal < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > I tried using the RelJsonWriter and tried to add my custom > > > > > implementation > > > > > > > in the my custom Rel Nodes but I am not getting all the data in > > my > > > > > child > > > > > > > operators. We are trying to use Calcite for a columnar data and > > we > > > > need > > > > > > the > > > > > > > information of the projections in the TableScan as well because > > we > > > > need > > > > > > to > > > > > > > pass those projections to read specific columns but I am not > able > > > to > > > > > find > > > > > > > the projections in my Table scan. As per my understanding > Calcite > > > > > should > > > > > > be > > > > > > > passing this information as it is required for me reading the > > > table. > > > > > > Please > > > > > > > let me know if I need to do something special to get that > > > information > > > > > in > > > > > > > the child operator. > > > > > > > > > > > > > > Regards > > > > > > > Bhavya > > > > > > > > > > > > > > On Sun, Dec 13, 2020 at 11:32 PM Michael Mior < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > >> That would likely be the best approach if you have some > specific > > > > JSON > > > > > > >> format you're trying to generate. If you're happy with a JSON > > > format > > > > > > >> defined by Calcite, you can look at RelJsonWriter. > > > > > > >> > > > > > > >> -- > > > > > > >> Michael Mior > > > > > > >> [email protected] > > > > > > >> > > > > > > >> Le dim. 13 déc. 2020 à 05:36, Muhammad Gelbana < > > > [email protected] > > > > > > > > > > a > > > > > > >> écrit : > > > > > > >> > > > > > > > >> > I would use a visitor to traverse the optimized/physical > plan. > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > On Sun, Dec 13, 2020 at 6:42 AM Bhavya Aggarwal < > > > > [email protected] > > > > > > > > > > > > >> wrote: > > > > > > >> > > > > > > > >> > > Hi All, > > > > > > >> > > > > > > > > >> > > We need to generate a JSON object for the physical > execution > > > > tree > > > > > > >> that has > > > > > > >> > > been created. Is there an option in Calcite that we can > use > > to > > > > do > > > > > > >> this. I > > > > > > >> > > am not sure what is the right approach to do it. Please > let > > me > > > > > know > > > > > > if > > > > > > >> > > there are different ways to achieve this. > > > > > > >> > > > > > > > > >> > > Regards > > > > > > >> > > Bhavya > > > > > > >> > > > > > > > > >> > > -- > > > > > > >> > > Your feedback matters - At Knoldus we aim to be very > > > > professional > > > > > in > > > > > > >> our > > > > > > >> > > quality of work, commitment to results, and proactive > > > > > communication. > > > > > > >> If > > > > > > >> > > you > > > > > > >> > > feel otherwise please share your feedback > > > > > > >> > > <https://forms.gle/Ax1Te1DDpirAQuQ8A> and we would work > on > > > it. > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > *Bhavya Aggarwal* > > > > > > > CTO & Partner > > > > > > > Knoldus Inc. <http://www.knoldus.com/> > > > > > > > +91-9910483067 > > > > > > > Canada - USA - India - Singapore > > > > > > > <https://in.linkedin.com/company/knoldus> < > > > > > https://twitter.com/Knolspeak > > > > > > > > > > > > > > <https://www.facebook.com/KnoldusSoftware/> < > > > > https://blog.knoldus.com/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > *Bhavya Aggarwal* > > > > > > CTO & Partner > > > > > > Knoldus Inc. <http://www.knoldus.com/> > > > > > > +91-9910483067 > > > > > > Canada - USA - India - Singapore > > > > > > <https://in.linkedin.com/company/knoldus> < > > > > https://twitter.com/Knolspeak > > > > > > > > > > > > <https://www.facebook.com/KnoldusSoftware/> < > > > https://blog.knoldus.com/ > > > > > > > > > > > > > > > > > -- > > > > > > Your feedback matters - At Knoldus we aim to be very professional > > in > > > > our > > > > > > quality of work, commitment to results, and proactive > > communication. > > > If > > > > > > you > > > > > > feel otherwise please share your feedback > > > > > > <https://forms.gle/Ax1Te1DDpirAQuQ8A> and we would work on it. > > > > > > > > > > > > > > > > > > > > > > > -- > > > > *Bhavya Aggarwal* > > > > CTO & Partner > > > > Knoldus Inc. <http://www.knoldus.com/> > > > > +91-9910483067 > > > > Canada - USA - India - Singapore > > > > <https://in.linkedin.com/company/knoldus> < > > https://twitter.com/Knolspeak > > > > > > > > <https://www.facebook.com/KnoldusSoftware/> < > https://blog.knoldus.com/ > > > > > > > > > > > -- > > > > Your feedback matters - At Knoldus we aim to be very professional in > > our > > > > quality of work, commitment to results, and proactive communication. > If > > > > you > > > > feel otherwise please share your feedback > > > > <https://forms.gle/Ax1Te1DDpirAQuQ8A> and we would work on it. > > > > > > > > > > > > > -- > > *Bhavya Aggarwal* > > CTO & Partner > > Knoldus Inc. <http://www.knoldus.com/> > > +91-9910483067 > > Canada - USA - India - Singapore > > <https://in.linkedin.com/company/knoldus> <https://twitter.com/Knolspeak > > > > <https://www.facebook.com/KnoldusSoftware/> <https://blog.knoldus.com/> > > > > -- > > Your feedback matters - At Knoldus we aim to be very professional in our > > quality of work, commitment to results, and proactive communication. If > > you > > feel otherwise please share your feedback > > <https://forms.gle/Ax1Te1DDpirAQuQ8A> and we would work on it. > > >
