It’s possible thst you’re running into a performance bug - such as a loop that is O[n ^ 2) in the number of SqlNodes - that has never been noticed before.
Can you do some investigation? Say, double the size or depth of the query and see if the running time more than doubles. Or run the conversion repeatedly, send sig-kill to your JVM to generate stack traces, and see whether it is often inside the same piece of code. If those approaches yield results, log a bug. We would rather fix the bug, and have people use SubqueryRemoveRule, rather than turning on isExpand. Julian > On Jun 11, 2024, at 7:28 PM, JinxTheKid <[email protected]> wrote: > > Hi Calcite community, > > I'm running into some challenges using SubqueryRemoveRule and was hoping > the community could help me understand any gaps in my understanding. > > I am attempting to remove subqueries from row expressions in a relational > plan. My motivation for doing this is that I want to call metadata queries > such GetColumnOrigins on my relational plan, but these tools don't seem to > work unless subqueries are removed. To remove subqueries, I am... > - Using SqlToRelConverter to convert from SqlNode to RelRoot. > - After obtaining the RelRoot, using a HepProgram with subquery remove > rules to remove the subqueries. > > For small programs, this process works fine, however when testing against > larger queries, ~2MB, the process of removing subqueries takes about as > much time as converting a SqlNode to a RelRoot, effectively doubling the > time to analyze a query. I've tried using a deprecated setting, > SqlToRelConverter.Config.isExpand, which seems to provide results similar > to what I want with minimal performance impact. > > Since SqlToRelConverter.Config.isExpand is deprecated, I assume that the > non-deprecated way to expand subqueries would be as performant or better, > but I am experiencing a worse performance. This leads me to believe that > I'm using the APIs incorrectly, but I'm not sure what I am missing. > > Q: Is there a more efficient way in Calcite to remove subqueries than what > I've described above? Is there risk to using > SqlToRelConverter.Config.isExpand even if it's deprecated? > > Q: Are there any other resources besides Java docs and GitHub projects I > could refer to to learn more about my problem? > > Thanks for any help, > Logan
