Hi Nikolay, Let me add my 5- cent here.
Ignite SQL layer has some issues that can't be fix with changes in Ignite only, and we are blocked with H2. To resolve these issues we can: 1. Donate some changes to H2 and wait for it's next release. But there are more cons than pros and I think we can't rely on H2 project anymore. - There is no guarantee our changes will be approved by H2 community. - We definitely won't to depend on H2 product lifecycle. - New H2 features (like parallel multi-statement query processing in latest release) force Ignite for significant changes\refactoring in Ignite SQL layer with no visible benefits. Every next release it becomes harder to upgrade H2 dependency. - Latest H2 versions causes questions about their stability. Hot issues are - Large intermediate results inside H2 internals can cause OOM for some kind of queries. Ignite can't handle this anyhow for now without reworking H2 code. - HashJoins - Ignite can't start multi-step queries, but 2-step (map-reduce) only. - It is not possible to apply optimizations on query plan as no logical plan actually doen't exists. H2 execution plan is hard-wired with H2 internals and can't be easily transformed. Implementing a new good planner over H2 looks like a huge task. 2. Fork H2. We already done this in GridGain (you can found H2 module in GridGain community edition) as fastest way to unblock work on SQL improvements. But this way doesn't look like a good one for Ignite, regarding our experience. - H2 code can't be included into Ignite at all. H2 license are MIT and EPL. From one side they can't be changed to Apache Licence. From other side Apache Foundation don't want to host any code licensed with other than Apache License. GridGain is ok with this, but Apache Foundation won't. - We can made separate H2 fork project with it's own lifecycle with full control over it and publish it in Maven Central. This doen't seem like a big deal. But will causes additional difficulties in development, test and release processes of Ignite. This way seems bring much pain for every contributor. 3. Replace H2 with smth else. E.g. with Apache Calcite. - Calcite is a framework and it is designed very flexible and extendable. - Every it's part can be replaced with our own implementation. - Apache License is out of the box =) So, summary: 1-st way of pain we have now and it slows down Ignite SQL layer developing. 2-nd looks few better, but seems bring Ignite to nowhere in prospect. 3-rd is a risky, but promissory way. On Fri, Sep 27, 2019 at 12:16 PM Nikolay Izhikov <nizhi...@apache.org> wrote: > Hello, Igor. > > Thanks for starting this discussion. > > I think we should take a step back in it and answer the following > questions: > > 1. What the exact issues with the H2 integration? > Can you send a tickets links? > Can we label all H2 integration issues in JIRA? I propose to use "h2" > label. > > 2. What are the requirements for the new SQL engine? > We should write it down and discuss. > > 3. What options do we have? > Are there any alternatives to Calcite on the market? > We did the wrong choice that looked obvious one time. > So we should carefully avoid it at this time. > > 4. What is improvements of Ignite we want to make with the new engine? > > > В Пт, 27/09/2019 в 08:44 +0000, Igor Seliverstov пишет: > > Hi Igniters! > > > > As you might know currently we have many open issues relating to current > H2 based engine and its execution flow. > > > > Some of them are critical (like impossibility to execute particular > queries), some of them are majors (like impossibility to execute particular > queries without pre-preparation your data to have a collocation) and many > minors. > > > > Most of the issues cannot be solved without whole engine redesign. > > > > So, here the proposal: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=130028084 > > > > I'll appreciate if you share your thoughts on top of that. > > > > Regards, > > Igor > -- Best regards, Andrey V. Mashenkov
