Paul King created GROOVY-12113:
----------------------------------
Summary: GINQ roadmap: SQL backend (GEP-26), GEP-18 concurrency
alignment, query continuation, and async sources
Key: GROOVY-12113
URL: https://issues.apache.org/jira/browse/GROOVY-12113
Project: Groovy
Issue Type: Umbrella
Reporter: Paul King
h2. Overview
Umbrella tracking the remaining GINQ (Groovy-Integrated Query) roadmap. GINQ's
SQL-like
surface and pluggable code-generation seam make several enhancements natural;
this issue
collects the outstanding workstreams in one place. Detailed designs are linked
where they
have their own proposals.
Already shipped (context):
* (/) [GROOVY-11915] -- {{groupby...into}} redesign with first-class
{{GroupResult<K,T>}} group variables (the foundation piece for both the SQL
backend and parallel improvements)
* (/) Set operations in the DSL -- {{union}}, {{unionall}}, {{intersect}},
{{minus}} with left-associative chaining
* (/) GEP-18 -- Integrated Concurrency and Parallel Processing: the bundled
core runtime GINQ will align to ({{groovy.concurrent.Pool}} /
{{ParallelScope}}, async/await, channels, {{ParallelCollectionExtensions}},
{{Cancellable}})
h2. Workstreams
h3. 1. SQL backend (native + JOOQ) -- specified in GEP-26
Execute an *unchanged* GINQ query against a relational database, pushing
filtering, joining,
grouping, and aggregation down to the database. Delivered as two providers over
one shared
front-end:
* *Native SQL provider* -- dependency-free, ANSI SQL via {{groovy.sql.Sql}},
fully ASL2
* *JOOQ provider* -- optional, dialect-aware, window functions, via JOOQ's
plain-SQL DSL
Both consume a shared, provider-independent {{SqlQuery}} intermediate model
produced once from
the GINQ AST. A new {{provider}} config key ({{'native-sql'}} / {{'jooq-sql'}}
/ {{'sql'}} auto)
selects the backend, sugar over the existing {{astWalker}} hook. New optional
{{groovy-ginq-sql}}
module. Estimated ~2,400-3,650 lines across five phases.
Full specification: *GEP-26 -- GINQ SQL Backend (Native and JOOQ Providers)*
(https://groovy.apache.org/wiki/GEP-26.html), target Groovy 7.0.
Phases:
# Shared walker + {{SqlQuery}} IR + native provider (single-table
SELECT/FROM/WHERE/ORDER BY/LIMIT)
# Joins + GROUP BY/aggregates + set operations (native)
# JOOQ provider (reuses the shared IR -> dialects + window functions)
# Subqueries, IN-subquery, EXISTS
# Mixed-mode fallback, function mapping, polish
h3. 2. GINQ <-> GEP-18 concurrency alignment
GINQ's parallel support ({{@GQ(parallel: true)}}) predates GEP-18 and uses a
private
{{QueryableHelper.ThreadPoolHolder}} ({{ginq-fj-thread}} {{ForkJoinPool}} +
{{-Dgroovy.ginq.parallelism}} + a separate virtual-thread executor). GEP-18 now
bundles a
general {{Pool}} / {{ParallelScope}} runtime, so GINQ's parallelism should
become a thin adapter
over it rather than a bespoke pool.
Scope:
* Replace {{ThreadPoolHolder}} with pool resolution: explicit {{poolSize:}} ->
{{Pool.current()}} (an enclosing {{ParallelScope.withPool}}) ->
{{ConcurrentConfig.getDefaultPool()}}
* Use the resolved pool's {{asForkJoinPool()}} for the parallel-stream work;
surface {{poolSize:}} and {{ParallelScope.withPool}} composition
* Unify config on {{groovy.concurrent.poolsize}} (keep
{{groovy.ginq.parallelism}} as a deprecated override one release)
* Drop bespoke pool shutdown for shared/borrowed pools; only close pools GINQ
itself creates
* Wire cooperative cancellation via {{Cancellable}}; consider an
ordered-parallel mode
The three subtleties that are the real work (not the wiring):
# *Virtual pools can't back parallel streams* -- {{asForkJoinPool()}} throws on
a virtual pool; GINQ must resolve a FJP-backed pool ({{cpu()}}/{{fixed}}) for
stream/groupby work and use a virtual/IO pool only for the {{supplyAsync}}
transform path. (Decision: what to do if a user binds a virtual pool.)
# *Scoped binding vs InheritableThreadLocal* -- {{Pool.current()}} is
scope-bound while GINQ's {{isParallel()}} flag is an
{{InheritableThreadLocal}}; capture the pool before {{submit}} and re-establish
it on the worker via {{Pool.withCurrent(...)}}.
# *Nested-parallelism deadlock* -- {{QueryableCollection}} already
force-materializes a hash table to avoid nested-parallel deadlock; verify the
guard still holds when a single bounded pool is shared across nested queries.
Files: {{QueryableHelper.groovy}}, {{GinqAstWalker.groovy}},
{{QueryableCollection.java}},
{{GinqGroovyMethods.groovy}}, optionally {{GQ.java}} (additive {{poolSize}}
member).
Effort: Medium, ~300-600 lines + tests. Best tracked as a minor-GEP integration
aligned to GEP-18.
h3. 3. Query continuation
Allow {{where}}/{{orderby}}/etc. *after* {{groupby...into g}} (operating on
{{g}} as a new data
source), matching C# LINQ's {{into}} continuation. Currently {{where}} after
{{groupby...into}}
is rejected ("reserved for future use"); users must use {{having}}. Requires a
chainable/nested
{{GinqExpression}} model (a {{GinqExpression}} containing a
sub-{{GinqExpression}}). Could
piggy-back on a general language continuations feature. Effort: High;
aspirational LINQ parity.
h3. 4. Async streaming data sources (GEP-18 addendum)
Awaitable GINQ results already work today (wrap {{GQL{...}}} in {{async {
}}}}). The future piece
is querying over asynchronous streams -- {{from e in someAsyncChannel}} with a
lazy
{{AsyncQueryable}} pulling from {{groovy.concurrent.AsyncChannel}} (with
backpressure) consumed via
{{for await}}. Now buildable against the *merged* GEP-18 primitives, so best
handled as a small
GEP-18 addendum rather than a standalone GEP. Effort: High.
h2. Suggested sequencing
|| # || Item || Effort || Depends on ||
| 1 | SQL backend Phase 1 (GEP-26) -- shared walker + native, single-table |
Medium | -- |
| 2 | SQL backend Phase 2 (GEP-26) -- joins + GROUP BY + set ops, native |
Medium | Phase 1 |
| 3 | SQL backend Phase 3 (GEP-26) -- JOOQ provider | Medium | Phase 1 |
| 4 | GINQ <-> GEP-18 alignment -- pool resolution, poolSize, withPool | Medium
| GEP-18 (done) |
| 5 | Cooperative cancellation + ordered parallel | Medium | #4 |
| 6 | SQL backend Phases 4-5 (GEP-26) -- subqueries, mixed-mode, functions |
Medium-High | Phase 2 |
| 7 | Query continuation | High | continuations (beneficial, not required) |
| 8 | Async streaming data sources (GEP-18 addendum) | High | for await over
AsyncChannel |
SQL phases 2 and 3 depend only on Phase 1 (not each other) and can proceed in
parallel. The
GEP-18 alignment (#4) is independent of the SQL work and can proceed at any
time.
h2. External dependencies
|| Feature || Dependency || License || Required? ||
| Native SQL provider (GEP-26) | none (uses groovy-sql) | ASL2 | built-in |
| JOOQ SQL provider (GEP-26) | JOOQ OSS edition | ASL2 (OSS DBs), commercial
(proprietary DBs) | optional, user-supplied |
| Parallel / async runtime | groovy.concurrent (GEP-18) | ASL2 | bundled in
core |
| Async data sources | GEP-18 AsyncChannel / for await | ASL2 | bundled in core
|
| SQL backend tests | H2 Database | EPL/MPL | test-only |
h2. References
* GEP-26: GINQ SQL Backend (Native and JOOQ Providers) --
https://groovy.apache.org/wiki/GEP-26.html
* GEP-18: Integrated Concurrency and Parallel Processing --
https://groovy.apache.org/wiki/GEP-18.html
* [GROOVY-11915] -- groupby...into redesign
* [GROOVY-8258] -- original GINQ proposal
* [GROOVY-12042] -- graduate groovy-ginq from incubating to stable (the
parallel/provider config keys should stay open to additive growth)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
