ptrdom opened a new pull request, #671:
URL: https://github.com/apache/pekko-projection/pull/671
### Motivation
apache/pekko-persistence-r2dbc#534 added the `mariadb` dialect to the
persistence plugin. R2DBC projections can't use it yet:
- `R2dbcOffsetStore.fromConfig` has no offset store for `Dialect.MariaDB`.
- The MySQL offset store upserts with the `VALUES (...) AS excluded` row
alias, which MariaDB doesn't support (`[1064] ... near 'AS excluded ON
DUPLICATE KEY UPDATE ...'`).
**Draft until pekko-persistence-r2dbc is released.** This builds against
`2.0.0-M1+120-5a212bab-SNAPSHOT`, which includes
apache/pekko-persistence-r2dbc#534 (and the `r2dbc-mariadb` 1.3.2 bump in
apache/pekko-persistence-r2dbc#538). It is stacked on #668, which needs the
same snapshot handling (the `close-calls-exceeding` executor change) and adds
`Dialect.H2` to the same matches. Until #668 is merged, the diff here includes
its commits; the MariaDB change is the last commit.
### Modification
- **Dialect:** `R2dbcOffsetStore.fromConfig` creates a
`MariaDBR2dbcOffsetStore` for `Dialect.MariaDB`. It extends
`MySQLR2dbcOffsetStore` and only overrides the offset and management upserts.
These use `ON DUPLICATE KEY UPDATE col = VALUES(col)`, the same approach as
`MariaDBSnapshotDao` in apache/pekko-persistence-r2dbc#534. Everything else is
the MySQL store:
- the `NOW(6)` consumed timestamp,
- binding each batch row before `add()`,
- the `CONCAT(...) NOT IN (...)` delete.
- **Timestamps:** the MariaDB driver doesn't support `java.time.Instant`.
Nothing in the projection handles that: `ConnectionFactoryProvider` in the
plugin wraps the MariaDB connection pool so that `Instant` is bound and read as
UTC `LocalDateTime`, with a UTC session. The projection gets its connections
from that provider, so the offset store and the statements of an `R2dbcSession`
are converted the same way.
- **Schema:** MariaDB uses the MySQL DDL (`create_tables_mysql.sql`), like
the persistence plugin.
- **Dependencies:** `org.mariadb:r2dbc-mariadb:1.3.2` (Apache 2.0), the
version pekko-persistence-r2dbc is built against, `test` only in
`r2dbc-int-test`. The `r2dbc` module doesn't reference driver classes.
- **CI:** a new "Build and Test Integration for R2DBC with MariaDB" job
(Scala 2.13 and Scala 3 next). It uses
`docker-files/docker-compose-mariadb.yml` (MariaDB 12.3.3, same as the plugin),
with the server in `America/New_York` so that a non-UTC session would show up
as wrong timestamps.
- **Docs:** a MariaDB entry in the schema list and a MariaDB section in
`r2dbc.md` that links to the plugin's MariaDB docs. `reference.conf` lists
`mariadb` as a dialect.
- **Tests:**
- `TestConfig` gets a `mariadb` case.
- `R2dbcProjectionSpec`'s test repository gets a `VALUES()` upsert for
MariaDB.
- `R2dbcTimestampOffsetProjectionSpec`'s spy offset store gets a MariaDB
case.
- New `MariaDBR2dbcOffsetStoreSpec`.
### Result
R2DBC projections can store offsets in MariaDB 10.11 or later with the
`mariadb` dialect.
### Tests
- New `MariaDBR2dbcOffsetStoreSpec` (MariaDB 12.3.3 in `America/New_York`,
Windows 11, JDK 25): 3 passed.
- Dialect selection and the upsert SQL. This test makes no database calls,
so it runs in every dialect's job.
- Offset and management state updates.
- Timestamp offsets at two instants whose UTC local times are ambiguous
(`2026-11-01T01:30Z`) or don't exist (`2027-03-14T02:30Z`) in the server zone.
The test checks the stored instant with `UNIX_TIMESTAMP`, which doesn't depend
on the session time zone, and then reads the offset back.
- With a New York session, MariaDB stores `2026-11-01 01:30` as `06:30Z`
but still reads it back as `01:30`, so only `UNIX_TIMESTAMP` catches it. It
rejects `2027-03-14 02:30` (`ERROR 1292`). Both checked with the `mariadb` CLI
against the test image.
- Not run through the plugin with a non-UTC session: the plugin rejects
a non-UTC `timezone` option, and `sessionVariables=time_zone=SYSTEM` in the URL
left the session in UTC.
- The other two tests are canceled with other dialects.
- With the MySQL upserts for MariaDB, 2 of 3 fail: the SQL check, and the
update test with `R2dbcBadGrammarException ... near 'AS excluded ...'`.
- `sbt r2dbc-int-test/test` against PostgreSQL: 137 succeeded, 0 failed, 2
canceled.
- `sbt -Dpekko.persistence.r2dbc.dialect=h2
-Dpekko.projection.r2dbc.dialect=h2 r2dbc-int-test/test`: 137 succeeded, 0
failed, 2 canceled.
- `sbt -Dpekko.persistence.r2dbc.dialect=mariadb
-Dpekko.projection.r2dbc.dialect=mariadb r2dbc-int-test/test` (MariaDB 12.3.3),
3 runs: 137 to 138 of 139 succeeded.
- The failures were `EventSourcedPubSubSpec`, `EventSourcedEndToEndSpec`
and once a `R2dbcTimestampOffsetProjectionSpec` test.
- All of them pass when run on their own. The log shows `Too many events
stored with the same timestamp ..., buffer size [10]`, from app timestamps on
the Windows clock.
- With MySQL 9.5.0, `EventSourcedPubSubSpec` fails the same way locally,
on this branch and on #668. CI passes it on #668.
- `sbt -Dpekko.persistence.r2dbc.dialect=mysql
-Dpekko.projection.r2dbc.dialect=mysql r2dbc-int-test/test` (MySQL 9.5.0): 136
succeeded, 1 failed (`EventSourcedPubSubSpec`, see above), 2 canceled.
- `++3.3.8!` and `++3.9.0!` `r2dbc/compile r2dbc-int-test/Test/compile`:
pass.
- `sbt "checkCodeStyle; mimaReportBinaryIssues"`, `headerCheckAll`,
`Test/headerCheckAll`, `docs/paradox`: pass. `sbt headerCreateAll` added the
ASF headers.
- scalafmt (native, diff against #668's branch plus the new files):
formatted. `git diff --check`: clean.
- Yugabyte not run locally; relying on CI.
### Notes
- Copied code: none from outside this repo. The `VALUES()` upserts follow
`MariaDBSnapshotDao` and `docker-compose-mariadb.yml` follows the plugin's file
from apache/pekko-persistence-r2dbc#534 (same ASF project).
- No check that the MariaDB driver and the `mariadb` dialect are used
together, unlike the plugin's `MariaDBConnections.requireMatchingDialect`,
which is `private[r2dbc]` in the plugin's package. The projection dialect
defaults to the plugin's, and the plugin's DAOs already check the pairing. A
projection set to `mysql` on the MariaDB driver fails on the first offset save
with the syntax error above.
- The snapshot bump from `+113` to `+120` also brings in
apache/pekko-persistence-r2dbc#515 (snapshots as starting points for
`eventsBySlices` and `eventsByPersistenceId`). The PostgreSQL and H2 suites
pass with it.
- After the pekko-persistence-r2dbc release: bump `currentVersion` to the
release, rebase onto `main` once #668 is merged, re-run CI and mark ready.
### References
Refs apache/pekko-persistence-r2dbc#534, Refs #668
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
