Hi everyone,
I'd like to start a discussion on FLIP-616: Materialized Tables support for
Table API [1].
Materialized tables (FLIP-435 [2]) have become a convenient way to define
pipelines declaratively. So far, though, this only works via SQL against
the SQL Gateway. Table API users can list materialized tables, but they
cannot define or manage them, and running the DDL through executeSql()
only gets them halfway, since the table is registered but no refresh job is
ever started.
The FLIP closes this gap. A Table can be turned into a materialized table
via Table#materializeInto(...), optionally with a descriptor for
freshness, refresh strategy, schema details or start mode. The result is a
MaterializedTable handle that lets users change, inspect, and control it.
To avoid two diverging implementations, the logic currently living in the
SQL Gateway moves into the planner, so the gateway and the Table
API behave the same. From a regular Table API program, continuous refreshes
work against a session cluster. Full refresh mode still
needs the gateway for now, since it depends on a workflow scheduler. As a
side effect, executeSql() would then actually manage refresh jobs
for materialized table statements outside the gateway.
There are two points I'd especially like your input on:
1. Should redeploying an unchanged definition restart the refresh job?
Today, CREATE OR ALTER always stops the job and starts a new one
without state, even if nothing changed. Since the list of changes is already
computed, skipping the restart when it is empty would be a small check.
On the other hand, always restarting keeps the semantics simple for now
and matches what SQL does today.
2. How should users get out of "stuck" tables?
If the cluster no longer "knows" the refresh job of an ACTIVATED table,
a redeploy is rejected, even though it would start a fresh job without state
anyway. That leaves DROP as the only way out, and for catalogs that own
the storage, DROP also deletes the data. If the cluster is not reachable
at all, the only workaround is to start a cluster under the recorded
cluster ID so that DROP sees the job as gone. Is that acceptable, or should
a
redeploy be able to recover such tables?
Looking forward to your thoughts and feedback!
Best,
Timo
[1]
https://cwiki.apache.org/confluence/spaces/FLINK/pages/451979479/FLIP-616+Materialized+Tables+support+for+Table+API
[2]
https://cwiki.apache.org/confluence/spaces/FLINK/pages/294816797/FLIP-435+Introduce+a+New+Materialized+Table+for+Simplifying+Data+Pipelines
