Hi all,
After some experimentation, we felt no problem putting the dynamic
storage outside of flink, and it also allowed us to design the
interface in more depth.
What do you think? If there is no problem, I am asking for PMC's help
here: we want to propose flink-dynamic-storage as a flink subproject,
and we want to build the project under apache.
Best,
Jingsong
On Wed, Nov 24, 2021 at 8:10 PM Jingsong Li <jingsongl...@gmail.com>
wrote:
Hi Stephan,
Thanks for your reply.
Data never expires automatically.
If there is a need for data retention, the user can choose one of the
following options:
- In the SQL for querying the managed table, users filter the data by
themselves
- Define the time partition, and users can delete the expired
partition by themselves. (DROP PARTITION ...)
- In the future version, we will support the "DELETE FROM" statement,
users can delete the expired data according to the conditions.
So to answer your question:
Will the VMQ send retractions so that the data will be removed from
the table (via compactions)?
The current implementation is not sending retraction, which I think
theoretically should be sent, currently the user can filter by
subsequent conditions.
And yes, the subscriber would not see strictly a correct result. I
think this is something we can improve for Flink SQL.
Do we want time retention semantics handled by the compaction?
Currently, no, Data never expires automatically.
Do we want to declare those types of queries "out of scope" initially?
I think we want users to be able to use three options above to
accomplish their requirements.
I will update FLIP to make the definition clearer and more explicit.
Best,
Jingsong
On Wed, Nov 24, 2021 at 5:01 AM Stephan Ewen <ewenstep...@gmail.com>
wrote:
Thanks for digging into this.
Regarding this query:
INSERT INTO the_table
SELECT window_end, COUNT(*)
FROM (TUMBLE(TABLE interactions, DESCRIPTOR(ts), INTERVAL '5'
MINUTES))
GROUP BY window_end
HAVING now() - window_end <= INTERVAL '14' DAYS;
I am not sure I understand what the conclusion is on the data
retention question, where the continuous streaming SQL query has retention
semantics. I think we would need to answer the following questions (I will
call the query that computed the managed table the "view materializer
query" - VMQ).
(1) I guess the VMQ will send no updates for windows beyond the
"retention period" is over (14 days), as you said. That makes sense.
(2) Will the VMQ send retractions so that the data will be removed
from the table (via compactions)?
- if yes, this seems semantically better for users, but it will be
expensive to keep the timers for retractions.
- if not, we can still solve this by adding filters to queries
against the managed table, as long as these queries are in Flink.
- any subscriber to the changelog stream would not see strictly a
correct result if we are not doing the retractions
(3) Do we want time retention semantics handled by the compaction?
- if we say that we lazily apply the deletes in the queries that
read the managed tables, then we could also age out the old data during
compaction.
- that is cheap, but it might be too much of a special case to be
very relevant here.
(4) Do we want to declare those types of queries "out of scope"
initially?
- if yes, how many users are we affecting? (I guess probably not
many, but would be good to hear some thoughts from others on this)
- should we simply reject such queries in the optimizer as "not
possible to support in managed tables"? I would suggest that, always better
to tell users exactly what works and what not, rather than letting them be
surprised in the end. Users can still remove the HAVING clause if they want
the query to run, and that would be better than if the VMQ just silently
ignores those semantics.
Thanks,
Stephan
--
Best, Jingsong Lee
--
Best, Jingsong Lee
