[
https://issues.apache.org/jira/browse/FLINK-36039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
RocMarshal updated FLINK-36039:
-------------------------------
Description:
Currently, the autoscaler generates a large amount of historical data for event
handlers. As the system runs for a long time, the volume of historical data
will continue to grow. It is necessary to support automatic cleanup of data
within a fixed period.
Based on the creation time timestamp, the following approach for cleaning up
historical data might be a way:
* Introduce the parameter {{autoscaler.standalone.jdbc-event-handler.ttl}}
** Type: Duration
** Default value: 90 days
** Setting it to 0 means disabling the cleanup functionality.
* In the {{JdbcAutoScalerEventHandler}} constructor, introduce a scheduled
job. Also, add an internal interface method {{close}} for
{{AutoScalerEventHandler}} & {{JobAutoScaler}} to stop and clean up related
logic.
* Cleanup logic:
**
### Query the messages with {{create_time}} less than {{(currentTime - ttl)}}
and find the maximum {{maxId}} in this collection.
**
### Delete 4096 messages at a time from the collection with IDs less than
{{{}maxId{}}}.
**
### Wait 10 ms between each deletion until the cleanup is complete.
was:
Currently, the autoscaler generates a large amount of historical data for event
handlers. As the system runs for a long time, the volume of historical data
will continue to grow. It is necessary to support automatic cleanup of data
within a fixed period.
Based on the creation time timestamp, the following approach for cleaning up
historical data might be a way:
* Introduce the parameter
{{job.autoscaler.event-handler.historical-ttl-duration}} (of type duration),
which represents the storage time for historical data.
* Introduce the parameter
{{job.autoscaler.event-handler.historical-clean-interval}} (of type duration),
which represents the interval between each data cleanup.
In the constructor of {{{}JdbcAutoScalerEventHandler{}}}, introduce a scheduled
task. Also, add an internal interface method, {{{}close{}}}, to
{{AutoScalerEventHandler}} and {{JobAutoScaler}} for stopping and cleaning up
the relevant logic.
> Support clean historical event handler records in autoscaler standalone
> -----------------------------------------------------------------------
>
> Key: FLINK-36039
> URL: https://issues.apache.org/jira/browse/FLINK-36039
> Project: Flink
> Issue Type: Improvement
> Components: Autoscaler
> Reporter: RocMarshal
> Assignee: RocMarshal
> Priority: Minor
>
> Currently, the autoscaler generates a large amount of historical data for
> event handlers. As the system runs for a long time, the volume of historical
> data will continue to grow. It is necessary to support automatic cleanup of
> data within a fixed period.
> Based on the creation time timestamp, the following approach for cleaning up
> historical data might be a way:
> * Introduce the parameter {{autoscaler.standalone.jdbc-event-handler.ttl}}
> ** Type: Duration
> ** Default value: 90 days
> ** Setting it to 0 means disabling the cleanup functionality.
> * In the {{JdbcAutoScalerEventHandler}} constructor, introduce a scheduled
> job. Also, add an internal interface method {{close}} for
> {{AutoScalerEventHandler}} & {{JobAutoScaler}} to stop and clean up related
> logic.
> * Cleanup logic:
> **
> ### Query the messages with {{create_time}} less than {{(currentTime -
> ttl)}} and find the maximum {{maxId}} in this collection.
> **
> ### Delete 4096 messages at a time from the collection with IDs less than
> {{{}maxId{}}}.
> **
> ### Wait 10 ms between each deletion until the cleanup is complete.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
