tzulitai opened a new pull request #257:
URL: https://github.com/apache/flink-statefun/pull/257
This PR intends to add a few configurations to the base `flink-conf.yaml`:
- Activate local recovery. This includes setting a local directory for
RocksDB to flush its sst / metadata files, as well as setting a local directory
for recovery.
- Activate partitioned indexes by default
- Use more threads for transferring checkpoint files from DFS
- Increase RocksDB compaction threads
---
## Testing the changes
The new configuration has been set in the `flink-conf.yaml` we use in the
E2E tests as well. After doing that, I can see logs such as:
```
13:23:41,274 INFO org.apache.flink.statefun.e2e.smoke.SmokeRunner
- 2021-08-16 05:23:41,278 INFO
org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation
[] - Finished restoring from state handle:
IncrementalLocalKeyedStateHandle{metaDataState=File State:
file:/local/state/recovery/ ......
```
and
```
13:23:41,198 INFO org.apache.flink.statefun.e2e.smoke.SmokeRunner
- 2021-08-16 05:23:41,202 INFO
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder [] -
Finished building RocksDB keyed state-backend at /local/state/rocksdb/ ......
```
which confirms that local recovery is working.
Also, other configurations have been picked up as well as confirmed by the
logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
