Jzjsnow commented on issue #3815:
URL: https://github.com/apache/amoro/issues/3815#issuecomment-3393048109
Bug 1 and 2 can be resolved in #3816. But 3 appears to have inconsistent
data results after the update:
The `table_config` column information in table `table_runtime` inserted from
my old tableRuntime table is in JSON format:
> {
> "expireSnapshotEnabled": true,
> "snapshotTTLMinutes": 720,
> "snapshotMinCount": 1,
> "changeDataTTLMinutes": 10080,
> "cleanOrphanEnabled": false,
> "orphanExistingMinutes": 2880,
> "deleteDanglingDeleteFilesEnabled": true,
> "optimizingConfig": {
> "enabled": true,
> "targetQuota": 0.5,
> "optimizerGroup": "flinkgroup-new",
> "maxExecuteRetryCount": 5,
> "targetSize": 134217728,
> "maxTaskSize": 134217728,
> "maxFileCount": 10000,
> "openFileCost": 4194304,
> "fragmentRatio": 8,
> "minTargetSizeRatio": 0.75,
> "minorLeastFileCount": 12,
> "minorLeastInterval": 3600000,
> "majorDuplicateRatio": 0.1,
> "fullTriggerInterval": -1,
> "fullRewriteAllFiles": true,
> "filter": null,
> "baseHashBucket": 4,
> "baseRefreshInterval": -1,
> "hiveRefreshInterval": -1,
> "minPlanInterval": 60000
> },
> "expiringDataConfig": {
> "enabled": false,
> "expirationField": null,
> "expirationLevel": "PARTITION",
> "retentionTime": 0,
> "dateTimePattern": "yyyy-MM-dd",
> "numberDateFormat": "TIMESTAMP_MS",
> "baseOnRule": "LAST_COMMIT_TIME"
> },
> "tagConfiguration": {
> "autoCreateTag": false,
> "tagFormat": "'tag-'yyyyMMdd",
> "triggerPeriod": "DAILY",
> "triggerOffsetMinutes": 0,
> "maxDelayMinutes": 60,
> "tagMaxAgeMs": -1
> }
> }
While the ` table_config` column information inserted in the new run is in
map format, e.g.:
> {
> "self-optimizing.group": "flinkgroup-new",
> "upsert-enabled": "false",
> "write-format": "parquet",
> "write.parquet.compression-codec": "snappy"
> }
It seems the old tableRuntime table information may require some adjustments
before it can be inserted into the new table. @baiyangtx PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
