This is an automated email from the ASF dual-hosted git repository.
jinsongzhou pushed a change to branch 0.8.x
in repository https://gitbox.apache.org/repos/asf/amoro.git
from 7d2b88c99 Merge branch 'master' into 0.8.x
new 5a6d1c207 [Bug]: The flink optimizer often triggers OOM exceptions. In
fact, the files processed by each task are not large. (#3521)
new 9d67b3009 [AMORO-3533] Avoid bulk table removal when synchronizing
external catalogs and fetching database table listings fails (#3534)
new 2d2664179 (hotfix) enhance log for iceberg table maintainer (#3543)
new ccbaf466e fix some doc error (#3551)
new 96b14bb71 [AMORO-3538] Resolve Hive 4 Metastore listing failure on
Hadoop2 (#3539)
new 33aa81566 [AMORO-3559] Extract abstract class for table metrics (#3560)
new b377f3842 [hotfix]update upgrade.sql (#3563)
new 2e179bf0b [Hotfix] Fix HudiTableDesctiptor error (#3572)
new 87908da19 [AMORO-3549] Fix AMS helm chart cannot set log4j2 because of
wrong indentation (#3550)
new 1ad20d99f [Hotfix] update upgrade.sql for postgres after 0.8.0
release. (#3574)
new d655d2e45 [AMORO-3540] Increase the precision of
optimizing_status_start_time from s to ms. (#3542)
new 183635b59 Optimize the performance of optimizer pollTask (#3583)
new ca00dcdfb [AMORO-3578] Improving Error Prompts for Login Failures
(#3579)
new 8790b741c [Hotfix] Bump parquet-avro to 1.15.2 to fix CVE (#3580)
new 3b740c4fc [AMORO-3545] Prevent empty tasks list during planning (#3546)
new 8b53d39e0 Optimize log output for SchedulingPolicy (#3592)
new dae0e589f [AMORO-3594] Improve table identifiers display (#3595)
new f660d3c99 [AMORO-3587] Modify the default value of hive catalog
clients to 20 (#3590)
new 8fce30182 [AMORO-3599] remove com.esotericsoftware.kryo dependencies
in flink optimizer package (#3600)
new 8c75c62d2 [AMORO-3299] Fix the problem that the commit time of the
file is displayed incorrectly (#3604)
new 3ac36310a [AMORO-3570] Add Rest Catalog configuration documentation
(#3597)
new 61f952703 [hotfix] Throw the exception when the tableRuntime disposal
fails. (#3557)
new 5c504d1a6 [AMORO-3616][Improvement] Correct the spelling errors in doc
of Managing Optimizers page (#3618)
new 85c5eaa6e [AMORO-3612][Improvement]: Add the log output function of
the StandaloneOptimizer module (#3613)
new 5e777dc89 [AMORO-3614][Improvement]: Add a document about deploying
Amoro in IDE (#3615)
new c814c06dc [AMORO-3620][Improvement]: Fix the error in the calculation
of the health score of table (#3621)
new a736fa797 [AMORO-3622] Fix expiring data when file without upperbounds
(#3623)
new 76cbaa7fd [AMORO-3633][improvement]: Fix the sorting rules of
topTables order by healthScore (#3634)
new 9d504d01d [Hotfix] Handle null partition values in
IcebergTableMaintainer to prevent expiration errors (#3635)
new 347b399fe [AMORO-3631][Improvement]: Update the doc for optimizer
management (#3641)
new fdff0b398 [AMORO-3645][Improvement]: Optimize target file size after
self-optimizing (#3646)
new 2a229b459 [AMORO-3664] Trim the leading and trailing whitespace in
ResourceGroup properties (#3664)
new 01cc1de8d Fixed the memory display error in Spark optimizer resource
allocation.
new 49a959280 [AMORO-3637] Harden the task completion logic to avoid the
task state in ACKED forever
new cdd250b91 [Hotfix] Replace Jersey Guava imports with Amoro shaded
Guava (#3674)
new f1a36194d Fix continually creating unused keep_file when committing
for an unpartitioned mixed_hive table optimization (#3690)
new 717d09a09 [AMORO-2519] Improve the display of Duration in the page of
Optimizing>Tables to make it more user-friendly (#3694)
new af986791d [AMORO-3638] Add support for task retries when task
execution times out
new e451f7389 [Improvement] Use fileIndex="nomax" to avoid log loss.
(#3701)
new 36ee3f4c9 [AMORO-3245] Display comment on Table page. (#3672)
new dcfb3cb71 [AMORO-3703] Use dataFileFormat for deleteFile if not set
(#3704)
new fcd539654 [AMORO-3682]Fix table creation failure when comment contains
`PRIMARY KEY` text (#3683)
new fb2752059 [AMORO-3695][Improvement]: Reduce unnecessary details in the
log output (#3696)
new af2ec1263 [AMORO-3677] Fill catalogProperties in creating catalog
rather than building (#3678)
new 90929ca78 [AMORO-3685] Optimize the performance of fillSkipSet for
planning (#3688)
new 2a7c690b5 [AMORO-3718]Inconsistency in capitalization between iceberg
and hive … (#3719)
new 04bd1a96c [AMORO-3716]Enhance FlinkUnifiedCatalog getTable efficiency
(#3717)
new 078d7ec2d [AMORO-3720] fix mixed_format does not support explain for
merge into SQL (#3721)
new a7b3f15fc [AMORO-3713] fix bug of merging into when selecting less
columns from souce table than target mixed table (#3722)
new beddc84b4 [AMORO-3732] Upgrade commons-beanutils version to 1.11.0
(#3733)
new de4fd4c96 [AMORO-3739][Bug]: Add iceberg-arrow to ams package (#3740)
new 56b686abb [hotfix] Fix doc error in displaying "Using Customized
Encryption Method for Configurations" on admin guidance page (#3741)
The 52 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CONTRIBUTING.md | 86 ++++++
amoro-ams/pom.xml | 9 +
.../apache/amoro/server/AmoroManagementConf.java | 6 +
.../amoro/server/DefaultOptimizingService.java | 31 +-
.../amoro/server/catalog/CatalogBuilder.java | 11 -
.../server/catalog/DefaultCatalogManager.java | 27 ++
.../dashboard/MixedAndIcebergTableDescriptor.java | 12 +-
.../dashboard/controller/LoginController.java | 2 +-
.../controller/OptimizerGroupController.java | 5 +-
.../dashboard/controller/OverviewController.java | 2 +-
.../dashboard/controller/SettingController.java | 2 +-
.../server/dashboard/utils/PropertiesUtil.java | 32 +++
.../amoro/server/optimizing/KeyedTableCommit.java | 18 +-
.../amoro/server/optimizing/OptimizingQueue.java | 54 +++-
.../amoro/server/optimizing/SchedulingPolicy.java | 4 +-
.../server/optimizing/UnKeyedTableCommit.java | 11 +-
.../maintainer/IcebergTableMaintainer.java | 57 ++--
.../persistence/mapper/TableBlockerMapper.java | 23 ++
.../amoro/server/table/AbstractTableMetrics.java | 71 +++++
.../amoro/server/table/DefaultTableService.java | 34 ++-
.../amoro/server/table/TableOptimizingMetrics.java | 40 +--
.../table/TableOrphanFilesCleaningMetrics.java | 42 +--
.../apache/amoro/server/table/TableRuntime.java | 4 +-
.../amoro/server/table/TableSummaryMetrics.java | 38 +--
.../amoro/server/table/blocker/TableBlocker.java | 7 +-
.../src/main/resources/mysql/ams-mysql-init.sql | 2 +-
.../{upgrade.sql => upgrade-0.7.0-to-0.8.0.sql} | 3 -
amoro-ams/src/main/resources/mysql/upgrade.sql | 61 +---
.../main/resources/postgres/ams-postgres-init.sql | 2 +-
.../{upgrade.sql => upgrade-0.7.0-to-0.8.0.sql} | 0
amoro-ams/src/main/resources/postgres/upgrade.sql | 108 +------
.../apache/amoro/server/AMSServiceTestBase.java | 2 +
.../amoro/server/TestDefaultOptimizingService.java | 30 ++
.../server/dashboard/utils/TestPropertiesUtil.java | 42 +++
.../amoro/server/table/TestCatalogService.java | 12 +
.../table/TestSyncTableOfExternalCatalog.java | 320 +++++++++++++++++++++
amoro-common/pom.xml | 5 +
.../amoro/properties/CatalogMetaProperties.java | 2 +-
.../amoro/table/descriptor/ServerTableMeta.java | 16 +-
.../amoro/formats/hudi/HudiTableDescriptor.java | 10 +-
...he.amoro.table.descriptor.FormatTableDescriptor | 2 +-
amoro-format-iceberg/pom.xml | 5 -
.../optimizing/AbstractRewriteFilesExecutor.java | 4 +-
.../amoro/optimizing/IcebergRewriteExecutor.java | 13 +
.../plan/AbstractOptimizingEvaluator.java | 2 +-
.../optimizing/plan/AbstractOptimizingPlanner.java | 36 ++-
.../optimizing/plan/CommonPartitionEvaluator.java | 4 +-
.../org/apache/amoro/table/TableProperties.java | 3 +
.../amoro/flink/catalog/FlinkUnifiedCatalog.java | 57 ++--
.../apache/amoro/flink/catalog/MixedCatalog.java | 31 +-
.../org/apache/amoro/hive/op/UpdateHiveFiles.java | 20 +-
.../amoro/hive/utils/HiveMetaSynchronizer.java | 36 ++-
.../v3.2/amoro-mixed-spark-3.2/pom.xml | 4 +
.../v3.3/amoro-mixed-spark-3.3/pom.xml | 4 +
.../RewriteMixedFormatMergeIntoTable.scala | 13 +-
.../parser/MixedFormatSqlExtensionsParser.scala | 3 +
.../parser/MixedFormatSqlExtendAstBuilder.scala | 6 +-
.../spark/test/suites/sql/TestCreateTableSQL.java | 97 +++++++
.../v3.5/amoro-mixed-spark-3.5/pom.xml | 4 +
.../parser/MixedFormatSqlExtendAstBuilder.scala | 6 +-
.../spark/test/suites/sql/TestCreateTableSQL.java | 97 +++++++
.../apache/amoro/trino/mixed/TestKeyedTable.java | 2 +-
.../formats/paimon/PaimonTableDescriptor.java | 3 +
.../amoro/optimizer/common/OptimizerExecutor.java | 33 ++-
amoro-optimizer/amoro-optimizer-flink/pom.xml | 2 +-
.../amoro/optimizer/flink/FlinkExecutor.java | 18 +-
.../amoro/optimizer/spark/SparkOptimizer.java | 5 +-
amoro-optimizer/amoro-optimizer-standalone/pom.xml | 33 +++
.../optimizer/standalone/StandaloneOptimizer.java | 1 +
.../src/main/resources/log4j2.xml | 13 -
amoro-web/src/types/common.type.ts | 1 +
amoro-web/src/views/optimize/components/List.vue | 6 +-
amoro-web/src/views/tables/components/Details.vue | 4 +-
amoro-web/src/views/tables/components/Files.vue | 2 +-
amoro-web/src/views/tables/index.vue | 6 +-
charts/amoro/templates/_pod.tpl | 2 +-
charts/amoro/tests/amoro-deployment_test.yaml | 10 +-
dist/src/main/amoro-bin/conf/config.yaml | 1 +
dist/src/main/amoro-bin/conf/log4j2.xml | 13 +-
dist/src/main/amoro-bin/conf/optimize/log4j2.xml | 7 +-
docs/_index.md | 2 +-
docs/admin-guides/deployment-on-kubernetes.md | 15 +-
docs/admin-guides/deployment.md | 3 +-
docs/admin-guides/managing-catalogs.md | 11 +
docs/admin-guides/managing-optimizers.md | 20 +-
...tomized-encryption-method-for-configurations.md | 8 +-
docs/engines/flink/flink-cdc-ingestion.md | 4 +-
docs/engines/flink/flink-dml.md | 14 +-
docs/engines/flink/flink-get-started.md | 2 +-
docs/images/admin/create-rest_catalog.png | Bin 0 -> 49616 bytes
docs/images/admin/optimizer_create.png | Bin 0 -> 54835 bytes
docs/images/admin/optimizer_scale.png | Bin 484488 -> 0 bytes
docs/user-guides/cdc-ingestion.md | 4 +-
pom.xml | 26 +-
tools/maven/checkstyle.xml | 5 +
95 files changed, 1439 insertions(+), 527 deletions(-)
create mode 100644
amoro-ams/src/main/java/org/apache/amoro/server/table/AbstractTableMetrics.java
copy amoro-ams/src/main/resources/mysql/{upgrade.sql =>
upgrade-0.7.0-to-0.8.0.sql} (96%)
copy amoro-ams/src/main/resources/postgres/{upgrade.sql =>
upgrade-0.7.0-to-0.8.0.sql} (100%)
copy {amoro-ams =>
amoro-optimizer/amoro-optimizer-standalone}/src/main/resources/log4j2.xml (71%)
create mode 100644 docs/images/admin/create-rest_catalog.png
create mode 100644 docs/images/admin/optimizer_create.png
delete mode 100644 docs/images/admin/optimizer_scale.png
