This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch dependabot/gradle/calcite-1.37.0
in repository https://gitbox.apache.org/repos/asf/iceberg.git
discard cc321c0591 Build: Bump calcite from 1.10.0 to 1.37.0
add 485ce3470b Build: Bump org.testcontainers:testcontainers from 1.19.7
to 1.19.8 (#10322)
add b752b742e4 docs: Update Quickstart to Hive 4.0.0 (#10325)
add d0dbc9cba2 Build: Bump software.amazon.awssdk:bom from 2.25.45 to
2.25.50 (#10323)
add b6236302c4 Parquet: Add Bloom filter FPP config (#10149)
add 02b1ff968d Spark 3.5: Add support for enums in SparkConfParser (#10311)
add d23c4902eb Spark: Backport tests for struct aggregation pushdown to
3.3/3.4, cleanup assertion (#10333)
add ea916c1700 Docs: Update vendor information for Cloudera (#10278)
add 5e08f886f3 Make proxy endpoint configurable for s3 Http clients
(#10332)
add a6fb9cd2d3 Spark 3.4: Add support for enums in SparkConfParser (#10330)
add 2058053b0c AWS: Retain Glue Catalog table description after updating
Iceberg table (#10199)
add 4c9f47d208 Kafka-connect: Handle namespace creation for auto table
creation (#10186)
add 2cd6d0d475 Avoid adding a closed client to the pool (#10337)
add 090fe2eca9 Build: Bump nessie from 0.81.1 to 0.82.0 (#10318)
add 788bea2695 Spark 3.5: Fix the setting of equalAuthorities in
RemoveOrphanFilesProcedure (#10334)
add f31315e935 Docs: Fix Apache Doris documentation link (#10263)
add f4aaa375e9 Build: Bump io.delta:delta-spark_2.12 from 3.1.0 to 3.2.0
(#10320)
add 139721fee6 Remove unused manifest predicate (#10339)
add bd046f844a Spark: Fix issue when partitioning by UUID (#8250)
add 6abb99f0a7 Spark 3.4, 3.3: Fix the setting of equalAuthorities in
RemoveOrphanFilesProcedure (#10342)
add 2a68edc04e Use a unique field-id for delete files elements (#10347)
add 2886ef4bf6 Core, Spark 3.4: Remove redundant output in tests (#10348)
add 236f6255b1 Core: Replace deprecated Roaring64Bitmap#add call with
addRange (#10350)
add 8d6bee7368 Spark: Coerce shorts and bytes into ints in Parquet Writer
(#10349)
add fcd07d91aa Build: Bump mkdocs-material from 9.5.21 to 9.5.23 (#10353)
add f1a548f972 Build: Bump org.springframework:spring-web from 5.3.34 to
5.3.35 (#10354)
add b3c25fb760 Build: Bump software.amazon.awssdk:bom from 2.25.50 to
2.25.57 (#10367)
add 9114cc87ac Hive: Use base table metadata to create HiveLock (#10016)
add dd2197f835 API: Fix aggregate pushdown when optional DataFile stats
are null (#10273)
add d4c2ef8950 Spark 3.5: Support camel case session configs and options
(#10310)
add 311dbbb10d AWS: Support S3 DSSE-KMS encryption (#8370)
add af9b9ee8a5 Docs: add metrics-reporting back (#10377)
add 2a35e23960 Build: Bump software.amazon.awssdk:bom from 2.25.57 to
2.25.60 (#10385)
add 9a3db37d39 Build: Bump io.netty:netty-buffer from 4.1.109.Final to
4.1.110.Final (#10384)
add ca8af31f7d Build: Bump io.airlift:aircompressor from 0.26 to 0.27
(#10383)
add 957cb0d67f Build: Bump org.springframework:spring-web from 5.3.35 to
5.3.36 (#10382)
add 6f4b19516b Prevent deadlock in Jackson (#10379)
add 580df62d4b Build: Bump nessie from 0.82.0 to 0.83.2 (#10381)
add 795fea9446 Url encode field names for partition paths (#10329)
add f9cdde2516 Build: Bump net.snowflake:snowflake-jdbc from 3.15.1 to
3.16.0 (#10269)
add d723f9fc63 Spark 3.5: Only traverse ancestors of current snapshot when
building changelog scan (#10252)
add 2843f32332 docs: Add archive for documentations older than 1.4.0
(#10374)
add 46732b876a Flink 1.19: Fix flaky TestIcebergSourceFailover >
testBoundedWithSavepoint (#10393)
add 2722290a77 docs: deploy on changes in `docs/` (#10394)
add 6a594546b0 Spark 3.4: Only traverse ancestors of current snapshot when
building changelog scan (#10405)
add 23eb5941cd Bump Azurite test-container to `3.30.0`
add b4ffbf4f18 Build: Bump mkdocs-material from 9.5.23 to 9.5.25 (#10413)
add 1837c81758 Build: Bump org.assertj:assertj-core from 3.25.3 to 3.26.0
(#10416)
add ee11de91c9 Build: Bump guava from 33.2.0-jre to 33.2.1-jre (#10414)
add 2dfc0c66fa Build: Bump org.xerial:sqlite-jdbc from 3.45.3.0 to
3.46.0.0 (#10415)
add 252168419f Docs: Refer to the README.md in `site/` for the docs
(#10402)
add 7d75f823ac Build: Require approving review (#10424)
add 134345dd20 Parquet: Remove TestHelpers in parquet module (#10428)
add 67e181ea9b Core: Introduce AuthConfig (#10161)
add 45bdf3fd43 Build: Bump software.amazon.awssdk:bom from 2.25.60 to
2.25.64 (#10421)
add ab476abfdb Build: Bump com.google.errorprone:error_prone_annotations
(#10418)
add 40da6f1e43 Core: Use TestTemplate instead of Test annotation in
TestPartitionSpecParser/Info (#10435)
add 0a26f02876 Docs: Point links in metrics-reporting.md to GitHub Java
source (#10397)
add a642a9350e Build: Clean up Jackson dependency usages (#10448)
add cbe391d1fa Flink: refactor sink shuffling statistics collection
(#10331)
add fb88e8ea96 Build: Bump calcite from 1.10.0 to 1.37.0
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (cc321c0591)
\
N -- N -- N refs/heads/dependabot/gradle/calcite-1.37.0
(fb88e8ea96)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 3 +
.github/workflows/site-ci.yml | 1 +
.../java/org/apache/iceberg/PartitionSpec.java | 2 +-
.../apache/iceberg/expressions/BoundAggregate.java | 7 +
.../apache/iceberg/expressions/CountNonNull.java | 3 +-
.../apache/iceberg/expressions/MaxAggregate.java | 2 +-
.../apache/iceberg/expressions/MinAggregate.java | 2 +-
.../org/apache/iceberg/TestPartitionPaths.java | 12 +-
.../expressions/TestAggregateEvaluator.java | 52 ++
.../org/apache/iceberg/aws/glue/GlueTestBase.java | 29 ++
.../iceberg/aws/glue/TestGlueCatalogTable.java | 13 +
.../iceberg/aws/s3/TestS3FileIOIntegration.java | 15 +
.../aws/ApacheHttpClientConfigurations.java | 10 +
.../apache/iceberg/aws/HttpClientProperties.java | 7 +
.../aws/UrlConnectionHttpClientConfigurations.java | 10 +
.../iceberg/aws/glue/GlueTableOperations.java | 3 +
.../apache/iceberg/aws/s3/S3FileIOProperties.java | 14 +-
.../org/apache/iceberg/aws/s3/S3RequestUtil.java | 5 +
.../iceberg/aws/s3/signer/S3ObjectMapper.java | 4 +-
.../aws/s3/signer/S3V4RestSignerClient.java | 26 +-
.../iceberg/aws/TestHttpClientConfigurations.java | 13 +
.../apache/iceberg/aws/s3/TestS3RequestUtil.java | 19 +
.../iceberg/azure/adlsv2/AzuriteContainer.java | 2 +-
build.gradle | 18 +-
.../java/org/apache/iceberg/ClientPoolImpl.java | 46 +-
.../java/org/apache/iceberg/ManifestGroup.java | 10 -
.../java/org/apache/iceberg/TableProperties.java | 4 +
.../iceberg/deletes/BitmapPositionDeleteIndex.java | 2 +-
.../apache/iceberg/rest/RESTMetricsReporter.java | 4 +
.../org/apache/iceberg/rest/RESTObjectMapper.java | 4 +-
.../apache/iceberg/rest/RESTSessionCatalog.java | 9 +-
.../auth/AuthConfig.java} | 58 ++-
.../org/apache/iceberg/rest/auth/OAuth2Util.java | 151 +++---
.../org/apache/iceberg/TestClientPoolImpl.java | 15 +-
.../org/apache/iceberg/TestLocationProvider.java | 19 +
.../org/apache/iceberg/TestPartitionSpecInfo.java | 12 +-
.../apache/iceberg/TestPartitionSpecParser.java | 10 +-
.../metrics/TestCommitMetricsResultParser.java | 1 -
.../metrics/TestScanMetricsResultParser.java | 1 -
.../java/org/apache/iceberg/RecordWrapperTest.java | 4 +-
docs/docs/aws.md | 14 +-
docs/docs/configuration.md | 3 +-
docs/docs/metrics-reporting.md | 18 +-
docs/mkdocs.yml | 3 +-
.../flink/source/TestIcebergSourceFailover.java | 36 +-
...cebergSourceFailoverWithWatermarkExtractor.java | 13 +-
.../flink/source/TestIcebergSourceFailover.java | 36 +-
...cebergSourceFailoverWithWatermarkExtractor.java | 13 +-
flink/v1.19/build.gradle | 2 +
.../flink/sink/shuffle/AggregatedStatistics.java | 93 ++--
.../shuffle/AggregatedStatisticsSerializer.java | 175 +++++++
.../sink/shuffle/AggregatedStatisticsTracker.java | 271 ++++++++---
.../iceberg/flink/sink/shuffle/DataStatistics.java | 25 +-
.../sink/shuffle/DataStatisticsCoordinator.java | 110 +++--
.../shuffle/DataStatisticsCoordinatorProvider.java | 24 +-
.../flink/sink/shuffle/DataStatisticsOperator.java | 150 +++---
.../sink/shuffle/DataStatisticsSerializer.java | 204 ++++++++
.../flink/sink/shuffle/DataStatisticsUtil.java | 96 ----
.../flink/sink/shuffle/MapDataStatistics.java | 52 +-
.../sink/shuffle/MapDataStatisticsSerializer.java | 187 -------
.../flink/sink/shuffle/MapRangePartitioner.java | 22 +-
.../flink/sink/shuffle/SketchDataStatistics.java | 87 ++++
.../iceberg/flink/sink/shuffle/SketchUtil.java | 148 ++++++
.../flink/sink/shuffle/SortKeySerializer.java | 17 +-
.../sink/shuffle/SortKeySketchSerializer.java | 143 ++++++
...taStatisticsEvent.java => StatisticsEvent.java} | 22 +-
...isticsOrRecord.java => StatisticsOrRecord.java} | 43 +-
...izer.java => StatisticsOrRecordSerializer.java} | 102 ++--
.../iceberg/flink/sink/shuffle/StatisticsType.java | 55 +++
.../iceberg/flink/sink/shuffle/StatisticsUtil.java | 97 ++++
.../iceberg/flink/sink/shuffle/Fixtures.java | 98 ++++
.../sink/shuffle/TestAggregatedStatistics.java | 63 ---
...ava => TestAggregatedStatisticsSerializer.java} | 41 +-
.../shuffle/TestAggregatedStatisticsTracker.java | 535 +++++++++++++++------
.../shuffle/TestDataStatisticsCoordinator.java | 195 ++++----
.../TestDataStatisticsCoordinatorProvider.java | 182 ++++---
.../sink/shuffle/TestDataStatisticsOperator.java | 332 +++++++------
.../shuffle/TestDataStatisticsSerializer.java} | 44 +-
.../flink/sink/shuffle/TestMapDataStatistics.java | 63 +--
.../sink/shuffle/TestMapRangePartitioner.java | 53 +-
.../sink/shuffle/TestSketchDataStatistics.java | 60 +++
.../iceberg/flink/sink/shuffle/TestSketchUtil.java | 133 +++++
.../shuffle/TestSortKeySerializerPrimitives.java | 33 ++
.../flink/source/TestIcebergSourceFailover.java | 36 +-
...cebergSourceFailoverWithWatermarkExtractor.java | 13 +-
gradle/libs.versions.toml | 33 +-
.../apache/iceberg/hive/HiveTableOperations.java | 6 +-
.../org/apache/iceberg/hive/TestHiveCommits.java | 39 +-
kafka-connect/build.gradle | 4 +-
.../apache/iceberg/connect/events/DataWritten.java | 2 +-
.../iceberg/connect/data/IcebergWriterFactory.java | 24 +
.../connect/data/IcebergWriterFactoryTest.java | 19 +-
mr/build.gradle | 2 +-
.../java/org/apache/iceberg/parquet/Parquet.java | 26 +
.../test/java/org/apache/iceberg/TestHelpers.java | 85 ----
.../iceberg/parquet/TestBloomRowGroupFilter.java | 36 +-
.../iceberg/parquet/TestParquetEncryption.java | 37 +-
site/README.md | 18 +-
site/docs/archive.md | 34 ++
site/docs/hive-quickstart.md | 6 +-
site/docs/how-to-release.md | 70 +--
site/docs/vendors.md | 25 +-
site/nav.yml | 1 +
site/requirements.txt | 2 +-
.../extensions/TestRemoveOrphanFilesProcedure.java | 83 ++++
.../iceberg/spark/source/WritersBenchmark.java | 9 +-
.../iceberg/spark/data/SparkParquetWriters.java | 13 +-
.../procedures/RemoveOrphanFilesProcedure.java | 2 +-
.../apache/iceberg/spark/source/BaseReader.java | 4 +-
.../iceberg/spark/source/InternalRowWrapper.java | 26 +-
.../spark/source/SparkPartitionedFanoutWriter.java | 2 +-
.../spark/source/SparkPartitionedWriter.java | 2 +-
.../spark/source/SparkPositionDeltaWrite.java | 5 +-
.../apache/iceberg/spark/source/SparkWrite.java | 2 +-
.../source/TestDataFrameWriterV2Coercion.java | 69 +++
.../spark/source/TestInternalRowWrapper.java | 3 +-
.../spark/source/TestSparkAppenderFactory.java | 2 +-
.../spark/source/TestSparkFileWriterFactory.java | 2 +-
.../spark/source/TestSparkPartitioningWriters.java | 2 +-
.../source/TestSparkPositionDeltaWriters.java | 2 +-
.../spark/source/TestSparkReaderDeletes.java | 3 +-
.../iceberg/spark/sql/TestAggregatePushDown.java | 122 +++++
.../apache/iceberg/spark/sql/TestCreateTable.java | 27 ++
.../spark/extensions/TestChangelogTable.java | 63 +++
.../extensions/TestRemoveOrphanFilesProcedure.java | 83 ++++
...SystemFunctionPushDownInRowLevelOperations.java | 2 -
.../iceberg/spark/source/WritersBenchmark.java | 9 +-
.../org/apache/iceberg/spark/SparkConfParser.java | 37 ++
.../org/apache/iceberg/spark/SparkReadConf.java | 14 +-
.../iceberg/spark/data/SparkParquetWriters.java | 13 +-
.../procedures/RemoveOrphanFilesProcedure.java | 2 +-
.../apache/iceberg/spark/source/BaseReader.java | 4 +-
.../iceberg/spark/source/InternalRowWrapper.java | 26 +-
.../spark/source/SparkPartitionedFanoutWriter.java | 2 +-
.../spark/source/SparkPartitionedWriter.java | 2 +-
.../spark/source/SparkPositionDeltaWrite.java | 5 +-
.../iceberg/spark/source/SparkScanBuilder.java | 13 +-
.../apache/iceberg/spark/source/SparkWrite.java | 2 +-
.../source/TestDataFrameWriterV2Coercion.java | 69 +++
.../spark/source/TestInternalRowWrapper.java | 3 +-
.../spark/source/TestSparkAppenderFactory.java | 2 +-
.../spark/source/TestSparkFileWriterFactory.java | 2 +-
.../spark/source/TestSparkPartitioningWriters.java | 2 +-
.../source/TestSparkPositionDeltaWriters.java | 2 +-
.../spark/source/TestSparkReaderDeletes.java | 3 +-
.../iceberg/spark/sql/TestAggregatePushDown.java | 122 +++++
.../apache/iceberg/spark/sql/TestCreateTable.java | 26 +
.../spark/extensions/TestChangelogTable.java | 63 +++
.../extensions/TestRemoveOrphanFilesProcedure.java | 83 ++++
.../iceberg/spark/source/WritersBenchmark.java | 9 +-
.../org/apache/iceberg/spark/SparkConfParser.java | 65 +++
.../org/apache/iceberg/spark/SparkReadConf.java | 14 +-
.../iceberg/spark/data/SparkParquetWriters.java | 13 +-
.../procedures/RemoveOrphanFilesProcedure.java | 2 +-
.../apache/iceberg/spark/source/BaseReader.java | 4 +-
.../iceberg/spark/source/InternalRowWrapper.java | 26 +-
.../spark/source/SparkPartitionedFanoutWriter.java | 2 +-
.../spark/source/SparkPartitionedWriter.java | 2 +-
.../spark/source/SparkPositionDeltaWrite.java | 5 +-
.../iceberg/spark/source/SparkScanBuilder.java | 13 +-
.../apache/iceberg/spark/source/SparkWrite.java | 2 +-
.../apache/iceberg/spark/TestSparkWriteConf.java | 26 +
.../iceberg/spark/data/TestSparkParquetWriter.java | 35 ++
.../source/TestDataFrameWriterV2Coercion.java | 81 ++++
.../spark/source/TestInternalRowWrapper.java | 3 +-
.../spark/source/TestSparkAppenderFactory.java | 2 +-
.../spark/source/TestSparkFileWriterFactory.java | 2 +-
.../spark/source/TestSparkPartitioningWriters.java | 2 +-
.../source/TestSparkPositionDeltaWriters.java | 2 +-
.../spark/source/TestSparkReaderDeletes.java | 3 +-
.../iceberg/spark/sql/TestAggregatePushDown.java | 6 +-
.../apache/iceberg/spark/sql/TestCreateTable.java | 26 +
172 files changed, 4763 insertions(+), 1849 deletions(-)
copy core/src/main/java/org/apache/iceberg/{view/BaseViewVersion.java =>
rest/auth/AuthConfig.java} (51%)
create mode 100644
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/AggregatedStatisticsSerializer.java
create mode 100644
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/DataStatisticsSerializer.java
delete mode 100644
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/DataStatisticsUtil.java
delete mode 100644
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/MapDataStatisticsSerializer.java
create mode 100644
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SketchDataStatistics.java
create mode 100644
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SketchUtil.java
create mode 100644
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/SortKeySketchSerializer.java
rename
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/{DataStatisticsEvent.java
=> StatisticsEvent.java} (68%)
rename
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/{DataStatisticsOrRecord.java
=> StatisticsOrRecord.java} (66%)
rename
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/{DataStatisticsOrRecordSerializer.java
=> StatisticsOrRecordSerializer.java} (54%)
create mode 100644
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/StatisticsType.java
create mode 100644
flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/StatisticsUtil.java
create mode 100644
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/shuffle/Fixtures.java
delete mode 100644
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/shuffle/TestAggregatedStatistics.java
copy
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/shuffle/{TestSortKeySerializerBase.java
=> TestAggregatedStatisticsSerializer.java} (53%)
copy
flink/{v1.17/flink/src/test/java/org/apache/iceberg/flink/sink/shuffle/TestSortKeySerializerBase.java
=>
v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/shuffle/TestDataStatisticsSerializer.java}
(51%)
create mode 100644
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/shuffle/TestSketchDataStatistics.java
create mode 100644
flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/shuffle/TestSketchUtil.java
delete mode 100644 parquet/src/test/java/org/apache/iceberg/TestHelpers.java
create mode 100644 site/docs/archive.md
create mode 100644
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/source/TestDataFrameWriterV2Coercion.java
create mode 100644
spark/v3.4/spark/src/test/java/org/apache/iceberg/spark/source/TestDataFrameWriterV2Coercion.java
create mode 100644
spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataFrameWriterV2Coercion.java
