This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch
dependabot/gradle/com.fasterxml.jackson.core-jackson-annotations-2.14.2
in repository https://gitbox.apache.org/repos/asf/iceberg.git
discard 7e19b8e3ca Build: Bump jackson-annotations from 2.14.1 to 2.14.2
add 32a8ef52dd Core: Prevent RESTCatalog AuthSession from expiring (#6749)
add 8bba0b1d71 Python: Bump pre-commit plugins to the latest version
(#6738)
add 9421ba0f11 Python: Avoid local imports in test (#6737)
add cce25865cf Python: Publish the docs by hand (#6719)
add 7dc48008be Python: Allow to pass in a string as filter (#6657)
add 10df74af01 Python: Remove the DNF conversion (#6721)
add eeb055a377 Delta: Support Snapshot Delta Lake Table to Iceberg Table
(#6449)
add 3a0686d88c Build: Bump cryptography from 39.0.0 to 39.0.1 in /python
(#6767)
add 505368ad3f Parquet: Improve Test Coverage of RowGroupFilter Code with
Nans #6518 (#6554)
add 2c9c5f8274 Spark: Test register table procedure with all Spark
catalogs (#6787)
add 2a7186f4a9 Build: Update `httpclient5` for thread interrupt fix (#6782)
add a180d63e18 Python: Loosen the version requirements for dependencies
(#6745)
add 99659a15c8 Flink: improve metrics
(elapsedSecondsSinceLastSuccessfulCommit) and logging for IcebergFilesCommitter
(#6764)
add 4d43c25447 AWS: Load HttpClientBuilder dynamically to avoid runtime
deps of both urlconnection and apache client (#6746)
add 44ff2a1049 Spark 3.3: Improve log messages in scans (#6776)
add 20fd2df1e9 Spark: Backport handling ResolvingFileIO in determining
locality (#6744)
add 038f831d5b Core: TableMetadata Always Strips Trailing Slash From
Location (#6777)
add e3c18e297e Build: Bump Nessie to 0.48.2 (#6790)
add 03443f5efd Core: Allow configuring metrics reporter by catalog
properties (#6410)
add e5846a573f Core: View representation core implementation (#6598)
add 370c135144 AWS: Remove unused validateTableIdentifier method in
IcebergToGlueConverter (#6795)
add 824c6d8ad3 Spark 3.3: Remove redundant vars in ChangelogRowReader
(#6792)
add d365e78e33 Spark 3.3: Fix comment formatting (#6793)
add 4c06b4c35d Spark 3.3: SQL Extensions for CREATE TAG (#6637)
add 970aeaedbd Spark 3.3: DROP BRANCH SQL implementation (#6752)
add b6b9972538 Doc: update Flink doc for sink metrics (#6765)
add 1521296dd2 Core: Support lazy snapshot loading in TableMetadata (#6811)
add a7a6d354b1 Flink: Support branch writes in Flink sink (#6660)
add 96dc0c6a6e Python: TypeVar for bounding TableScan (#6819)
add e5a93e83b5 API: Revert to using stephenc findbugs dependency for
Nullable (#6815)
add 1e57760394 API: Add new KeyManagementClient interface (#6485)
add 775b3ce3a7 Python: Add support for static table (#6644)
add ef7e20e12e Core: Refactor validation in TableScanUtil (#6791)
add 6cd3d240fb Flink: use tag or branch to scan data (#5029)
add 38d7602d86 Parquet: deprecate Decimal Metadata usage in favor of
DecimalLogicalTypeAnnotation (#6735)
add 72adc7c0ed Spark 3.3: SQL Extensions for DROP TAG (#6807)
add 634215fcd4 Spark 3.3: Return partition stats for AddFiles procedure
(#6797)
add becba79d85 Build: Bump
com.palantir.gradle.gitversion:gradle-git-version (#6812)
add 1ab43a9192 Python: Inline the PyArrow import (#6827)
add 3bbea4aee9 Python: Set PyArrow as the default FileIO (#6822)
add 18d45b42b6 Python: Support for DynamoDB Catalog (#6646)
add 14663ed633 Python: Fix the CI (#6842)
add 49d833aa83 API,Core,Spark: Add rewritten bytes to rewrite data files
procedure results (#6801)
add d79acd4d68 Add application identifier for Snowflake JDBC driver (#6740)
add ec85314266 Spark 3.3: Make manifest file names unique during imports
(#6818)
add 8c4fd98e15 Spark 3.2: Make manifest file names unique during imports
(#6845)
add a6ad1d1312 Spark 3.3: Change default distribution modes (#6828)
add 7fff07c6e2 Data: Add a select collections for IcebergGenerics (#6834)
add 630201e8da Build: Bump werkzeug from 2.2.2 to 2.2.3 in /python (#6854)
add 7d08f4f6fa Python: Add String to Boolean literal conversion (#6851)
add 12feef188a Build: Bump jackson-annotations from 2.14.1 to 2.14.2
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (7e19b8e3ca)
\
N -- N -- N
refs/heads/dependabot/gradle/com.fasterxml.jackson.core-jackson-annotations-2.14.2
(12feef188a)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../{spark-ci.yml => delta-conversion-ci.yml} | 92 +--
.github/workflows/python-ci-docs.yml | 7 +-
.../apache/iceberg/actions/RewriteDataFiles.java | 8 +
.../org/apache/iceberg/encryption/KmsClient.java | 2 +
.../apache/iceberg/view/SQLViewRepresentation.java | 16 +-
.../apache/iceberg/view/ViewRepresentation.java | 17 +-
.../aws/ApacheHttpClientConfigurations.java | 115 +++
.../java/org/apache/iceberg/aws/AwsProperties.java | 154 ++---
.../aws/UrlConnectionHttpClientConfigurations.java | 71 ++
.../iceberg/aws/glue/IcebergToGlueConverter.java | 10 -
.../org/apache/iceberg/aws/TestAwsProperties.java | 327 +--------
.../iceberg/aws/TestHttpClientConfigurations.java | 133 ++++
build.gradle | 78 ++-
.../org/apache/iceberg/BaseMetastoreCatalog.java | 18 +-
.../java/org/apache/iceberg/TableMetadata.java | 63 +-
.../org/apache/iceberg/TableMetadataParser.java | 1 +
.../actions/BaseFileGroupRewriteResult.java | 18 +
.../apache/iceberg/actions/RewriteFileGroup.java | 4 +-
.../iceberg/encryption/KeyManagementClient.java | 30 +-
.../java/org/apache/iceberg/jdbc/JdbcCatalog.java | 2 +-
.../apache/iceberg/rest/RESTSessionCatalog.java | 15 +-
.../java/org/apache/iceberg/util/PropertyUtil.java | 21 +
.../org/apache/iceberg/util/TableScanUtil.java | 27 +-
.../iceberg/view/SQLViewRepresentationParser.java | 119 ++++
.../iceberg/view/UnknownViewRepresentation.java | 9 +-
.../iceberg/view/ViewRepresentationParser.java | 68 ++
.../org/apache/iceberg/TestSnapshotLoading.java | 241 +++++++
.../java/org/apache/iceberg/TestSplitPlanning.java | 6 +-
.../java/org/apache/iceberg/TestTableMetadata.java | 25 +
.../encryption/{kms => }/KeyStoreKmsClient.java | 8 +-
.../encryption/{kms => }/MemoryMockKMS.java | 16 +-
.../org/apache/iceberg/jdbc/TestJdbcCatalog.java | 45 ++
.../view/TestSQLViewRepresentationParser.java | 123 ++++
.../iceberg/view/TestViewRepresentationParser.java | 54 ++
.../org/apache/iceberg/data/IcebergGenerics.java | 9 +-
.../apache/iceberg/data/GenericAppenderHelper.java | 21 +-
.../iceberg/data/TestMetricsRowGroupFilter.java | 29 +-
...eltaLakeToIcebergMigrationSparkIntegration.java | 63 ++
.../delta/SparkDeltaLakeSnapshotTestBase.java | 66 +-
.../iceberg/delta/TestSnapshotDeltaLakeTable.java | 501 ++++++++++++++
.../delta/BaseSnapshotDeltaLakeTableAction.java | 390 +++++++++++
.../BaseSnapshotDeltaLakeTableActionResult.java | 24 +-
.../iceberg/delta/DeltaLakeDataTypeVisitor.java | 50 +-
...DeltaLakeToIcebergMigrationActionsProvider.java | 58 ++
.../apache/iceberg/delta/DeltaLakeTypeToType.java | 73 +-
.../iceberg/delta/SnapshotDeltaLakeTable.java | 89 +++
.../TestBaseSnapshotDeltaLakeTableAction.java | 130 ++++
.../iceberg/delta/TestDeltaLakeTypeToType.java | 187 +++++
docs/flink-getting-started.md | 44 ++
.../org/apache/iceberg/flink/FlinkReadConf.java | 16 +
.../org/apache/iceberg/flink/FlinkReadOptions.java | 12 +
.../org/apache/iceberg/flink/FlinkWriteConf.java | 8 +
.../apache/iceberg/flink/FlinkWriteOptions.java | 5 +
.../org/apache/iceberg/flink/sink/FlinkSink.java | 8 +-
.../iceberg/flink/sink/IcebergFilesCommitter.java | 19 +-
.../flink/sink/IcebergFilesCommitterMetrics.java | 31 +
.../apache/iceberg/flink/source/FlinkSource.java | 20 +
.../iceberg/flink/source/FlinkSplitPlanner.java | 29 +-
.../apache/iceberg/flink/source/IcebergSource.java | 20 +
.../apache/iceberg/flink/source/ScanContext.java | 89 ++-
.../flink/source/StreamingMonitorFunction.java | 27 +-
.../org/apache/iceberg/flink/SimpleDataUtil.java | 46 +-
.../iceberg/flink/sink/TestFlinkIcebergSink.java | 35 +-
.../flink/sink/TestFlinkIcebergSinkBase.java | 64 ++
.../flink/sink/TestFlinkIcebergSinkBranch.java | 137 ++++
.../iceberg/flink/sink/TestFlinkIcebergSinkV2.java | 310 +--------
.../flink/sink/TestFlinkIcebergSinkV2Base.java} | 505 +++++---------
.../flink/sink/TestFlinkIcebergSinkV2Branch.java | 134 ++++
.../flink/sink/TestIcebergFilesCommitter.java | 87 +--
.../apache/iceberg/flink/source/TestFlinkScan.java | 144 ++++
.../iceberg/flink/source/TestFlinkSource.java | 4 +
.../iceberg/flink/source/TestStreamScanSql.java | 71 +-
.../flink/source/TestStreamingMonitorFunction.java | 36 +
.../iceberg/data/parquet/BaseParquetReaders.java | 5 +-
.../iceberg/parquet/ParquetAvroValueReaders.java | 5 +-
.../apache/iceberg/parquet/ParquetAvroWriter.java | 5 +-
.../parquet/ParquetBloomRowGroupFilter.java | 5 +-
.../apache/iceberg/parquet/ParquetConversions.java | 5 +-
.../org/apache/iceberg/pig/PigParquetReader.java | 5 +-
python/.pre-commit-config.yaml | 6 +-
python/mkdocs/docs/api.md | 28 +
python/mkdocs/docs/configuration.md | 36 +
python/mkdocs/docs/how-to-release.md | 4 +
python/mkdocs/docs/index.md | 19 +-
python/poetry.lock | 65 +-
python/pyiceberg/avro/reader.py | 2 +-
python/pyiceberg/catalog/__init__.py | 145 +++-
python/pyiceberg/catalog/dynamodb.py | 769 +++++++++++++++++++++
python/pyiceberg/catalog/glue.py | 228 ++----
python/pyiceberg/catalog/hive.py | 76 +-
python/pyiceberg/catalog/rest.py | 6 +-
python/pyiceberg/exceptions.py | 12 +
python/pyiceberg/expressions/literals.py | 8 +
python/pyiceberg/io/__init__.py | 8 +-
python/pyiceberg/io/pyarrow.py | 6 +-
python/pyiceberg/table/__init__.py | 76 +-
python/pyiceberg/utils/bin_packing.py | 1 -
python/pyproject.toml | 21 +-
...n_test_glue.py => integration_test_dynamodb.py} | 157 ++---
python/tests/catalog/integration_test_glue.py | 128 +---
python/tests/catalog/test_base.py | 7 +-
.../catalog/{test_glue.py => test_dynamodb.py} | 413 ++++++-----
python/tests/catalog/test_glue.py | 264 +++----
python/tests/catalog/test_hive.py | 14 +-
python/tests/cli/test_console.py | 64 +-
python/tests/conftest.py | 109 ++-
python/tests/expressions/test_literals.py | 16 +-
python/tests/expressions/test_visitors.py | 18 +-
python/tests/io/test_io.py | 3 +-
python/tests/table/test_init.py | 19 +-
settings.gradle | 2 +
.../apache/iceberg/snowflake/SnowflakeCatalog.java | 8 +
.../iceberg/snowflake/SnowflakeCatalogTest.java | 8 +-
.../org/apache/iceberg/spark/SparkReadConf.java | 18 +-
.../spark/actions/TestRemoveOrphanFilesAction.java | 2 +-
.../org/apache/iceberg/spark/SparkReadConf.java | 18 +-
.../iceberg/spark/data/SparkParquetReaders.java | 5 +-
.../iceberg/spark/data/SparkParquetWriters.java | 5 +-
.../spark/actions/TestRemoveOrphanFilesAction.java | 2 +-
.../spark/extensions/TestAddFilesProcedure.java | 35 +
.../extensions/TestRegisterTableProcedure.java | 9 +-
.../org/apache/iceberg/spark/SparkReadConf.java | 18 +-
.../org/apache/iceberg/spark/SparkTableUtil.java | 5 +-
.../iceberg/spark/data/SparkParquetReaders.java | 5 +-
.../iceberg/spark/data/SparkParquetWriters.java | 5 +-
.../spark/actions/TestRemoveOrphanFilesAction.java | 2 +-
.../IcebergSqlExtensions.g4 | 20 +-
.../IcebergSparkSqlExtensionsParser.scala | 7 +-
.../IcebergSqlExtensionsAstBuilder.scala | 47 ++
.../plans/logical/CreateOrReplaceBranch.scala | 9 +-
.../plans/logical/CreateOrReplaceTag.scala} | 11 +-
.../sql/catalyst/plans/logical/DropBranch.scala} | 7 +-
.../sql/catalyst/plans/logical/DropTag.scala} | 7 +-
.../sql/catalyst/plans/logical/TagOptions.scala | 7 +-
.../datasources/v2/CreateOrReplaceTagExec.scala} | 40 +-
.../execution/datasources/v2/DropBranchExec.scala} | 18 +-
.../execution/datasources/v2/DropTagExec.scala} | 18 +-
.../v2/ExtendedDataSourceV2Strategy.scala | 12 +
.../spark/extensions/TestAddFilesProcedure.java | 189 ++---
.../{TestCreateBranch.java => TestBranchDDL.java} | 110 ++-
.../extensions/TestRegisterTableProcedure.java | 9 +-
.../extensions/TestRewriteDataFilesProcedure.java | 134 +++-
.../iceberg/spark/extensions/TestTagDDL.java | 364 ++++++++++
.../org/apache/iceberg/spark/SparkCatalog.java | 8 +-
.../org/apache/iceberg/spark/SparkTableUtil.java | 5 +-
.../java/org/apache/iceberg/spark/SparkUtil.java | 15 +-
.../org/apache/iceberg/spark/SparkWriteConf.java | 21 +-
.../iceberg/spark/data/SparkParquetReaders.java | 5 +-
.../iceberg/spark/data/SparkParquetWriters.java | 5 +-
.../spark/procedures/AddFilesProcedure.java | 21 +-
.../procedures/RewriteDataFilesProcedure.java | 7 +-
.../iceberg/spark/source/ChangelogRowReader.java | 10 +-
.../iceberg/spark/source/SparkBatchQueryScan.java | 13 +-
.../iceberg/spark/source/SparkCopyOnWriteScan.java | 19 +-
.../spark/source/SparkFileWriterFactory.java | 3 +-
.../spark/source/SparkMicroBatchStream.java | 5 +-
.../spark/source/SparkPartitioningAwareScan.java | 25 +-
.../spark/source/SparkPositionDeltaWrite.java | 6 +-
.../org/apache/iceberg/spark/source/SparkScan.java | 5 +-
.../apache/iceberg/spark/SparkCatalogConfig.java | 5 +-
.../apache/iceberg/spark/SparkTestHelperBase.java | 2 +-
.../TestSparkDistributionAndOrderingUtil.java | 227 +++++-
.../spark/actions/TestRemoveOrphanFilesAction.java | 2 +-
.../spark/actions/TestRewriteDataFilesAction.java | 55 ++
.../spark/actions/TestRewriteManifestsAction.java | 8 +-
.../spark/source/TestIcebergSourceTablesBase.java | 2 +
.../iceberg/spark/source/TestPartitionValues.java | 1 +
.../iceberg/spark/source/TestSparkDataWrite.java | 7 +-
.../apache/iceberg/spark/sql/TestRefreshTable.java | 4 +-
versions.props | 6 +-
170 files changed, 7243 insertions(+), 2697 deletions(-)
copy .github/workflows/{spark-ci.yml => delta-conversion-ci.yml} (52%)
create mode 100644
aws/src/main/java/org/apache/iceberg/aws/ApacheHttpClientConfigurations.java
create mode 100644
aws/src/main/java/org/apache/iceberg/aws/UrlConnectionHttpClientConfigurations.java
create mode 100644
aws/src/test/java/org/apache/iceberg/aws/TestHttpClientConfigurations.java
copy api/src/main/java/org/apache/iceberg/encryption/KmsClient.java =>
core/src/main/java/org/apache/iceberg/encryption/KeyManagementClient.java (75%)
create mode 100644
core/src/main/java/org/apache/iceberg/view/SQLViewRepresentationParser.java
copy api/src/main/java/org/apache/iceberg/io/DelegatingInputStream.java =>
core/src/main/java/org/apache/iceberg/view/UnknownViewRepresentation.java (83%)
create mode 100644
core/src/main/java/org/apache/iceberg/view/ViewRepresentationParser.java
create mode 100644
core/src/test/java/org/apache/iceberg/TestSnapshotLoading.java
rename core/src/test/java/org/apache/iceberg/encryption/{kms =>
}/KeyStoreKmsClient.java (93%)
rename core/src/test/java/org/apache/iceberg/encryption/{kms =>
}/MemoryMockKMS.java (75%)
create mode 100644
core/src/test/java/org/apache/iceberg/view/TestSQLViewRepresentationParser.java
create mode 100644
core/src/test/java/org/apache/iceberg/view/TestViewRepresentationParser.java
create mode 100644
delta-lake/src/integration/java/org/apache/iceberg/delta/DeltaLakeToIcebergMigrationSparkIntegration.java
copy
spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkExtensionsTestBase.java
=>
delta-lake/src/integration/java/org/apache/iceberg/delta/SparkDeltaLakeSnapshotTestBase.java
(50%)
create mode 100644
delta-lake/src/integration/java/org/apache/iceberg/delta/TestSnapshotDeltaLakeTable.java
create mode 100644
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeTableAction.java
copy aliyun/src/test/java/org/apache/iceberg/aliyun/oss/mock/Range.java =>
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeTableActionResult.java
(69%)
copy
spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/SparkTypeVisitor.java
=>
delta-lake/src/main/java/org/apache/iceberg/delta/DeltaLakeDataTypeVisitor.java
(54%)
create mode 100644
delta-lake/src/main/java/org/apache/iceberg/delta/DeltaLakeToIcebergMigrationActionsProvider.java
copy
spark/v3.1/spark/src/main/java/org/apache/iceberg/spark/SparkTypeToType.java =>
delta-lake/src/main/java/org/apache/iceberg/delta/DeltaLakeTypeToType.java (66%)
create mode 100644
delta-lake/src/main/java/org/apache/iceberg/delta/SnapshotDeltaLakeTable.java
create mode 100644
delta-lake/src/test/java/org/apache/iceberg/delta/TestBaseSnapshotDeltaLakeTableAction.java
create mode 100644
delta-lake/src/test/java/org/apache/iceberg/delta/TestDeltaLakeTypeToType.java
create mode 100644
flink/v1.16/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSinkBase.java
create mode 100644
flink/v1.16/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSinkBranch.java
copy
flink/{v1.14/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSinkV2.java
=>
v1.16/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSinkV2Base.java}
(62%)
create mode 100644
flink/v1.16/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSinkV2Branch.java
create mode 100644 python/pyiceberg/catalog/dynamodb.py
copy python/tests/catalog/{integration_test_glue.py =>
integration_test_dynamodb.py} (65%)
copy python/tests/catalog/{test_glue.py => test_dynamodb.py} (53%)
copy
spark/{v3.1/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/SetIdentifierFields.scala
=>
v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/CreateOrReplaceTag.scala}
(84%)
copy
spark/{v3.1/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/DropIdentifierFields.scala
=>
v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/DropBranch.scala}
(87%)
copy
spark/{v3.1/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/DropIdentifierFields.scala
=>
v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/DropTag.scala}
(87%)
copy api/src/main/java/org/apache/iceberg/io/DelegatingInputStream.java =>
spark/v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/TagOptions.scala
(85%)
copy
spark/{v3.1/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/AddPartitionFieldExec.scala
=>
v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateOrReplaceTagExec.scala}
(57%)
copy
spark/{v3.1/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/SetIdentifierFieldsExec.scala
=>
v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropBranchExec.scala}
(77%)
copy
spark/{v3.1/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/SetIdentifierFieldsExec.scala
=>
v3.3/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropTagExec.scala}
(78%)
rename
spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/{TestCreateBranch.java
=> TestBranchDDL.java} (73%)
create mode 100644
spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestTagDDL.java
