>From Ian Maxon <[email protected]>:
Attention is currently required from: Ian Maxon, hari.
Ian Maxon has uploaded a new patch set (#13) to the change originally created
by hari. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/17419 )
Change subject: [ASTERIXDB-3154] Adds apache iceberg table read support
......................................................................
[ASTERIXDB-3154] Adds apache iceberg table read support
- user model changes: yes
- Introduced two new DDL parameters “table-format” and
“metadata-path” specific to iceberg
- storage format changes: no
- interface changes: no
Details:
These changes add support in AsterixDB to read apache iceberg tables
(version 1) as external data.
Iceberg table details can be specified in the external dataset DDL and
all queries fetch the data from the
latest iceberg snapshot.
APE link:
https://cwiki.apache.org/confluence/display/ASTERIXDB/APE+1%3A+Iceberg+API+Integration
* External Adapters modified: AWS S3 and Hadoop
* Removed kite-sdk dependency and extracted the schema inferrence
utilities from kite-sdk
Change-Id: I12df589a6dffdc5af4a5cace68a11729995ea9af
---
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/s3/iceberg-metadata-specific-location/iceberg-load-selective-metadata.00.ddl.sqlpp
M asterixdb/asterix-external-data/pom.xml
A
asterixdb/asterix-app/src/test/resources/runtimets/results/hdfs/iceberg/read-data.2.adm
M
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/s3/iceberg-mixed-data-format/iceberg-mixed-data-format.00.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/s3/iceberg-modified-data/iceberg-modified-data.01.query.sqlpp
M asterixdb/asterix-server/pom.xml
M
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/SqlppHdfsExecutionTest.java
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/s3/iceberg-mixed-data-format/iceberg-mixed-data-format.01.query.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/hdfs/iceberg/read-data.1.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/s3/iceberg-metadata-specific-location/iceberg-load-selective-metadata.01.query.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/s3/iceberg/iceberg-read-from-latest-snapshot.01.query.sqlpp
M hyracks-fullstack/hyracks/hyracks-hdfs/pom.xml
A
asterixdb/asterix-app/src/test/resources/runtimets/results/external-dataset/s3/iceberg/iceberg-read-from-latest-snapshot.01.adm
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/hdfs/iceberg/read-data.2.query.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/results/external-dataset/s3/iceberg-empty/iceberg-empty.01.adm
M asterixdb/src/main/appended-resources/supplemental-models.xml
A
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/parquet/JsonUtil.java
M
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/aws/s3/S3Constants.java
M
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/parquet/BinaryFileConverterUtil.java
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/s3/iceberg-modified-data/iceberg-modified-data.00.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/s3/iceberg-unsupported-version/iceberg-unsupported-version.01.query.sqlpp
M asterixdb/asterix-app/pom.xml
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/s3/iceberg-multiple-data-files/iceberg-multiple-data-files.01.query.sqlpp
M
asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml
M
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
M asterixdb/pom.xml
M
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/aws/parquet/AwsS3ParquetReaderFactory.java
A
asterixdb/src/main/licenses/content/raw.githubusercontent.com_apache_orc_v1.8.0_NOTICE.txt
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/s3/iceberg-multiple-data-files/iceberg-multiple-data-files.00.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/s3/iceberg-empty/iceberg-empty.00.ddl.sqlpp
A
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/parquet/SchemaUtil.java
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/s3/iceberg/iceberg-read-from-latest-snapshot.00.ddl.sqlpp
M
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
A
asterixdb/asterix-app/src/test/resources/runtimets/results/external-dataset/s3/iceberg-modified-data/iceberg-modified-data.01.adm
A
asterixdb/asterix-app/src/test/resources/runtimets/results/external-dataset/s3/iceberg-multiple-data-files/iceberg-multiple-data-files.01.adm
M
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/aws/AwsS3ExternalDatasetTest.java
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/s3/iceberg-empty/iceberg-empty.01.query.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp_hdfs.xml
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/s3/iceberg-metadata-invalid-location/iceberg-metadata-invalid-location.00.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/s3/iceberg-unsupported-version/iceberg-unsupported-version.00.ddl.sqlpp
M asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
A
asterixdb/src/main/licenses/content/raw.githubusercontent.com_airlift_aircompressor_0.21_notice.md.txt
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/s3/iceberg-metadata-invalid-location/iceberg-metadata-invalid-location.01.query.sqlpp
44 files changed, 2,434 insertions(+), 26 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/19/17419/13
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/17419
To unsubscribe, or for help writing mail filters, visit
https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I12df589a6dffdc5af4a5cace68a11729995ea9af
Gerrit-Change-Number: 17419
Gerrit-PatchSet: 13
Gerrit-Owner: hari <[email protected]>
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins <[email protected]>
Gerrit-Attention: Ian Maxon <[email protected]>
Gerrit-Attention: hari <[email protected]>
Gerrit-MessageType: newpatchset
