>From Peeyush Gupta <[email protected]>:
Attention is currently required from: Peeyush Gupta.
Hello Jenkins, Anon. E. Moose #1000171,
I'd like you to reexamine a change. Please visit
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19047
to look at the new patch set (#7).
Change subject: [ASTERIXDB-3503][EXT] Use delta kernel api instead of delta
standalone api
......................................................................
[ASTERIXDB-3503][EXT] Use delta kernel api instead of delta standalone api
- user model changes: no
- storage format changes: no
- interface changes: yes
Details:
Delta standalone api does not support reading delta tables created using newer
delta table write protocols. Furthermore, the current implementation does not
support other delta table feautres like column mapping and deletion vectors.
With this change we move to delta kernel api to read delta tables.
Also, to support column mapping and deletion vector features, instead of just
collecting the set of parquet files to read and then reading them through
our own parquet reader implementation, we use the delta kernel api itself
to read the parquet files.
Ext-ref: MB-63840
Change-Id: I0017e63ac0bddcfa0b342d9380d55934a76c12ec
---
A
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/aws/delta/RowSerDe.java
M
asterixdb/asterix-external-data/src/main/resources/META-INF/services/org.apache.asterix.external.api.IDataParserFactory
M asterixdb/asterix-external-data/pom.xml
M
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/aws/delta/AwsS3DeltaReaderFactory.java
M
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/StreamRecordReaderProvider.java
A
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/DeltaDataParser.java
M
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
A
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/aws/delta/DeltaFileRecordReader.java
A
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/aws/delta/DataTypeJsonSerDe.java
A
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/DeltaTableDataParserFactory.java
M
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/external_dataset/deltalake/DeltaTableGenerator.java
M
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/common/deltalake-empty/deltalake-empty.00.ddl.sqlpp
12 files changed, 1,340 insertions(+), 34 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/47/19047/7
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19047
To unsubscribe, or for help writing mail filters, visit
https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I0017e63ac0bddcfa0b342d9380d55934a76c12ec
Gerrit-Change-Number: 19047
Gerrit-PatchSet: 7
Gerrit-Owner: Peeyush Gupta <[email protected]>
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins <[email protected]>
Gerrit-CC: Ayush Tripathi <[email protected]>
Gerrit-CC: Murtadha Hubail <[email protected]>
Gerrit-Attention: Peeyush Gupta <[email protected]>
Gerrit-MessageType: newpatchset
