Change in asterixdb[master]: [WIP] Support COPY TO in parquet
Attention is currently required from: preetham.polupar...@couchbase.com. Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18209 ) Change subject: [WIP] Support COPY TO in parquet .. Patch Set 1: Contrib-2 Analytics Compatibility Tests Failed https://cbjenkins.page.link/UtwQWwXpRKW9rzcM9 : UNSTABLE -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18209 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I40dc16969e66af09cde04b460f441af666b39d51 Gerrit-Change-Number: 18209 Gerrit-PatchSet: 1 Gerrit-Owner: preetham.polupar...@couchbase.com Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Attention: preetham.polupar...@couchbase.com Gerrit-Comment-Date: Tue, 26 Mar 2024 21:38:28 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3361][CONF] Add Containerfile and Podman plugin
Attention is currently required from: Ian Maxon, Michael Blow. Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18183 ) Change subject: [ASTERIXDB-3361][CONF] Add Containerfile and Podman plugin .. Patch Set 7: Contrib+1 Analytics Compatibility Tests Successful https://cbjenkins.page.link/MKjtE8MTt2QUEeYR7 : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18183 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Ibdacf4e6b156a3b6ef15b4420a4102c122f8af1d Gerrit-Change-Number: 18183 Gerrit-PatchSet: 7 Gerrit-Owner: Ian Maxon Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Attention: Ian Maxon Gerrit-Attention: Michael Blow Gerrit-Comment-Date: Tue, 26 Mar 2024 21:38:19 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [WIP] Support COPY TO in parquet
Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18209 ) Change subject: [WIP] Support COPY TO in parquet .. Patch Set 1: Analytics Compatibility Compilation Successful https://cbjenkins.page.link/1p7GrUJiF45g1bSb8 : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18209 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I40dc16969e66af09cde04b460f441af666b39d51 Gerrit-Change-Number: 18209 Gerrit-PatchSet: 1 Gerrit-Owner: preetham.polupar...@couchbase.com Gerrit-CC: Anon. E. Moose #1000171 Gerrit-CC: Jenkins Gerrit-Comment-Date: Tue, 26 Mar 2024 19:30:52 + Gerrit-HasComments: No Gerrit-Has-Labels: No Gerrit-MessageType: comment
Change in asterixdb[neo]: [WIP] median
Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18210 ) Change subject: [WIP] median .. Patch Set 1: Analytics Compatibility Compilation Successful https://cbjenkins.page.link/3MM1DTUTK6MP4X1S6 : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18210 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: neo Gerrit-Change-Id: Id8d03f42d54b5ed4cf316c31f0b4cce9dd7c1dc0 Gerrit-Change-Number: 18210 Gerrit-PatchSet: 1 Gerrit-Owner: Ali Alsuliman Gerrit-CC: Anon. E. Moose #1000171 Gerrit-CC: Jenkins Gerrit-Comment-Date: Tue, 26 Mar 2024 21:32:32 + Gerrit-HasComments: No Gerrit-Has-Labels: No Gerrit-MessageType: comment
Change in asterixdb[trinity]: Merge branch 'gerrit/neo' into 'gerrit/trinity'
>From Jenkins : Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18208 ) Change subject: Merge branch 'gerrit/neo' into 'gerrit/trinity' .. Patch Set 1: Integration-Tests+1 Integration Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/237/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18208 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: trinity Gerrit-Change-Id: I511c46368cb0dbf06103d4bd9cf4d2e9c2f558d9 Gerrit-Change-Number: 18208 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Blow Gerrit-Reviewer: Jenkins Gerrit-Comment-Date: Tue, 26 Mar 2024 16:17:57 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3361][CONF] Add Containerfile and Podman plugin
Attention is currently required from: Ian Maxon, Michael Blow. Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18183 ) Change subject: [ASTERIXDB-3361][CONF] Add Containerfile and Podman plugin .. Patch Set 7: Analytics Compatibility Compilation Successful https://cbjenkins.page.link/gHPZSd64bgZk3mMHA : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18183 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Ibdacf4e6b156a3b6ef15b4420a4102c122f8af1d Gerrit-Change-Number: 18183 Gerrit-PatchSet: 7 Gerrit-Owner: Ian Maxon Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Attention: Ian Maxon Gerrit-Attention: Michael Blow Gerrit-Comment-Date: Tue, 26 Mar 2024 19:27:56 + Gerrit-HasComments: No Gerrit-Has-Labels: No Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3361][CONF] Add Containerfile and Podman plugin
>From Jenkins : Attention is currently required from: Ian Maxon, Michael Blow. Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18183 ) Change subject: [ASTERIXDB-3361][CONF] Add Containerfile and Podman plugin .. Patch Set 7: Integration-Tests+1 Integration Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/238/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18183 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: Ibdacf4e6b156a3b6ef15b4420a4102c122f8af1d Gerrit-Change-Number: 18183 Gerrit-PatchSet: 7 Gerrit-Owner: Ian Maxon Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Attention: Ian Maxon Gerrit-Attention: Michael Blow Gerrit-Comment-Date: Tue, 26 Mar 2024 20:33:29 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [WIP] Support COPY TO in parquet
>From :
preetham.polupar...@couchbase.com has uploaded this change for review. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18209 )
Change subject: [WIP] Support COPY TO in parquet
..
[WIP] Support COPY TO in parquet
Change-Id: I40dc16969e66af09cde04b460f441af666b39d51
---
A
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/writer/printer/TextualExternalFileParquetPrinter.java
M
asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml
A
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/pointables/printer/parquet/ObjectWriteSupport.java
M
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/provider/ExternalWriterProvider.java
M asterixdb/asterix-om/pom.xml
A
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/writer/printer/TextualExternalFileParquetPrinterFactory.java
A
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/pointables/printer/parquet/AsterixParquetWriter.java
A
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/pointables/printer/parquet/ParquetRecordVisitorUtils.java
M
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
A
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/pointables/printer/parquet/ParquetRecordLazyVisitor.java
10 files changed, 861 insertions(+), 4 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/09/18209/1
diff --git
a/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml
b/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml
index 1e90f98..fd88b86 100644
---
a/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml
+++
b/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml
@@ -80,7 +80,7 @@
supported-adapter-format-compression
ASX1188: Unsupported writing adapter 'AZUREBLOB'.
Supported adapters: [localfs, s3]
-ASX1189: Unsupported writing format 'csv'. Supported
formats: [json]
+ASX1189: Unsupported writing format 'csv'. Supported
formats: [json, parquet]
ASX1096: Unknown compression scheme rar. Supported
schemes are [gzip]
diff --git
a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
index 79252ad..86c8ef3 100644
---
a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
+++
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
@@ -82,6 +82,7 @@
public static final String KEY_EXPRESSION = "expression";
public static final String KEY_LOCAL_SOCKET_PATH = "local-socket-path";
public static final String KEY_FORMAT = "format";
+public static final String SCHEMA_FORMAT = "schema";
public static final String KEY_INCLUDE = "include";
public static final String KEY_EXCLUDE = "exclude";
public static final String KEY_QUOTE = "quote";
@@ -317,7 +318,7 @@
public static final Set WRITER_SUPPORTED_COMPRESSION;
static {
-WRITER_SUPPORTED_FORMATS = Set.of(FORMAT_JSON_LOWER_CASE);
+WRITER_SUPPORTED_FORMATS = Set.of(FORMAT_JSON_LOWER_CASE,
FORMAT_PARQUET);
WRITER_SUPPORTED_ADAPTERS =
Set.of(ALIAS_LOCALFS_ADAPTER.toLowerCase(),
KEY_ADAPTER_NAME_AWS_S3.toLowerCase());
WRITER_SUPPORTED_COMPRESSION = Set.of(KEY_COMPRESSION_GZIP);
}
diff --git
a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/writer/printer/TextualExternalFileParquetPrinter.java
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/writer/printer/TextualExternalFileParquetPrinter.java
new file mode 100644
index 000..59783e5
--- /dev/null
+++
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/writer/printer/TextualExternalFileParquetPrinter.java
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific
Change in asterixdb[master]: [WIP] Support COPY TO in parquet
>From Jenkins : Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18209 ) Change subject: [WIP] Support COPY TO in parquet .. Patch Set 1: Integration-Tests+1 Integration Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/239/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18209 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I40dc16969e66af09cde04b460f441af666b39d51 Gerrit-Change-Number: 18209 Gerrit-PatchSet: 1 Gerrit-Owner: preetham.polupar...@couchbase.com Gerrit-Reviewer: Jenkins Gerrit-CC: Anon. E. Moose #1000171 Gerrit-Comment-Date: Tue, 26 Mar 2024 20:26:37 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[neo]: [WIP] median
>From Jenkins : Attention is currently required from: Ali Alsuliman. Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18210 ) Change subject: [WIP] median .. Patch Set 1: Integration-Tests-1 Integration Tests Failed https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/240/ : UNSTABLE -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18210 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: neo Gerrit-Change-Id: Id8d03f42d54b5ed4cf316c31f0b4cce9dd7c1dc0 Gerrit-Change-Number: 18210 Gerrit-PatchSet: 1 Gerrit-Owner: Ali Alsuliman Gerrit-Reviewer: Jenkins Gerrit-CC: Anon. E. Moose #1000171 Gerrit-Attention: Ali Alsuliman Gerrit-Comment-Date: Tue, 26 Mar 2024 22:00:42 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[trinity]: Merge branch 'gerrit/neo' into 'gerrit/trinity'
>From Michael Blow : Michael Blow has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18208 ) Change subject: Merge branch 'gerrit/neo' into 'gerrit/trinity' .. Patch Set 1: Verified+1 Code-Review+2 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18208 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: trinity Gerrit-Change-Id: I511c46368cb0dbf06103d4bd9cf4d2e9c2f558d9 Gerrit-Change-Number: 18208 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Blow Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Comment-Date: Wed, 27 Mar 2024 00:35:26 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[trinity]: Merge branch 'gerrit/neo' into 'gerrit/trinity'
>From Michael Blow : Michael Blow has submitted this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18208 ) Change subject: Merge branch 'gerrit/neo' into 'gerrit/trinity' .. Merge branch 'gerrit/neo' into 'gerrit/trinity' Change-Id: I511c46368cb0dbf06103d4bd9cf4d2e9c2f558d9 --- 1 file changed, 13 insertions(+), 0 deletions(-) Approvals: Jenkins: Verified; Verified Michael Blow: Looks good to me, approved; Verified -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18208 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: trinity Gerrit-Change-Id: I511c46368cb0dbf06103d4bd9cf4d2e9c2f558d9 Gerrit-Change-Number: 18208 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Blow Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-MessageType: merged
Change in asterixdb[neo]: [WIP] median
Attention is currently required from: Ali Alsuliman. Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18210 ) Change subject: [WIP] median .. Patch Set 1: Contrib-2 Analytics Compatibility Tests Failed https://cbjenkins.page.link/2p636L8DhBj7XmgY7 : UNSTABLE -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18210 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: neo Gerrit-Change-Id: Id8d03f42d54b5ed4cf316c31f0b4cce9dd7c1dc0 Gerrit-Change-Number: 18210 Gerrit-PatchSet: 1 Gerrit-Owner: Ali Alsuliman Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Attention: Ali Alsuliman Gerrit-Comment-Date: Tue, 26 Mar 2024 23:25:52 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[trinity]: Merge branch 'gerrit/neo' into 'gerrit/trinity'
>From Michael Blow : Michael Blow has uploaded this change for review. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18208 ) Change subject: Merge branch 'gerrit/neo' into 'gerrit/trinity' .. Merge branch 'gerrit/neo' into 'gerrit/trinity' Change-Id: I511c46368cb0dbf06103d4bd9cf4d2e9c2f558d9 --- 1 file changed, 13 insertions(+), 0 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/08/18208/1 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18208 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: trinity Gerrit-Change-Id: I511c46368cb0dbf06103d4bd9cf4d2e9c2f558d9 Gerrit-Change-Number: 18208 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Blow Gerrit-MessageType: newchange
Change in asterixdb[neo]: [ASTERIXDB-3368][EXT] Abort S3 streams before closing
>From Michael Blow : Attention is currently required from: Murtadha Hubail. Michael Blow has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18207 ) Change subject: [ASTERIXDB-3368][EXT] Abort S3 streams before closing .. Patch Set 1: Code-Review+2 Integration-Tests+1 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18207 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: neo Gerrit-Change-Id: I5e85ab19734f417e6a38b522db5298534951687e Gerrit-Change-Number: 18207 Gerrit-PatchSet: 1 Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Attention: Murtadha Hubail Gerrit-Comment-Date: Tue, 26 Mar 2024 13:21:14 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[neo]: [ASTERIXDB-3368][EXT] Abort S3 streams before closing
>From Michael Blow : Attention is currently required from: Murtadha Hubail. Michael Blow has removed a vote from this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18207 ) Change subject: [ASTERIXDB-3368][EXT] Abort S3 streams before closing .. Removed Integration-Tests-1 by Jenkins -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18207 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: neo Gerrit-Change-Id: I5e85ab19734f417e6a38b522db5298534951687e Gerrit-Change-Number: 18207 Gerrit-PatchSet: 1 Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Attention: Murtadha Hubail Gerrit-MessageType: deleteVote
Change in asterixdb[neo]: [ASTERIXDB-3368][EXT] Abort S3 streams before closing
>From Murtadha Hubail :
Murtadha Hubail has submitted this change. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18207 )
Change subject: [ASTERIXDB-3368][EXT] Abort S3 streams before closing
..
[ASTERIXDB-3368][EXT] Abort S3 streams before closing
- user model changes: no
- storage format changes: no
- interface changes: no
Details:
- Abort the S3 stream to avoid fully consuming the stream
in cases where we would like to close the stream early.
Change-Id: I5e85ab19734f417e6a38b522db5298534951687e
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18207
Tested-by: Jenkins
Reviewed-by: Michael Blow
Integration-Tests: Michael Blow
---
M
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/aws/AwsS3InputStream.java
1 file changed, 29 insertions(+), 1 deletion(-)
Approvals:
Michael Blow: Looks good to me, approved; Verified
Jenkins: Verified
Objections:
Anon. E. Moose #1000171: Violations found
diff --git
a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/aws/AwsS3InputStream.java
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/aws/AwsS3InputStream.java
index f14af53..6386813 100644
---
a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/aws/AwsS3InputStream.java
+++
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/aws/AwsS3InputStream.java
@@ -38,6 +38,7 @@
import org.apache.hyracks.api.util.CleanupUtils;
import org.apache.hyracks.util.LogRedactionUtil;
+import software.amazon.awssdk.core.ResponseInputStream;
import software.amazon.awssdk.core.exception.SdkException;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
@@ -49,6 +50,7 @@
// Configuration
private final String bucket;
private final S3Client s3Client;
+private ResponseInputStream s3InStream;
private static final int MAX_RETRIES = 5; // We will retry 5 times in case
of internal error from AWS S3 service
public AwsS3InputStream(Map configuration, List
filePaths) throws HyracksDataException {
@@ -83,7 +85,8 @@
int retries = 0;
while (retries < MAX_RETRIES) {
try {
-in = s3Client.getObject(request);
+s3InStream = s3Client.getObject(request);
+in = s3InStream;
break;
} catch (NoSuchKeyException ex) {
LOGGER.debug(() -> "Key " +
LogRedactionUtil.userData(request.key()) + " was not found in bucket "
@@ -115,6 +118,9 @@
@Override
public void close() throws IOException {
if (in != null) {
+if (s3InStream != null) {
+s3InStream.abort();
+}
CleanupUtils.close(in, null);
}
if (s3Client != null) {
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18207
To unsubscribe, or for help writing mail filters, visit
https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: neo
Gerrit-Change-Id: I5e85ab19734f417e6a38b522db5298534951687e
Gerrit-Change-Number: 18207
Gerrit-PatchSet: 2
Gerrit-Owner: Murtadha Hubail
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins
Gerrit-Reviewer: Michael Blow
Gerrit-Reviewer: Murtadha Hubail
Gerrit-MessageType: merged
