Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Savyasach Reddy : Attention is currently required from: Hussain Towaileb, Savyasach Reddy. Hello Jenkins, Anon. E. Moose #1000171, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 to look at the new patch set (#7). Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS - user model changes: add dynamic prefixes for HDFS datasets - storage format changes: no - interface changes: no details: - Support dynamic prefixes on HDFS - Support include, exclude and definition parameters in HDFS - Parameters 'path' or 'definition' can be used to specify prefixes Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 --- M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.16.update.sqlpp M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/SqlppHdfsExecutionTest.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/quote-escape/quote-escape.03.ddl.sqlpp M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/hdfs/parquet/ParquetFileRecordReader.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.12.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/header/header.20.update.sqlpp M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/hdfs/HDFSRecordReader.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/simple-write/simple-write.02.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/simple-csv/simple-csv.03.update.sqlpp M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/writer/HDFSExternalFileWriterFactory.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/bad-max-objects-per-file/empty-over.01.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.08.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/delimiter/delimiter.04.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/non-empty-folder/non-empty-folder.01.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.05.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.04.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/early-missing/early-missing.03.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/parquet-utf8/parquet-utf8.03.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/header/header.04.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/csv-error-checks/csv-error-checks.06.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.09.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/supported-adapter-format-compression/supported-adapters.04.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/simple-csv/simple-csv.04.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/quote-escape/quote-escape.11.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/simple-csv/simple-csv.12.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/csv-error-checks/csv-error-checks.08.ddl.sqlpp M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataPrefix.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/HDFSUtils.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/non-empty-folder/non-empty-folder.02.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/order-by/order-by.02.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/csv-error-checks/csv-error-checks.05.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/simple-write/simple-write.01.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimet
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Hussain Towaileb : Attention is currently required from: Savyasach Reddy. Hussain Towaileb has removed a vote from this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Removed Verified-1 by Jenkins -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 8 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Hussain Towaileb Gerrit-Reviewer: Jenkins Gerrit-Attention: Savyasach Reddy Gerrit-MessageType: deleteVote
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Hussain Towaileb : Attention is currently required from: Savyasach Reddy. Hussain Towaileb has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 8: Verified+1 Code-Review+2 (1 comment) File asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/ff8d132f_79e87954 PS6, Line 5766: String adapterModifiedCase = ExternalDataConstants.EXTERNAL_READ_ADAPTERS.stream() : .filter(k -> k.equalsIgnoreCase(adapter)).findFirst().orElse(adapter); : externalDetails.setAdapter(adapterModifiedCase); Let's move this to normalize and include all adapters. Put a comment there stating that this will check the provided adapter with case-sensitivity, and if valid, will use the constant we have for it. -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 8 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Hussain Towaileb Gerrit-Reviewer: Jenkins Gerrit-Attention: Savyasach Reddy Gerrit-Comment-Date: Wed, 20 Nov 2024 16:00:40 + Gerrit-HasComments: Yes Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Hussain Towaileb : Attention is currently required from: Savyasach Reddy. Hussain Towaileb has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 8: (1 comment) File asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/15ca366f_e1650fd0 PS6, Line 5766: String adapterModifiedCase = ExternalDataConstants.EXTERNAL_READ_ADAPTERS.stream() : .filter(k -> k.equalsIgnoreCase(adapter)).findFirst().orElse(adapter); : externalDetails.setAdapter(adapterModifiedCase); > Let's move this to normalize and include all adapters. […] Done -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 8 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Hussain Towaileb Gerrit-Reviewer: Jenkins Gerrit-Attention: Savyasach Reddy Gerrit-Comment-Date: Wed, 20 Nov 2024 16:00:55 + Gerrit-HasComments: Yes Gerrit-Has-Labels: No Comment-In-Reply-To: Hussain Towaileb Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
Attention is currently required from: Hussain Towaileb, Savyasach Reddy. Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 8: Analytics Compatibility Compilation Successful https://cbjenkins.page.link/aS6cWqfTaZeyMQ9YA : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 8 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-CC: Hussain Towaileb Gerrit-Attention: Hussain Towaileb Gerrit-Attention: Savyasach Reddy Gerrit-Comment-Date: Wed, 20 Nov 2024 14:40:46 + Gerrit-HasComments: No Gerrit-Has-Labels: No Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Jenkins : Attention is currently required from: Hussain Towaileb, Savyasach Reddy. Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 8: Integration-Tests+1 Integration Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/1853/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 8 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-CC: Hussain Towaileb Gerrit-Attention: Hussain Towaileb Gerrit-Attention: Savyasach Reddy Gerrit-Comment-Date: Wed, 20 Nov 2024 15:13:57 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
Attention is currently required from: Hussain Towaileb, Savyasach Reddy. Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 7: Analytics Compatibility Tests Failed https://cbjenkins.page.link/Exc1ckJQbvCk7LSq9 : UNSTABLE -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 7 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-CC: Hussain Towaileb Gerrit-Attention: Hussain Towaileb Gerrit-Attention: Savyasach Reddy Gerrit-Comment-Date: Wed, 20 Nov 2024 14:20:22 + Gerrit-HasComments: No Gerrit-Has-Labels: No Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Jenkins : Attention is currently required from: Hussain Towaileb. Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 7: Integration-Tests+1 Integration Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/1850/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 7 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-CC: Hussain Towaileb Gerrit-Attention: Hussain Towaileb Gerrit-Comment-Date: Wed, 20 Nov 2024 13:29:47 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
Attention is currently required from: Hussain Towaileb, Savyasach Reddy. Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 7: Contrib-2 Analytics Compatibility Tests Failed https://cbjenkins.page.link/9jUpqGNoEBedXiR1A : UNSTABLE -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 7 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-CC: Hussain Towaileb Gerrit-Attention: Hussain Towaileb Gerrit-Attention: Savyasach Reddy Gerrit-Comment-Date: Wed, 20 Nov 2024 13:52:49 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
Attention is currently required from: Hussain Towaileb. Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 7: Analytics Compatibility Compilation Successful https://cbjenkins.page.link/cEVi3JKH3Gkfy8tcA : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 7 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-CC: Hussain Towaileb Gerrit-Attention: Hussain Towaileb Gerrit-Comment-Date: Wed, 20 Nov 2024 11:28:02 + Gerrit-HasComments: No Gerrit-Has-Labels: No Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Savyasach Reddy : Attention is currently required from: Hussain Towaileb. Savyasach Reddy has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 7: (2 comments) File asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/3619ed69_477e9227 PS4, Line 30: : > You probably could reduce these by implementing that a missing value of a > placeholder means it passe […] Done File asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/HDFSUtils.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/25372790_099c9b03 PS4, Line 590: Objects.equals(ExternalDataConstants.KEY_ADAPTER_NAME_HDFS, : configuration.get(ExternalDataConstants.KEY_EXTERNAL_SOURCE_TYPE)); > Instead do: […] Done -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 7 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-CC: Hussain Towaileb Gerrit-Attention: Hussain Towaileb Gerrit-Comment-Date: Wed, 20 Nov 2024 11:16:06 + Gerrit-HasComments: Yes Gerrit-Has-Labels: No Comment-In-Reply-To: Hussain Towaileb Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
Attention is currently required from: Hussain Towaileb, Savyasach Reddy. Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 6: Contrib+1 Analytics Compatibility Tests Successful https://cbjenkins.page.link/DguKLcDZWRF8xK968 : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 6 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-CC: Hussain Towaileb Gerrit-Attention: Hussain Towaileb Gerrit-Attention: Savyasach Reddy Gerrit-Comment-Date: Wed, 20 Nov 2024 07:32:00 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
Attention is currently required from: Hussain Towaileb, Savyasach Reddy. Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 6: Contrib-2 Analytics Compatibility Tests Failed https://cbjenkins.page.link/HKEEmXBVcSu4x4bN8 : UNSTABLE -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 6 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-CC: Hussain Towaileb Gerrit-Attention: Hussain Towaileb Gerrit-Attention: Savyasach Reddy Gerrit-Comment-Date: Tue, 19 Nov 2024 21:36:25 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
Attention is currently required from: Hussain Towaileb. Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 6: Analytics Compatibility Compilation Successful https://cbjenkins.page.link/1TeC8rH6pKXQrpuJ7 : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 6 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-CC: Hussain Towaileb Gerrit-Attention: Hussain Towaileb Gerrit-Comment-Date: Tue, 19 Nov 2024 19:17:50 + Gerrit-HasComments: No Gerrit-Has-Labels: No Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Jenkins : Attention is currently required from: Hussain Towaileb. Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 6: Integration-Tests+1 Integration Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/1843/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 6 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-CC: Hussain Towaileb Gerrit-Attention: Hussain Towaileb Gerrit-Comment-Date: Tue, 19 Nov 2024 20:03:10 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Savyasach Reddy : Attention is currently required from: Hussain Towaileb. Hello Jenkins, Anon. E. Moose #1000171, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 to look at the new patch set (#6). Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS - user model changes: add dynamic prefixes for HDFS datasets - storage format changes: no - interface changes: no details: - Support dynamic prefixes on HDFS - Support include, exclude and definition parameters in HDFS - Parameters 'path' or 'definition' can be used to specify prefixes Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 --- M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.16.update.sqlpp M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/SqlppHdfsExecutionTest.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/quote-escape/quote-escape.03.ddl.sqlpp M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/hdfs/parquet/ParquetFileRecordReader.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.12.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/header/header.20.update.sqlpp M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/hdfs/HDFSRecordReader.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/simple-write/simple-write.02.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/simple-csv/simple-csv.03.update.sqlpp M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/writer/HDFSExternalFileWriterFactory.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/bad-max-objects-per-file/empty-over.01.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.08.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/delimiter/delimiter.04.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/non-empty-folder/non-empty-folder.01.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.05.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.04.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/early-missing/early-missing.03.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/parquet-utf8/parquet-utf8.03.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/header/header.04.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/csv-error-checks/csv-error-checks.06.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.09.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/supported-adapter-format-compression/supported-adapters.04.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/simple-csv/simple-csv.04.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/quote-escape/quote-escape.11.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/simple-csv/simple-csv.12.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/csv-error-checks/csv-error-checks.08.ddl.sqlpp M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataPrefix.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/HDFSUtils.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/non-empty-folder/non-empty-folder.02.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/order-by/order-by.02.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/csv-error-checks/csv-error-checks.05.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/simple-write/simple-write.01.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/c
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Savyasach Reddy : Attention is currently required from: Hussain Towaileb. Savyasach Reddy has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 5: (3 comments) File asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/2d0e3639_874b2938 PS3, Line 153: (ExternalDataConstants.KEY_HADOOP_INPUT_DIR, ""); > hdfsConf.set(ExternalDataConstants.KEY_HADOOP_INPUT_DIR, "/") is returning > all the files. […] Done https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/a2918f38_1fc18b0f PS3, Line 158: hdfsConf.set(ExternalDataConstants.KEY_HADOOP_INPUT_DIR, ""); > Ideally, we should do the same, but wouldn't the continuation of the > execution fail? After all, HDFS […] Done File asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/a5dddff3_84764640 PS4, Line 162: "No files found at the given path" > Why not use ErrorCode. […] Done -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 5 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-CC: Hussain Towaileb Gerrit-Attention: Hussain Towaileb Gerrit-Comment-Date: Tue, 19 Nov 2024 19:02:20 + Gerrit-HasComments: Yes Gerrit-Has-Labels: No Comment-In-Reply-To: Hussain Towaileb Comment-In-Reply-To: Savyasach Reddy Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Jenkins : Attention is currently required from: Savyasach Reddy. Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 5: Integration-Tests-1 Integration Tests Failed https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/1836/ : UNSTABLE -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 5 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-CC: Hussain Towaileb Gerrit-Attention: Savyasach Reddy Gerrit-Comment-Date: Tue, 19 Nov 2024 12:11:30 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
Attention is currently required from: Savyasach Reddy. Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 5: Contrib-2 Analytics Compatibility Tests Failed https://cbjenkins.page.link/NgvJQDDuq29eBrJX9 : UNSTABLE -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 5 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-CC: Hussain Towaileb Gerrit-Attention: Savyasach Reddy Gerrit-Comment-Date: Tue, 19 Nov 2024 13:55:35 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
Attention is currently required from: Savyasach Reddy. Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 5: Analytics Compatibility Compilation Successful https://cbjenkins.page.link/uNtfUaXB3bqfi9646 : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 5 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-CC: Hussain Towaileb Gerrit-Attention: Savyasach Reddy Gerrit-Comment-Date: Tue, 19 Nov 2024 11:29:09 + Gerrit-HasComments: No Gerrit-Has-Labels: No Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Savyasach Reddy : Attention is currently required from: Savyasach Reddy. Hello Jenkins, Anon. E. Moose #1000171, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 to look at the new patch set (#5). Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS - user model changes: add dynamic prefixes for HDFS datasets - storage format changes: no - interface changes: no details: - Support dynamic prefixes on HDFS - Support include, exclude and definition parameters in HDFS - Parameters 'path' or 'definition' can be used to specify prefixes Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 --- M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.16.update.sqlpp M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/SqlppHdfsExecutionTest.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/quote-escape/quote-escape.03.ddl.sqlpp M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/hdfs/parquet/ParquetFileRecordReader.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.12.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/header/header.20.update.sqlpp M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/hdfs/HDFSRecordReader.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/simple-write/simple-write.02.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/simple-csv/simple-csv.03.update.sqlpp M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/writer/HDFSExternalFileWriterFactory.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/bad-max-objects-per-file/empty-over.01.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.08.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/delimiter/delimiter.04.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/non-empty-folder/non-empty-folder.01.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.05.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.04.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/early-missing/early-missing.03.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/parquet-utf8/parquet-utf8.03.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/header/header.04.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/csv-error-checks/csv-error-checks.06.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.09.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/supported-adapter-format-compression/supported-adapters.04.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/simple-csv/simple-csv.04.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/quote-escape/quote-escape.11.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/simple-csv/simple-csv.12.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/csv-error-checks/csv-error-checks.08.ddl.sqlpp M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataPrefix.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/HDFSUtils.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/non-empty-folder/non-empty-folder.02.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/order-by/order-by.02.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/csv-error-checks/csv-error-checks.05.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/simple-write/simple-write.01.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/co
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
Attention is currently required from: Hussain Towaileb, Savyasach Reddy. Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 4: Contrib+1 Analytics Compatibility Tests Successful https://cbjenkins.page.link/PRVEBt24QntQGtTn9 : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 4 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-CC: Hussain Towaileb Gerrit-Attention: Hussain Towaileb Gerrit-Attention: Savyasach Reddy Gerrit-Comment-Date: Mon, 18 Nov 2024 16:20:29 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Hussain Towaileb : Attention is currently required from: Savyasach Reddy. Hussain Towaileb has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 4: (4 comments) File asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/c1661fd8_327ae2c2 PS4, Line 30: : You probably could reduce these by implementing that a missing value of a placeholder means it passes nothing. File asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/93dd5656_d5ae419c PS3, Line 158: hdfsConf.set(ExternalDataConstants.KEY_HADOOP_INPUT_DIR, ""); > I've updated with the same. […] Ideally, we should do the same, but wouldn't the continuation of the execution fail? After all, HDFS itself is throwing an exception. AWS doesn't throw an exception by itself, just returns empty, so we translate that into a warning. File asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/e76a7614_1fcd51fc PS4, Line 162: "No files found at the given path" Why not use ErrorCode.EXTERNAL_SOURCE_CONFIGURATION_RETURNED_NO_FILES? Although the current one is fine as well. File asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/HDFSUtils.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/5f7aae17_1efd6335 PS4, Line 590: Objects.equals(ExternalDataConstants.KEY_ADAPTER_NAME_HDFS, : configuration.get(ExternalDataConstants.KEY_EXTERNAL_SOURCE_TYPE)); Instead do: return ExternalDataConstants.KEY_ADAPTER_NAME_HDFS.equalsIgnoreCase(configuration.get(ExternalDataConstants.KEY_EXTERNAL_SOURCE_TYPE)); And you can revert the HDFS to hdfs in the test cases, it becomes case insensitive. -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 4 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-CC: Hussain Towaileb Gerrit-Attention: Savyasach Reddy Gerrit-Comment-Date: Mon, 18 Nov 2024 16:49:48 + Gerrit-HasComments: Yes Gerrit-Has-Labels: No Comment-In-Reply-To: Hussain Towaileb Comment-In-Reply-To: Savyasach Reddy Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Jenkins : Attention is currently required from: Hussain Towaileb, Savyasach Reddy. Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 4: Integration Tests Failed https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/1800/ : UNSTABLE -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 4 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-CC: Hussain Towaileb Gerrit-Attention: Hussain Towaileb Gerrit-Attention: Savyasach Reddy Gerrit-Comment-Date: Mon, 18 Nov 2024 15:58:29 + Gerrit-HasComments: No Gerrit-Has-Labels: No Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Jenkins : Attention is currently required from: Hussain Towaileb, Savyasach Reddy. Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 4: Integration Tests Failed https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/1798/ : UNSTABLE -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 4 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-CC: Hussain Towaileb Gerrit-Attention: Hussain Towaileb Gerrit-Attention: Savyasach Reddy Gerrit-Comment-Date: Mon, 18 Nov 2024 15:12:12 + Gerrit-HasComments: No Gerrit-Has-Labels: No Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Savyasach Reddy :
Attention is currently required from: Hussain Towaileb.
Savyasach Reddy has posted comments on this change. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 )
Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
..
Patch Set 4:
(2 comments)
File
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/default-namespace/default-namespace.05.ddl.sqlpp:
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/4b1a8a4d_269398da
PS3, Line 22: ("container"="playground"),
> Same as below.
Done
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/ed2238cf_95d83ffd
PS3, Line 24: ("input-format" = "text-input-format"),
> ("input-format" = "text-input-format"), change to %additionalProperties%
Done
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066
To unsubscribe, or for help writing mail filters, visit
https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2
Gerrit-Change-Number: 19066
Gerrit-PatchSet: 4
Gerrit-Owner: Savyasach Reddy
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins
Gerrit-CC: Hussain Towaileb
Gerrit-Attention: Hussain Towaileb
Gerrit-Comment-Date: Mon, 18 Nov 2024 13:45:12 +
Gerrit-HasComments: Yes
Gerrit-Has-Labels: No
Comment-In-Reply-To: Hussain Towaileb
Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Savyasach Reddy : Attention is currently required from: Savyasach Reddy. Hello Jenkins, Anon. E. Moose #1000171, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 to look at the new patch set (#4). Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS - user model changes: add dynamic prefixes for HDFS datasets - storage format changes: no - interface changes: no details: - Support dynamic prefixes on HDFS - Support include, exclude and definition parameters in HDFS - Parameters 'path' or 'definition' can be used to specify prefixes Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 --- M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.16.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to-hdfs/parquet-simple/parquet-simple.03.ddl.sqlpp M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/SqlppHdfsExecutionTest.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/quote-escape/quote-escape.03.ddl.sqlpp M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/hdfs/parquet/ParquetFileRecordReader.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.12.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/header/header.20.update.sqlpp M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/hdfs/HDFSRecordReader.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/simple-write/simple-write.02.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/simple-csv/simple-csv.03.update.sqlpp M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/writer/HDFSExternalFileWriterFactory.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/bad-max-objects-per-file/empty-over.01.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.08.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/delimiter/delimiter.04.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/non-empty-folder/non-empty-folder.01.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.05.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.04.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/early-missing/early-missing.03.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/hdfs/hdfs_02/hdfs_02.1.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/parquet-utf8/parquet-utf8.03.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/header/header.04.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/csv-error-checks/csv-error-checks.06.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/parquet-error-checks/parquet-error-checks.09.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/supported-adapter-format-compression/supported-adapters.04.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/simple-csv/simple-csv.04.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/hdfs/issue_245_hdfs/issue_245_hdfs.1.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/quote-escape/quote-escape.11.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/csv/simple-csv/simple-csv.12.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/csv-error-checks/csv-error-checks.08.ddl.sqlpp M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataPrefix.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/HDFSUtils.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to-hdfs/parquet-tweet/parquet-tweet.04.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/negative/non-empty-folde
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
Attention is currently required from: Hussain Towaileb. Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 4: Analytics Compatibility Compilation Successful https://cbjenkins.page.link/n9ps2KrQqKD2tCJM8 : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 4 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-CC: Hussain Towaileb Gerrit-Attention: Hussain Towaileb Gerrit-Comment-Date: Mon, 18 Nov 2024 13:54:04 + Gerrit-HasComments: No Gerrit-Has-Labels: No Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Jenkins : Attention is currently required from: Hussain Towaileb, Savyasach Reddy. Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 4: Integration-Tests-1 Integration Tests Failed https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/1797/ : UNSTABLE -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 4 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-CC: Hussain Towaileb Gerrit-Attention: Hussain Towaileb Gerrit-Attention: Savyasach Reddy Gerrit-Comment-Date: Mon, 18 Nov 2024 13:54:33 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Savyasach Reddy :
Attention is currently required from: Hussain Towaileb.
Savyasach Reddy has posted comments on this change. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 )
Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
..
Patch Set 4:
(11 comments)
File
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestConstants.java:
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/0b0f808f_7f6eed05
PS3, Line 37: S3_DDL_TEMPLATE_DEFAULT
> S3_TEMPLATE_DEFAULT_NO_PARANTHESES_WITH_COLONS
Done
File
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestExecutor.java:
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/4fc90e64_41d90a36
PS3, Line 2567: else
> Same as above
Done
File
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java:
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/e0ab026b_62f59f1e
PS3, Line 152: LOGGER.warn("The provided external dataset configuration
returned no files from the external source");
> Use warning collector to warn, see examples in S3Utils.
Done
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/392c2c73_98aa957f
PS3, Line 153: hdfsConf
> Let's have a method to do this in the HDFS utils, and here instead set the
> KEY_PATH to empty string.
Done
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/cc3da553_454aa950
PS3, Line 153: (ExternalDataConstants.KEY_HADOOP_INPUT_DIR, "");
> Let's see what happens if we pass "" vs "/". Will it return everything or
> empty, .. […]
hdfsConf.set(ExternalDataConstants.KEY_HADOOP_INPUT_DIR, "/") is returning all
the files.
hdfsConf.set(ExternalDataConstants.KEY_HADOOP_INPUT_DIR, "") doesn't return any
files due to the function HDFSDataSourceFactory#getInputSplits
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/c916c817_eef09793
PS3, Line 158: hdfsConf.set(ExternalDataConstants.KEY_HADOOP_INPUT_DIR, "");
> Throw external source error exception.
I've updated with the same.
But for S3, if the prefix doesn't match, we aren't throwing an error, just a
warning. Shouldn't we do the same over here instead of an error?
File
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataPrefix.java:
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/2fbee68b_837fcece
PS3, Line 80:
Objects.equals(configuration.get(ExternalDataConstants.KEY_EXTERNAL_SOURCE_TYPE),
: ExternalDataConstants.KEY_HDFS_URL)
> Let's have a function for this in HDFSUtils. […]
Done
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/49e72340_0ce960cd
PS3, Line 81: KEY_HDFS_URL
> Compare it with KEY_ADAPTER_NAME_HDFS, use . […]
Done
File
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/HDFSUtils.java:
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/af74494d_cfdd1d8a
PS3, Line 236:
> Add this comment: […]
Done
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/73a921f6_1e2b4b2a
PS3, Line 404: updateRoot
> updateRootPath
Done
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/b85670ad_1e3f0a8b
PS3, Line 577: try {
> Add a comment here that this is for validation purpose for external data
> prefix stuff.
Done
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066
To unsubscribe, or for help writing mail filters, visit
https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2
Gerrit-Change-Number: 19066
Gerrit-PatchSet: 4
Gerrit-Owner: Savyasach Reddy
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins
Gerrit-CC: Hussain Towaileb
Gerrit-Attention: Hussain Towaileb
Gerrit-Comment-Date: Mon, 18 Nov 2024 13:44:49 +
Gerrit-HasComments: Yes
Gerrit-Has-Labels: No
Comment-In-Reply-To: Hussain Towaileb
Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Hussain Towaileb :
Attention is currently required from: Savyasach Reddy.
Hussain Towaileb has posted comments on this change. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 )
Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
..
Patch Set 3:
(13 comments)
File
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestConstants.java:
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/6b48ecd7_9ce43e6f
PS3, Line 37: S3_DDL_TEMPLATE_DEFAULT
S3_TEMPLATE_DEFAULT_NO_PARANTHESES_WITH_COLONS
File
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestExecutor.java:
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/b7e3e388_ed0c8915
PS3, Line 2567: else
Same as above
File
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/default-namespace/default-namespace.05.ddl.sqlpp:
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/27743719_eb982b72
PS3, Line 22: ("container"="playground"),
Same as below.
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/71622ddb_b2ce25d8
PS3, Line 24: ("input-format" = "text-input-format"),
("input-format" = "text-input-format"), change to %additionalProperties%
File
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java:
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/5f0212c2_53451cef
PS3, Line 152: LOGGER.warn("The provided external dataset configuration
returned no files from the external source");
Use warning collector to warn, see examples in S3Utils.
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/6c17ab73_564d0b02
PS3, Line 153: hdfsConf
Let's have a method to do this in the HDFS utils, and here instead set the
KEY_PATH to empty string.
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/6455506f_16769350
PS3, Line 153: (ExternalDataConstants.KEY_HADOOP_INPUT_DIR, "");
Let's see what happens if we pass "" vs "/". Will it return everything or
empty, .. etc
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/6be45446_b604b126
PS3, Line 158: hdfsConf.set(ExternalDataConstants.KEY_HADOOP_INPUT_DIR, "");
Throw external source error exception.
File
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataPrefix.java:
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/3a68875f_516db8ef
PS3, Line 80:
Objects.equals(configuration.get(ExternalDataConstants.KEY_EXTERNAL_SOURCE_TYPE),
: ExternalDataConstants.KEY_HDFS_URL)
Let's have a function for this in HDFSUtils.
boolean isHdfs = ...;
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/4226b370_0962bf19
PS3, Line 81: KEY_HDFS_URL
Compare it with KEY_ADAPTER_NAME_HDFS, use .equalsIgnoreCase
File
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/HDFSUtils.java:
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/ff0ff9fc_430197bb
PS3, Line 236:
Add this comment:
In case KEY_PATH is not being used, it means definition is being used, and the
property KEY_HADOOP_INPUT_DIR will be set in HDFSDatasourceFactory based on the
provided definition.
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/a0643f5d_3bfae0c8
PS3, Line 404: updateRoot
updateRootPath
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/2fd7c968_255c18bd
PS3, Line 577: try {
Add a comment here that this is for validation purpose for external data prefix
stuff.
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066
To unsubscribe, or for help writing mail filters, visit
https://asterix-gerrit.ics.uci.edu/settings
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2
Gerrit-Change-Number: 19066
Gerrit-PatchSet: 3
Gerrit-Owner: Savyasach Reddy
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins
Gerrit-CC: Hussain Towaileb
Gerrit-Attention: Savyasach Reddy
Gerrit-Comment-Date: Fri, 15 Nov 2024 15:42:00 +
Gerrit-HasComments: Yes
Gerrit-Has-Labels: No
Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
Attention is currently required from: Savyasach Reddy. Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 3: Contrib+1 Analytics Compatibility Tests Successful https://cbjenkins.page.link/apwoxdYEXu3EtMsQ6 : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 3 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Attention: Savyasach Reddy Gerrit-Comment-Date: Thu, 14 Nov 2024 18:11:57 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
Attention is currently required from: Savyasach Reddy. Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 3: Contrib-2 Analytics Compatibility Tests Failed https://cbjenkins.page.link/zWqkKVw6NM2An8tn8 : UNSTABLE -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 3 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Attention: Savyasach Reddy Gerrit-Comment-Date: Thu, 14 Nov 2024 14:09:29 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Jenkins : Attention is currently required from: Savyasach Reddy. Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 3: Integration-Tests-1 Integration Tests Failed https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/1766/ : UNSTABLE -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 3 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Attention: Savyasach Reddy Gerrit-Comment-Date: Thu, 14 Nov 2024 12:53:32 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
Attention is currently required from: Savyasach Reddy. Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 3: Analytics Compatibility Compilation Successful https://cbjenkins.page.link/28BF4TmczjFaHZBUA : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 3 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Attention: Savyasach Reddy Gerrit-Comment-Date: Thu, 14 Nov 2024 11:48:52 + Gerrit-HasComments: No Gerrit-Has-Labels: No Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Savyasach Reddy : Attention is currently required from: Savyasach Reddy. Hello Jenkins, Anon. E. Moose #1000171, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 to look at the new patch set (#3). Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS - user model changes: add dynamic prefixes for HDFS datasets - storage format changes: no - interface changes: no details: - Support dynamic prefixes on HDFS - Support include, exclude and definition parameters in HDFS - Parameters 'path' or 'definition' can be used to specify prefixes Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 --- M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/default-namespace/default-namespace.01.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/SqlppHdfsExecutionTest.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/writer/HDFSExternalFileWriter.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestExecutor.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataPrefix.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/HDFSUtils.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/default-namespace/default-namespace.02.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/default-namespace/default-namespace.05.ddl.sqlpp M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/AnalyzingTestExecutor.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestConstants.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java M asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp_hdfs.xml M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/default-namespace/default-namespace.03.update.sqlpp M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/writer/HDFSExternalFileWriterFactory.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/default-namespace/default-namespace.04.update.sqlpp 16 files changed, 255 insertions(+), 64 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/66/19066/3 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 3 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Attention: Savyasach Reddy Gerrit-MessageType: newpatchset
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
Attention is currently required from: Savyasach Reddy. Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 2: Contrib+1 Analytics Compatibility Tests Successful https://cbjenkins.page.link/5KeUnfVWf4MFWa6B9 : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 2 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins Gerrit-Attention: Savyasach Reddy Gerrit-Comment-Date: Wed, 13 Nov 2024 22:49:46 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Jenkins : Attention is currently required from: Savyasach Reddy. Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 2: Integration-Tests+1 Integration Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/1755/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 2 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Jenkins Gerrit-CC: Anon. E. Moose #1000171 Gerrit-Attention: Savyasach Reddy Gerrit-Comment-Date: Wed, 13 Nov 2024 21:26:07 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
Attention is currently required from: Savyasach Reddy. Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 2: Analytics Compatibility Compilation Successful https://cbjenkins.page.link/1Vg68f3Q52BEavpv9 : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 2 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Jenkins Gerrit-CC: Anon. E. Moose #1000171 Gerrit-Attention: Savyasach Reddy Gerrit-Comment-Date: Wed, 13 Nov 2024 20:29:20 + Gerrit-HasComments: No Gerrit-Has-Labels: No Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Savyasach Reddy : Hello Jenkins, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 to look at the new patch set (#2). Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS - user model changes: add dynamic prefixes for HDFS datasets - storage format changes: no - interface changes: no details: - Support dynamic prefixes on HDFS - Support include, exclude and definition parameters in HDFS - Parameters 'path' or 'definition' can be used to specify prefixes Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 --- M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/default-namespace/default-namespace.01.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/hdfs/parquet/parquet.1.ddl.sqlpp M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/SqlppHdfsExecutionTest.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/hdfs/issue_245_hdfs/issue_245_hdfs.1.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/hdfs/hdfs_02/hdfs_02.1.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/hdfs/hdfs_03/hdfs_03.1.ddl.sqlpp M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestExecutor.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataPrefix.java M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/HDFSUtils.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/default-namespace/default-namespace.02.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/hdfs/hdfs_shortcircuit/hdfs_shortcircuit.1.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/default-namespace/default-namespace.05.ddl.sqlpp M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestConstants.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/hints/issue_251_dataset_hint_6/issue_251_dataset_hint_6.1.ddl.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/subquery/decorrelate_with_unique_id_2/decorrelate_with_unique_id_2.1.ddl.sqlpp M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java M asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp_hdfs.xml M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/default-namespace/default-namespace.03.update.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/default-namespace/default-namespace.04.update.sqlpp 20 files changed, 228 insertions(+), 66 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/66/19066/2 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 2 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Jenkins Gerrit-CC: Anon. E. Moose #1000171 Gerrit-MessageType: newpatchset
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Jenkins : Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 1: Integration-Tests+1 Integration Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/1752/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 1 Gerrit-Owner: Savyasach Reddy Gerrit-Reviewer: Jenkins Gerrit-CC: Anon. E. Moose #1000171 Gerrit-Comment-Date: Wed, 13 Nov 2024 19:29:19 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 ) Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS .. Patch Set 1: Analytics Compatibility Compilation Successful https://cbjenkins.page.link/fQ9ziKZsZo69p2YX9 : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 1 Gerrit-Owner: Savyasach Reddy Gerrit-CC: Anon. E. Moose #1000171 Gerrit-CC: Jenkins Gerrit-Comment-Date: Wed, 13 Nov 2024 19:00:36 + Gerrit-HasComments: No Gerrit-Has-Labels: No Gerrit-MessageType: comment
Change in asterixdb[master]: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
>From Savyasach Reddy :
Savyasach Reddy has uploaded this change for review. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 )
Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
..
[ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS
- user model changes: add dynamic prefixes for HDFS datasets
- storage format changes: no
- interface changes: no
details:
- Support dynamic prefixes on HDFS
- Support include/exclude parameters
- Parameter 'path' is used to specify prefixes
Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2
---
M
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/default-namespace/default-namespace.01.ddl.sqlpp
M
asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml
M
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/hdfs/parquet/parquet.1.ddl.sqlpp
M
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/SqlppHdfsExecutionTest.java
M
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
M
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/hdfs/issue_245_hdfs/issue_245_hdfs.1.ddl.sqlpp
M
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/hdfs/hdfs_02/hdfs_02.1.ddl.sqlpp
M
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/hdfs/hdfs_03/hdfs_03.1.ddl.sqlpp
M
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestExecutor.java
M
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataPrefix.java
M
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/HDFSUtils.java
M
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/default-namespace/default-namespace.02.update.sqlpp
M
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/hdfs/hdfs_shortcircuit/hdfs_shortcircuit.1.ddl.sqlpp
M
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/default-namespace/default-namespace.05.ddl.sqlpp
M
asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestConstants.java
M
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/hints/issue_251_dataset_hint_6/issue_251_dataset_hint_6.1.ddl.sqlpp
M
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/subquery/decorrelate_with_unique_id_2/decorrelate_with_unique_id_2.1.ddl.sqlpp
M
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp_hdfs.xml
M
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/default-namespace/default-namespace.03.update.sqlpp
M
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/default-namespace/default-namespace.04.update.sqlpp
21 files changed, 231 insertions(+), 62 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/66/19066/1
diff --git
a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestConstants.java
b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestConstants.java
index 0a96943..2be6a7d 100644
---
a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestConstants.java
+++
b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestConstants.java
@@ -34,6 +34,9 @@
public static final String S3_TEMPLATE_DEFAULT = "(\"accessKeyId\"=\"" +
S3_ACCESS_KEY_ID_DEFAULT + "\"),\n"
+ "(\"secretAccessKey\"=\"" + S3_SECRET_ACCESS_KEY_DEFAULT +
"\"),\n" + "(\"region\"=\"" + S3_REGION_DEFAULT
+ "\"),\n" + "(\"serviceEndpoint\"=\"" +
S3_SERVICE_ENDPOINT_DEFAULT + "\")";
+public static final String S3_DDL_TEMPLATE_DEFAULT = "\"accessKeyId\":\""
+ S3_ACCESS_KEY_ID_DEFAULT + "\",\n"
++ "\"secretAccessKey\":\"" + S3_SECRET_ACCESS_KEY_DEFAULT +
"\",\n" + "\"region\":\"" + S3_REGION_DEFAULT
++ "\",\n" + "\"serviceEndpoint\":\"" + S3_SERVICE_ENDPOINT_DEFAULT
+ "\"";
// Azure blob storage constants and placeholders
public static class Azure {
@@ -89,5 +92,9 @@
public static final String KERBEROS_PASSWORD_DEFAULT = "hdfspassword";
public static final String KERBEROS_REALM_DEFAULT = "EXAMPLE.COM";
public static final String KERBEROS_KDC_DEFAULT = "localhost:8800";
+public static final String HDFS_ENDPOINT_DEFAULT =
"hdfs://localhost:31888";
+
+public static final String HDFS_TEMPLATE_DEFAULT = "(\"hdfs\"=\"" +
HDFS_ENDPOINT_DEFAULT + "\")";
+public static final String HDFS_DDL_TEMPLATE_DEFAULT = "\"hdfs\":\"" +
HDFS_ENDPOINT_DEFAULT + "\"";
}
}
diff --git
a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestExecutor.java
b/asterixdb/asterix-app/src/test/java/org/apache/asterix/tes
