>From Hussain Towaileb <[email protected]>: Attention is currently required from: Savyasach Reddy. Hussain Towaileb has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 )
Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS ...................................................................... Patch Set 3: (13 comments) File asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestConstants.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/6b48ecd7_9ce43e6f PS3, Line 37: S3_DDL_TEMPLATE_DEFAULT S3_TEMPLATE_DEFAULT_NO_PARANTHESES_WITH_COLONS File asterixdb/asterix-app/src/test/java/org/apache/asterix/test/common/TestExecutor.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/b7e3e388_ed0c8915 PS3, Line 2567: else Same as above File asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/copy-to/default-namespace/default-namespace.05.ddl.sqlpp: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/27743719_eb982b72 PS3, Line 22: ("container"="playground"), Same as below. https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/71622ddb_b2ce25d8 PS3, Line 24: ("input-format" = "text-input-format"), ("input-format" = "text-input-format"), change to %additionalProperties% File asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/5f0212c2_53451cef PS3, Line 152: LOGGER.warn("The provided external dataset configuration returned no files from the external source"); Use warning collector to warn, see examples in S3Utils. https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/6c17ab73_564d0b02 PS3, Line 153: hdfsConf Let's have a method to do this in the HDFS utils, and here instead set the KEY_PATH to empty string. https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/6455506f_16769350 PS3, Line 153: (ExternalDataConstants.KEY_HADOOP_INPUT_DIR, ""); Let's see what happens if we pass "" vs "/". Will it return everything or empty, .. etc https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/6be45446_b604b126 PS3, Line 158: hdfsConf.set(ExternalDataConstants.KEY_HADOOP_INPUT_DIR, ""); Throw external source error exception. File asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataPrefix.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/3a68875f_516db8ef PS3, Line 80: Objects.equals(configuration.get(ExternalDataConstants.KEY_EXTERNAL_SOURCE_TYPE), : ExternalDataConstants.KEY_HDFS_URL) Let's have a function for this in HDFSUtils. boolean isHdfs = ...; https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/4226b370_0962bf19 PS3, Line 81: KEY_HDFS_URL Compare it with KEY_ADAPTER_NAME_HDFS, use .equalsIgnoreCase File asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/HDFSUtils.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/ff0ff9fc_430197bb PS3, Line 236: Add this comment: In case KEY_PATH is not being used, it means definition is being used, and the property KEY_HADOOP_INPUT_DIR will be set in HDFSDatasourceFactory based on the provided definition. https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/a0643f5d_3bfae0c8 PS3, Line 404: updateRoot updateRootPath https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/2fd7c968_255c18bd PS3, Line 577: try { Add a comment here that this is for validation purpose for external data prefix stuff. -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 3 Gerrit-Owner: Savyasach Reddy <[email protected]> Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins <[email protected]> Gerrit-CC: Hussain Towaileb <[email protected]> Gerrit-Attention: Savyasach Reddy <[email protected]> Gerrit-Comment-Date: Fri, 15 Nov 2024 15:42:00 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: No Gerrit-MessageType: comment
