>From Hussain Towaileb <[email protected]>: Attention is currently required from: Savyasach Reddy. Hussain Towaileb has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 )
Change subject: [ASTERIXDB-3519][EXT]: Support dynamic prefixes on HDFS ...................................................................... Patch Set 4: (4 comments) File asterixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_s3.xml: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/c1661fd8_327ae2c2 PS4, Line 30: <placeholder name="pathprefix" value="" /> : <placeholder name="path_prefix" value="" /> You probably could reduce these by implementing that a missing value of a placeholder means it passes nothing. File asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/93dd5656_d5ae419c PS3, Line 158: hdfsConf.set(ExternalDataConstants.KEY_HADOOP_INPUT_DIR, ""); > I've updated with the same. […] Ideally, we should do the same, but wouldn't the continuation of the execution fail? After all, HDFS itself is throwing an exception. AWS doesn't throw an exception by itself, just returns empty, so we translate that into a warning. File asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/e76a7614_1fcd51fc PS4, Line 162: "No files found at the given path" Why not use ErrorCode.EXTERNAL_SOURCE_CONFIGURATION_RETURNED_NO_FILES? Although the current one is fine as well. File asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/HDFSUtils.java: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066/comment/5f7aae17_1efd6335 PS4, Line 590: Objects.equals(ExternalDataConstants.KEY_ADAPTER_NAME_HDFS, : configuration.get(ExternalDataConstants.KEY_EXTERNAL_SOURCE_TYPE)); Instead do: return ExternalDataConstants.KEY_ADAPTER_NAME_HDFS.equalsIgnoreCase(configuration.get(ExternalDataConstants.KEY_EXTERNAL_SOURCE_TYPE)); And you can revert the HDFS to hdfs in the test cases, it becomes case insensitive. -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/19066 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I1bdbcd44c059f64f2da436a40ac3f59293442cf2 Gerrit-Change-Number: 19066 Gerrit-PatchSet: 4 Gerrit-Owner: Savyasach Reddy <[email protected]> Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins <[email protected]> Gerrit-CC: Hussain Towaileb <[email protected]> Gerrit-Attention: Savyasach Reddy <[email protected]> Gerrit-Comment-Date: Mon, 18 Nov 2024 16:49:48 +0000 Gerrit-HasComments: Yes Gerrit-Has-Labels: No Comment-In-Reply-To: Hussain Towaileb <[email protected]> Comment-In-Reply-To: Savyasach Reddy <[email protected]> Gerrit-MessageType: comment
