Xiang Yang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )
Change subject: IMPALA-11629: Support for huawei OBS FileSystem ...................................................................... Patch Set 9: > Uploaded patch set 9. Hi all, Patch#9 add OBS related configurations to core-site.xml to reduce manual work, and export 'WAREHOUSE_LOCATION_PREFIX' env variable. I figured out that when the 'fs.defaultFS' in core-site.xml set to 'obs://...', the table's location will defaulted to 'obs://...' but 'hdfs://localhost:20500', then some failed cases in Patch#8 will be passed, thus the '$NAMENODE' variable I mentioned doesn't need to change, see https://github.com/apache/impala/blob/4.2.0/tests/util/hdfs_util.py#L55 . Another challenge I face is some EE tests rely on hbase, but hbase table store data on hdfs currently, so I need to start the hdfs cluster. But namenode will fail to start when 'fs.defaultFS' set to 'obs://...'. So I need to initialize 'fs.defaultFS' to 'hdfs://localhost:20500' and start hdfs cluster, then change it to 'obs://...' to start hive and impala cluster. It'll then cause some EE tests failed because of unique_database or table location defaults to 'hdfs://localhost:20500/...' for unknown reason. So I export 'WAREHOUSE_LOCATION_PREFIX' env variable so the 'hive.metastore.warehouse.dir' and 'hive.metastore.warehouse.external.dir' in hive-site.xml fixed with explicit filesystem schema, or I have to research how to modify hbase table's location to point to obs filesystem. -- To view, visit http://gerrit.cloudera.org:8080/19110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1 Gerrit-Change-Number: 19110 Gerrit-PatchSet: 9 Gerrit-Owner: Xiang Yang <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Xiang Yang <[email protected]> Gerrit-Reviewer: Yida Wu <[email protected]> Gerrit-Comment-Date: Sun, 29 Jan 2023 09:56:09 +0000 Gerrit-HasComments: No
