Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/19110 )
Change subject: IMPALA-11629: Support for huawei OBS FileSystem ...................................................................... Patch Set 9: (3 comments) > Patch Set 9: > > > Uploaded patch set 9. > > Hi all, Patch#9 add OBS related configurations to core-site.xml to reduce > manual work, and export 'WAREHOUSE_LOCATION_PREFIX' env variable. > > I figured out that when the 'fs.defaultFS' in core-site.xml set to > 'obs://...', the table's location will defaulted to 'obs://...' but > 'hdfs://localhost:20500', then some failed cases in Patch#8 will be passed, > thus the '$NAMENODE' variable I mentioned doesn't need to change, see > https://github.com/apache/impala/blob/4.2.0/tests/util/hdfs_util.py#L55 . > > Another challenge I face is some EE tests rely on hbase, but hbase table > store data on hdfs currently, so I need to start the hdfs cluster. But > namenode will fail to start when 'fs.defaultFS' set to 'obs://...'. So I need > to initialize 'fs.defaultFS' to 'hdfs://localhost:20500' and start hdfs > cluster, then change it to 'obs://...' to start hive and impala cluster. > It'll then cause some EE tests failed because of unique_database or table > location defaults to 'hdfs://localhost:20500/...' for unknown reason. So I > export 'WAREHOUSE_LOCATION_PREFIX' env variable so the > 'hive.metastore.warehouse.dir' and 'hive.metastore.warehouse.external.dir' in > hive-site.xml fixed with explicit filesystem schema, or I have to research > how to modify hbase table's location to point to obs filesystem. Tests that use hbase should be skipped via https://github.com/apache/impala/blob/master/tests/common/skip.py#L78, and hbase shouldn't be started - https://github.com/apache/impala/blob/master/testdata/bin/run-all.sh#L62 - unless DEFAULT_FS == 'hdfs://localhost:20500'. http://gerrit.cloudera.org:8080/#/c/19110/9/bin/impala-config.sh File bin/impala-config.sh: http://gerrit.cloudera.org:8080/#/c/19110/9/bin/impala-config.sh@710 PS9, Line 710: WAREHOUSE_LOCATION_PREFIX=${DEFAULT_FS} This shouldn't need to be set. It's only used if your warehouse is not "obs://${OBS_BUCKET}/test-warehouse". http://gerrit.cloudera.org:8080/#/c/19110/9/tests/custom_cluster/test_metastore_service.py File tests/custom_cluster/test_metastore_service.py: http://gerrit.cloudera.org:8080/#/c/19110/9/tests/custom_cluster/test_metastore_service.py@1214 PS9, Line 1214: if IS_HDFS: This should be IS_HDFS or IS_OZONE now. Probably should have updated the comment. http://gerrit.cloudera.org:8080/#/c/19110/9/tests/custom_cluster/test_metastore_service.py@1242 PS9, Line 1242: if IS_HDFS: Same as above. -- To view, visit http://gerrit.cloudera.org:8080/19110 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I84a54dbebcc5b71e9bcdd141dae9e95104d98cb1 Gerrit-Change-Number: 19110 Gerrit-PatchSet: 9 Gerrit-Owner: Xiang Yang <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Xiang Yang <[email protected]> Gerrit-Reviewer: Yida Wu <[email protected]> Gerrit-Comment-Date: Mon, 30 Jan 2023 17:47:51 +0000 Gerrit-HasComments: Yes
