[
https://issues.apache.org/jira/browse/HIVE-24342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17824661#comment-17824661
]
Palakur Eshwitha Sai commented on HIVE-24342:
---------------------------------------------
Hi [~umamaheswararao],
[https://blog.cloudera.com/global-view-distributed-file-system-with-mount-points/]
I have been referring to the above article of yours and testing hive operations
with hdfs and s3 using viewfs overload scheme.
However, without encryption enabled, Hive insert into fails during hive staging
with below error:
{quote}Error: Error while processing statement: FAILED: Execution Error, return
code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to move source
hdfs://s3cluster/warehouse/tablespace/external/hive/test_tablehdfs/state=CA/.hive-staging_hive_2024-02-23_10-20-55_735_9094747688901732278-1/-ext-10002
to destination
hdfs://s3cluster/warehouse/tablespace/external/hive/test_tablehdfs/state=CA/.hive-staging_hive_2024-02-23_10-20-55_735_9094747688901732278-1/-ext-10000
(state=,code=0)
Caused by: java.io.FileNotFoundException: File does not exist:
/warehouse/tablespace/external/hive/test_tablehdfs/state=CA/.hive-staging_hive_2024-02-23_10-20-55_735_9094747688901732278-1/-ext-10000
at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1833)
{quote}
isPathEncrypted call in needToCopy (in Hive.java), leads to resolvePath call
which checks for file existence even for dest file hence the error thrown. In
HDFS, rename strategy is used to move files during hive staging.
> isPathEncrypted should make sure resolved path also from HDFS
> -------------------------------------------------------------
>
> Key: HIVE-24342
> URL: https://issues.apache.org/jira/browse/HIVE-24342
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Shims
> Affects Versions: 3.1.2
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Currently isPathEncrypted will make sure path is from hdfs by checking the
> path scheme is "hdfs"
> In the case if mounted ViewFileSystem based files systems like
> ViewFSOverloadScheme or ViewHDFS (HDFS-15289) may need o check resolved path
> is really hdfs.
> In ViewHDFS case, we can mount hdfs://ns1/test ---> o3fs://b.v.ozone1/test
> When user calling queries with the path hdfs://ns1/test, isPathEncrypted will
> think the path is from hdfs only as its checking path scheme.
>
> {code:java}
> 0: jdbc:hive2://umag-1.umag.root.xxx.site:218> select * from test30;
> Error: Error while compiling statement: FAILED: SemanticException Unable to
> determine if hdfs://ns1/test is encrypted:
> java.lang.UnsupportedOperationException: This API:getEZForPath is specific to
> DFS. Can't run on other fs:o3fs://bucket.volume.ozone1
> (state=42000,code=40000)
> 0: jdbc:hive2://umag-1.umag.root.xxx.site:218> cd Closing: 0:
> jdbc:hive2://umag-1.umag.root.xxx.site:2181,umag-2.umag.root.xxx.site:2181,umag-5.umag.root.xxx.site:2181/default;password=root;principal=hive/[email protected];retries=5;serviceDiscoveryMode=zooKeeper;user=root;zooKeeperNamespace=hiveserver2
> {code}
>
> So, here we should use resolvePath to make sure the resolved path really in
> hdfs. If the resolved path is not from hdfs (in above case, it o3fs path),
> then it will return false.
> After fixing this, the query is passing.:
>
> {code:java}
> 0: jdbc:hive2://umag-1.umag.root.xxx.site:218> select * from test30;
> INFO : Compiling
> command(queryId=hive_20201031002253_1691548f-6fa8-4ea9-9cd4-87b70fe8f6bb):
> select * from test30
> INFO : No Stats for default@test30, Columns: item, user_id, state, order_id
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:test30.order_id, type:bigint,
> comment:null), FieldSchema(name:test30.user_id, type:string, comment:null),
> FieldSchema(name:test30.item, type:string, comment:null),
> FieldSchema(name:test30.state, type:string, comment:null)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20201031002253_1691548f-6fa8-4ea9-9cd4-87b70fe8f6bb);
> Time taken: 4.47 seconds
> INFO : Executing
> command(queryId=hive_20201031002253_1691548f-6fa8-4ea9-9cd4-87b70fe8f6bb):
> select * from test30
> INFO : Completed executing
> command(queryId=hive_20201031002253_1691548f-6fa8-4ea9-9cd4-87b70fe8f6bb);
> Time taken: 0.09 seconds
> INFO : OK
> +------------------+-----------------+--------------+---------------+
> | test30.order_id | test30.user_id | test30.item | test30.state |
> +------------------+-----------------+--------------+---------------+
> | 1234 | u1 | iphone7 | CA |
> | 2345 | u1 | ipad | CA |
> | 3456 | u2 | desktop | NY |
>
>
> +------------------+-----------------+--------------+---------------+
> 11 rows selected (6.975 seconds)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
