[
https://issues.apache.org/jira/browse/HIVE-28473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liang yu updated HIVE-28473:
----------------------------
Description:
using HIVE 3.1.3 ; mr engine; HADOOP 3.3.4
When I try to insert data into the local directory "/path/to/local", Hive
usually first creates an intermediate HDFS directory like
"hdfs:/session/execution/.staging-hive-xx", which is based on sessionId and
executionId. After that, it moves the results to the local filesystem at
"/path/to/local".
However, it’s currently trying to create an intermediate HDFS directory at
"hdfs:/path/to/local/.staging-hive-xx", which incorrectly uses the local
filesystem path. This causes an error because it's attempting to create a new
path starting from {{{}/root{}}}, where we don't have sufficient permissions.
It can be reproduced by:
{code:java}
INSERT OVERWRITE LOCAL DIRECTORY "/path/to/local/dir"
select a
from table
group by a; {code}
when I run this sql in hive, it throws exception:
{code:java}
RuntimeException: cannot create staging directory
"hdfs:/path/to/local/dir/.hive-staging-xx":
Permission denied: user=aaa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x {code}
It tried to write the staging file to a hdfs path which is generated from LOCAL
DIRECTORY.
but the staging file should be written to a temp hdfs directory which is
created from hiveSessionPath and executionId
was:
using HIVE 3.1.3 ; mr engine; HADOOP 3.3.4
When I try to insert into local directory "/path/to/local", hive should first
creates a intermediate hdfs directory
"hdfs:/session/execution/.staging-hive-xx" whose path is generated from
sessionId and executionId, and then put the result to local filesystem
"/path/to/local".
But now it tries to create an intermediate hdfs directory
"hdfs:/path/to/local/.staging-hive-xx" whose path is generated from local
filesystem, this will cause an exception because we try to create a new path
from /root and we don't have enough permission.
It can be reproduced by:
{code:java}
INSERT OVERWRITE LOCAL DIRECTORY "/path/to/local/dir"
select a
from table
group by a; {code}
when I run this sql in hive, it throws exception:
{code:java}
RuntimeException: cannot create staging directory
"hdfs:/path/to/local/dir/.hive-staging-xx":
Permission denied: user=aaa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x {code}
It tried to write the staging file to a hdfs path which is generated from LOCAL
DIRECTORY.
but the staging file should be written to a temp hdfs directory which is
created from hiveSessionPath and executionId
> INSERT OVERWRITE LOCAL DIRECTORY writes staging files to wrong hdfs directory
> -----------------------------------------------------------------------------
>
> Key: HIVE-28473
> URL: https://issues.apache.org/jira/browse/HIVE-28473
> Project: Hive
> Issue Type: Bug
> Affects Versions: 3.1.3
> Environment: Hadoop 3.3.4
> HIVE 3.1.3
> mapreduce engine
> Reporter: liang yu
> Priority: Major
>
> using HIVE 3.1.3 ; mr engine; HADOOP 3.3.4
>
> When I try to insert data into the local directory "/path/to/local", Hive
> usually first creates an intermediate HDFS directory like
> "hdfs:/session/execution/.staging-hive-xx", which is based on sessionId and
> executionId. After that, it moves the results to the local filesystem at
> "/path/to/local".
> However, it’s currently trying to create an intermediate HDFS directory at
> "hdfs:/path/to/local/.staging-hive-xx", which incorrectly uses the local
> filesystem path. This causes an error because it's attempting to create a new
> path starting from {{{}/root{}}}, where we don't have sufficient permissions.
>
> It can be reproduced by:
> {code:java}
> INSERT OVERWRITE LOCAL DIRECTORY "/path/to/local/dir"
> select a
> from table
> group by a; {code}
>
> when I run this sql in hive, it throws exception:
> {code:java}
> RuntimeException: cannot create staging directory
> "hdfs:/path/to/local/dir/.hive-staging-xx":
> Permission denied: user=aaa, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x
> {code}
> It tried to write the staging file to a hdfs path which is generated from
> LOCAL DIRECTORY.
> but the staging file should be written to a temp hdfs directory which is
> created from hiveSessionPath and executionId
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
