shamim created SPARK-43109:
------------------------------
Summary: JavaRDD.saveAsTextFile Directory Creation issue using
Spark 3.3.2, with hadoop3
Key: SPARK-43109
URL: https://issues.apache.org/jira/browse/SPARK-43109
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.3.2
Reporter: shamim
We are using Spark 3.3.2, with hadoop3 in Java Appliaction .
While Saving file using JavaRDD.saveAsTextFile(DIR_NAME/Sample.txt).
Spark is creating DIR_NAME with the same user by which Application is Running
ex vs.
However we have provide another user to create the DIR_NAME like below in our
Spark code .
UserGroupInformation theSparkUser=
UserGroupInformation.createRemoteUser("spark");
UserGroupInformation.setLoginUser(theSparkUser);
Below is the dir structure created by the Application.
drwxrwxrwx+ 2 vs vs 6 Apr 12 17:09 main
We want Dir should be created by the user that we have provided, Like below
drwxrwxrwx+ 2 spark spark 6 Apr 12 17:09 main
When we are verifying the user by printing its showing same user as we pass in
UserGroupInformation both below printing same user "spark"
UserGroupInformation.getLoginUser().getUserName()); --> printing user as
"spark"
UserGroupInformation.getCurrentUser().getUserName()); --> printing user as
"spark"
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
