[
https://issues.apache.org/jira/browse/FLINK-23169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17377350#comment-17377350
]
jinfeng edited comment on FLINK-23169 at 7/8/21, 12:22 PM:
-----------------------------------------------------------
[~fly_in_gis] thanks for your reply.
When we use a Flink job management platform to submit jobs from different
users, we config the 'yarn.staging-directory' ,for example. /flink/staging/
1. if user1 submit the job, and user1 will create the
/flink/staging/.flink/application_xxxx user1 supergroup drwx--x--x
2. if user2 submit the job, and user2 can't use /flink/staging/.flink , becase
directory `.flink` belongs to user1.
3. if we create /flink/staging/.flink first, there is also another problem,
user1, user2, user3.. 's application staging directory will be created in the
same directory .
The reason why we don't use the default home directory is that we want to
provided a common directory to different users. If not, every new user need to
create their own home directory and apply quota.
I notice that the spark also provided a config to specific the yarn staging
directory, and it will be divided by user directory by default.
was (Author: hackergin):
[~fly_in_gis] thank for you reply.
When we use a Flink job management platform to submit jobs from different
users, we config the 'yarn.staging-directory' ,for example. /flink/staging/
1. if user1 submit the job, and user1 will create the
/flink/staging/.flink/application_xxxx user1 supergroup drwx--x--x
2. if user2 submit the job, and user2 can't use /flink/staging/.flink , becase
directory `.flink` belongs to user1.
3. if we create /flink/staging/.flink first, there is also another problem,
user1, user2, user3.. 's application staging directory will be created in the
same directory .
The reason why we don't use the default home directory is that we want to
provided a common directory to different users. If not, every new user need to
create their own home directory and apply quota.
I notice that the spark also provided a config to specific the yarn staging
directory, and it will be divided by user directory by default.
> Support user-level app staging directory when yarn.staging-directory is
> specified
> ---------------------------------------------------------------------------------
>
> Key: FLINK-23169
> URL: https://issues.apache.org/jira/browse/FLINK-23169
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
> Reporter: jinfeng
> Priority: Major
>
> When yarn.staging-directory is specified, different users will use the same
> directory as the staging directory. It may not friendly for a job platform
> to submit job for different users. I propose to use the user-level directory
> by default when yarn.staging-directory is specified. We only need to make
> small changes for `getStagingDir` function in
> YarnClusterDescriptor
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
