[
https://issues.apache.org/jira/browse/FLINK-18362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17140297#comment-17140297
]
gramo lee edited comment on FLINK-18362 at 6/19/20, 7:29 AM:
-------------------------------------------------------------
Hi, [~fly_in_gis]
I've done this and verified it works fine as well. All files shipped in Flink
have been considered as "LocalResourceType.FILE". So
I added "yarn.ship-archive-directories" as a new option and overloaded
'Utils.registerLocalResource()' so that it can get 'LocalResourceType'. In case
of files in "ship-archive-directories", those are considered
"LocalResourecType.ARCHIVE". May I attach my work at this issue? I need a
little time because of checking policy of my company regarding contributing
opensource, though.
was (Author: gramo):
Hi, [~fly_in_gis]
I've done this and verified it works fine as well. All files shipped in Flink
have been considered as "LocalResourceType.FILE". So
I added "yarn.ship-archive-directories" as a new option and overloaded
'Utils.registerLocalResource()' so that it can get 'LocalResourceType'. In case
of files in "ship-archive-directories", those are considered
"LocalResourecType.ARCHIVE". May I attach my work at this issue? I need a
little time because of checking policy of my company regarding to contribute
opensource, though.
> Shipping jdk by shipping archive
> --------------------------------
>
> Key: FLINK-18362
> URL: https://issues.apache.org/jira/browse/FLINK-18362
> Project: Flink
> Issue Type: Wish
> Affects Versions: 1.10.1
> Reporter: Noah
> Priority: Minor
>
> Hello,
> Our team are running flink cluster on YARN, and it works so well 👍
> h4. Functional requirements
> Is there any option to ship archive to YARN applications?
> h4. Backgrounds
> Recently, one of job has been shut down with jdk8 version related issues.
> https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6675699
> It's easy problem if we could set latest jdk on
> `containerized.taskmanager.env.JAVA_HOME`.
> However, cluster administrator said it's difficult to install the latest jdk
> on all cluster machines.
>
> So, we planned to run a job on latest jdk that is shipped via shared
> resources.
> There's an option `yarn.ship-directories` but it's quite slow because jdk has
> large number of files.
> If Flink supports to ship archive such as `yarn.ship-archive`, we can ship
> jdk archive to remote machines and use shipped jdk location as JAVA_HOME
> (using `yarn.container-start-command-template` )
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
