[jira] [Comment Edited] (FLINK-17745) PackagedProgram' extractedTempLibraries and jarfiles may be duplicate
[ https://issues.apache.org/jira/browse/FLINK-17745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111903#comment-17111903 ] Kostas Kloudas edited comment on FLINK-17745 at 5/20/20, 8:44 AM: -- [~Echo Lee] Given the discussion here, I opened a discussion in the ML with title "[DISCUSS] Remove dependency shipping through nested jars during job submission". Let's move the discussion there so that the whole community can participate. was (Author: kkl0u): [~Echo Lee] Given the discussion here, I opened a discussion in the ML with title "[DISCUSS] Remove dependency shipping through nested jars during job submission. ". Let's move the discussion there so that the whole community can participate. > PackagedProgram' extractedTempLibraries and jarfiles may be duplicate > - > > Key: FLINK-17745 > URL: https://issues.apache.org/jira/browse/FLINK-17745 > Project: Flink > Issue Type: Improvement > Components: Client / Job Submission >Reporter: Echo Lee >Assignee: Kostas Kloudas >Priority: Major > Labels: pull-request-available > > When i submit a flink app with a fat jar, PackagedProgram will extracted temp > libraries by the fat jar, and add to pipeline.jars, and the pipeline.jars > contains fat jar and temp libraries. I don't think we should add fat jar to > the pipeline.jars if extractedTempLibraries is not empty. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-17745) PackagedProgram' extractedTempLibraries and jarfiles may be duplicate
[ https://issues.apache.org/jira/browse/FLINK-17745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111903#comment-17111903 ] Kostas Kloudas edited comment on FLINK-17745 at 5/20/20, 8:44 AM: -- [~Echo Lee] Given the discussion here, I opened a discussion in the ML with title "[DISCUSS] Remove dependency shipping through nested jars during job submission. ". Let's move the discussion there so that the whole community can participate. was (Author: kkl0u): [~Echo Lee] Is it ok if we open a discussion in the ML to see also what other members of the community have to say about it? As you also mentioned, this way of submission has some problems related to redundant transfers and it is (so far) a hidden way of submitting jobs. > PackagedProgram' extractedTempLibraries and jarfiles may be duplicate > - > > Key: FLINK-17745 > URL: https://issues.apache.org/jira/browse/FLINK-17745 > Project: Flink > Issue Type: Improvement > Components: Client / Job Submission >Reporter: Echo Lee >Assignee: Kostas Kloudas >Priority: Major > Labels: pull-request-available > > When i submit a flink app with a fat jar, PackagedProgram will extracted temp > libraries by the fat jar, and add to pipeline.jars, and the pipeline.jars > contains fat jar and temp libraries. I don't think we should add fat jar to > the pipeline.jars if extractedTempLibraries is not empty. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-17745) PackagedProgram' extractedTempLibraries and jarfiles may be duplicate
[
https://issues.apache.org/jira/browse/FLINK-17745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111007#comment-17111007
]
Kostas Kloudas edited comment on FLINK-17745 at 5/19/20, 9:27 AM:
--
Hi all, from the removed code in the PR, we were traversing the user jar and we
were adding the found nested jars in classpath *only* if they were packaged in
a {{lib/}} folder. We were expecting a *specific structure* of the jar. See
the {{getContainedJarEntries()}} in the removed code.
Given that this feature was not documented, I am not sure if anyone is actually
using it. Do you have any such cases? If yes, then I agree that the code should
not be removed and I will close the PR.
Of course, if we allow such type of submission, we must include the nested jars
in the classpath as they are not included by default. We transfer them twice
though, but this another story.
was (Author: kkl0u):
Hi all, from the removed code in the PR, we were traversing the user jar and we
were adding the found nested jars in classpath *only* if they were packaged in
a {{lib/}} folder. We were expecting a *specific structure* of the jar. See
the {{getContainedJarEntries()}} in the removed code.
Given that this feature was not documented, I am not sure if anyone is actually
using it. Do you have any such cases? If yes, then I agree that the code should
not be removed and I will close the PR.
> PackagedProgram' extractedTempLibraries and jarfiles may be duplicate
> -
>
> Key: FLINK-17745
> URL: https://issues.apache.org/jira/browse/FLINK-17745
> Project: Flink
> Issue Type: Improvement
> Components: Client / Job Submission
>Affects Versions: 1.11.0
>Reporter: lisen
>Assignee: Kostas Kloudas
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> When i submit a flink app with a fat jar, PackagedProgram will extracted temp
> libraries by the fat jar, and add to pipeline.jars, and the pipeline.jars
> contains fat jar and temp libraries. I don't think we should add fat jar to
> the pipeline.jars if extractedTempLibraries is not empty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-17745) PackagedProgram' extractedTempLibraries and jarfiles may be duplicate
[ https://issues.apache.org/jira/browse/FLINK-17745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110805#comment-17110805 ] Echo Lee edited comment on FLINK-17745 at 5/19/20, 3:08 AM: [~kkl0u] I don't think nested jar should be removed, because in standalone mode, we need it. It can extracted dependent jar and upload to the blob server and each task can be access it. If the nested jar is removed, I'm not sure whether it works. was (Author: leeecho): [~kkl0u] I don't think nested jar should be removed, because in standalone mode, we need it. It can extracted dependent jar and upload to the blob file and each task can be access it. If the nested jar is removed, I'm not sure whether it works. > PackagedProgram' extractedTempLibraries and jarfiles may be duplicate > - > > Key: FLINK-17745 > URL: https://issues.apache.org/jira/browse/FLINK-17745 > Project: Flink > Issue Type: Improvement > Components: Client / Job Submission >Affects Versions: 1.11.0 >Reporter: lisen >Assignee: Kostas Kloudas >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > > When i submit a flink app with a fat jar, PackagedProgram will extracted temp > libraries by the fat jar, and add to pipeline.jars, and the pipeline.jars > contains fat jar and temp libraries. I don't think we should add fat jar to > the pipeline.jars if extractedTempLibraries is not empty. -- This message was sent by Atlassian Jira (v8.3.4#803005)
