[jira] [Created] (FLINK-17858) Yarn mode, windows and linux environment should be interlinked

[ChengWei Ye (Jira)]

ChengWei Ye created FLINK-17858:
---

 Summary: Yarn mode, windows and linux environment should be 
interlinked
 Key: FLINK-17858
 URL: https://issues.apache.org/jira/browse/FLINK-17858
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / YARN
Affects Versions: 1.10.0
Reporter: ChengWei Ye


my flink version: 1.10.0

my yarn version: 2.7.6

When I use windows idea to try to submit a job to Linux's yarn cluster using 
java, am container throws an exception "Error: Could not find or load main 
class org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint".

Then find out that the CLASSPATH and _FLINK_CLASSPATH delimiter of the 
container startup script "launch_container.sh" are different, windows is ";", 
linux is ":", so the container cannot get the dependent packages.

I think the operating environment should be left to Yarn to judge.

Locate in the flink-yarn module
1. The startAppMaster method of org.apache.flink.yarn.YarnClusterDescriptor
Change "File.pathSeparator" to "ApplicationConstants.CLASS_PATH_SEPARATOR"
Yarn will replace special characters according to the environment, but this 
only solves _FLINK_CLASSPATH

 
{code:java}
private ApplicationReport startAppMaster(.) {
.
StringBuilder classPathBuilder = new StringBuilder();
if (userJarInclusion == YarnConfigOptions.UserJarInclusion.FIRST) {
   for (String userClassPath : userClassPaths) {
  // here
  classPathBuilder.append(userClassPath).append(File.pathSeparator);
   }
}
for (String classPath : systemClassPaths) {
   // here
   classPathBuilder.append(classPath).append(File.pathSeparator);
}
// Setup jar for ApplicationMaster
.
// here 
classPathBuilder.append(flinkJarPath.getName()).append(File.pathSeparator);
.
}{code}
 

2. The addToEnvironment method of org.apache.flink.yarn.Utils
Change "File.pathSeparator" to "ApplicationConstants.CLASS_PATH_SEPARATOR"
Can solve the CLASSPATH
{code:java}
public static void addToEnvironment(Map environment,
  String variable, String value) {
   String val = environment.get(variable);
   if (val == null) {
  val = value;
   } else {
  // here
  val = val + File.pathSeparator + value;
   }
   environment.put(StringInterner.weakIntern(variable),
 StringInterner.weakIntern(val));
}
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-13710) JarListHandler always extract the jar package

[ChengWei Ye (JIRA)]

ChengWei Ye created FLINK-13710:
---

 Summary: JarListHandler always extract the jar package
 Key: FLINK-13710
 URL: https://issues.apache.org/jira/browse/FLINK-13710
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.8.1
Reporter: ChengWei Ye


 
{code:java}
// JarListHandler class
// handleRequest method

for (String clazz : classes) {
   clazz = clazz.trim();

   PackagedProgram program = null;
   try {
  // here
  program = new PackagedProgram(f, clazz, new String[0]);
   } catch (Exception ignored) {
  // ignore jar files which throw an error upon creating a PackagedProgram
   }
   if (program != null) {
  JarListInfo.JarEntryInfo jarEntryInfo = new 
JarListInfo.JarEntryInfo(clazz, program.getDescription());
  jarEntryList.add(jarEntryInfo);
   }
}
{code}
When I open the submit page of the jm web 
([http://localhost:7081/#/submit|http://localhost:8081/#/submit]), the 
background always decompresses the lib directory in the job jar package until 
the temp directory is full.

If the jobmanager just gets the jar information, the submit page should not 
extract the jar package.

And I think the same jar only needs to be decompressed once, and should not be 
decompressed every time it is submitted.

 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (FLINK-13710) JarListHandler always extract the jar package

[ChengWei Ye (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-13710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ChengWei Ye updated FLINK-13710:

Affects Version/s: (was: 1.8.1)

> JarListHandler always extract the jar package
> -
>
> Key: FLINK-13710
> URL: https://issues.apache.org/jira/browse/FLINK-13710
> Project: Flink
>  Issue Type: Bug
>Reporter: ChengWei Ye
>Priority: Minor
>
>  
> {code:java}
> // JarListHandler class
> // handleRequest method
> for (String clazz : classes) {
>clazz = clazz.trim();
>PackagedProgram program = null;
>try {
>   // here
>   program = new PackagedProgram(f, clazz, new String[0]);
>} catch (Exception ignored) {
>   // ignore jar files which throw an error upon creating a PackagedProgram
>}
>if (program != null) {
>   JarListInfo.JarEntryInfo jarEntryInfo = new 
> JarListInfo.JarEntryInfo(clazz, program.getDescription());
>   jarEntryList.add(jarEntryInfo);
>}
> }
> {code}
> When I open the submit page of the jm web 
> ([http://localhost:7081/#/submit|http://localhost:8081/#/submit]), the 
> background always decompresses the lib directory in the job jar package until 
> the temp directory is full.
> If the jobmanager just gets the jar information, the submit page should not 
> extract the jar package.
> And I think the same jar only needs to be decompressed once, and should not 
> be decompressed every time it is submitted.
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)