[jira] [Created] (FLINK-27130) unable to read custom System properties in job class
Adrian Zhong created FLINK-27130:
Summary: unable to read custom System properties in job class
Key: FLINK-27130
URL: https://issues.apache.org/jira/browse/FLINK-27130
Project: Flink
Issue Type: Bug
Components: Client / Job Submission
Affects Versions: 1.13.6, 1.13.0
Reporter: Adrian Zhong
I'm using Flink YARN-PER-JOB mode to submit a job.
I'm wondering what is wrong, and how flink CliFrontend prevent job class to
read system properties which seems to be per-jvm.
I have searched all related issues, read unit tests for CliFrontend and
DynamicProperties, however, I can't figure out.
Here is my job class:
```
public static void main(String[] args) {
String property = System.getProperty("kafka.start_from_timestamp");
if (property == null) {
//-Dkafka.start_from_timestamp=164882880
System.err.println("-Dkafka.start_from_timestamp Not found");
System.err.println("This are Properties Found in this JVM:");
System.err.println(System.getProperties().stringPropertyNames());
} else {
System.err.println("-Dkafka.start_from_timestamp is" + property);
}
//
}
```
outputs:
```
-Dkafka.start_from_timestamp Not found
This are Properties Found in this JVM:
[zookeeper.sasl.client, java.runtime.name, sun.boot.library.path,
java.vm.version, java.vm.vendor, java.vendor.url, path.separator, java.vm.name,
file.encoding.pkg, user.country, sun.java.launcher, sun.os.patch.level,
java.vm.specification.name, user.dir, java.runtime.version,
java.awt.graphicsenv, java.endorsed.dirs, os.arch, java.io.tmpdir,
line.separator, java.vm.specification.vendor, os.name, log4j.configuration,
sun.jnu.encoding, java.library.path, java.specification.name,
java.class.version, sun.management.compiler, os.version, user.home,
user.timezone, java.awt.printerjob, file.encoding, java.specification.version,
log4j.configurationFile, user.name, java.class.path, log.file,
java.vm.specification.version, sun.arch.data.model, java.home,
sun.java.command, java.specification.vendor, user.language, awt.toolkit,
java.vm.info, java.version, java.ext.dirs, sun.boot.class.path, java.vendor,
logback.configurationFile, java.security.auth.login.config, file.separator,
java.vendor.url.bug, sun.cpu.endian, sun.io.unicode.encoding, sun.cpu.isalist]
```
Environment:
JDK: Oracle 1.8/25.121-b13
Flink flink-1.13.0
What I have tried:
```
-Denv.java.opts.client="-Dkafka.start_from_timestamp=164882880"
-Denv.java.opts="-Dkafka.start_from_timestamp=164882881"
-Dkafka.start_from_timestamp=164882882
-yD env.java.opts="kafka.start_from_timestamp=164882883"
```
submit command:
```
bin/flink run --yarnjobManagerMemory 1G --yarntaskManagerMemory 1G --yarnqueue
root.users.appuser --yarnslots 1 --yarnname SocketWindowWordCount -m
yarn-cluster --class com.slankka.learn.rtc.SocketWindowWordCount
-Denv.java.opts="-Dkafka.start_from_timestamp=164882881"
-Dkafka.start_from_timestamp=164882882 -yD
env.java.opts="kafka.start_from_timestamp=164882883" -d
/data/files_upload/socketWindowWordCount.jar --hostname 10.11.159.156 --port
7890
```
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-22769) yarnship do not support symbolic directory
Adrian Zhong created FLINK-22769:
Summary: yarnship do not support symbolic directory
Key: FLINK-22769
URL: https://issues.apache.org/jira/browse/FLINK-22769
Project: Flink
Issue Type: Improvement
Components: Deployment / YARN
Affects Versions: 1.13.0, 1.12.2, 1.13.1
Reporter: Adrian Zhong
If we pass `-yt ` a symbolic directory, we will get an exception:
{code:java}
Caused by: java.lang.IllegalArgumentException: File to copy cannot be a
directory: file:/path/to/sql/connectors/Caused by:
java.lang.IllegalArgumentException: File to copy cannot be a directory:
file:/path/to/sql/connectors/ at
org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:138) at
org.apache.flink.yarn.YarnApplicationFileUploader.uploadLocalFileToRemote(YarnApplicationFileUploader.java:197)
at
org.apache.flink.yarn.YarnApplicationFileUploader.registerSingleLocalResource(YarnApplicationFileUploader.java:179)
at
org.apache.flink.yarn.YarnApplicationFileUploader.registerMultipleLocalResources(YarnApplicationFileUploader.java:289)
at
org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:870)
{code}
The root cause is not here but in :
{code:java}
final java.nio.file.Path shipPath = file.toPath();
final java.nio.file.Path parentPath = shipPath.getParent();
Files.walkFileTree(
shipPath.toRealPath(),
new SimpleFileVisitor() {
@Override
public FileVisitResult visitFile(
java.nio.file.Path file, BasicFileAttributes attrs) {
localPaths.add(new Path(file.toUri())); //root cause: localPath
is an symbolic file, without any file under real path.
relativePaths.add(
new Path(
localResourcesDirectory,
parentPath.relativize(file).toString()));
return FileVisitResult.CONTINUE;
}
});
{code}
Please assign to me, I'd like to be a contributor.
Any other please *do not implement* or create pr before this issue is assigned.
Note: this bug is not os platform specific.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22639) ClassLoaderUtil cannot print classpath of FlinkUserCodeClassLoader
Adrian Zhong created FLINK-22639:
Summary: ClassLoaderUtil cannot print classpath of
FlinkUserCodeClassLoader
Key: FLINK-22639
URL: https://issues.apache.org/jira/browse/FLINK-22639
Project: Flink
Issue Type: Improvement
Components: Runtime / Configuration, Table SQL / Runtime
Affects Versions: 1.13.0
Reporter: Adrian Zhong
Hello, community.
I found FlinkUserCodeClassLoader is wrapping by
SafetyNetWrapperClassLoader, but it cut getURL invoking chain.
ClassLoaderUtil.getUserCodeClassLoaderInfo:
{code:java}
public static String getUserCodeClassLoaderInfo(ClassLoader loader) {
if (loader instanceof URLClassLoader) {
URLClassLoader cl = (URLClassLoader) loader;
try {
StringBuilder bld = new StringBuilder();
if (cl == ClassLoader.getSystemClassLoader()) {
bld.append("System ClassLoader: ");
} else {
bld.append("URL ClassLoader:");
}
for (URL url : cl.getURLs()) {
}
}{code}
{code:java}
SafetyNetWrapperClassLoader(FlinkUserCodeClassLoader inner, ClassLoader parent)
{
super(new URL[0], parent);
this.inner = inner;
}
{code}
The url passing empty array, SafetyNetWrapperClassLoader.getURL should dispatch
this invocation.
{code:java}
@Override
public URL[] getURLs() {
return inner.getURLs();
}
{code}
Otherwise,
{code:java}
ClassLoaderUtil.getUserCodeClassLoaderInfo(theJarClassLoader);
{code}
will print empty, like below:
{code:java}
URL ClassLoader:
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
