liuxinjie5211990 opened a new issue, #3524:
URL: https://github.com/apache/incubator-streampark/issues/3524

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-streampark/issues?q=is%3Aissue+label%3A%22bug%22)
 and found no similar issues.
   
   
   ### Java Version
   
   java 8
   
   ### Scala Version
   
   2.11.x
   
   ### StreamPark Version
   
   2.1.1
   
   ### Flink Version
   
   1.16.2
   
   ### deploy mode
   
   yarn-application
   
   ### What happened
   
   while we have configured a flink home before and use it to start a flink job 
with our jar using yarn-application mode.At the same time, this JAR will use a 
new connector. Since we are building a source code JAR package, we are 
uploading the connector to the local lib directory of Flink.In this case, when 
executing the Flink JAR, an error occurs indicating that the class for the 
connector cannot be found.
   
   The issue arises because the EnvInitializer.checkFlinkEnv method uploads the 
Flink directory to the HDFS StreamPark workspace only on certain occasions, 
such as when starting or deploying the application. In the YARN application 
mode, the /streampark/flink/flink-1.16.2/lib directory is included in the 
classpath. However, since the checkFlinkEnv method checks whether the directory 
exists before uploading files (if (!fsOperator.exists(flinkHome))), any 
subsequent modifications to the local Flink lib directory will not be updated 
in HDFS.
   
   We can only manually upload any added JARs to HDFS, which is not very 
user-friendly. This is because, without inspecting the source code, users may 
not be aware that uploading to this 
directory(/streampark/flink/flink-1.16.2/lib) is required for working nomally.
   
   ### Error Exception
   
   ```log
   Caused by: java.lang.NoClassDefFoundError: 
com/ververica/cdc/connectors/postgres/PostgreSQLSource
   
   if (!fsOperator.exists(flinkHome)) {
         log.info("{} is not exists,upload beginning....", flinkHome);
         fsOperator.upload(flinkLocalHome, flinkHome, false, true);
    }
   ```
   
   
   ### Screenshots
   
   
![image](https://github.com/apache/incubator-streampark/assets/57188699/6573f97c-0abb-4d5c-94e2-7b3c56791a72)
   
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!(您是否要贡献这个PR?)
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to