Hi Joel, There is a few uncertainties in my reply so I am including Timo who should be able to confirm or deny what I wrote.
Generally speaking once a JobGraph is created it should contain the UDF. I might be wrong here, but the UDF must be available while the JobGraph is created. I believe you're submitting the job via the `flink run` command. Is that right? Therefore the UDF must be available on the classpath of this client, as that is the moment the JobGraph is create. If you use the web ui to submit the job, then the JobGraph is created on the JobManager and it must be available there. Best, Dawid On 07/09/2021 16:31, Joel Edwards wrote: > Good day, > > I have been attempting to submit a job to a session cluster. This job > involves a pair of dynamic tables and a SQL query. The SQL query is > calling a UDF which I register via the table API's > createTemporarySystemFunction() method. The job runs locally, but when > I attempt to submit it to a remote session cluster, the job fails with > the error: > > `Cannot load user class: <fully-qualified-class-name>` > > If I place a fat jar containing all of my local dependencies on the > JobManagers and TaskManagers, the UDF will be loaded. However, I would > expect the UDF to be serialized and sent with the rest of the job. I > have looked over the UDF documentation, and I don't see a reason why > it would not be serialized with the rest of the job. However, seeing > as there is no error related to serializing the UDF, my assumptions > related to UDF serialization must be incorrect. Is there a hint I can > use to cause the closure cleaner to identify the UDF for > serialization? I suspect the reason it is not being included is that > it is referenced only in the SQL query, and not streams feeding the > input table or the stream consuming the output table. > > Summary of questions: > - Will UDF be serialized with the job? Or are they never included? > - Is it possible to hint at what should be serialized and sent along > with the job? > > Thank you, > Joel > > > -- > Joel Edwards > Software Architect > Ed-Craft Software Solutions
OpenPGP_signature
Description: OpenPGP digital signature
