You are Right Chesnay but I'm doing this stuff in parallel with other 2
things and I messed up the jar name, sorry for that.
For the code after env.execute I'll try to use the new JobListener
interface next days.. I hope it could be sufficient (I just have to call an
external service to update the
Yes, it is definitely way easier to upload jars instead of
submitting JobGraphs.
But I thought this was not viable for you because you cannot execute
anything after env.execute()? I believe this limitation still exists. Or
are you referring here to error-handling in case env.execute() throws
I just discovered that I was using the "slim" jar instead of the "fat"
one...sorry. Now I'm able to successfully run the program on the remote
cluster.
However, the fact of generating the job graph on the client side it's
something I really don't like at allbecause it requires access both to
flink
Can you give me more information on your packaging setup / project
structure? Is "it.test.MyOb" a test class? Does the dependency
containing this class have a "test" scope?
On 10/30/2020 11:34 AM, Chesnay Schepler wrote:
It is irrelevant whether it contains transitive dependencies or not;
It is irrelevant whether it contains transitive dependencies or not;
that's a maven concept, not a classloading one.
The WordCount main class, which is only contained in that jar, could be
found, so the classloading is working. If any other class that is
supposed to be in jar cannot be found,
Yes, with the WordCount it works but that jar is not a "fat" jar (it does
not include transitive dependencies).
Actually I was able to use the REST API without creating the JobGraph, you
just have to tell the run API the jar id, the main cluss to run and the
optional parameters.
For this don't use
If you aren't setting up the classpath correctly then you cannot create
a JobGraph, and cannot use the REST API (outside of uploading jars).
In other words, you _will_ have to solve this issue, one way or another.
FYI, I just tried your code to submit a WordCount jar to a cluster (the
one in
For "REST only client" I mean using only the REST API to interact with the
Flink cluster, i.e. without creating any PackagedProgram and thus incurring
into classpath problems.
I've implemented a running job server that was using the REST API to upload
the job jar and execute the run command but
To clarify, if the job creation fails on the JM side, in 1.11 the job
submission will fail, in 1.12 it will succeed but the job will be in a
failed state.
On 10/30/2020 10:23 AM, Chesnay Schepler wrote:
1) the job still reaches a failed state, which you can poll for, see 2)
2) polling is your
1) the job still reaches a failed state, which you can poll for, see 2)
2) polling is your only way.
What do you mean with "REST only client"? Do you mean a plain http
client, not something that Flink provides?
On 10/30/2020 10:02 AM, Flavio Pompermaier wrote:
Nothing to do also with
Nothing to do also with IntelliJ..do you have any sample project I
can reuse to test the job submission to a cluster?
I can't really understand why the classes within the fat jar are not found
when generating the PackagedProgram.
Ideally, I'd prefer to use REST only client (so no need to build
I'm runnin the code from Eclipse, the jar exists and it contains the
classes Flink is not finding..maybe I can try to use IntelliJ in the
afternoon
On Wed, Oct 28, 2020 at 12:13 PM Chesnay Schepler
wrote:
> @Kostas: Ah, I missed that.
>
> @Flavio: the only alternative I can think your jar does
@Kostas: Ah, I missed that.
@Flavio: the only alternative I can think your jar does not contain the
classes, or does not exist at all on the machine your application is run on.
On 10/28/2020 12:08 PM, Kostas Kloudas wrote:
Hi all,
I will have a look in the whole stack trace in a bit.
Hi all,
I will have a look in the whole stack trace in a bit.
@Chesnay Schepler I think that we are setting the correct classloader
during jobgraph creation [1]. Is that what you mean?
Cheers,
Kostas
[1]
Always the same problem.
Caused by: java.lang.ClassNotFoundException: it.test.XXX
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
at
hmm..it appears as if PackagedProgramUtils#createJobGraph does some
things outside the usercode classlodaer (getPipelineFromProgram()),
specifically the call to the main method.
@klou This seems like wrong behavior?
@Flavio What you could try in the meantime is wrap the call to
Any help here? How can I understand why the classes inside the jar are not
found when creating the PackagedProgram?
On Tue, Oct 27, 2020 at 11:04 AM Flavio Pompermaier
wrote:
> In the logs I see that the jar is the classpath (I'm trying to debug the
> program from the IDE)..isn'it?
>
>
In the logs I see that the jar is the classpath (I'm trying to debug the
program from the IDE)..isn'it?
Classpath: [file:/tmp/job-bundle.jar]
...
Best,
Flavio
On Tue, Oct 27, 2020 at 10:39 AM Chesnay Schepler
wrote:
> * your JobExecutor is _not_ putting it on the classpath.
>
> On 10/27/2020
* your JobExecutor is _not_ putting it on the classpath.
On 10/27/2020 10:36 AM, Chesnay Schepler wrote:
Well it happens on the client before you even hit the
RestClusterClient, so I assume that either your jar is not packaged
correctly or you your JobExecutor is putting it on the classpath.
Well it happens on the client before you even hit the RestClusterClient,
so I assume that either your jar is not packaged correctly or you your
JobExecutor is putting it on the classpath.
On 10/27/2020 9:42 AM, Flavio Pompermaier wrote:
Sure. Here it is (org.apache.flink.client.cli.JobExecutor
Sure. Here it is (org.apache.flink.client.cli.JobExecutor is my main class
I'm trying to use as a client towards the Flink cluster - session mode).
it/test/MyOb is within the fat jar (/tmp/job-bundle.jar).
The code of getBatchEnv is:
@Deprecated
public static BatchEnv getBatchEnv() {
//
Hi Flavio,
can you share the full stacktrace you are seeing? I'm wondering if the
error happens on the client or server side (among other questions I have).
On Mon, Oct 26, 2020 at 5:58 PM Flavio Pompermaier
wrote:
> Hi to all,
> I was trying to use the RestClusterClient to submit my job to the
Hi to all,
I was trying to use the RestClusterClient to submit my job to the Flink
cluster.
However when I submit the job Flink cannot find the classes contained in
the "fat" jar..what should I do? Am I missing something in my code?
This is the current client code I'm testing:
public static void
23 matches
Mail list logo
