Re: ClasNotFound when submitting job from command line

[Flavio Pompermaier]

At the moment my 2 PRs about jdbc batch connector and input format fix are
pending so I have my customized implementation of the inputformat in my jar
andas jdbc backend I need ijdbc6.jar as dependency (that I shade on the
jar).
Thus the class.forName() is called on the inputformat open().
On 20 Apr 2016 09:32, "Robert Metzger"  wrote:

Hi Flavio,

in which class are you calling Class.forName()? Is the class where the
Class.forName() call is loaded from the user jar or is it a class from the
Flink distribution?
I'm asking because Class.forName() is using the classloader of the class
where the call is located. So if the class has been loaded from the system
class loader, it can not access classes from the user jar.



On Tue, Apr 19, 2016 at 6:13 PM, Flavio Pompermaier 
wrote:

> I use maven to generate the shaded jar (and the classes are inside it) but
> when the job starts it can load those classes using Class.forName()
> (required to instantiate the JDBC connections).
> I think it's probably a problem related to class loading of Flink
>
> On Tue, Apr 19, 2016 at 6:02 PM, Balaji Rajagopalan <
> balaji.rajagopa...@olacabs.com> wrote:
>
>> In your pom.xml add the maven.plugins like this, and you will have to add
>> all the dependent artifacts, this works for me, if you fire mvn clean
>> compile package, the created jar is a fat jar.
>>
>> 
>>
>> org.apache.maven.plugins
>>
>> maven-dependency-plugin
>>
>> 2.9
>>
>> 
>>
>> 
>>
>> unpack
>>
>> 
>>
>> prepare-package
>>
>> 
>>
>> unpack
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>> org.apache.flink
>>
>>
>> flink-connector-kafka-0.8_${scala.version}
>>
>> ${flink.version}
>>
>> jar
>>
>> false
>>
>>
>> ${project.build.directory}/classes
>>
>>
>> org/apache/flink/**
>>
>> 
>>
>>...
>>
>>   ...
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>> On Tue, Apr 19, 2016 at 9:10 PM, Flavio Pompermaier > > wrote:
>>
>>> Hi to all,
>>>
>>> I just tied to dubmit my application to the Flink cluster (1.0.1) but I
>>> get ClassNotFound exceptions for classes inside my shaded jar (like
>>> oracle.jdbc.OracleDriver or org.apache.commons.pool2.PooledObjectFactory).
>>> Those classes are in the shaded jar but aren't found.
>>> If I put the jars in the flink's lib dir (for every node of the cluster)
>>> things work.
>>> How can I solve that?
>>>
>>> Best,
>>> Flavio
>>>
>>
>>
>
>

Re: ClasNotFound when submitting job from command line

[Robert Metzger]

Hi Flavio,

in which class are you calling Class.forName()? Is the class where the
Class.forName() call is loaded from the user jar or is it a class from the
Flink distribution?
I'm asking because Class.forName() is using the classloader of the class
where the call is located. So if the class has been loaded from the system
class loader, it can not access classes from the user jar.



On Tue, Apr 19, 2016 at 6:13 PM, Flavio Pompermaier 
wrote:

> I use maven to generate the shaded jar (and the classes are inside it) but
> when the job starts it can load those classes using Class.forName()
> (required to instantiate the JDBC connections).
> I think it's probably a problem related to class loading of Flink
>
> On Tue, Apr 19, 2016 at 6:02 PM, Balaji Rajagopalan <
> balaji.rajagopa...@olacabs.com> wrote:
>
>> In your pom.xml add the maven.plugins like this, and you will have to add
>> all the dependent artifacts, this works for me, if you fire mvn clean
>> compile package, the created jar is a fat jar.
>>
>> 
>>
>> org.apache.maven.plugins
>>
>> maven-dependency-plugin
>>
>> 2.9
>>
>> 
>>
>> 
>>
>> unpack
>>
>> 
>>
>> prepare-package
>>
>> 
>>
>> unpack
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>> org.apache.flink
>>
>>
>> flink-connector-kafka-0.8_${scala.version}
>>
>> ${flink.version}
>>
>> jar
>>
>> false
>>
>>
>> ${project.build.directory}/classes
>>
>>
>> org/apache/flink/**
>>
>> 
>>
>>...
>>
>>   ...
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>> On Tue, Apr 19, 2016 at 9:10 PM, Flavio Pompermaier > > wrote:
>>
>>> Hi to all,
>>>
>>> I just tied to dubmit my application to the Flink cluster (1.0.1) but I
>>> get ClassNotFound exceptions for classes inside my shaded jar (like
>>> oracle.jdbc.OracleDriver or org.apache.commons.pool2.PooledObjectFactory).
>>> Those classes are in the shaded jar but aren't found.
>>> If I put the jars in the flink's lib dir (for every node of the cluster)
>>> things work.
>>> How can I solve that?
>>>
>>> Best,
>>> Flavio
>>>
>>
>>
>
>

Re: ClasNotFound when submitting job from command line

[Balaji Rajagopalan]

In your pom.xml add the maven.plugins like this, and you will have to add
all the dependent artifacts, this works for me, if you fire mvn clean
compile package, the created jar is a fat jar.



org.apache.maven.plugins

maven-dependency-plugin

2.9





unpack



prepare-package



unpack











org.apache.flink


flink-connector-kafka-0.8_${scala.version}

${flink.version}

jar

false


${project.build.directory}/classes

org/apache/flink/**



   ...

  ...











On Tue, Apr 19, 2016 at 9:10 PM, Flavio Pompermaier 
wrote:

> Hi to all,
>
> I just tied to dubmit my application to the Flink cluster (1.0.1) but I
> get ClassNotFound exceptions for classes inside my shaded jar (like
> oracle.jdbc.OracleDriver or org.apache.commons.pool2.PooledObjectFactory).
> Those classes are in the shaded jar but aren't found.
> If I put the jars in the flink's lib dir (for every node of the cluster)
> things work.
> How can I solve that?
>
> Best,
> Flavio
>

RE: ClasNotFound when submitting job from command line

[Radu Tudoran]

Hi,

In my case the root cause for this was mainly that I was using eclipse to 
package the jar. Try using mvn instead. Additioanlly you can copy the 
dependency jars in the lib of the task managers and restart them

Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R Division

[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München

E-mail: radu.tudo...@huawei.com
Mobile: +49 15209084330
Telephone: +49 891588344173

HUAWEI TECHNOLOGIES Duesseldorf GmbH
Hansaallee 205, 40549 Düsseldorf, Germany, 
www.huawei.com
Registered Office: Düsseldorf, Register Court Düsseldorf, HRB 56063,
Managing Director: Bo PENG, Wanzhou MENG, Lifang CHEN
Sitz der Gesellschaft: Düsseldorf, Amtsgericht Düsseldorf, HRB 56063,
Geschäftsführer: Bo PENG, Wanzhou MENG, Lifang CHEN
This e-mail and its attachments contain confidential information from HUAWEI, 
which is intended only for the person or entity whose address is listed above. 
Any use of the information contained herein in any way (including, but not 
limited to, total or partial disclosure, reproduction, or dissemination) by 
persons other than the intended recipient(s) is prohibited. If you receive this 
e-mail in error, please notify the sender by phone or email immediately and 
delete it!

From: Flavio Pompermaier [mailto:pomperma...@okkam.it]
Sent: Tuesday, April 19, 2016 5:40 PM
To: user
Subject: ClasNotFound when submitting job from command line

Hi to all,
I just tied to dubmit my application to the Flink cluster (1.0.1) but I get 
ClassNotFound exceptions for classes inside my shaded jar (like 
oracle.jdbc.OracleDriver or org.apache.commons.pool2.PooledObjectFactory). 
Those classes are in the shaded jar but aren't found.
If I put the jars in the flink's lib dir (for every node of the cluster) things 
work.
How can I solve that?

Best,
Flavio