Re: Classloader issue using AvroParquetInputFormat via HadoopInputFormat
I've started a vote for 1.1.1 containing hopefully fixed artifacts. If you have any spare time, would you mind checking whether it fixes your problem? The artifacts are here: http://home.apache.org/~uce/flink-1.1.1-rc1/ You would have to add the following repository to your Maven project and update the Flink version to 1.1.1: flink-rc flink-rc https://repository.apache.org/content/repositories/orgapacheflink-1101 true false Would really appreciate it! On Tue, Aug 9, 2016 at 2:11 PM, Ufuk Celebi wrote: > This is a problem with the Maven artifacts of 1.1.0 :-( I've added a > warning to the release note and will start a emergency vote for 1.1.1 > which only updates the Maven artifacts. > > On Tue, Aug 9, 2016 at 1:20 PM, LINZ, Arnaud wrote: >> Okay, >> >> That would also solve my issue. >> >> Greetings, >> >> Arnaud >> >> >> >> De : Stephan Ewen [mailto:se...@apache.org] >> Envoyé : mardi 9 août 2016 12:41 >> À : user@flink.apache.org >> Objet : Re: Classloader issue using AvroParquetInputFormat via >> HadoopInputFormat >> >> >> >> Hi Shannon! >> >> >> >> It seams that the something in the maven deployment went wrong with this >> release. >> >> >> >> There should be: >> >> - flink-java (the default, with a transitive dependency to hadoop 2.x for >> hadoop compatibility features) >> >> - flink-java-hadoop1 (with a transitive dependency for hadoop 1.x fir >> older hadoop compatibility features) >> >> >> >> Apparently the "flink-java" artifact git overwritten with the >> "flink-java-hadoop1" artifact. Damn. >> >> >> >> I think we need to release new artifacts that fix these dependency >> descriptors. >> >> >> >> That needs to be a 1.1.1 release, because maven artifacts cannot be changed >> after they were deployed. >> >> >> >> Greetings, >> Stephan >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Mon, Aug 8, 2016 at 11:08 PM, Shannon Carey wrote: >> >> Correction: I cannot work around the problem. If I exclude hadoop1, I get >> the following exception which appears to be due to flink-java-1.1.0's >> dependency on Hadoop1. >> >> >> >> Failed to submit job 4b6366d101877d38ef33454acc6ca500 >> (com.expedia.www.flink.jobs.DestinationCountsHistoryJob$) >> >> org.apache.flink.runtime.client.JobExecutionException: Failed to submit job >> 4b6366d101877d38ef33454acc6ca500 >> (com.expedia.www.flink.jobs.DestinationCountsHistoryJob$) >> >> at >> org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1281) >> >> at >> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:478) >> >> at >> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36) >> >> at >> org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:36) >> >> at >> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36) >> >> at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33) >> >> at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28) >> >> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) >> >> at >> org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28) >> >> at akka.actor.Actor$class.aroundReceive(Actor.scala:465) >> >> at >> org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:121) >> >> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) >> >> at akka.actor.ActorCell.invoke(ActorCell.scala:487) >> >> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254) >> >> at akka.dispatch.Mailbox.run(Mailbox.scala:221) >> >> at akka.dispatch.Mailbox.exec(Mailbox.scala:231) >> >> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >> >> at >> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >> >> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >> >> at >> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >> >> Caused by: org.apache.flink.runtime.JobException: Creating the
Re: Classloader issue using AvroParquetInputFormat via HadoopInputFormat
This is a problem with the Maven artifacts of 1.1.0 :-( I've added a warning to the release note and will start a emergency vote for 1.1.1 which only updates the Maven artifacts. On Tue, Aug 9, 2016 at 1:20 PM, LINZ, Arnaud wrote: > Okay, > > That would also solve my issue. > > Greetings, > > Arnaud > > > > De : Stephan Ewen [mailto:se...@apache.org] > Envoyé : mardi 9 août 2016 12:41 > À : user@flink.apache.org > Objet : Re: Classloader issue using AvroParquetInputFormat via > HadoopInputFormat > > > > Hi Shannon! > > > > It seams that the something in the maven deployment went wrong with this > release. > > > > There should be: > > - flink-java (the default, with a transitive dependency to hadoop 2.x for > hadoop compatibility features) > > - flink-java-hadoop1 (with a transitive dependency for hadoop 1.x fir > older hadoop compatibility features) > > > > Apparently the "flink-java" artifact git overwritten with the > "flink-java-hadoop1" artifact. Damn. > > > > I think we need to release new artifacts that fix these dependency > descriptors. > > > > That needs to be a 1.1.1 release, because maven artifacts cannot be changed > after they were deployed. > > > > Greetings, > Stephan > > > > > > > > > > > > > > On Mon, Aug 8, 2016 at 11:08 PM, Shannon Carey wrote: > > Correction: I cannot work around the problem. If I exclude hadoop1, I get > the following exception which appears to be due to flink-java-1.1.0's > dependency on Hadoop1. > > > > Failed to submit job 4b6366d101877d38ef33454acc6ca500 > (com.expedia.www.flink.jobs.DestinationCountsHistoryJob$) > > org.apache.flink.runtime.client.JobExecutionException: Failed to submit job > 4b6366d101877d38ef33454acc6ca500 > (com.expedia.www.flink.jobs.DestinationCountsHistoryJob$) > > at > org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1281) > > at > org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:478) > > at > scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36) > > at > org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:36) > > at > scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36) > > at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33) > > at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28) > > at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) > > at > org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28) > > at akka.actor.Actor$class.aroundReceive(Actor.scala:465) > > at > org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:121) > > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) > > at akka.actor.ActorCell.invoke(ActorCell.scala:487) > > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254) > > at akka.dispatch.Mailbox.run(Mailbox.scala:221) > > at akka.dispatch.Mailbox.exec(Mailbox.scala:231) > > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > > at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > > Caused by: org.apache.flink.runtime.JobException: Creating the input splits > caused an error: Found interface org.apache.hadoop.mapreduce.JobContext, but > class was expected > > at > org.apache.flink.runtime.executiongraph.ExecutionJobVertex.(ExecutionJobVertex.java:172) > > at > org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:695) > > at > org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1178) > > ... 19 more > > Caused by: java.lang.IncompatibleClassChangeError: Found interface > org.apache.hadoop.mapreduce.JobContext, but class was expected > > at > org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.createInputSplits(HadoopInputFormatBase.java:158) > > at > org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.createInputSplits(HadoopInputFormatBase.java:56) > > at > org.apache.flink.runtime.executiongraph.ExecutionJobVertex.(ExecutionJobVertex.java:156) > > ... 21 more > > > > And if I exclude had
RE: Classloader issue using AvroParquetInputFormat via HadoopInputFormat
Okay,
That would also solve my issue.
Greetings,
Arnaud
De : Stephan Ewen [mailto:se...@apache.org]
Envoyé : mardi 9 août 2016 12:41
À : user@flink.apache.org
Objet : Re: Classloader issue using AvroParquetInputFormat via HadoopInputFormat
Hi Shannon!
It seams that the something in the maven deployment went wrong with this
release.
There should be:
- flink-java (the default, with a transitive dependency to hadoop 2.x for
hadoop compatibility features)
- flink-java-hadoop1 (with a transitive dependency for hadoop 1.x fir older
hadoop compatibility features)
Apparently the "flink-java" artifact git overwritten with the
"flink-java-hadoop1" artifact. Damn.
I think we need to release new artifacts that fix these dependency descriptors.
That needs to be a 1.1.1 release, because maven artifacts cannot be changed
after they were deployed.
Greetings,
Stephan
On Mon, Aug 8, 2016 at 11:08 PM, Shannon Carey
mailto:sca...@expedia.com>> wrote:
Correction: I cannot work around the problem. If I exclude hadoop1, I get the
following exception which appears to be due to flink-java-1.1.0's dependency on
Hadoop1.
Failed to submit job 4b6366d101877d38ef33454acc6ca500
(com.expedia.www.flink.jobs<http://com.expedia.www.flink.jobs>.DestinationCountsHistoryJob$)
org.apache.flink.runtime.client.JobExecutionException: Failed to submit job
4b6366d101877d38ef33454acc6ca500
(com.expedia.www.flink.jobs<http://com.expedia.www.flink.jobs>.DestinationCountsHistoryJob$)
at
org.apache.flink.runtime.jobmanager.JobManager.org<http://org.apache.flink.runtime.jobmanager.JobManager.org>$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1281)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:478)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at
org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:36)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at
org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at
org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:121)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
at akka.dispatch.Mailbox.run(Mailbox.scala:221)
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.flink.runtime.JobException: Creating the input splits
caused an error: Found interface org.apache.hadoop.mapreduce.JobContext, but
class was expected
at
org.apache.flink.runtime.executiongraph.ExecutionJobVertex.(ExecutionJobVertex.java:172)
at
org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:695)
at
org.apache.flink.runtime.jobmanager.JobManager.org<http://org.apache.flink.runtime.jobmanager.JobManager.org>$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1178)
... 19 more
Caused by: java.lang.IncompatibleClassChangeError: Found interface
org.apache.hadoop.mapreduce.JobContext, but class was expected
at
org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.createInputSplits(HadoopInputFormatBase.java:158)
at
org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.createInputSplits(HadoopInputFormatBase.java:56)
at
org.apache.flink.runtime.executiongraph.ExecutionJobVertex.(ExecutionJobVertex.java:156)
... 21 more
And if I exclude hadoop2, I get the exception from my previous email with
AvroParquetInputFormat.
From: Shannon Carey mailto:sca...@expedia.com>>
Date: Monday, August 8, 2016 at 2:46 PM
To: "user@flink.apache.org<mailto:user@flink.apache.org>"
mailto:user@flink.apache.org>>
Subject: Classloader issue using AvroParquetInputFormat via HadoopInputFormat
Hi folks, congrats on 1.1.0!
FYI, after updating to Flink 1.1.0 I get the exception at bottom when
attempting to run a job that uses AvroParquetInputFormat wrapped in a Flink
HadoopInputFormat. The ContextUtil.java:71 is trying to execute:
Class.forName("org.apache.hadoop.mapreduce.task.JobContextImpl");
I am using Scala 2.11.7. JobContextImpl is coming from
flink-shaded-hadoop2
Re: Classloader issue using AvroParquetInputFormat via HadoopInputFormat
Hi Shannon!
It seams that the something in the maven deployment went wrong with this
release.
There should be:
- *flink-java* (the default, with a transitive dependency to hadoop 2.x
for hadoop compatibility features)
- *flink-java-hadoop1* (with a transitive dependency for hadoop 1.x fir
older hadoop compatibility features)
Apparently the "flink-java" artifact git overwritten with the
"flink-java-hadoop1" artifact. Damn.
I think we need to release new artifacts that fix these dependency
descriptors.
That needs to be a 1.1.1 release, because maven artifacts cannot be changed
after they were deployed.
Greetings,
Stephan
On Mon, Aug 8, 2016 at 11:08 PM, Shannon Carey wrote:
> Correction: I cannot work around the problem. If I exclude hadoop1, I get
> the following exception which appears to be due to flink-java-1.1.0's
> dependency on Hadoop1.
>
> Failed to submit job 4b6366d101877d38ef33454acc6ca500 (
> com.expedia.www.flink.jobs.DestinationCountsHistoryJob$)
> org.apache.flink.runtime.client.JobExecutionException: Failed to submit
> job 4b6366d101877d38ef33454acc6ca500 (com.expedia.www.flink.jobs.
> DestinationCountsHistoryJob$)
> at org.apache.flink.runtime.jobmanager.JobManager.org$
> apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:
> 1281)
> at org.apache.flink.runtime.jobmanager.JobManager$$
> anonfun$handleMessage$1.applyOrElse(JobManager.scala:478)
> at scala.runtime.AbstractPartialFunction.apply(
> AbstractPartialFunction.scala:36)
> at org.apache.flink.runtime.LeaderSessionMessageFilter$$
> anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:36)
> at scala.runtime.AbstractPartialFunction.apply(
> AbstractPartialFunction.scala:36)
> at org.apache.flink.runtime.LogMessages$$anon$1.apply(
> LogMessages.scala:33)
> at org.apache.flink.runtime.LogMessages$$anon$1.apply(
> LogMessages.scala:28)
> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> at org.apache.flink.runtime.LogMessages$$anon$1.
> applyOrElse(LogMessages.scala:28)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
> at org.apache.flink.runtime.jobmanager.JobManager.
> aroundReceive(JobManager.scala:121)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
> at akka.actor.ActorCell.invoke(ActorCell.scala:487)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
> at akka.dispatch.Mailbox.run(Mailbox.scala:221)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.
> runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(
> ForkJoinPool.java:1979)
> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(
> ForkJoinWorkerThread.java:107)
> Caused by: org.apache.flink.runtime.JobException: Creating the input
> splits caused an error: Found interface
> org.apache.hadoop.mapreduce.JobContext,
> but class was expected
> at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.(
> ExecutionJobVertex.java:172)
> at org.apache.flink.runtime.executiongraph.ExecutionGraph.
> attachJobGraph(ExecutionGraph.java:695)
> at org.apache.flink.runtime.jobmanager.JobManager.org$
> apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:
> 1178)
> ... 19 more
> Caused by: java.lang.IncompatibleClassChangeError: Found interface
> org.apache.hadoop.mapreduce.JobContext, but class was expected
> at org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.
> createInputSplits(HadoopInputFormatBase.java:158)
> at org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.
> createInputSplits(HadoopInputFormatBase.java:56)
> at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.(
> ExecutionJobVertex.java:156)
> ... 21 more
>
> And if I exclude hadoop2, I get the exception from my previous email with
> AvroParquetInputFormat.
>
>
>
> From: Shannon Carey
> Date: Monday, August 8, 2016 at 2:46 PM
> To: "user@flink.apache.org"
> Subject: Classloader issue using AvroParquetInputFormat via
> HadoopInputFormat
>
> Hi folks, congrats on 1.1.0!
>
> FYI, after updating to Flink 1.1.0 I get the exception at bottom when
> attempting to run a job that uses AvroParquetInputFormat wrapped in a Flink
> HadoopInputFormat. The ContextUtil.java:71 is trying to execute:
>
> Class.forName("org.apache.hadoop.mapreduce.task.JobContextImpl");
>
> I am using Scala * 2.11*.7. JobContextImpl is coming from flink-shaded-
> *hadoop2*:1.1.0. However, its parent class (JobContext) is actually being
> loaded (according to output with JVM param "-verbose:class") from the
> flink-shaded-
Re: Classloader issue using AvroParquetInputFormat via HadoopInputFormat
Correction: I cannot work around the problem. If I exclude hadoop1, I get the
following exception which appears to be due to flink-java-1.1.0's dependency on
Hadoop1.
Failed to submit job 4b6366d101877d38ef33454acc6ca500
(com.expedia.www.flink.jobs.DestinationCountsHistoryJob$)
org.apache.flink.runtime.client.JobExecutionException: Failed to submit job
4b6366d101877d38ef33454acc6ca500
(com.expedia.www.flink.jobs.DestinationCountsHistoryJob$)
at
org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1281)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:478)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at
org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:36)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at
org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at
org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:121)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
at akka.dispatch.Mailbox.run(Mailbox.scala:221)
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.apache.flink.runtime.JobException: Creating the input splits
caused an error: Found interface org.apache.hadoop.mapreduce.JobContext, but
class was expected
at
org.apache.flink.runtime.executiongraph.ExecutionJobVertex.(ExecutionJobVertex.java:172)
at
org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:695)
at
org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1178)
... 19 more
Caused by: java.lang.IncompatibleClassChangeError: Found interface
org.apache.hadoop.mapreduce.JobContext, but class was expected
at
org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.createInputSplits(HadoopInputFormatBase.java:158)
at
org.apache.flink.api.java.hadoop.mapreduce.HadoopInputFormatBase.createInputSplits(HadoopInputFormatBase.java:56)
at
org.apache.flink.runtime.executiongraph.ExecutionJobVertex.(ExecutionJobVertex.java:156)
... 21 more
And if I exclude hadoop2, I get the exception from my previous email with
AvroParquetInputFormat.
From: Shannon Carey mailto:sca...@expedia.com>>
Date: Monday, August 8, 2016 at 2:46 PM
To: "user@flink.apache.org<mailto:user@flink.apache.org>"
mailto:user@flink.apache.org>>
Subject: Classloader issue using AvroParquetInputFormat via HadoopInputFormat
Hi folks, congrats on 1.1.0!
FYI, after updating to Flink 1.1.0 I get the exception at bottom when
attempting to run a job that uses AvroParquetInputFormat wrapped in a Flink
HadoopInputFormat. The ContextUtil.java:71 is trying to execute:
Class.forName("org.apache.hadoop.mapreduce.task.JobContextImpl");
I am using Scala 2.11.7. JobContextImpl is coming from
flink-shaded-hadoop2:1.1.0. However, its parent class (JobContext) is actually
being loaded (according to output with JVM param "-verbose:class") from the
flink-shaded-hadoop1_2.10 jar.
After adding an exclusion on flink-shaded-hadoop1_2.10, the problem appears to
be resolved. Is that the right way to fix the problem?
From what I can tell, the problem is that the JARs that are deployed to Maven
Central were built with different versions of Hadoop (as controlled by
hadoop.profile):
flink-runtime_2.11 depends on Hadoop 2
flink-java depends on Hadoop 1 (Scala 2.10)
flink-core depends on Hadoop 1 (Scala 2.10)
This seems like a problem with Flink's build process.
As an aside: would it be possible to change the interface of HadoopInputFormat
to take a Configuration instead of a Job? That would reduce the dependence on
the Hadoop API somewhat. It doesn't look like the Job itself is ever actually
used for anything. I'm glad to see you already have
https://issues.apache.org/jira/browse/FLINK-4316 and
https://issues.apache.org/jira/browse/FLINK-4315
Thanks,
Shannon
java.lang.IncompatibleClassChangeError: Implementing class
at java.lang.ClassLoader.defineClass1(Native M
Classloader issue using AvroParquetInputFormat via HadoopInputFormat
Hi folks, congrats on 1.1.0!
FYI, after updating to Flink 1.1.0 I get the exception at bottom when
attempting to run a job that uses AvroParquetInputFormat wrapped in a Flink
HadoopInputFormat. The ContextUtil.java:71 is trying to execute:
Class.forName("org.apache.hadoop.mapreduce.task.JobContextImpl");
I am using Scala 2.11.7. JobContextImpl is coming from
flink-shaded-hadoop2:1.1.0. However, its parent class (JobContext) is actually
being loaded (according to output with JVM param "-verbose:class") from the
flink-shaded-hadoop1_2.10 jar.
After adding an exclusion on flink-shaded-hadoop1_2.10, the problem appears to
be resolved. Is that the right way to fix the problem?
From what I can tell, the problem is that the JARs that are deployed to Maven
Central were built with different versions of Hadoop (as controlled by
hadoop.profile):
flink-runtime_2.11 depends on Hadoop 2
flink-java depends on Hadoop 1 (Scala 2.10)
flink-core depends on Hadoop 1 (Scala 2.10)
This seems like a problem with Flink's build process.
As an aside: would it be possible to change the interface of HadoopInputFormat
to take a Configuration instead of a Job? That would reduce the dependence on
the Hadoop API somewhat. It doesn't look like the Job itself is ever actually
used for anything. I'm glad to see you already have
https://issues.apache.org/jira/browse/FLINK-4316 and
https://issues.apache.org/jira/browse/FLINK-4315
Thanks,
Shannon
java.lang.IncompatibleClassChangeError: Implementing class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.parquet.hadoop.util.ContextUtil.(ContextUtil.java:71)
at
org.apache.parquet.avro.AvroParquetInputFormat.setRequestedProjection(AvroParquetInputFormat.java:54)
at
com.expedia.www.sdk.flink.HistoricalDataIngestionJob.readHistoricalParquetFile(HistoricalDataIngestionJob.scala:63)
at
com.expedia.www.flink.jobs.DestinationCountsHistoryJob$.main(DestinationCountsHistoryJob.scala:25)
at
com.expedia.www.flink.jobs.DestinationCountsHistoryTest$$anonfun$1.apply$mcV$sp(DestinationCountsHistoryTest.scala:23)
at
com.expedia.www.flink.jobs.DestinationCountsHistoryTest$$anonfun$1.apply(DestinationCountsHistoryTest.scala:20)
at
com.expedia.www.flink.jobs.DestinationCountsHistoryTest$$anonfun$1.apply(DestinationCountsHistoryTest.scala:20)
at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1647)
at org.scalatest.Suite$class.withFixture(Suite.scala:1122)
at org.scalatest.FlatSpec.withFixture(FlatSpec.scala:1683)
at org.scalatest.FlatSpecLike$class.invokeWithFixture$1(FlatSpecLike.scala:1644)
at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1656)
at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1656)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
at org.scalatest.FlatSpecLike$class.runTest(FlatSpecLike.scala:1656)
at org.scalatest.FlatSpec.runTest(FlatSpec.scala:1683)
at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1714)
at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1714)
at
org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
at
org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
at
org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:390)
at
org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:427)
at
org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
at
org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
at org.scalatest.SuperEngine.runTestsImpl(Engine.scala
