[
https://issues.apache.org/jira/browse/FLINK-8186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276432#comment-16276432
]
Timo Walther commented on FLINK-8186:
-------------------------------------
I think we should rework the dependencies of our quickstarts. What is in the
jars seems pretty random to me.
Without -Pbuild-jar:
{code}
[INFO] Excluding org.apache.flink:flink-java:jar:1.4-SNAPSHOT from the shaded
jar.
[INFO] Excluding org.apache.flink:flink-core:jar:1.4-SNAPSHOT from the shaded
jar.
[INFO] Excluding org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT from the
shaded jar.
[INFO] Excluding org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT from the
shaded jar.
[INFO] Excluding com.esotericsoftware.kryo:kryo:jar:2.24.0 from the shaded jar.
[INFO] Excluding com.esotericsoftware.minlog:minlog:jar:1.2 from the shaded jar.
[INFO] Excluding org.objenesis:objenesis:jar:2.1 from the shaded jar.
[INFO] Excluding commons-collections:commons-collections:jar:3.2.2 from the
shaded jar.
[INFO] Excluding org.apache.commons:commons-compress:jar:1.4.1 from the shaded
jar.
[INFO] Excluding org.xerial.snappy:snappy-java:jar:1.1.1.3 from the shaded jar.
[INFO] Including org.apache.flink:flink-shaded-asm:jar:5.0.4-2.0 in the shaded
jar.
[INFO] Excluding org.apache.commons:commons-lang3:jar:3.3.2 from the shaded jar.
[INFO] Including org.apache.commons:commons-math3:jar:3.5 in the shaded jar.
[INFO] Excluding org.slf4j:slf4j-api:jar:1.7.7 from the shaded jar.
[INFO] Including com.google.code.findbugs:jsr305:jar:1.3.9 in the shaded jar.
[INFO] Including org.apache.flink:force-shading:jar:1.4-SNAPSHOT in the shaded
jar.
[INFO] Excluding org.apache.flink:flink-clients_2.11:jar:1.4-SNAPSHOT from the
shaded jar.
[INFO] Excluding org.apache.flink:flink-runtime_2.11:jar:1.4-SNAPSHOT from the
shaded jar.
[INFO] Including
org.apache.flink:flink-queryable-state-client-java_2.11:jar:1.4-SNAPSHOT in the
shaded jar.
[INFO] Including org.apache.flink:flink-hadoop-fs:jar:1.4-SNAPSHOT in the
shaded jar.
[INFO] Excluding commons-io:commons-io:jar:2.4 from the shaded jar.
[INFO] Including org.apache.flink:flink-shaded-netty:jar:4.0.27.Final-2.0 in
the shaded jar.
[INFO] Including org.apache.flink:flink-shaded-guava:jar:18.0-2.0 in the shaded
jar.
[INFO] Excluding org.apache.flink:flink-shaded-jackson:jar:2.7.9-2.0 from the
shaded jar.
[INFO] Including org.javassist:javassist:jar:3.18.2-GA in the shaded jar.
[INFO] Excluding org.scala-lang:scala-library:jar:2.11.11 from the shaded jar.
[INFO] Excluding com.typesafe.akka:akka-actor_2.11:jar:2.4.20 from the shaded
jar.
[INFO] Excluding com.typesafe:config:jar:1.3.0 from the shaded jar.
[INFO] Including org.scala-lang.modules:scala-java8-compat_2.11:jar:0.7.0 in
the shaded jar.
[INFO] Including com.typesafe.akka:akka-stream_2.11:jar:2.4.20 in the shaded
jar.
[INFO] Including org.reactivestreams:reactive-streams:jar:1.0.0 in the shaded
jar.
[INFO] Including com.typesafe:ssl-config-core_2.11:jar:0.2.1 in the shaded jar.
[INFO] Including org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4
in the shaded jar.
[INFO] Including com.typesafe.akka:akka-protobuf_2.11:jar:2.4.20 in the shaded
jar.
[INFO] Excluding com.typesafe.akka:akka-slf4j_2.11:jar:2.4.20 from the shaded
jar.
[INFO] Including org.clapper:grizzled-slf4j_2.11:jar:1.0.2 in the shaded jar.
[INFO] Excluding com.github.scopt:scopt_2.11:jar:3.5.0 from the shaded jar.
[INFO] Excluding com.twitter:chill_2.11:jar:0.7.4 from the shaded jar.
[INFO] Excluding com.twitter:chill-java:jar:0.7.4 from the shaded jar.
[INFO] Excluding org.apache.flink:flink-optimizer_2.11:jar:1.4-SNAPSHOT from
the shaded jar.
[INFO] Excluding commons-cli:commons-cli:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.apache.flink:flink-avro:jar:1.4-SNAPSHOT from the shaded
jar.
[INFO] Excluding org.apache.avro:avro:jar:1.8.2 from the shaded jar.
[INFO] Excluding org.codehaus.jackson:jackson-core-asl:jar:1.9.13 from the
shaded jar.
[INFO] Excluding org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 from the
shaded jar.
[INFO] Excluding com.thoughtworks.paranamer:paranamer:jar:2.7 from the shaded
jar.
[INFO] Excluding org.tukaani:xz:jar:1.5 from the shaded jar.
[INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.7 from the shaded jar.
[INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded jar.
{code}
With -Pbuild-jar:
{code}
[INFO] Including org.apache.commons:commons-compress:jar:1.4.1 in the shaded
jar.
[INFO] Including org.xerial.snappy:snappy-java:jar:1.1.1.3 in the shaded jar.
[INFO] Including org.slf4j:slf4j-api:jar:1.7.7 in the shaded jar.
[INFO] Including com.google.code.findbugs:jsr305:jar:1.3.9 in the shaded jar.
[INFO] Including org.apache.flink:force-shading:jar:1.4-SNAPSHOT in the shaded
jar.
[INFO] Including org.apache.flink:flink-avro:jar:1.4-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.avro:avro:jar:1.8.2 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-core-asl:jar:1.9.13 in the shaded
jar.
[INFO] Including org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 in the
shaded jar.
[INFO] Including com.thoughtworks.paranamer:paranamer:jar:2.7 in the shaded jar.
[INFO] Including org.tukaani:xz:jar:1.5 in the shaded jar.
{code}
> AvroInputFormat regression: fails to deserialize GenericRecords on standalone
> cluster with hadoop27 compat
> ----------------------------------------------------------------------------------------------------------
>
> Key: FLINK-8186
> URL: https://issues.apache.org/jira/browse/FLINK-8186
> Project: Flink
> Issue Type: Bug
> Affects Versions: 1.4.0
> Reporter: Sebastian Klemke
> Priority: Minor
> Attachments: GenericRecordCount.java, pom.xml
>
>
> The following job runs fine on a Flink 1.3.2 cluster, but fails on a Flink
> 1.4.0 RC2 standalone cluster, "hadoop27" flavour:
> {code}
> public class GenericRecordCount {
> public static void main(String[] args) throws Exception {
> String input = ParameterTool.fromArgs(args).getRequired("input");
> ExecutionEnvironment env =
> ExecutionEnvironment.getExecutionEnvironment();
> long count = env.readFile(new AvroInputFormat<>(new Path(input),
> GenericRecord.class), input)
> .count();
> System.out.printf("Counted %d records\n", count);
> }
> }
> {code}
> Runs fine in LocalExecutionEnvironment and also on no-hadoop flavour
> standalone cluster, though. Exception thrown in Flink 1.4.0 hadoop27:
> {code}
> 12/01/2017 13:22:09 DataSource (at
> readFile(ExecutionEnvironment.java:514)
> (org.apache.flink.formats.avro.AvroInputFormat))(4/4) switched to FAILED
> java.lang.RuntimeException: java.lang.NoSuchMethodException:
> org.apache.avro.generic.GenericRecord.<init>()
> at
> org.apache.avro.specific.SpecificData.newInstance(SpecificData.java:353)
> at
> org.apache.avro.specific.SpecificData.newRecord(SpecificData.java:369)
> at org.apache.avro.reflect.ReflectData.newRecord(ReflectData.java:901)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:212)
> at
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145)
> at org.apache.avro.file.DataFileStream.next(DataFileStream.java:233)
> at
> org.apache.flink.formats.avro.AvroInputFormat.nextRecord(AvroInputFormat.java:165)
> at
> org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:167)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NoSuchMethodException:
> org.apache.avro.generic.GenericRecord.<init>()
> at java.lang.Class.getConstructor0(Class.java:3082)
> at java.lang.Class.getDeclaredConstructor(Class.java:2178)
> at
> org.apache.avro.specific.SpecificData.newInstance(SpecificData.java:347)
> ... 11 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
