Hello,
We recently refactored our large avro schema into separate schemas to make them
more composable, since many of them get incorporated into other objects. We
were glad to see that the avro-maven-plugin allows this composable model rather
than having to do one large schema with everything embedded.
However, we have hit a problem - the Parser cannot always resolve the
cross-references. I upgraded to avro-1.7.7 (both the core avro and the
avro-maven-plugin project), but that does not solve the issue.
The problem is that names/schemas are not always resolved. Worse, we see
different behavior on Windows vs. Linux. Below I show a set up with a dummy
schema that works on Windows 7, but fails on Linux (tested on Centos 6 and
Ubuntu 14.04). We also have a more complicated schema with many objects which
have a number of cross references (but not circular ones) that fails on both
Windows and Linux.
Is this behavior a defect? Should the Schema.Parser be able to resolve these
cross-dependencies?
A larger question is why we need to put things in two directories (top level
and "imports"). Couldn't the Parser be made to resolve references of things
when they are all in the same directory?
Here is a detailed example - again this one works on Windows but fails on Linux:
u070072@TST imports$ tree
imports
├── pom.xml (1.4 KiB)
├── src
│ ├── main
│ ├── java
│ ├── quux00
│ ├── App.java (182 B)
│ ├── resources
│ ├── avro
│ ├── Top.avsc (473 B)
│ ├── import
│ ├── Bottom.avsc (239 B)
│ ├── Foo.avsc (307 B)
│ ├── Middle.avsc (393 B)
$ mvn -X generate-sources
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.786 s
[INFO] Finished at: 2014-08-04T15:06:02-05:00
[INFO] Final Memory: 19M/843M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.avro:avro-maven-plugin:1.7.7:schema
(default) on project imports: Execution default of goal
org.apache.avro:avro-maven-plugin:1.7.7:schema failed: Undefined name: "Bottom"
-> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal
org.apache.avro:avro-maven-plugin:1.7.7:schema (default) on project imports:
Execution default of goal org.apache.avro:avro-maven-plugin:1.7.7:schema
failed: Undefined name: "Bottom"
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:224)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:76)
at
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:116)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:361)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:213)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution default
of goal org.apache.avro:avro-maven-plugin:1.7.7:schema failed: Undefined name:
"Bottom"
at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:144)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
... 19 more
Caused by: org.apache.avro.SchemaParseException: Undefined name: "Bottom"
at org.apache.avro.Schema.parse(Schema.java:1162)
at org.apache.avro.Schema.parse(Schema.java:1272)
at org.apache.avro.Schema.parse(Schema.java:1203)
at org.apache.avro.Schema$Parser.parse(Schema.java:965)
at org.apache.avro.Schema$Parser.parse(Schema.java:932)
at org.apache.avro.mojo.SchemaMojo.doCompile(SchemaMojo.java:73)
at
org.apache.avro.mojo.AbstractAvroMojo.compileFiles(AbstractAvroMojo.java:216)
at
org.apache.avro.mojo.AbstractAvroMojo.execute(AbstractAvroMojo.java:154)
at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:133)
... 20 more
Here are the schema sections:
$ cat src/main/resources/avro/Top.avsc
{
"namespace": "quux00.avro",
"type": "record",
"name": "Top",
"fields": [
{"name": "TopId", "type": ["null", "string"], "default": null},
{"name": "TopCode", "type": ["null", "string"], "default": null},
{"name": "Middles", "type": ["null", {"type": "array", "items":
"Middle"}], "default": null},
{"name": "Bottom", "type": ["null", "Bottom"], "default": null},
{"name": "AFoo", "type": ["null", "Foo"], "default": null}
]
}
$ cat src/main/resources/avro/import/Middle.avsc
{
"namespace": "quux00.avro",
"type": "record",
"name": "Middle",
"fields": [
{"name": "Bottom", "type": ["null", "Bottom"], "default": null},
{"name": "MiddleId", "type": ["null", "string"], "default": null},
{"name": "MiddleCode", "type": ["null", "string"], "default": null},
{"name": "MyFoo", "type": ["null", "Foo"], "default": null}
]
}
$ cat src/main/resources/avro/import/Foo.avsc
{
"namespace": "quux00.avro",
"type": "record",
"name": "Foo",
"fields": [
{"name": "FooId", "type": ["null", "string"], "default": null},
{"name": "FooCode", "type": ["null", "string"], "default": null},
{"name": "BottomObj", "type": ["null", "Bottom"], "default": null}
]
}
$ cat src/main/resources/avro/import/Bottom.avsc
{
"namespace": "quux00.avro",
"type": "record",
"name": "Bottom",
"fields": [
{"name": "BottomId", "type": ["null", "string"], "default": null},
{"name": "BottomCode", "type": ["null", "string"], "default": null}
]
}
And the pom:
$ cat pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd";>
<modelVersion>4.0.0</modelVersion>
<groupId>quux00</groupId>
<artifactId>imports</artifactId>
<packaging>jar</packaging>
<version>1.0-SNAPSHOT</version>
<name>imports</name>
<url>http://maven.apache.org</url>
<build>
<plugins>
<plugin>
<groupId>org.apache.avro</groupId>
<artifactId>avro-maven-plugin</artifactId>
<version>1.7.7</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>schema</goal>
</goals>
<configuration>
<sourceDirectory>${project.basedir}/src/main/resources/avro/</sourceDirectory>
<outputDirectory>${project.basedir}/src/main/java/</outputDirectory>
<imports>
<import>${basedir}/src/main/resources/avro/import</import>
</imports>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>1.7.7</version>
</dependency>
</dependencies>
</project>
Finally, this doesn't seem to be an issue with the avro-maven-plugin. I see
the same behavior with the avro-tools jar. It works on Windows:
$ ls input/
Bottom.avsc Foo.avsc Middle.avsc Top.avsc
$ java -jar ~/bin/avro-tools-1.7.6.jar compile schema input/ out/
Input files to compile:
input\Bottom.avsc
input\Dog.avsc
input\Middle.avsc
input\Top.avsc
$ tree out/
out
├── quux00
│ ├── avro
│ ├── Bottom.java (6.4 KiB)
│ ├── Foo.java (8.1 KiB)
│ ├── Middle.java (10.1 KiB)
│ ├── Top.java (12.0 KiB)
but fails on Linux:
$ java -jar ~/bin/avro-tools-1.7.6.jar compile schema input/ out/
Input files to compile:
input/Foo.avsc
input/Top.avsc
input/Middle.avsc
input/Bottom.avsc
Exception in thread "main" org.apache.avro.SchemaParseException: Undefined
name: "Bottom"
at org.apache.avro.Schema.parse(Schema.java:1078)
at org.apache.avro.Schema.parse(Schema.java:1188)
at org.apache.avro.Schema.parse(Schema.java:1119)
at org.apache.avro.Schema$Parser.parse(Schema.java:953)
at org.apache.avro.Schema$Parser.parse(Schema.java:922)
at
org.apache.avro.tool.SpecificCompilerTool.run(SpecificCompilerTool.java:73)
at org.apache.avro.tool.Main.run(Main.java:84)
at org.apache.avro.tool.Main.main(Main.java:73)
Thank you,
Michael
