You will probably want to explicitly define the order of the imports. I
have something like this:
<plugin>
<groupId>org.apache.avro</groupId>
<artifactId>avro-maven-plugin</artifactId>
<version>${avro.version}</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>schema</goal>
</goals>
<configuration>
<imports>
<import>${project.basedir}/src/main/avro/imports/MySchema1.avsc</import>
<import>${project.basedir}/src/main/avro/imports/MySchema2.avsc</import>
<import>${project.basedir}/src/main/avro/imports/MySchema3.avsc</import>
</imports>
<sourceDirectory>${project.basedir}/src/main/avro/src</sourceDirectory>
<outputDirectory>${project.basedir}/src/main/java/</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
-a
On Mon, Aug 4, 2014 at 12:28 PM, Peterson, Michael <
[email protected]> wrote:
> Hello,
>
>
>
> We recently refactored our large avro schema into separate schemas to make
> them more composable, since many of them get incorporated into other
> objects. We were glad to see that the avro-maven-plugin allows this
> composable model rather than having to do one large schema with everything
> embedded.
>
>
>
> However, we have hit a problem – the Parser cannot always resolve the
> cross-references. I upgraded to avro-1.7.7 (both the core avro and the
> avro-maven-plugin project), but that does not solve the issue.
>
>
>
> The problem is that names/schemas are not always resolved. Worse, we see
> different behavior on Windows vs. Linux. Below I show a set up with a
> dummy schema that works on Windows 7, but fails on Linux (tested on Centos
> 6 and Ubuntu 14.04). We also have a more complicated schema with many
> objects which have a number of cross references (but not circular ones)
> that fails on both Windows and Linux.
>
>
>
> Is this behavior a defect? Should the Schema.Parser be able to resolve
> these cross-dependencies?
>
>
>
> A larger question is why we need to put things in two directories (top
> level and “imports”). Couldn’t the Parser be made to resolve references of
> things when they are all in the same directory?
>
>
>
>
>
>
>
> Here is a detailed example – again this one works on Windows but fails on
> Linux:
>
>
>
> u070072@TST imports$ tree
>
> imports
>
> ├── pom.xml (1.4 KiB)
>
> ├── src
>
> │ ├── main
>
> │ ├── java
>
> │ ├── quux00
>
> │ ├── App.java (182 B)
>
> │ ├── resources
>
> │ ├── avro
>
> │ ├── Top.avsc (473 B)
>
> │ ├── import
>
> │ ├── Bottom.avsc (239 B)
>
> │ ├── Foo.avsc (307 B)
>
> │ ├── Middle.avsc (393 B)
>
>
>
>
>
> $ mvn -X generate-sources
>
> [INFO] BUILD FAILURE
>
> [INFO]
> ------------------------------------------------------------------------
>
> [INFO] Total time: 0.786 s
>
> [INFO] Finished at: 2014-08-04T15:06:02-05:00
>
> [INFO] Final Memory: 19M/843M
>
> [INFO]
> ------------------------------------------------------------------------
>
> [ERROR] Failed to execute goal
> org.apache.avro:avro-maven-plugin:1.7.7:schema (default) on project
> imports: Execution default of goal
> org.apache.avro:avro-maven-plugin:1.7.7:schema failed: Undefined name:
> "Bottom" -> [Help 1]
>
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.apache.avro:avro-maven-plugin:1.7.7:schema (default) on project
> imports: Execution default of goal
> org.apache.avro:avro-maven-plugin:1.7.7:schema failed: Undefined name:
> "Bottom"
>
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:224)
>
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108)
>
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:76)
>
> at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:116)
>
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:361)
>
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
>
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
>
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:213)
>
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:157)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
>
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
>
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
>
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
>
> Caused by: org.apache.maven.plugin.PluginExecutionException: Execution
> default of goal org.apache.avro:avro-maven-plugin:1.7.7:schema failed:
> *Undefined
> name: "Bottom"*
>
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:144)
>
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
>
> ... 19 more
>
> Caused by: org.apache.avro.SchemaParseException: Undefined name: "Bottom"
>
> at org.apache.avro.Schema.parse(Schema.java:1162)
>
> at org.apache.avro.Schema.parse(Schema.java:1272)
>
> at org.apache.avro.Schema.parse(Schema.java:1203)
>
> at org.apache.avro.Schema$Parser.parse(Schema.java:965)
>
> at org.apache.avro.Schema$Parser.parse(Schema.java:932)
>
> at org.apache.avro.mojo.SchemaMojo.doCompile(SchemaMojo.java:73)
>
> at
> org.apache.avro.mojo.AbstractAvroMojo.compileFiles(AbstractAvroMojo.java:216)
>
> at
> org.apache.avro.mojo.AbstractAvroMojo.execute(AbstractAvroMojo.java:154)
>
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:133)
>
> ... 20 more
>
>
>
>
>
> Here are the schema sections:
>
>
>
> $ cat src/main/resources/avro/Top.avsc
>
> {
>
> "namespace": "quux00.avro",
>
> "type": "record",
>
> "name": "Top",
>
> "fields": [
>
> {"name": "TopId", "type": ["null", "string"], "default": null},
>
> {"name": "TopCode", "type": ["null", "string"], "default": null},
>
> {"name": "Middles", "type": ["null", {"type": "array", "items":
> "Middle"}], "default": null},
>
> {"name": "Bottom", "type": ["null", "Bottom"], "default": null},
>
> {"name": "AFoo", "type": ["null", "Foo"], "default": null}
>
> ]
>
> }
>
>
>
> $ cat src/main/resources/avro/import/Middle.avsc
>
> {
>
> "namespace": "quux00.avro",
>
> "type": "record",
>
> "name": "Middle",
>
> "fields": [
>
> {"name": "Bottom", "type": ["null", "Bottom"], "default": null},
>
> {"name": "MiddleId", "type": ["null", "string"], "default": null},
>
> {"name": "MiddleCode", "type": ["null", "string"], "default": null},
>
> {"name": "MyFoo", "type": ["null", "Foo"], "default": null}
>
> ]
>
> }
>
>
>
> $ cat src/main/resources/avro/import/Foo.avsc
>
> {
>
> "namespace": "quux00.avro",
>
> "type": "record",
>
> "name": "Foo",
>
> "fields": [
>
> {"name": "FooId", "type": ["null", "string"], "default": null},
>
> {"name": "FooCode", "type": ["null", "string"], "default": null},
>
> {"name": "BottomObj", "type": ["null", "Bottom"], "default": null}
>
> ]
>
> }
>
>
>
> $ cat src/main/resources/avro/import/Bottom.avsc
>
> {
>
> "namespace": "quux00.avro",
>
> "type": "record",
>
> "name": "Bottom",
>
> "fields": [
>
> {"name": "BottomId", "type": ["null", "string"], "default": null},
>
> {"name": "BottomCode", "type": ["null", "string"], "default": null}
>
> ]
>
> }
>
>
>
>
>
> And the pom:
>
>
>
> $ cat pom.xml
>
> <project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="
> http://www.w3.org/2001/XMLSchema-instance";
>
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/maven-v4_0_0.xsd";>
>
> <modelVersion>4.0.0</modelVersion>
>
> <groupId>quux00</groupId>
>
> <artifactId>imports</artifactId>
>
> <packaging>jar</packaging>
>
> <version>1.0-SNAPSHOT</version>
>
> <name>imports</name>
>
> <url>http://maven.apache.org</url>
>
>
>
> <build>
>
> <plugins>
>
> <plugin>
>
> <groupId>org.apache.avro</groupId>
>
> <artifactId>avro-maven-plugin</artifactId>
>
> <version>1.7.7</version>
>
> <executions>
>
> <execution>
>
> <phase>generate-sources</phase>
>
> <goals>
>
> <goal>schema</goal>
>
> </goals>
>
> <configuration>
>
>
> <sourceDirectory>${project.basedir}/src/main/resources/avro/</sourceDirectory>
>
>
> <outputDirectory>${project.basedir}/src/main/java/</outputDirectory>
>
> <imports>
>
> <import>${basedir}/src/main/resources/avro/import</import>
>
> </imports>
>
> </configuration>
>
> </execution>
>
> </executions>
>
> </plugin>
>
> </plugins>
>
> </build>
>
>
>
> <dependencies>
>
> <dependency>
>
> <groupId>org.apache.avro</groupId>
>
> <artifactId>avro</artifactId>
>
> <version>1.7.7</version>
>
> </dependency>
>
> </dependencies>
>
> </project>
>
>
>
>
>
>
>
>
>
> Finally, this doesn’t seem to be an issue with the avro-maven-plugin. I
> see the same behavior with the avro-tools jar. It works on Windows:
>
>
>
> $ ls input/
>
> Bottom.avsc Foo.avsc Middle.avsc Top.avsc
>
>
>
> $ java -jar ~/bin/avro-tools-1.7.6.jar compile schema input/ out/
>
> Input files to compile:
>
> input\Bottom.avsc
>
> input\Dog.avsc
>
> input\Middle.avsc
>
> input\Top.avsc
>
>
>
> $ tree out/
>
> out
>
> ├── quux00
>
> │ ├── avro
>
> │ ├── Bottom.java (6.4 KiB)
>
> │ ├── Foo.java (8.1 KiB)
>
> │ ├── Middle.java (10.1 KiB)
>
> │ ├── Top.java (12.0 KiB)
>
>
>
>
>
>
>
> but fails on Linux:
>
>
>
> $ java -jar ~/bin/avro-tools-1.7.6.jar compile schema input/ out/
>
> Input files to compile:
>
> input/Foo.avsc
>
> input/Top.avsc
>
> input/Middle.avsc
>
> input/Bottom.avsc
>
> Exception in thread "main" org.apache.avro.SchemaParseException: Undefined
> name: "Bottom"
>
> at org.apache.avro.Schema.parse(Schema.java:1078)
>
> at org.apache.avro.Schema.parse(Schema.java:1188)
>
> at org.apache.avro.Schema.parse(Schema.java:1119)
>
> at org.apache.avro.Schema$Parser.parse(Schema.java:953)
>
> at org.apache.avro.Schema$Parser.parse(Schema.java:922)
>
> at
> org.apache.avro.tool.SpecificCompilerTool.run(SpecificCompilerTool.java:73)
>
> at org.apache.avro.tool.Main.run(Main.java:84)
>
> at org.apache.avro.tool.Main.main(Main.java:73)
>
>
>
>
>
>
>
>
>
>
>
> Thank you,
>
> Michael
>
>
>
