I think it works on Windows because the directory there is listed in an order that happens to work, whereas in Linux the directory ordering doesn't happen to work. We could perhaps always sort directory listings alphabetically so this is deterministic, but requiring folks to create schema names so that their import order corresponds to alphabetical order doesn't seem ideal.
Order matters because Avro doesn't support forward declarations. When includes are interdependent, specifying includes as directories is error prone. For independent includes, an import directory is reasonable. We should better document this. Where should we add such documentation? Adding support for forward declarations is a topic that has been discussed a bit previously: http://s.apache.org/OKV Forward declarations might be implemented for the schema parser used by the compiler, but the parser used when parsing schemas from data files, RPC handshakes, etc. should probably not permit forward declarations. Doug On Mon, Aug 4, 2014 at 12:28 PM, Peterson, Michael <[email protected]> wrote: > Hello, > > > > We recently refactored our large avro schema into separate schemas to make > them more composable, since many of them get incorporated into other > objects. We were glad to see that the avro-maven-plugin allows this > composable model rather than having to do one large schema with everything > embedded. > > > > However, we have hit a problem – the Parser cannot always resolve the > cross-references. I upgraded to avro-1.7.7 (both the core avro and the > avro-maven-plugin project), but that does not solve the issue. > > > > The problem is that names/schemas are not always resolved. Worse, we see > different behavior on Windows vs. Linux. Below I show a set up with a dummy > schema that works on Windows 7, but fails on Linux (tested on Centos 6 and > Ubuntu 14.04). We also have a more complicated schema with many objects > which have a number of cross references (but not circular ones) that fails > on both Windows and Linux. > > > > Is this behavior a defect? Should the Schema.Parser be able to resolve > these cross-dependencies? > > > > A larger question is why we need to put things in two directories (top level > and “imports”). Couldn’t the Parser be made to resolve references of things > when they are all in the same directory? > > > > > > > > Here is a detailed example – again this one works on Windows but fails on > Linux: > > > > u070072@TST imports$ tree > > imports > > ├── pom.xml (1.4 KiB) > > ├── src > > │ ├── main > > │ ├── java > > │ ├── quux00 > > │ ├── App.java (182 B) > > │ ├── resources > > │ ├── avro > > │ ├── Top.avsc (473 B) > > │ ├── import > > │ ├── Bottom.avsc (239 B) > > │ ├── Foo.avsc (307 B) > > │ ├── Middle.avsc (393 B) > > > > > > $ mvn -X generate-sources > > [INFO] BUILD FAILURE > > [INFO] > ------------------------------------------------------------------------ > > [INFO] Total time: 0.786 s > > [INFO] Finished at: 2014-08-04T15:06:02-05:00 > > [INFO] Final Memory: 19M/843M > > [INFO] > ------------------------------------------------------------------------ > > [ERROR] Failed to execute goal > org.apache.avro:avro-maven-plugin:1.7.7:schema (default) on project imports: > Execution default of goal org.apache.avro:avro-maven-plugin:1.7.7:schema > failed: Undefined name: "Bottom" -> [Help 1] > > org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute > goal org.apache.avro:avro-maven-plugin:1.7.7:schema (default) on project > imports: Execution default of goal > org.apache.avro:avro-maven-plugin:1.7.7:schema failed: Undefined name: > "Bottom" > > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:224) > > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) > > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) > > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108) > > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:76) > > at > org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) > > at > org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:116) > > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:361) > > at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155) > > at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584) > > at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:213) > > at org.apache.maven.cli.MavenCli.main(MavenCli.java:157) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) > > at > org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) > > at > org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) > > at > org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) > > Caused by: org.apache.maven.plugin.PluginExecutionException: Execution > default of goal org.apache.avro:avro-maven-plugin:1.7.7:schema failed: > Undefined name: "Bottom" > > at > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:144) > > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) > > ... 19 more > > Caused by: org.apache.avro.SchemaParseException: Undefined name: "Bottom" > > at org.apache.avro.Schema.parse(Schema.java:1162) > > at org.apache.avro.Schema.parse(Schema.java:1272) > > at org.apache.avro.Schema.parse(Schema.java:1203) > > at org.apache.avro.Schema$Parser.parse(Schema.java:965) > > at org.apache.avro.Schema$Parser.parse(Schema.java:932) > > at org.apache.avro.mojo.SchemaMojo.doCompile(SchemaMojo.java:73) > > at > org.apache.avro.mojo.AbstractAvroMojo.compileFiles(AbstractAvroMojo.java:216) > > at > org.apache.avro.mojo.AbstractAvroMojo.execute(AbstractAvroMojo.java:154) > > at > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:133) > > ... 20 more > > > > > > Here are the schema sections: > > > > $ cat src/main/resources/avro/Top.avsc > > { > > "namespace": "quux00.avro", > > "type": "record", > > "name": "Top", > > "fields": [ > > {"name": "TopId", "type": ["null", "string"], "default": null}, > > {"name": "TopCode", "type": ["null", "string"], "default": null}, > > {"name": "Middles", "type": ["null", {"type": "array", "items": > "Middle"}], "default": null}, > > {"name": "Bottom", "type": ["null", "Bottom"], "default": null}, > > {"name": "AFoo", "type": ["null", "Foo"], "default": null} > > ] > > } > > > > $ cat src/main/resources/avro/import/Middle.avsc > > { > > "namespace": "quux00.avro", > > "type": "record", > > "name": "Middle", > > "fields": [ > > {"name": "Bottom", "type": ["null", "Bottom"], "default": null}, > > {"name": "MiddleId", "type": ["null", "string"], "default": null}, > > {"name": "MiddleCode", "type": ["null", "string"], "default": null}, > > {"name": "MyFoo", "type": ["null", "Foo"], "default": null} > > ] > > } > > > > $ cat src/main/resources/avro/import/Foo.avsc > > { > > "namespace": "quux00.avro", > > "type": "record", > > "name": "Foo", > > "fields": [ > > {"name": "FooId", "type": ["null", "string"], "default": null}, > > {"name": "FooCode", "type": ["null", "string"], "default": null}, > > {"name": "BottomObj", "type": ["null", "Bottom"], "default": null} > > ] > > } > > > > $ cat src/main/resources/avro/import/Bottom.avsc > > { > > "namespace": "quux00.avro", > > "type": "record", > > "name": "Bottom", > > "fields": [ > > {"name": "BottomId", "type": ["null", "string"], "default": null}, > > {"name": "BottomCode", "type": ["null", "string"], "default": null} > > ] > > } > > > > > > And the pom: > > > > $ cat pom.xml > > <project xmlns="http://maven.apache.org/POM/4.0.0"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 > http://maven.apache.org/maven-v4_0_0.xsd";> > > <modelVersion>4.0.0</modelVersion> > > <groupId>quux00</groupId> > > <artifactId>imports</artifactId> > > <packaging>jar</packaging> > > <version>1.0-SNAPSHOT</version> > > <name>imports</name> > > <url>http://maven.apache.org</url> > > > > <build> > > <plugins> > > <plugin> > > <groupId>org.apache.avro</groupId> > > <artifactId>avro-maven-plugin</artifactId> > > <version>1.7.7</version> > > <executions> > > <execution> > > <phase>generate-sources</phase> > > <goals> > > <goal>schema</goal> > > </goals> > > <configuration> > > > <sourceDirectory>${project.basedir}/src/main/resources/avro/</sourceDirectory> > > > <outputDirectory>${project.basedir}/src/main/java/</outputDirectory> > > <imports> > > <import>${basedir}/src/main/resources/avro/import</import> > > </imports> > > </configuration> > > </execution> > > </executions> > > </plugin> > > </plugins> > > </build> > > > > <dependencies> > > <dependency> > > <groupId>org.apache.avro</groupId> > > <artifactId>avro</artifactId> > > <version>1.7.7</version> > > </dependency> > > </dependencies> > > </project> > > > > > > > > > > Finally, this doesn’t seem to be an issue with the avro-maven-plugin. I see > the same behavior with the avro-tools jar. It works on Windows: > > > > $ ls input/ > > Bottom.avsc Foo.avsc Middle.avsc Top.avsc > > > > $ java -jar ~/bin/avro-tools-1.7.6.jar compile schema input/ out/ > > Input files to compile: > > input\Bottom.avsc > > input\Dog.avsc > > input\Middle.avsc > > input\Top.avsc > > > > $ tree out/ > > out > > ├── quux00 > > │ ├── avro > > │ ├── Bottom.java (6.4 KiB) > > │ ├── Foo.java (8.1 KiB) > > │ ├── Middle.java (10.1 KiB) > > │ ├── Top.java (12.0 KiB) > > > > > > > > but fails on Linux: > > > > $ java -jar ~/bin/avro-tools-1.7.6.jar compile schema input/ out/ > > Input files to compile: > > input/Foo.avsc > > input/Top.avsc > > input/Middle.avsc > > input/Bottom.avsc > > Exception in thread "main" org.apache.avro.SchemaParseException: Undefined > name: "Bottom" > > at org.apache.avro.Schema.parse(Schema.java:1078) > > at org.apache.avro.Schema.parse(Schema.java:1188) > > at org.apache.avro.Schema.parse(Schema.java:1119) > > at org.apache.avro.Schema$Parser.parse(Schema.java:953) > > at org.apache.avro.Schema$Parser.parse(Schema.java:922) > > at > org.apache.avro.tool.SpecificCompilerTool.run(SpecificCompilerTool.java:73) > > at org.apache.avro.tool.Main.run(Main.java:84) > > at org.apache.avro.tool.Main.main(Main.java:73) > > > > > > > > > > > > Thank you, > > Michael > >
