[
https://issues.apache.org/jira/browse/AVRO-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Austin Cawley-Edwards updated AVRO-2644:
----------------------------------------
Description:
{color:#222222}We're trying to use the `compile \{src dir} \{output dir}`
command in{color}
{color:#222222}`avro-tools` and finding that there are some
non-deterministic{color}
{color:#222222}behaviors between systems, depending on how the OS sorts
files.{color}
{color:#222222}Example:{color}
{color:#222222}schemas/Component.avsc{color}
{color:#222222} - defines Component record type in the namespace
`com.test`{color}
{color:#222222}schemas/Parent.avsc{color}
{color:#222222} - defines a Parent record, in the same `com.test` namespace,
with a{color}
{color:#222222}field of type `com.test.Component`{color}
{color:#222222}With the command, `java -jar avro-tools-1.9.1.jar compile
schemas/{color}
{color:#222222}out-dir/`, some systems compile the directory in the order
Component,{color}
{color:#222222}Parent while others compile in the order Parent, Component. The
latter{color}
{color:#222222}fails as Component has not been defined when it is referenced
by{color}
{color:#222222}Parent.{color}
{color:#222222}We have also tried using the IDL and importing the dependency
types,{color}
{color:#222222}and then converting them to avsc, and finally compiling the
entire{color}
{color:#222222}directory, but that fails as the generated avsc files embed/
duplicate{color}
{color:#222222}the "Component" types each time it is used.{color}
{color:#222222}OS:{color}
{color:#222222}Linux 857aaf92e059 4.15.0-70-generic #79-Ubuntu SMP Tue Nov
12{color}
{color:#222222}10:36:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux{color}
{color:#222222}Avro:{color}
{color:#222222}version 1.9.1{color}
Would a PR be accepted that enforces LANG=C semantics or would that have to be
shipped as a breaking change?
Coming from this thread in the mailing list:
[http://mail-archives.apache.org/mod_mbox/avro-user/201911.mbox/%3CCALGL%2BUDH03pCyKAQ5a%2B_fvwnUVougwwEXe8%2BHFAuR8Q%3D2cqYmw%40mail.gmail.com%3E]
was:
{color:#222222}We're trying to use the `compile \{src dir} \{output dir}`
command in{color}
{color:#222222}`avro-tools` and finding that there are some
non-deterministic{color}
{color:#222222}behaviors between systems, depending on how the OS sorts
files.{color}
{color:#222222}Example:{color}
{color:#222222}schemas/Component.avsc{color}
{color:#222222} - defines Component record type in the namespace
`com.test`{color}
{color:#222222}schemas/Parent.avsc{color}
{color:#222222} - defines a Parent record, in the same `com.test` namespace,
with a{color}
{color:#222222}field of type `com.test.Component`{color}
{color:#222222}With the command, `java -jar avro-tools-1.9.1.jar compile
schemas/{color}
{color:#222222}out-dir/`, some systems compile the directory in the order
Component,{color}
{color:#222222}Parent while others compile in the order Parent, Component. The
latter{color}
{color:#222222}fails as Component has not been defined when it is referenced
by{color}
{color:#222222}Parent.{color}
{color:#222222}We have also tried using the IDL and importing the dependency
types,{color}
{color:#222222}and then converting them to avsc, and finally compiling the
entire{color}
{color:#222222}directory, but that fails as the generated avsc files embed/
duplicate{color}
{color:#222222}the "Component" types each time it is used.{color}
{color:#222222}Is there a way to deterministically compile a directory? Or
compile{color}
{color:#222222}directly from IDL to java?{color}
{color:#222222}OS:{color}
{color:#222222}Linux 857aaf92e059 4.15.0-70-generic #79-Ubuntu SMP Tue Nov
12{color}
{color:#222222}10:36:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux{color}
{color:#222222}Avro:{color}
{color:#222222}version 1.9.1{color}
Coming from this thread in the mailing list:
[http://mail-archives.apache.org/mod_mbox/avro-user/201911.mbox/%3CCALGL%2BUDH03pCyKAQ5a%2B_fvwnUVougwwEXe8%2BHFAuR8Q%3D2cqYmw%40mail.gmail.com%3E]
> Non-Deterministic avsc Directory Compilation
> --------------------------------------------
>
> Key: AVRO-2644
> URL: https://issues.apache.org/jira/browse/AVRO-2644
> Project: Apache Avro
> Issue Type: Bug
> Reporter: Austin Cawley-Edwards
> Priority: Minor
>
> {color:#222222}We're trying to use the `compile \{src dir} \{output dir}`
> command in{color}
> {color:#222222}`avro-tools` and finding that there are some
> non-deterministic{color}
> {color:#222222}behaviors between systems, depending on how the OS sorts
> files.{color}
> {color:#222222}Example:{color}
> {color:#222222}schemas/Component.avsc{color}
> {color:#222222} - defines Component record type in the namespace
> `com.test`{color}
> {color:#222222}schemas/Parent.avsc{color}
> {color:#222222} - defines a Parent record, in the same `com.test`
> namespace, with a{color}
> {color:#222222}field of type `com.test.Component`{color}
> {color:#222222}With the command, `java -jar avro-tools-1.9.1.jar compile
> schemas/{color}
> {color:#222222}out-dir/`, some systems compile the directory in the order
> Component,{color}
> {color:#222222}Parent while others compile in the order Parent, Component.
> The latter{color}
> {color:#222222}fails as Component has not been defined when it is referenced
> by{color}
> {color:#222222}Parent.{color}
> {color:#222222}We have also tried using the IDL and importing the dependency
> types,{color}
> {color:#222222}and then converting them to avsc, and finally compiling the
> entire{color}
> {color:#222222}directory, but that fails as the generated avsc files embed/
> duplicate{color}
> {color:#222222}the "Component" types each time it is used.{color}
> {color:#222222}OS:{color}
> {color:#222222}Linux 857aaf92e059 4.15.0-70-generic #79-Ubuntu SMP Tue Nov
> 12{color}
> {color:#222222}10:36:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux{color}
> {color:#222222}Avro:{color}
> {color:#222222}version 1.9.1{color}
>
>
> Would a PR be accepted that enforces LANG=C semantics or would that have to
> be shipped as a breaking change?
>
> Coming from this thread in the mailing list:
> [http://mail-archives.apache.org/mod_mbox/avro-user/201911.mbox/%3CCALGL%2BUDH03pCyKAQ5a%2B_fvwnUVougwwEXe8%2BHFAuR8Q%3D2cqYmw%40mail.gmail.com%3E]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
