Re: Avro 1.9.2
Hello! It looks like this has already been done -- the commit was already cherrry-picked into branch-1.9 and should appear in release 1.9.2 (see https://github.com/apache/avro/blob/branch-1.9/lang/java/pom.xml#L42). A few months ago, there was some question about whether the jackson upgrade was accidental (causing a build failure fixed by AVRO-2632). I've updated that JIRA and the Fix Version of AVRO-2587. Best regards, Ryan On Sat, Jan 18, 2020 at 2:25 AM Robert Yokota wrote: > Hi, > > Here's the upgrade to Jackson 2.10.0 that I was hoping to get cherry-picked > to 1.9.2: > > > https://github.com/apache/avro/commit/c50974094d95d1dbd9c8b3bb6c4133a43b44fd03 > > Thanks, > Robert > > On Fri, Jan 17, 2020 at 9:52 AM Robert Yokota wrote: > > > Hi, > > > > Any chance we can get the following upgrade to Jackson 2.10.0 > cherry-picked to 1.9.2? > > > > Thanks, > > > > Robert > > > > > > On Thu, Jan 9, 2020 at 9:48 AM Driesprong, Fokko > wrote: > > > > > > Hi all, > > > > > > I'm working on bumping Apache Avro on the Apache Iceberg project: > > > https://github.com/apache/incubator-iceberg/pull/297, I've discovered > a > > > regression bug. I've managed to create a unit test that catches the > issue: > > > https://github.com/apache/avro/pull/752. The current fix isn't the > correct > > > one. I hope to fix it somewhere this weekend, and afterward, start the > > > release process for 1.9.2. > > > > > > If there is anything that you need to have cherry-picked onto the 1.9 > > > branch, please let me know. Also, if there is anyone who likes to > shepherd > > > the release, or wants to help out, please feel free to let me know :-) > > > > > > Thanks all, > > > > > > Cheers, Fokko > > > > >
[jira] [Resolved] (AVRO-2632) getJsonStringEncoder() method not found on branch-1.9
[
https://issues.apache.org/jira/browse/AVRO-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Skraba resolved AVRO-2632.
---
Resolution: Fixed
> getJsonStringEncoder() method not found on branch-1.9
> -
>
> Key: AVRO-2632
> URL: https://issues.apache.org/jira/browse/AVRO-2632
> Project: Apache Avro
> Issue Type: Bug
> Components: build, java
>Reporter: Brian Lachniet
>Assignee: Brian Lachniet
>Priority: Major
> Fix For: 1.9.2
>
>
> {{branch-1.9}} began failing Travis builds in [Build
> #1382|https://travis-ci.org/apache/avro/builds/607297589] with the following
> errors:
> {code}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/travis/build/apache/avro/lang/java/avro/src/main/java/org/apache/avro/SchemaBuilder.java:[2714,40]
> cannot find symbol
> symbol: method getJsonStringEncoder()
> location: class com.fasterxml.jackson.core.util.BufferRecyclers
> [ERROR]
> /home/travis/build/apache/avro/lang/java/avro/src/main/java/org/apache/avro/SchemaBuilder.java:[2718,40]
> cannot find symbol
> symbol: method getJsonStringEncoder()
> location: class com.fasterxml.jackson.core.util.BufferRecyclers
> {code}
> The Jackson version was bumped [in this set of
> changes|https://github.com/apache/avro/compare/fea6888bc211...f156063edb17].
> This seems like it could be related to the cause of the problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (AVRO-2707) union reading does not use first matching member
Roger created AVRO-2707:
---
Summary: union reading does not use first matching member
Key: AVRO-2707
URL: https://issues.apache.org/jira/browse/AVRO-2707
Project: Apache Avro
Issue Type: Bug
Reporter: Roger
[Here|https://gist.github.com/rogpeppe/5faf971eec51f3dbd316bb71d6d5dccc] is a
JSON representation of a test case that produces unexpected behaviour under the
Java Avro implementation.
The test verifies that the JSON data in inData with encoding inSchema produces
outData when read as outSchema.
The specification says: "The first schema in the reader's union that matches
the selected writer's union schema" which to me says that the output should be
a long, because long is before int in the reader's union and long matches int.
To reproduce on the command line, you'll need a version of avro-tools with
[this PR applied|https://github.com/apache/avro/pull/785], then run:
echo '{"F":{"int":999}}' |
avro fromjson --schema
'{"name":"R","type":"record","fields":[{"name":"F","type":["int","string"],"default":1234}]}'
- |
avro tojson --reader-schema
'{"name":"R","type":"record","fields":[{"name":"F","type":["long","int","string"],"default":1234}]}'
-
This prints:
{"F":{"int":999}}
but I'd expect to see this:
{"F":{"long":999}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AVRO-2587) Upgrade jackson to 2.10.0
[ https://issues.apache.org/jira/browse/AVRO-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Skraba updated AVRO-2587: -- Fix Version/s: 1.9.2 > Upgrade jackson to 2.10.0 > - > > Key: AVRO-2587 > URL: https://issues.apache.org/jira/browse/AVRO-2587 > Project: Apache Avro > Issue Type: Task >Affects Versions: 1.9.0 >Reporter: Ana >Assignee: Ana >Priority: Major > Fix For: 1.10.0, 1.9.2 > > > The new minor version of jackson is available, bumping the version will make > it easier to address future issues and vulnerabilities. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AVRO-2706) union reading does not use first matching member
Roger created AVRO-2706:
---
Summary: union reading does not use first matching member
Key: AVRO-2706
URL: https://issues.apache.org/jira/browse/AVRO-2706
Project: Apache Avro
Issue Type: Improvement
Components: java
Reporter: Roger
[Here|https://gist.github.com/rogpeppe/5faf971eec51f3dbd316bb71d6d5dccc] is a
JSON representation of a test case that fails with the Java implementation of
Avro.
To reproduce on the command line with avro-tools, you'll need a version of
avro-tools with https://github.com/apache/avro/pull/785 committed. I've used
the name "avro" as a synonym for running the avro-tools jar under the Java VM:
echo '{"F":{"int":999}}' |
avro fromjson --schema
'{"name":"R","type":"record","fields":[{"name":"F","type":["int","string"],"default":1234}]}'
- |
avro tojson --reader-schema
'{"name":"R","type":"record","fields":[{"name":"F","type":["long","int","string"],"default":1234}]}'
-
This produces:
{"F":{"int":999}}
whereas I'd expect to see "long", because the first matching item in the union
is "long".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AVRO-2632) getJsonStringEncoder() method not found on branch-1.9
[
https://issues.apache.org/jira/browse/AVRO-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Skraba updated AVRO-2632:
--
Fix Version/s: 1.9.2
> getJsonStringEncoder() method not found on branch-1.9
> -
>
> Key: AVRO-2632
> URL: https://issues.apache.org/jira/browse/AVRO-2632
> Project: Apache Avro
> Issue Type: Bug
> Components: build, java
>Reporter: Brian Lachniet
>Assignee: Brian Lachniet
>Priority: Major
> Fix For: 1.9.2
>
>
> {{branch-1.9}} began failing Travis builds in [Build
> #1382|https://travis-ci.org/apache/avro/builds/607297589] with the following
> errors:
> {code}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/travis/build/apache/avro/lang/java/avro/src/main/java/org/apache/avro/SchemaBuilder.java:[2714,40]
> cannot find symbol
> symbol: method getJsonStringEncoder()
> location: class com.fasterxml.jackson.core.util.BufferRecyclers
> [ERROR]
> /home/travis/build/apache/avro/lang/java/avro/src/main/java/org/apache/avro/SchemaBuilder.java:[2718,40]
> cannot find symbol
> symbol: method getJsonStringEncoder()
> location: class com.fasterxml.jackson.core.util.BufferRecyclers
> {code}
> The Jackson version was bumped [in this set of
> changes|https://github.com/apache/avro/compare/fea6888bc211...f156063edb17].
> This seems like it could be related to the cause of the problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2707) union reading does not use first matching member
[
https://issues.apache.org/jira/browse/AVRO-2707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019442#comment-17019442
]
Roger commented on AVRO-2707:
-
Sorry, I created this duplicate issue by accident because I thought JIRA had
discarded the first one I submitted. I don't seem to be able to delete it, so
please could someone do so in favour of AVRO-2706.
> union reading does not use first matching member
>
>
> Key: AVRO-2707
> URL: https://issues.apache.org/jira/browse/AVRO-2707
> Project: Apache Avro
> Issue Type: Bug
>Reporter: Roger
>Priority: Major
>
> [Here|https://gist.github.com/rogpeppe/5faf971eec51f3dbd316bb71d6d5dccc] is a
> JSON representation of a test case that produces unexpected behaviour under
> the Java Avro implementation.
> The test verifies that the JSON data in inData with encoding inSchema
> produces outData when read as outSchema.
> The specification says: "The first schema in the reader's union that matches
> the selected writer's union schema" which to me says that the output should
> be a long, because long is before int in the reader's union and long matches
> int.
> To reproduce on the command line, you'll need a version of avro-tools with
> [this PR applied|https://github.com/apache/avro/pull/785], then run:
> echo '{"F":{"int":999}}' |
> avro fromjson --schema
> '{"name":"R","type":"record","fields":[{"name":"F","type":["int","string"],"default":1234}]}'
> - |
> avro tojson --reader-schema
> '{"name":"R","type":"record","fields":[{"name":"F","type":["long","int","string"],"default":1234}]}'
> -
> This prints:
> {"F":{"int":999}}
> but I'd expect to see this:
> {"F":{"long":999}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AVRO-2706) union reading does not use first matching member
[
https://issues.apache.org/jira/browse/AVRO-2706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roger updated AVRO-2706:
Priority: Minor (was: Major)
> union reading does not use first matching member
>
>
> Key: AVRO-2706
> URL: https://issues.apache.org/jira/browse/AVRO-2706
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: Roger
>Priority: Minor
>
> [Here|https://gist.github.com/rogpeppe/5faf971eec51f3dbd316bb71d6d5dccc] is a
> JSON representation of a test case that produces unexpected behaviour under
> the Java Avro implementation.
> The test verifies that the JSON data in inData with encoding inSchema
> produces outData when read as outSchema.
> The specification says: "The first schema in the reader's union that matches
> the selected writer's union schema" which to me says that the output should
> be a long, because long is before int in the reader's union and long matches
> int.
> To reproduce on the command line, you'll need a version of avro-tools with
> [this PR applied|https://github.com/apache/avro/pull/785], then run:
> echo '{"F":{"int":999}}' |
> avro fromjson --schema
> '{"name":"R","type":"record","fields":[{"name":"F","type":["int","string"],"default":1234}]}'
> - |
> avro tojson --reader-schema
> '{"name":"R","type":"record","fields":[{"name":"F","type":["long","int","string"],"default":1234}]}'
> -
> This prints:
> {"F":{"int":999}}
> but I'd expect to see this:
> {"F":{"long":999}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AVRO-2706) union reading does not use first matching member
[
https://issues.apache.org/jira/browse/AVRO-2706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roger updated AVRO-2706:
Description:
[Here|https://gist.github.com/rogpeppe/5faf971eec51f3dbd316bb71d6d5dccc] is a
JSON representation of a test case that produces unexpected behaviour under the
Java Avro implementation.
The test verifies that the JSON data in inData with encoding inSchema produces
outData when read as outSchema.
The specification says: "The first schema in the reader's union that matches
the selected writer's union schema" which to me says that the output should be
a long, because long is before int in the reader's union and long matches int.
To reproduce on the command line, you'll need a version of avro-tools with
[this PR applied|https://github.com/apache/avro/pull/785], then run:
echo '{"F":{"int":999}}' |
avro fromjson --schema
'{"name":"R","type":"record","fields":[{"name":"F","type":["int","string"],"default":1234}]}'
- |
avro tojson --reader-schema
'{"name":"R","type":"record","fields":[{"name":"F","type":["long","int","string"],"default":1234}]}'
-
This prints:
{"F":{"int":999}}
but I'd expect to see this:
{"F":{"long":999}}
was:
[Here|https://gist.github.com/rogpeppe/5faf971eec51f3dbd316bb71d6d5dccc] is a
JSON representation of a test case that fails with the Java implementation of
Avro.
To reproduce on the command line with avro-tools, you'll need a version of
avro-tools with https://github.com/apache/avro/pull/785 committed. I've used
the name "avro" as a synonym for running the avro-tools jar under the Java VM:
echo '{"F":{"int":999}}' |
avro fromjson --schema
'{"name":"R","type":"record","fields":[{"name":"F","type":["int","string"],"default":1234}]}'
- |
avro tojson --reader-schema
'{"name":"R","type":"record","fields":[{"name":"F","type":["long","int","string"],"default":1234}]}'
-
This produces:
{"F":{"int":999}}
whereas I'd expect to see "long", because the first matching item in the union
is "long".
> union reading does not use first matching member
>
>
> Key: AVRO-2706
> URL: https://issues.apache.org/jira/browse/AVRO-2706
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: Roger
>Priority: Major
>
> [Here|https://gist.github.com/rogpeppe/5faf971eec51f3dbd316bb71d6d5dccc] is a
> JSON representation of a test case that produces unexpected behaviour under
> the Java Avro implementation.
> The test verifies that the JSON data in inData with encoding inSchema
> produces outData when read as outSchema.
> The specification says: "The first schema in the reader's union that matches
> the selected writer's union schema" which to me says that the output should
> be a long, because long is before int in the reader's union and long matches
> int.
> To reproduce on the command line, you'll need a version of avro-tools with
> [this PR applied|https://github.com/apache/avro/pull/785], then run:
> echo '{"F":{"int":999}}' |
> avro fromjson --schema
> '{"name":"R","type":"record","fields":[{"name":"F","type":["int","string"],"default":1234}]}'
> - |
> avro tojson --reader-schema
> '{"name":"R","type":"record","fields":[{"name":"F","type":["long","int","string"],"default":1234}]}'
> -
> This prints:
> {"F":{"int":999}}
> but I'd expect to see this:
> {"F":{"long":999}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AVRO-2689) Avro-tools should have a way to test schema evolution.
[
https://issues.apache.org/jira/browse/AVRO-2689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Skraba updated AVRO-2689:
--
Fix Version/s: 1.10.0
> Avro-tools should have a way to test schema evolution.
> --
>
> Key: AVRO-2689
> URL: https://issues.apache.org/jira/browse/AVRO-2689
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: Ryan Skraba
>Priority: Major
> Fix For: 1.10.0
>
>
> The avro-tools command line should have at least one way to test
> reading/writing with an evolved schema. This would be useful for testing
> schema evolution and reproducing errors or bug reports from the command line.
> A good candidate would be to add a {{--reader-schema}} option to the
> {{tojson}} tool (where the writer schema comes from the data file).
> Alternatively, the {{fragtojson}} already takes the writer schema and we
> could add {{--reader-schema-file}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2706) union reading does not use first matching member
[
https://issues.apache.org/jira/browse/AVRO-2706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019539#comment-17019539
]
Ryan Skraba commented on AVRO-2706:
---
I agree – the behaviour here is not matching the spec, which says that a
"matching" schema can include primitive promotion (i.e. {{"long"}} *does* match
{{"int"}} and it *is* the first in the list).
> union reading does not use first matching member
>
>
> Key: AVRO-2706
> URL: https://issues.apache.org/jira/browse/AVRO-2706
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: Roger
>Priority: Minor
>
> [Here|https://gist.github.com/rogpeppe/5faf971eec51f3dbd316bb71d6d5dccc] is a
> JSON representation of a test case that produces unexpected behaviour under
> the Java Avro implementation.
> The test verifies that the JSON data in inData with encoding inSchema
> produces outData when read as outSchema.
> The specification says: "The first schema in the reader's union that matches
> the selected writer's union schema" which to me says that the output should
> be a long, because long is before int in the reader's union and long matches
> int.
> To reproduce on the command line, you'll need a version of avro-tools with
> [this PR applied|https://github.com/apache/avro/pull/785], then run:
> echo '{"F":{"int":999}}' |
> avro fromjson --schema
> '{"name":"R","type":"record","fields":[{"name":"F","type":["int","string"],"default":1234}]}'
> - |
> avro tojson --reader-schema
> '{"name":"R","type":"record","fields":[{"name":"F","type":["long","int","string"],"default":1234}]}'
> -
> This prints:
> {"F":{"int":999}}
> but I'd expect to see this:
> {"F":{"long":999}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-2707) union reading does not use first matching member
[
https://issues.apache.org/jira/browse/AVRO-2707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Skraba resolved AVRO-2707.
---
Resolution: Duplicate
> union reading does not use first matching member
>
>
> Key: AVRO-2707
> URL: https://issues.apache.org/jira/browse/AVRO-2707
> Project: Apache Avro
> Issue Type: Bug
>Reporter: Roger
>Priority: Major
>
> [Here|https://gist.github.com/rogpeppe/5faf971eec51f3dbd316bb71d6d5dccc] is a
> JSON representation of a test case that produces unexpected behaviour under
> the Java Avro implementation.
> The test verifies that the JSON data in inData with encoding inSchema
> produces outData when read as outSchema.
> The specification says: "The first schema in the reader's union that matches
> the selected writer's union schema" which to me says that the output should
> be a long, because long is before int in the reader's union and long matches
> int.
> To reproduce on the command line, you'll need a version of avro-tools with
> [this PR applied|https://github.com/apache/avro/pull/785], then run:
> echo '{"F":{"int":999}}' |
> avro fromjson --schema
> '{"name":"R","type":"record","fields":[{"name":"F","type":["int","string"],"default":1234}]}'
> - |
> avro tojson --reader-schema
> '{"name":"R","type":"record","fields":[{"name":"F","type":["long","int","string"],"default":1234}]}'
> -
> This prints:
> {"F":{"int":999}}
> but I'd expect to see this:
> {"F":{"long":999}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (AVRO-2689) Avro-tools should have a way to test schema evolution.
[
https://issues.apache.org/jira/browse/AVRO-2689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Skraba reassigned AVRO-2689:
-
Assignee: Roger
> Avro-tools should have a way to test schema evolution.
> --
>
> Key: AVRO-2689
> URL: https://issues.apache.org/jira/browse/AVRO-2689
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: Ryan Skraba
>Assignee: Roger
>Priority: Major
> Fix For: 1.10.0
>
>
> The avro-tools command line should have at least one way to test
> reading/writing with an evolved schema. This would be useful for testing
> schema evolution and reproducing errors or bug reports from the command line.
> A good candidate would be to add a {{--reader-schema}} option to the
> {{tojson}} tool (where the writer schema comes from the data file).
> Alternatively, the {{fragtojson}} already takes the writer schema and we
> could add {{--reader-schema-file}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (AVRO-2708) Remove Unused Code
David Mollitor created AVRO-2708: Summary: Remove Unused Code Key: AVRO-2708 URL: https://issues.apache.org/jira/browse/AVRO-2708 Project: Apache Avro Issue Type: Improvement Reporter: David Mollitor Assignee: David Mollitor * Remove unused imports * Remove unused private methods * Remove unused private constructors * Remove unused private types * Remove unused private fields * Remove unused local variables -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AVRO-2709) Remove Superfluous Dependencies from Avro-Tools Shaded JAR
David Mollitor created AVRO-2709: Summary: Remove Superfluous Dependencies from Avro-Tools Shaded JAR Key: AVRO-2709 URL: https://issues.apache.org/jira/browse/AVRO-2709 Project: Apache Avro Issue Type: Improvement Reporter: David Mollitor Assignee: David Mollitor * Including org.apache.zookeeper:zookeeper:jar:3.4.6 in the shaded jar. * Including org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 in the shaded jar. * Including org.apache.curator:curator-client:jar:2.7.1 in the shaded jar * Including org.apache.curator:curator-recipes:jar:2.7.1 in the shaded jar Some weird stuff to be included in a CLI that doesn't talk to ZK and doesn't create a LevelDB database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
