Re: Including "fastavro" in release 1.9.2
unless the RM has an objection based on where they are in the RC creation process, I say go for it. (and if there is such an issue then let's get it ready for 1.9.3) On Fri, Jan 31, 2020 at 10:29 AM Ryan Skraba wrote: > > Hello! There's a long standing PR[1][2] that provides an impressive > performance boost to deserializing Avro. > > The author has been patient and reactive for a LONG time (thanks Martin!), > and it's inspired some really interesting discussion. > > I'd like to merge and cherry-pick into release 1.9.2 -- as long as it's > turned off there shouldn't be any visible effect at all, and having it in a > release would be a great way to find regressions (if any) and encourage > some of the ideas brought up in the PR discussions. > > Any objections? Is this too big a change for a minor release? > > I noted in the PR that it would be pretty cool if we had a "better" way of > noting experimental features (and that the Yetus annotations would be a > good start for 1.10!) > > All my best, Ryan > > [1]: https://issues.apache.org/jira/browse/AVRO-2247 > [2]: https://github.com/apache/avro/pull/391
Re: Including "fastavro" in release 1.9.2
Yes, it's time to get this in. On Fri, Jan 31, 2020 at 8:28 AM Ryan Skraba wrote: > > Hello! There's a long standing PR[1][2] that provides an impressive > performance boost to deserializing Avro. > > The author has been patient and reactive for a LONG time (thanks Martin!), > and it's inspired some really interesting discussion. > > I'd like to merge and cherry-pick into release 1.9.2 -- as long as it's > turned off there shouldn't be any visible effect at all, and having it in a > release would be a great way to find regressions (if any) and encourage > some of the ideas brought up in the PR discussions. > > Any objections? Is this too big a change for a minor release? > > I noted in the PR that it would be pretty cool if we had a "better" way of > noting experimental features (and that the Yetus annotations would be a > good start for 1.10!) > > All my best, Ryan > > [1]: https://issues.apache.org/jira/browse/AVRO-2247 > [2]: https://github.com/apache/avro/pull/391
[jira] [Commented] (AVRO-2247) Improve Java reading performance with a new reader
[ https://issues.apache.org/jira/browse/AVRO-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027873#comment-17027873 ] Raymie Stata commented on AVRO-2247: I'm in favor. > Improve Java reading performance with a new reader > -- > > Key: AVRO-2247 > URL: https://issues.apache.org/jira/browse/AVRO-2247 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Martin Jubelgas >Priority: Major > Fix For: 1.10.0 > > Attachments: Perf-Comparison.md > > > Complementary to AVRO-2090, I have been working on decoding of Avro objects > in Java and am suggesting a new implementation of a DatumReader that improves > read performance for both generic and specific records by approximately 20% > (and even more in cases of nested objects with defaults, a case I encounter a > lot in practical use). > Key concept is to create a detailed execution plan once at DatumReader. This > execution plan contains all required defaulting/lookup values so they need > not be looked up during object traversal while reading. > The reader implementation can be enabled and disabled per GenericData > instance. The system default is set via the system variable > "org.apache.avro.fastread" (defaults to "false"). > Attached a performance comparison of the existing implementation with the > proposed one. Will open a pull request with respective code in a bit (not > including interoperability with the optimizations of AVRO-2090 yet). Please > let me know your opinion of whether this is worth pursuing further. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AVRO-2057) JsonDecoder.skipChildren does not skip map/records correctly
[
https://issues.apache.org/jira/browse/AVRO-2057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Farkas resolved AVRO-2057.
-
Resolution: Implemented
Looks like this was resolved by another chamge
> JsonDecoder.skipChildren does not skip map/records correctly
>
>
> Key: AVRO-2057
> URL: https://issues.apache.org/jira/browse/AVRO-2057
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.2
>Reporter: Zoltan Farkas
>Priority: Critical
>
> at
> https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/io/JsonDecoder.java#L585
> {code}
> @Override
> public JsonParser skipChildren() throws IOException {
> JsonToken tkn = elements.get(pos).token;
> int level = (tkn == JsonToken.START_ARRAY || tkn ==
> JsonToken.END_ARRAY) ? 1 : 0;
> while (level > 0) {
> switch(elements.get(++pos).token) {
> case START_ARRAY:
> case START_OBJECT:
> level++;
> break;
> case END_ARRAY:
> case END_OBJECT:
> level--;
> break;
> }
> }
> return this;
> }
> {code}
> should be:
> {code}
> @Override
> public JsonParser skipChildren() throws IOException {
> JsonToken tkn = elements.get(pos).token;
> int level = (tkn == JsonToken.START_ARRAY || tkn ==
> JsonToken.START_OBJECT) ? 1 : 0;
> while (level > 0) {
> switch(elements.get(++pos).token) {
> case START_ARRAY:
> case START_OBJECT:
> level++;
> break;
> case END_ARRAY:
> case END_OBJECT:
> level--;
> break;
> }
> }
> return this;
> }
> {code}
> This results in de-serialization failures when the reader schema does not
> have fields that are present in the serialized object and the writer schema.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Including "fastavro" in release 1.9.2
Hello! There's a long standing PR[1][2] that provides an impressive performance boost to deserializing Avro. The author has been patient and reactive for a LONG time (thanks Martin!), and it's inspired some really interesting discussion. I'd like to merge and cherry-pick into release 1.9.2 -- as long as it's turned off there shouldn't be any visible effect at all, and having it in a release would be a great way to find regressions (if any) and encourage some of the ideas brought up in the PR discussions. Any objections? Is this too big a change for a minor release? I noted in the PR that it would be pretty cool if we had a "better" way of noting experimental features (and that the Yetus annotations would be a good start for 1.10!) All my best, Ryan [1]: https://issues.apache.org/jira/browse/AVRO-2247 [2]: https://github.com/apache/avro/pull/391
[jira] [Commented] (AVRO-2680) Bump grpc.version from 1.24.1 to 1.26.0
[ https://issues.apache.org/jira/browse/AVRO-2680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027637#comment-17027637 ] ASF subversion and git services commented on AVRO-2680: --- Commit 10a1d365979fe28b161b01ee2d53cbb0b408c0b8 in avro's branch refs/heads/branch-1.9 from Fokko Driesprong [ https://gitbox.apache.org/repos/asf?p=avro.git;h=10a1d36 ] AVRO-2680: Bump grpc.version from 1.24.1 to 1.26.0 (#767) > Bump grpc.version from 1.24.1 to 1.26.0 > --- > > Key: AVRO-2680 > URL: https://issues.apache.org/jira/browse/AVRO-2680 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.9.1 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-2685) Bump apache from 21 to 22
[ https://issues.apache.org/jira/browse/AVRO-2685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027634#comment-17027634 ] ASF subversion and git services commented on AVRO-2685: --- Commit fba35ae9a95484afb84fde7bd7a1d64d22ccfae3 in avro's branch refs/heads/branch-1.9 from Fokko Driesprong [ https://gitbox.apache.org/repos/asf?p=avro.git;h=fba35ae ] AVRO-2685: Bump apache from 21 to 22 (#768) The Maven parent project, provided by the ASF > Bump apache from 21 to 22 > - > > Key: AVRO-2685 > URL: https://issues.apache.org/jira/browse/AVRO-2685 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.9.1 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.0 > > > Bump parent project to 22 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-2540) Bump commons-lang3 from 3.8.1 to 3.9
[ https://issues.apache.org/jira/browse/AVRO-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027629#comment-17027629 ] ASF subversion and git services commented on AVRO-2540: --- Commit 437fbc9ee1b9c54d05fbadff40c676b9a7b52e7a in avro's branch refs/heads/branch-1.9 from Fokko Driesprong [ https://gitbox.apache.org/repos/asf?p=avro.git;h=437fbc9 ] AVRO-2540: Bump commons-lang3 from 3.8.1 to 3.9.0 (#633) * AVRO-2540: Bump commons-lang3 from 3.8.1 to 3.9 * Fix version > Bump commons-lang3 from 3.8.1 to 3.9 > > > Key: AVRO-2540 > URL: https://issues.apache.org/jira/browse/AVRO-2540 > Project: Apache Avro > Issue Type: Task >Affects Versions: 1.9.1 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-2540) Bump commons-lang3 from 3.8.1 to 3.9
[ https://issues.apache.org/jira/browse/AVRO-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027628#comment-17027628 ] ASF subversion and git services commented on AVRO-2540: --- Commit 437fbc9ee1b9c54d05fbadff40c676b9a7b52e7a in avro's branch refs/heads/branch-1.9 from Fokko Driesprong [ https://gitbox.apache.org/repos/asf?p=avro.git;h=437fbc9 ] AVRO-2540: Bump commons-lang3 from 3.8.1 to 3.9.0 (#633) * AVRO-2540: Bump commons-lang3 from 3.8.1 to 3.9 * Fix version > Bump commons-lang3 from 3.8.1 to 3.9 > > > Key: AVRO-2540 > URL: https://issues.apache.org/jira/browse/AVRO-2540 > Project: Apache Avro > Issue Type: Task >Affects Versions: 1.9.1 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-2543) Bump grpc.version from 1.19.0 to 1.23.0
[ https://issues.apache.org/jira/browse/AVRO-2543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027636#comment-17027636 ] ASF subversion and git services commented on AVRO-2543: --- Commit 1219f3e5e3904a02d3969ff8c0cd1cb485adf10e in avro's branch refs/heads/branch-1.9 from Fokko Driesprong [ https://gitbox.apache.org/repos/asf?p=avro.git;h=1219f3e ] AVRO-2543: Bump grpc.version from 1.19.0 to 1.23.0 (#635) * AVRO-2543: Bump grpc.version from 1.19.0 to 1.23.0 * Import it from Guava instead > Bump grpc.version from 1.19.0 to 1.23.0 > --- > > Key: AVRO-2543 > URL: https://issues.apache.org/jira/browse/AVRO-2543 > Project: Apache Avro > Issue Type: Task > Components: dependencies >Affects Versions: 1.9.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-2686) Bump jetty.version from 9.4.21.v20190926 to 9.4.25.v20191220
[ https://issues.apache.org/jira/browse/AVRO-2686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027632#comment-17027632 ] ASF subversion and git services commented on AVRO-2686: --- Commit 28e79218d0baf16f9cbee0786f89bf7b478998c0 in avro's branch refs/heads/branch-1.9 from Fokko Driesprong [ https://gitbox.apache.org/repos/asf?p=avro.git;h=28e7921 ] AVRO-2686: Bump jetty.version from 9.4.21.v20190926 to 9.4.25.v20191220 (#770) > Bump jetty.version from 9.4.21.v20190926 to 9.4.25.v20191220 > > > Key: AVRO-2686 > URL: https://issues.apache.org/jira/browse/AVRO-2686 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.9.1 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-2542) Bump various Maven plugins
[ https://issues.apache.org/jira/browse/AVRO-2542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027627#comment-17027627 ] ASF subversion and git services commented on AVRO-2542: --- Commit 464498534f10b5e9e04619d2b1bee2f668bac433 in avro's branch refs/heads/branch-1.9 from Fokko Driesprong [ https://gitbox.apache.org/repos/asf?p=avro.git;h=4644985 ] AVRO-2542: Bump various Apache Maven plugins (#634) > Bump various Maven plugins > -- > > Key: AVRO-2542 > URL: https://issues.apache.org/jira/browse/AVRO-2542 > Project: Apache Avro > Issue Type: Task > Components: dependencies >Affects Versions: 1.9.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-2673) Bump protobuf-java from 3.10.0 to 3.11.1
[ https://issues.apache.org/jira/browse/AVRO-2673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027630#comment-17027630 ] ASF subversion and git services commented on AVRO-2673: --- Commit fc090b2251e24000d9728c1a4138bd80836e4aac in avro's branch refs/heads/branch-1.9 from Fokko Driesprong [ https://gitbox.apache.org/repos/asf?p=avro.git;h=fc090b2 ] AVRO-2673: Bump protobuf-java from 3.10.0 to 3.11.1 (#759) > Bump protobuf-java from 3.10.0 to 3.11.1 > > > Key: AVRO-2673 > URL: https://issues.apache.org/jira/browse/AVRO-2673 > Project: Apache Avro > Issue Type: Improvement >Affects Versions: 1.9.1 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-2684) Bump netty-codec-http2 from 4.1.44.Final to 4.1.45.Final
[ https://issues.apache.org/jira/browse/AVRO-2684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027633#comment-17027633 ] ASF subversion and git services commented on AVRO-2684: --- Commit a59a1d47f6868df400719174333faf1a6dce8af5 in avro's branch refs/heads/branch-1.9 from Fokko Driesprong [ https://gitbox.apache.org/repos/asf?p=avro.git;h=a59a1d4 ] AVRO-2684: Bump netty-codec-http2 from 4.1.44.Final to 4.1.45.Final (#769) > Bump netty-codec-http2 from 4.1.44.Final to 4.1.45.Final > > > Key: AVRO-2684 > URL: https://issues.apache.org/jira/browse/AVRO-2684 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.9.1 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-2672) Bump spotless-maven-plugin from 1.25.1 to 1.27.0
[ https://issues.apache.org/jira/browse/AVRO-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027631#comment-17027631 ] ASF subversion and git services commented on AVRO-2672: --- Commit e534c37484bf69dfa733482265fdfb59aac7a557 in avro's branch refs/heads/branch-1.9 from Fokko Driesprong [ https://gitbox.apache.org/repos/asf?p=avro.git;h=e534c37 ] AVRO-2672: Bump spotless-maven-plugin from 1.25.1 to 1.27.0 (#757) > Bump spotless-maven-plugin from 1.25.1 to 1.27.0 > > > Key: AVRO-2672 > URL: https://issues.apache.org/jira/browse/AVRO-2672 > Project: Apache Avro > Issue Type: Improvement >Affects Versions: 1.9.1 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-2541) Bump velocity-engine-core from 2.0 to 2.1
[ https://issues.apache.org/jira/browse/AVRO-2541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027626#comment-17027626 ] ASF subversion and git services commented on AVRO-2541: --- Commit 8163c0d6439a0c26622a9fff30959f322a85398c in avro's branch refs/heads/branch-1.9 from Fokko Driesprong [ https://gitbox.apache.org/repos/asf?p=avro.git;h=8163c0d ] AVRO-2541: Bump velocity-engine-core from 2.0 to 2.1 (#632) > Bump velocity-engine-core from 2.0 to 2.1 > - > > Key: AVRO-2541 > URL: https://issues.apache.org/jira/browse/AVRO-2541 > Project: Apache Avro > Issue Type: Task > Components: dependencies >Affects Versions: 1.9.1 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-2543) Bump grpc.version from 1.19.0 to 1.23.0
[ https://issues.apache.org/jira/browse/AVRO-2543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027635#comment-17027635 ] ASF subversion and git services commented on AVRO-2543: --- Commit 1219f3e5e3904a02d3969ff8c0cd1cb485adf10e in avro's branch refs/heads/branch-1.9 from Fokko Driesprong [ https://gitbox.apache.org/repos/asf?p=avro.git;h=1219f3e ] AVRO-2543: Bump grpc.version from 1.19.0 to 1.23.0 (#635) * AVRO-2543: Bump grpc.version from 1.19.0 to 1.23.0 * Import it from Guava instead > Bump grpc.version from 1.19.0 to 1.23.0 > --- > > Key: AVRO-2543 > URL: https://issues.apache.org/jira/browse/AVRO-2543 > Project: Apache Avro > Issue Type: Task > Components: dependencies >Affects Versions: 1.9.0 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-2548) StringType of "String" causes logicalType converters to be ignored for field
[
https://issues.apache.org/jira/browse/AVRO-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027487#comment-17027487
]
ASF subversion and git services commented on AVRO-2548:
---
Commit 9b21e85bbb9db74604cbb30bcf2b3b4e8d69eee5 in avro's branch
refs/heads/branch-1.9 from ivangreene
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=9b21e85 ]
AVRO-2548: Handle logicalTypes with stringType (#655)
When using 'stringType', logical types whose
Avro type was string would not have that conversion
applied.
> StringType of "String" causes logicalType converters to be ignored for field
>
>
> Key: AVRO-2548
> URL: https://issues.apache.org/jira/browse/AVRO-2548
> Project: Apache Avro
> Issue Type: Bug
> Components: java, logical types
>Affects Versions: 1.9.0
>Reporter: Chris Woodham
>Assignee: Ivan Greene
>Priority: Major
> Fix For: 1.9.2
>
> Attachments: avro-issue-example.zip
>
>
> I have the avro maven plugin configured like this:
> {code:xml}
>
> org.apache.avro
> avro-maven-plugin
> ${avro.version}
>
>
>
> schema
>
>
> String
> false
>
> true
> private
>
>
> org.apache.avro.Conversions$UUIDConversion
>
>
>
>
>
>
> {code}
> With the intention of using the provided {{UUIDConversion}} class on string
> fields set with the {{logicalType}} {{uuid}}. However, it seems like having
> {{stringType}} specified as {{String}} means the converter is ignored.
> If I comment out the line {{String}} then the
> converter is used as expected and the java classes have {{UUID}} types for
> those fields, but obviously all normal strings are now {{CharSequence}}
> objects.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2429) Avro 1.9.0 fails when reading logical types other than "decimal"
[ https://issues.apache.org/jira/browse/AVRO-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027485#comment-17027485 ] ASF subversion and git services commented on AVRO-2429: --- Commit b39e0a998c7f1f3efc394148d236bd608ddbbb9a in avro's branch refs/heads/branch-1.9 from RyanSkraba [ https://gitbox.apache.org/repos/asf?p=avro.git;h=b39e0a9 ] AVRO-2429: Ignore unknown logical types in python2. (#687) * AVRO-2429: Ignore unknown logical types in python2. * AVRO-2429: Clean up math and unicode. * AVRO-2429: Fix max precision formula. If the maximum magnitude of a two's complement number of N bytes is: max_mag(N) = 2 ** (8 * n - 1) and the number of decimal digits required to represent M is: decimal_digits(M) = floor(log10(M)) + 1 Then the maximum number of decimal digits that a N byte number can contain: max_decimal_digits(N) = decimal_digits(max_mag(N)) = floor(log10(2 ** (8 * n - 1))) + 1 = floor( (8*n-1) * log10(2) ) + 1 If the biggest number has X decimal digits, the N bytes can represent all numbers with X-1 decimal digits. max_precision(N) = floor( (8*n-1) * log10(2) ) * AVRO-2429: Add unit tests for max precision. Fix lint. > Avro 1.9.0 fails when reading logical types other than "decimal" > > > Key: AVRO-2429 > URL: https://issues.apache.org/jira/browse/AVRO-2429 > Project: Apache Avro > Issue Type: Bug > Components: python >Affects Versions: 1.9.0 >Reporter: Chamikara Madhusanka Jayalath >Assignee: Ryan Skraba >Priority: Major > Fix For: 1.10.0, 1.9.2 > > Attachments: uuid.avro > > > [https://github.com/apache/avro/pull/82] added support for Avro "decimal" > logical type but also added an assertion that results in a reader failing for > other logical types. > [https://github.com/apache/avro/blob/master/lang/py/src/avro/schema.py#L821] > I believe this is a regression since previously avro library used to read the > underlying primitive type instead of failing. > Can we revert the behavior for logical types that are not "decimal" by > removing this assertion and reverting to the old (avro 1.8.1) behavior of > returning the primitive type ? > > cc: [~Fokko] [~mtth] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-2689) Avro-tools should have a way to test schema evolution.
[
https://issues.apache.org/jira/browse/AVRO-2689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027491#comment-17027491
]
ASF subversion and git services commented on AVRO-2689:
---
Commit aea89bf7816d0cd2631fcf8be04795d686ad95c6 in avro's branch
refs/heads/branch-1.9 from Roger Peppe
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=aea89bf ]
AVRO-2689: add reader-schema to DataFileReadTool (#785)
* AVRO-2689: add reader schema to DataFileReadTool
This PR adds --reader-schema and --reader-schema-file flags
to the tojson tool.
* AVRO-2689: Unit test for reader schema on tojson tool.
Co-authored-by: RyanSkraba
> Avro-tools should have a way to test schema evolution.
> --
>
> Key: AVRO-2689
> URL: https://issues.apache.org/jira/browse/AVRO-2689
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: Ryan Skraba
>Assignee: Roger
>Priority: Major
> Fix For: 1.10.0, 1.9.2
>
>
> The avro-tools command line should have at least one way to test
> reading/writing with an evolved schema. This would be useful for testing
> schema evolution and reproducing errors or bug reports from the command line.
> A good candidate would be to add a {{--reader-schema}} option to the
> {{tojson}} tool (where the writer schema comes from the data file).
> Alternatively, the {{fragtojson}} already takes the writer schema and we
> could add {{--reader-schema-file}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2466) Fix a malformed schema in the share/test/schemas directory
[
https://issues.apache.org/jira/browse/AVRO-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027492#comment-17027492
]
ASF subversion and git services commented on AVRO-2466:
---
Commit 299261e24d554be478965572d799c36fb7c09630 in avro's branch
refs/heads/branch-1.9 from Kengo Seki
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=299261e ]
AVRO-2466: Fix a malformed schema in the share/test/schemas directory (#578)
> Fix a malformed schema in the share/test/schemas directory
> --
>
> Key: AVRO-2466
> URL: https://issues.apache.org/jira/browse/AVRO-2466
> Project: Apache Avro
> Issue Type: Bug
> Components: misc
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Minor
> Fix For: 1.10.0, 1.9.2
>
>
> The schema defined in share/test/schemas/reserved.avsc is invalid because of
> its trailing comma:
> {code}
> $ python -c 'from avro.schema import parse;
> parse(open("../../share/test/schemas/reserved.avsc").read())'
> Traceback (most recent call last):
> File "", line 1, in
> File "/home/sekikn/repo/avro/lang/py/src/avro/schema.py", line 976, in parse
> json_data = json.loads(json_string)
> File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
> return _default_decoder.decode(s)
> File "/usr/lib/python2.7/json/decoder.py", line 367, in decode
> raise ValueError(errmsg("Extra data", s, end, len(s)))
> avro.schema.SchemaParseException: Error parsing JSON: {"name":
> "org.apache.avro.test.Reserved", "type": "enum",
> "symbols": ["default","class","int"]},
> , error = Extra data: line 2 column 39 - line 3 column 1 (char 96 - 98)
> $ cat ../../share/test/schemas/reserved.avsc
> {"name": "org.apache.avro.test.Reserved", "type": "enum",
> "symbols": ["default","class","int"]},
> {code}
> This file doesn't seem to be used as far as I investigated, but I'd rather
> fix this than remove since it might be useful for some test.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2689) Avro-tools should have a way to test schema evolution.
[
https://issues.apache.org/jira/browse/AVRO-2689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027490#comment-17027490
]
ASF subversion and git services commented on AVRO-2689:
---
Commit aea89bf7816d0cd2631fcf8be04795d686ad95c6 in avro's branch
refs/heads/branch-1.9 from Roger Peppe
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=aea89bf ]
AVRO-2689: add reader-schema to DataFileReadTool (#785)
* AVRO-2689: add reader schema to DataFileReadTool
This PR adds --reader-schema and --reader-schema-file flags
to the tojson tool.
* AVRO-2689: Unit test for reader schema on tojson tool.
Co-authored-by: RyanSkraba
> Avro-tools should have a way to test schema evolution.
> --
>
> Key: AVRO-2689
> URL: https://issues.apache.org/jira/browse/AVRO-2689
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: Ryan Skraba
>Assignee: Roger
>Priority: Major
> Fix For: 1.10.0, 1.9.2
>
>
> The avro-tools command line should have at least one way to test
> reading/writing with an evolved schema. This would be useful for testing
> schema evolution and reproducing errors or bug reports from the command line.
> A good candidate would be to add a {{--reader-schema}} option to the
> {{tojson}} tool (where the writer schema comes from the data file).
> Alternatively, the {{fragtojson}} already takes the writer schema and we
> could add {{--reader-schema-file}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2631) py3 extras_require installs the wrong snappy
[
https://issues.apache.org/jira/browse/AVRO-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027486#comment-17027486
]
ASF subversion and git services commented on AVRO-2631:
---
Commit 60a5b9d976c9ae1b0b8c3893b992f624970fc8ed in avro's branch
refs/heads/branch-1.9 from Michael A. Smith
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=60a5b9d ]
AVRO-2631: Install the Right Snappy (#713)
> py3 extras_require installs the wrong snappy
>
>
> Key: AVRO-2631
> URL: https://issues.apache.org/jira/browse/AVRO-2631
> Project: Apache Avro
> Issue Type: Bug
>Reporter: Michael A. Smith
>Assignee: Michael A. Smith
>Priority: Major
> Fix For: 1.10.0
>
>
> If you run {{pip install avro[snappy]}} or {{pip3 install
> avro-python3[snappy]}} you should get avro installed with the python-snappy
> library, but instead you get an unrelated project
> https://pypi.org/project/snappy/.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2664) Pin Ruby rdoc to <=6.2.0
[ https://issues.apache.org/jira/browse/AVRO-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027488#comment-17027488 ] ASF subversion and git services commented on AVRO-2664: --- Commit a24ba54c501ce3a78eab5c98dab9a92e9f2d67bb in avro's branch refs/heads/branch-1.9 from Fokko Driesprong [ https://gitbox.apache.org/repos/asf?p=avro.git;h=a24ba54 ] AVRO-2664: Pin Ruby rdoc to <=6.2.0 (#753) Ruby rdoc 6.2.1 has been released which requires Ruby 2.4: https://rubygems.org/gems/rdoc/versions/6.2.1 We would like to contrain this to 6.2.0: https://rubygems.org/gems/rdoc/versions/6.2.0 > Pin Ruby rdoc to <=6.2.0 > > > Key: AVRO-2664 > URL: https://issues.apache.org/jira/browse/AVRO-2664 > Project: Apache Avro > Issue Type: Bug >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Fix For: 1.10.0, 1.9.2 > > > Ruby rdoc 6.2.1 has been released which requires Ruby 2.4: > https://rubygems.org/gems/rdoc/versions/6.2.1 > We would like to contrain this to 6.2.0: > https://rubygems.org/gems/rdoc/versions/6.2.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-2689) Avro-tools should have a way to test schema evolution.
[
https://issues.apache.org/jira/browse/AVRO-2689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027489#comment-17027489
]
ASF subversion and git services commented on AVRO-2689:
---
Commit aea89bf7816d0cd2631fcf8be04795d686ad95c6 in avro's branch
refs/heads/branch-1.9 from Roger Peppe
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=aea89bf ]
AVRO-2689: add reader-schema to DataFileReadTool (#785)
* AVRO-2689: add reader schema to DataFileReadTool
This PR adds --reader-schema and --reader-schema-file flags
to the tojson tool.
* AVRO-2689: Unit test for reader schema on tojson tool.
Co-authored-by: RyanSkraba
> Avro-tools should have a way to test schema evolution.
> --
>
> Key: AVRO-2689
> URL: https://issues.apache.org/jira/browse/AVRO-2689
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: Ryan Skraba
>Assignee: Roger
>Priority: Major
> Fix For: 1.10.0, 1.9.2
>
>
> The avro-tools command line should have at least one way to test
> reading/writing with an evolved schema. This would be useful for testing
> schema evolution and reproducing errors or bug reports from the command line.
> A good candidate would be to add a {{--reader-schema}} option to the
> {{tojson}} tool (where the writer schema comes from the data file).
> Alternatively, the {{fragtojson}} already takes the writer schema and we
> could add {{--reader-schema-file}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2429) Avro 1.9.0 fails when reading logical types other than "decimal"
[ https://issues.apache.org/jira/browse/AVRO-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027483#comment-17027483 ] ASF subversion and git services commented on AVRO-2429: --- Commit b39e0a998c7f1f3efc394148d236bd608ddbbb9a in avro's branch refs/heads/branch-1.9 from RyanSkraba [ https://gitbox.apache.org/repos/asf?p=avro.git;h=b39e0a9 ] AVRO-2429: Ignore unknown logical types in python2. (#687) * AVRO-2429: Ignore unknown logical types in python2. * AVRO-2429: Clean up math and unicode. * AVRO-2429: Fix max precision formula. If the maximum magnitude of a two's complement number of N bytes is: max_mag(N) = 2 ** (8 * n - 1) and the number of decimal digits required to represent M is: decimal_digits(M) = floor(log10(M)) + 1 Then the maximum number of decimal digits that a N byte number can contain: max_decimal_digits(N) = decimal_digits(max_mag(N)) = floor(log10(2 ** (8 * n - 1))) + 1 = floor( (8*n-1) * log10(2) ) + 1 If the biggest number has X decimal digits, the N bytes can represent all numbers with X-1 decimal digits. max_precision(N) = floor( (8*n-1) * log10(2) ) * AVRO-2429: Add unit tests for max precision. Fix lint. > Avro 1.9.0 fails when reading logical types other than "decimal" > > > Key: AVRO-2429 > URL: https://issues.apache.org/jira/browse/AVRO-2429 > Project: Apache Avro > Issue Type: Bug > Components: python >Affects Versions: 1.9.0 >Reporter: Chamikara Madhusanka Jayalath >Assignee: Ryan Skraba >Priority: Major > Fix For: 1.10.0, 1.9.2 > > Attachments: uuid.avro > > > [https://github.com/apache/avro/pull/82] added support for Avro "decimal" > logical type but also added an assertion that results in a reader failing for > other logical types. > [https://github.com/apache/avro/blob/master/lang/py/src/avro/schema.py#L821] > I believe this is a regression since previously avro library used to read the > underlying primitive type instead of failing. > Can we revert the behavior for logical types that are not "decimal" by > removing this assertion and reverting to the old (avro 1.8.1) behavior of > returning the primitive type ? > > cc: [~Fokko] [~mtth] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-2429) Avro 1.9.0 fails when reading logical types other than "decimal"
[ https://issues.apache.org/jira/browse/AVRO-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027481#comment-17027481 ] ASF subversion and git services commented on AVRO-2429: --- Commit b39e0a998c7f1f3efc394148d236bd608ddbbb9a in avro's branch refs/heads/branch-1.9 from RyanSkraba [ https://gitbox.apache.org/repos/asf?p=avro.git;h=b39e0a9 ] AVRO-2429: Ignore unknown logical types in python2. (#687) * AVRO-2429: Ignore unknown logical types in python2. * AVRO-2429: Clean up math and unicode. * AVRO-2429: Fix max precision formula. If the maximum magnitude of a two's complement number of N bytes is: max_mag(N) = 2 ** (8 * n - 1) and the number of decimal digits required to represent M is: decimal_digits(M) = floor(log10(M)) + 1 Then the maximum number of decimal digits that a N byte number can contain: max_decimal_digits(N) = decimal_digits(max_mag(N)) = floor(log10(2 ** (8 * n - 1))) + 1 = floor( (8*n-1) * log10(2) ) + 1 If the biggest number has X decimal digits, the N bytes can represent all numbers with X-1 decimal digits. max_precision(N) = floor( (8*n-1) * log10(2) ) * AVRO-2429: Add unit tests for max precision. Fix lint. > Avro 1.9.0 fails when reading logical types other than "decimal" > > > Key: AVRO-2429 > URL: https://issues.apache.org/jira/browse/AVRO-2429 > Project: Apache Avro > Issue Type: Bug > Components: python >Affects Versions: 1.9.0 >Reporter: Chamikara Madhusanka Jayalath >Assignee: Ryan Skraba >Priority: Major > Fix For: 1.10.0, 1.9.2 > > Attachments: uuid.avro > > > [https://github.com/apache/avro/pull/82] added support for Avro "decimal" > logical type but also added an assertion that results in a reader failing for > other logical types. > [https://github.com/apache/avro/blob/master/lang/py/src/avro/schema.py#L821] > I believe this is a regression since previously avro library used to read the > underlying primitive type instead of failing. > Can we revert the behavior for logical types that are not "decimal" by > removing this assertion and reverting to the old (avro 1.8.1) behavior of > returning the primitive type ? > > cc: [~Fokko] [~mtth] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-2512) writerField is always null
[ https://issues.apache.org/jira/browse/AVRO-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027479#comment-17027479 ] ASF subversion and git services commented on AVRO-2512: --- Commit 426a6ee6e340255a45c51662886e7c8e4f69ee6e in avro's branch refs/heads/branch-1.9 from Fokko Driesprong [ https://gitbox.apache.org/repos/asf?p=avro.git;h=426a6ee ] AVRO-2512: Fix formatting > writerField is always null > -- > > Key: AVRO-2512 > URL: https://issues.apache.org/jira/browse/AVRO-2512 > Project: Apache Avro > Issue Type: Task >Affects Versions: 1.9.0 >Reporter: Fokko Driesprong >Priority: Major > Fix For: 1.10.0, 1.9.2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-2429) Avro 1.9.0 fails when reading logical types other than "decimal"
[ https://issues.apache.org/jira/browse/AVRO-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027484#comment-17027484 ] ASF subversion and git services commented on AVRO-2429: --- Commit b39e0a998c7f1f3efc394148d236bd608ddbbb9a in avro's branch refs/heads/branch-1.9 from RyanSkraba [ https://gitbox.apache.org/repos/asf?p=avro.git;h=b39e0a9 ] AVRO-2429: Ignore unknown logical types in python2. (#687) * AVRO-2429: Ignore unknown logical types in python2. * AVRO-2429: Clean up math and unicode. * AVRO-2429: Fix max precision formula. If the maximum magnitude of a two's complement number of N bytes is: max_mag(N) = 2 ** (8 * n - 1) and the number of decimal digits required to represent M is: decimal_digits(M) = floor(log10(M)) + 1 Then the maximum number of decimal digits that a N byte number can contain: max_decimal_digits(N) = decimal_digits(max_mag(N)) = floor(log10(2 ** (8 * n - 1))) + 1 = floor( (8*n-1) * log10(2) ) + 1 If the biggest number has X decimal digits, the N bytes can represent all numbers with X-1 decimal digits. max_precision(N) = floor( (8*n-1) * log10(2) ) * AVRO-2429: Add unit tests for max precision. Fix lint. > Avro 1.9.0 fails when reading logical types other than "decimal" > > > Key: AVRO-2429 > URL: https://issues.apache.org/jira/browse/AVRO-2429 > Project: Apache Avro > Issue Type: Bug > Components: python >Affects Versions: 1.9.0 >Reporter: Chamikara Madhusanka Jayalath >Assignee: Ryan Skraba >Priority: Major > Fix For: 1.10.0, 1.9.2 > > Attachments: uuid.avro > > > [https://github.com/apache/avro/pull/82] added support for Avro "decimal" > logical type but also added an assertion that results in a reader failing for > other logical types. > [https://github.com/apache/avro/blob/master/lang/py/src/avro/schema.py#L821] > I believe this is a regression since previously avro library used to read the > underlying primitive type instead of failing. > Can we revert the behavior for logical types that are not "decimal" by > removing this assertion and reverting to the old (avro 1.8.1) behavior of > returning the primitive type ? > > cc: [~Fokko] [~mtth] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-2512) writerField is always null
[ https://issues.apache.org/jira/browse/AVRO-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027478#comment-17027478 ] ASF subversion and git services commented on AVRO-2512: --- Commit b0194db23ee3c59ebf3d330f44d973ae186f1885 in avro's branch refs/heads/branch-1.9 from Fokko Driesprong [ https://gitbox.apache.org/repos/asf?p=avro.git;h=b0194db ] AVRO-2512: writerField is always null > writerField is always null > -- > > Key: AVRO-2512 > URL: https://issues.apache.org/jira/browse/AVRO-2512 > Project: Apache Avro > Issue Type: Task >Affects Versions: 1.9.0 >Reporter: Fokko Driesprong >Priority: Major > Fix For: 1.10.0, 1.9.2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-2468) Fix broken data interoperability on the Perl bindings
[
https://issues.apache.org/jira/browse/AVRO-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027477#comment-17027477
]
ASF subversion and git services commented on AVRO-2468:
---
Commit 5b466f556f4bc42e04e5fbbc373ba3dd3ffcce0f in avro's branch
refs/heads/branch-1.9 from Kengo Seki
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=5b466f5 ]
AVRO-2468: Fix broken data interoperability on the Perl bindings (#582)
* Fix Avro::Schema to take namespace into consideration
when parsing named types in arrays/maps/unions.
* Fix Avro::Schema to encode fixed type's size value
into numeric when it outputs Avro schema.
* To ensure the above fixes work, add data interop test
for the Perl bindings.
> Fix broken data interoperability on the Perl bindings
> -
>
> Key: AVRO-2468
> URL: https://issues.apache.org/jira/browse/AVRO-2468
> Project: Apache Avro
> Issue Type: Bug
> Components: interop, perl
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Critical
> Fix For: 1.10.0, 1.9.2
>
>
> I found some data interop problems on the Perl bindings.
> 1. They fail to parse a schema if there's an array/map/union which contains
> named types with a simple (not fully-qualified) name in it. For example, they
> can't parse {{share/test/schemas/interop.avsc}} or
> {{share/schemas/org/apache/avro/data/Json.avsc}}, because they have a named
> type called "Node" or "Json" respectively in arrays/maps. This seems because
> the parser doesn't take namespace into consideration in parsing
> array/map/union.
> {code}
> $ cd lang/perl
> $ perl -Ilib -de 1
> (snip)
> DB<1> open FH, '../../share/test/schemas/interop.avsc'; local $/ = undef;
> $s = ; close FH; print $s
> {"type": "record", "name":"Interop", "namespace": "org.apache.avro",
> "fields": [
> {"name": "intField", "type": "int"},
> {"name": "longField", "type": "long"},
> {"name": "stringField", "type": "string"},
> {"name": "boolField", "type": "boolean"},
> {"name": "floatField", "type": "float"},
> {"name": "doubleField", "type": "double"},
> {"name": "bytesField", "type": "bytes"},
> {"name": "nullField", "type": "null"},
> {"name": "arrayField", "type": {"type": "array", "items": "double"}},
> {"name": "mapField", "type":
>{"type": "map", "values":
> {"type": "record", "name": "Foo",
> "fields": [{"name": "label", "type": "string"}]}}},
> {"name": "unionField", "type":
>["boolean", "double", {"type": "array", "items": "bytes"}]},
> {"name": "enumField", "type":
>{"type": "enum", "name": "Kind", "symbols": ["A","B","C"]}},
> {"name": "fixedField", "type":
>{"type": "fixed", "name": "MD5", "size": 16}},
> {"name": "recordField", "type":
>{"type": "record", "name": "Node",
> "fields": [
> {"name": "label", "type": "string"},
> {"name": "children", "type": {"type": "array", "items":
> "Node"}}]}}
> ]
> }
> DB<2> use Avro::Schema; Avro::Schema->parse($s)
> Not a primitive type Node at lib/Avro/Schema.pm line 257.
> {code}
> 2. They encode the size for a fixed type as a string rather than a number, so
> other language bindings fail to parse it.
> {code}
> $ cd lang/perl
> $ perl -Ilib -de 1
> (snip)
> DB<1> use Avro::Schema; $s = Avro::Schema->parse('{"type": "fixed", "size":
> 16, "name": "md5"}')
> DB<2> open($fh, '>/tmp/output')
> DB<3> use Avro::DataFileWriter; $w = Avro::DataFileWriter->new(fh => $fh,
> writer_schema => $s)
> DB<4> $w->print('0123456789abcdef')
> DB<5> $w->close
> {code}
> {code}
> $ ipython
> (snip)
> In [1]: from avro.datafile import DataFileReader
> In [2]: from avro.io import DatumReader
> In [3]: DataFileReader(datum_reader=DatumReader(), reader=open("/tmp/output"))
> ---
> AvroException Traceback (most recent call last)
> in ()
> > 1 DataFileReader(datum_reader=DatumReader(), reader=open("/tmp/output"))
> /home/sekikn/repo/avro/lang/py/src/avro/datafile.pyc in __init__(self,
> reader, datum_reader)
> 255 # get ready to read
> 256 self._block_count = 0
> --> 257 self.datum_reader.writers_schema =
> schema.parse(self.get_meta(SCHEMA_KEY))
> 258
> 259 def __enter__(self):
> /home/sekikn/repo/avro/lang/py/src/avro/schema.pyc in parse(json_string)
> 984
> 985 # construct the Avro Schema object
> --> 986 return make_avsc_object(json_data, names)
> /home/sekikn/repo/avro/lang/py/src/avro/schema.pyc in
> make_avsc_object(json_data, names)
> 931 scale = 0 if json_data.get('scale') is None else
> json_data.get('scale')
> 932
[jira] [Commented] (AVRO-2564) The link address in javadoc-plugin has expired
[ https://issues.apache.org/jira/browse/AVRO-2564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027480#comment-17027480 ] ASF subversion and git services commented on AVRO-2564: --- Commit c1a210caf4a3422c79ac83e5e1adfa84ad0896bb in avro's branch refs/heads/branch-1.9 from Zezeng Wang [ https://gitbox.apache.org/repos/asf?p=avro.git;h=c1a210c ] AVRO-2564: Update link in javadoc-plugin (#653) > The link address in javadoc-plugin has expired > -- > > Key: AVRO-2564 > URL: https://issues.apache.org/jira/browse/AVRO-2564 > Project: Apache Avro > Issue Type: Bug > Components: java >Reporter: Zezeng Wang >Assignee: Zezeng Wang >Priority: Minor > Fix For: 1.10.0 > > > In the API documentation for lang/java/mapred/target/apidocs, there is no api > connection for hadoop. > Because the link address of maven-plugin-javadoc is > http://hadoop.apache.org/common/docs/current/api/ 404 no found , > the /common in the url does not exist and needs to be changed to > Https://hadoop.apache.org/docs/current/api/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-2429) Avro 1.9.0 fails when reading logical types other than "decimal"
[ https://issues.apache.org/jira/browse/AVRO-2429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027482#comment-17027482 ] ASF subversion and git services commented on AVRO-2429: --- Commit b39e0a998c7f1f3efc394148d236bd608ddbbb9a in avro's branch refs/heads/branch-1.9 from RyanSkraba [ https://gitbox.apache.org/repos/asf?p=avro.git;h=b39e0a9 ] AVRO-2429: Ignore unknown logical types in python2. (#687) * AVRO-2429: Ignore unknown logical types in python2. * AVRO-2429: Clean up math and unicode. * AVRO-2429: Fix max precision formula. If the maximum magnitude of a two's complement number of N bytes is: max_mag(N) = 2 ** (8 * n - 1) and the number of decimal digits required to represent M is: decimal_digits(M) = floor(log10(M)) + 1 Then the maximum number of decimal digits that a N byte number can contain: max_decimal_digits(N) = decimal_digits(max_mag(N)) = floor(log10(2 ** (8 * n - 1))) + 1 = floor( (8*n-1) * log10(2) ) + 1 If the biggest number has X decimal digits, the N bytes can represent all numbers with X-1 decimal digits. max_precision(N) = floor( (8*n-1) * log10(2) ) * AVRO-2429: Add unit tests for max precision. Fix lint. > Avro 1.9.0 fails when reading logical types other than "decimal" > > > Key: AVRO-2429 > URL: https://issues.apache.org/jira/browse/AVRO-2429 > Project: Apache Avro > Issue Type: Bug > Components: python >Affects Versions: 1.9.0 >Reporter: Chamikara Madhusanka Jayalath >Assignee: Ryan Skraba >Priority: Major > Fix For: 1.10.0, 1.9.2 > > Attachments: uuid.avro > > > [https://github.com/apache/avro/pull/82] added support for Avro "decimal" > logical type but also added an assertion that results in a reader failing for > other logical types. > [https://github.com/apache/avro/blob/master/lang/py/src/avro/schema.py#L821] > I believe this is a regression since previously avro library used to read the > underlying primitive type instead of failing. > Can we revert the behavior for logical types that are not "decimal" by > removing this assertion and reverting to the old (avro 1.8.1) behavior of > returning the primitive type ? > > cc: [~Fokko] [~mtth] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-2592) Avro decimal fails on certain conditions - ByteBuffer.position() is the root cause
[
https://issues.apache.org/jira/browse/AVRO-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027379#comment-17027379
]
Hudson commented on AVRO-2592:
--
SUCCESS: Integrated in Jenkins build AvroJava #812 (See
[https://builds.apache.org/job/AvroJava/812/])
AVRO-2592: Avoid consuming ByteBuffer for decimal. (iemejia:
[https://github.com/apache/avro/commit/ce9032a4123e262341a3e746e6eb2b8476eae1ae])
* (edit)
lang/java/avro/src/test/java/org/apache/avro/generic/TestGenericLogicalTypes.java
* (edit) lang/java/avro/src/main/java/org/apache/avro/Conversions.java
> Avro decimal fails on certain conditions - ByteBuffer.position() is the root
> cause
> --
>
> Key: AVRO-2592
> URL: https://issues.apache.org/jira/browse/AVRO-2592
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.1
>Reporter: Werner Daehn
>Assignee: Ryan Skraba
>Priority: Blocker
> Fix For: 1.9.2
>
>
> The Decimal Conversion from/to Bytebuffer is using the methods that consider
> the current position, e.g. remaining().
> [https://github.com/apache/avro/blob/release-1.9.1/lang/java/avro/src/main/java/org/apache/avro/Conversions.java#L82]
> At first sight that looks like a good idea. But actually it creates all sorts
> of problems.
> For example this code fails with "Zero Length BigInteger":
>
> {code:java}
> BigDecimal d = BigDecimal.valueOf(3.1415); BigDecimal d =
> BigDecimal.valueOf(3.1415);
> Decimal decimaltype = LogicalTypes.decimal(7, 4); ByteBuffer buffer =
> DECIMAL_CONVERTER.toBytes(d, null, decimaltype);
> System.out.println(DECIMAL_CONVERTER.fromBytes(buffer, null,
> decimaltype).toString());
> BigDecimal n = DECIMAL_CONVERTER.fromBytes(buffer, null, decimaltype);
> System.out.println(n.toString());{code}
>
> Reason is obvious. The first call to fromBytes() moves the position from 0 to
> the last byte. The second invocation reads from the last position, hence zero
> records.
> There are other situations this might cause issues, e.g. a user might create
> the ByteBuffer via other means and it is normal that the position is after
> the last byte. Then the serialization would not work either. And many other.
> As the ByteBuffer is used to wrap a single BigDecimal, I would suggest to
> remove all position-aware/setting methods and read/write from position zero.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2592) Avro decimal fails on certain conditions - ByteBuffer.position() is the root cause
[
https://issues.apache.org/jira/browse/AVRO-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027321#comment-17027321
]
ASF subversion and git services commented on AVRO-2592:
---
Commit 6ee4b6b90271ccc78ce3b51c422bcbdb9abab995 in avro's branch
refs/heads/branch-1.9 from Ryan Skraba
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=6ee4b6b ]
AVRO-2592: Avoid consuming ByteBuffer for decimal.
> Avro decimal fails on certain conditions - ByteBuffer.position() is the root
> cause
> --
>
> Key: AVRO-2592
> URL: https://issues.apache.org/jira/browse/AVRO-2592
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.1
>Reporter: Werner Daehn
>Assignee: Ryan Skraba
>Priority: Blocker
> Fix For: 1.9.2
>
>
> The Decimal Conversion from/to Bytebuffer is using the methods that consider
> the current position, e.g. remaining().
> [https://github.com/apache/avro/blob/release-1.9.1/lang/java/avro/src/main/java/org/apache/avro/Conversions.java#L82]
> At first sight that looks like a good idea. But actually it creates all sorts
> of problems.
> For example this code fails with "Zero Length BigInteger":
>
> {code:java}
> BigDecimal d = BigDecimal.valueOf(3.1415); BigDecimal d =
> BigDecimal.valueOf(3.1415);
> Decimal decimaltype = LogicalTypes.decimal(7, 4); ByteBuffer buffer =
> DECIMAL_CONVERTER.toBytes(d, null, decimaltype);
> System.out.println(DECIMAL_CONVERTER.fromBytes(buffer, null,
> decimaltype).toString());
> BigDecimal n = DECIMAL_CONVERTER.fromBytes(buffer, null, decimaltype);
> System.out.println(n.toString());{code}
>
> Reason is obvious. The first call to fromBytes() moves the position from 0 to
> the last byte. The second invocation reads from the last position, hence zero
> records.
> There are other situations this might cause issues, e.g. a user might create
> the ByteBuffer via other means and it is normal that the position is after
> the last byte. Then the serialization would not work either. And many other.
> As the ByteBuffer is used to wrap a single BigDecimal, I would suggest to
> remove all position-aware/setting methods and read/write from position zero.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AVRO-2592) Avro decimal fails on certain conditions - ByteBuffer.position() is the root cause
[
https://issues.apache.org/jira/browse/AVRO-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated AVRO-2592:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Avro decimal fails on certain conditions - ByteBuffer.position() is the root
> cause
> --
>
> Key: AVRO-2592
> URL: https://issues.apache.org/jira/browse/AVRO-2592
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.1
>Reporter: Werner Daehn
>Assignee: Ryan Skraba
>Priority: Blocker
> Fix For: 1.9.2
>
>
> The Decimal Conversion from/to Bytebuffer is using the methods that consider
> the current position, e.g. remaining().
> [https://github.com/apache/avro/blob/release-1.9.1/lang/java/avro/src/main/java/org/apache/avro/Conversions.java#L82]
> At first sight that looks like a good idea. But actually it creates all sorts
> of problems.
> For example this code fails with "Zero Length BigInteger":
>
> {code:java}
> BigDecimal d = BigDecimal.valueOf(3.1415); BigDecimal d =
> BigDecimal.valueOf(3.1415);
> Decimal decimaltype = LogicalTypes.decimal(7, 4); ByteBuffer buffer =
> DECIMAL_CONVERTER.toBytes(d, null, decimaltype);
> System.out.println(DECIMAL_CONVERTER.fromBytes(buffer, null,
> decimaltype).toString());
> BigDecimal n = DECIMAL_CONVERTER.fromBytes(buffer, null, decimaltype);
> System.out.println(n.toString());{code}
>
> Reason is obvious. The first call to fromBytes() moves the position from 0 to
> the last byte. The second invocation reads from the last position, hence zero
> records.
> There are other situations this might cause issues, e.g. a user might create
> the ByteBuffer via other means and it is normal that the position is after
> the last byte. Then the serialization would not work either. And many other.
> As the ByteBuffer is used to wrap a single BigDecimal, I would suggest to
> remove all position-aware/setting methods and read/write from position zero.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2592) Avro decimal fails on certain conditions - ByteBuffer.position() is the root cause
[
https://issues.apache.org/jira/browse/AVRO-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17027320#comment-17027320
]
ASF subversion and git services commented on AVRO-2592:
---
Commit ce9032a4123e262341a3e746e6eb2b8476eae1ae in avro's branch
refs/heads/master from Ryan Skraba
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=ce9032a ]
AVRO-2592: Avoid consuming ByteBuffer for decimal.
> Avro decimal fails on certain conditions - ByteBuffer.position() is the root
> cause
> --
>
> Key: AVRO-2592
> URL: https://issues.apache.org/jira/browse/AVRO-2592
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.1
>Reporter: Werner Daehn
>Assignee: Ryan Skraba
>Priority: Blocker
> Fix For: 1.9.2
>
>
> The Decimal Conversion from/to Bytebuffer is using the methods that consider
> the current position, e.g. remaining().
> [https://github.com/apache/avro/blob/release-1.9.1/lang/java/avro/src/main/java/org/apache/avro/Conversions.java#L82]
> At first sight that looks like a good idea. But actually it creates all sorts
> of problems.
> For example this code fails with "Zero Length BigInteger":
>
> {code:java}
> BigDecimal d = BigDecimal.valueOf(3.1415); BigDecimal d =
> BigDecimal.valueOf(3.1415);
> Decimal decimaltype = LogicalTypes.decimal(7, 4); ByteBuffer buffer =
> DECIMAL_CONVERTER.toBytes(d, null, decimaltype);
> System.out.println(DECIMAL_CONVERTER.fromBytes(buffer, null,
> decimaltype).toString());
> BigDecimal n = DECIMAL_CONVERTER.fromBytes(buffer, null, decimaltype);
> System.out.println(n.toString());{code}
>
> Reason is obvious. The first call to fromBytes() moves the position from 0 to
> the last byte. The second invocation reads from the last position, hence zero
> records.
> There are other situations this might cause issues, e.g. a user might create
> the ByteBuffer via other means and it is normal that the position is after
> the last byte. Then the serialization would not work either. And many other.
> As the ByteBuffer is used to wrap a single BigDecimal, I would suggest to
> remove all position-aware/setting methods and read/write from position zero.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
