[jira] [Created] (AVRO-2051) Thread contention accessing JsonProperties props
Daniel Kulp created AVRO-2051: - Summary: Thread contention accessing JsonProperties props Key: AVRO-2051 URL: https://issues.apache.org/jira/browse/AVRO-2051 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.8.2 Reporter: Daniel Kulp See https://lists.apache.org/thread.html/dd34ab8439137a81a9de29ad4161f37b17638394cea0806765689976@%3Cuser.avro.apache.org%3E Basically, the getJsonProp method, being synchronized, is causing thread contention issues when trying to share schemas between threads.My proposal (pull request forthcoming) is to treat "props" as an immutable map and do a copy+add+swap for the addProp method. This will make the addProp call slower (particularly for large maps of props), but would make the reads significantly faster as no locking will be needed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2051) Thread contention accessing JsonProperties props
[ https://issues.apache.org/jira/browse/AVRO-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090896#comment-16090896 ] Daniel Kulp commented on AVRO-2051: --- I'm trying to find something that will work for Avro 1.8.x as that's what we'll need.Thus, removing all of that is likely not an option. That said, I just discovered that we already have parts of guava shaded in as a dependency. Thus, I believe I can use the CacheBuilder to create the equivalent of a "ConcurrentLinkedHashMap" (there are some google links that mention this) that would work for this and not have the quadratic issue. I'll investigate more tomorrow. Another option would be to either add a dependency to something else (like caffeine) that has a ConcurrentLinkedHashMap or copy/shade an Apache licensed version (like https://github.com/ben-manes/concurrentlinkedhashmap/blob/master/src/main/java/com/googlecode/concurrentlinkedhashmap/ConcurrentLinkedHashMap.java) into the src and use it. > Thread contention accessing JsonProperties props > > > Key: AVRO-2051 > URL: https://issues.apache.org/jira/browse/AVRO-2051 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.2 >Reporter: Daniel Kulp > > See > https://lists.apache.org/thread.html/dd34ab8439137a81a9de29ad4161f37b17638394cea0806765689976@%3Cuser.avro.apache.org%3E > Basically, the getJsonProp method, being synchronized, is causing thread > contention issues when trying to share schemas between threads.My > proposal (pull request forthcoming) is to treat "props" as an immutable map > and do a copy+add+swap for the addProp method. This will make the addProp > call slower (particularly for large maps of props), but would make the reads > significantly faster as no locking will be needed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (AVRO-2051) Thread contention accessing JsonProperties props
[ https://issues.apache.org/jira/browse/AVRO-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090911#comment-16090911 ] Daniel Kulp edited comment on AVRO-2051 at 7/18/17 1:04 AM: Of course another option is to just surround the access to the props with a ReentrantReadWriteLock. Bunch of ideas to test and benchmark. was (Author: dkulp): Of course another option is to just surround the access to the props with a ReentrantReadWriteLock. > Thread contention accessing JsonProperties props > > > Key: AVRO-2051 > URL: https://issues.apache.org/jira/browse/AVRO-2051 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.2 >Reporter: Daniel Kulp > > See > https://lists.apache.org/thread.html/dd34ab8439137a81a9de29ad4161f37b17638394cea0806765689976@%3Cuser.avro.apache.org%3E > Basically, the getJsonProp method, being synchronized, is causing thread > contention issues when trying to share schemas between threads.My > proposal (pull request forthcoming) is to treat "props" as an immutable map > and do a copy+add+swap for the addProp method. This will make the addProp > call slower (particularly for large maps of props), but would make the reads > significantly faster as no locking will be needed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2051) Thread contention accessing JsonProperties props
[ https://issues.apache.org/jira/browse/AVRO-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16090911#comment-16090911 ] Daniel Kulp commented on AVRO-2051: --- Of course another option is to just surround the access to the props with a ReentrantReadWriteLock. > Thread contention accessing JsonProperties props > > > Key: AVRO-2051 > URL: https://issues.apache.org/jira/browse/AVRO-2051 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.2 >Reporter: Daniel Kulp > > See > https://lists.apache.org/thread.html/dd34ab8439137a81a9de29ad4161f37b17638394cea0806765689976@%3Cuser.avro.apache.org%3E > Basically, the getJsonProp method, being synchronized, is causing thread > contention issues when trying to share schemas between threads.My > proposal (pull request forthcoming) is to treat "props" as an immutable map > and do a copy+add+swap for the addProp method. This will make the addProp > call slower (particularly for large maps of props), but would make the reads > significantly faster as no locking will be needed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2051) Thread contention accessing JsonProperties props
[ https://issues.apache.org/jira/browse/AVRO-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091772#comment-16091772 ] Daniel Kulp commented on AVRO-2051: --- I just updated the pull request. Basically, since the JsonProperties props field is only ever iterated across and added to (values never change or are removed), it was simple enough to create a ConcurrentHashMap subclass that can also maintain a linked list type thing to maintain the order. This keeps the O(log N) lookups, O(log N) inserts, and O(N) iterator. > Thread contention accessing JsonProperties props > > > Key: AVRO-2051 > URL: https://issues.apache.org/jira/browse/AVRO-2051 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.2 >Reporter: Daniel Kulp > > See > https://lists.apache.org/thread.html/dd34ab8439137a81a9de29ad4161f37b17638394cea0806765689976@%3Cuser.avro.apache.org%3E > Basically, the getJsonProp method, being synchronized, is causing thread > contention issues when trying to share schemas between threads.My > proposal (pull request forthcoming) is to treat "props" as an immutable map > and do a copy+add+swap for the addProp method. This will make the addProp > call slower (particularly for large maps of props), but would make the reads > significantly faster as no locking will be needed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2051) Thread contention accessing JsonProperties props
[ https://issues.apache.org/jira/browse/AVRO-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16159144#comment-16159144 ] Daniel Kulp commented on AVRO-2051: --- Redid the pull request to base off a branch so I can keep my fork more in sync. Anyway, I made the hashCode and propsEquals package protected which should keep those from being exposed.I needed to add these methods as the subclass (Protocol) needs to know those, but I wanted to keep the "props" field private to make sure any subclass or other does not do any direct access to the props field. This forces subclasses (and others) to use the addProp method where we can make sure properties don't "change". (Properties can only be added, not modified) I don't think putting the ConcurrentHashMap into utils makes sense. This is a VERY particular use case and only certain methods are implemented for it. By keeping it anonymous and limiting access (via a private field), we can make sure the proper methods are called. For example, for a proper public class, keySet() would need to be implemented to return the keys in the proper order. I didn't implement it as, in this case, it's not ever called. Likewise, clear/remove/etc... are not implemented as they are not needed in this case. (the comment in the code mentions this) > Thread contention accessing JsonProperties props > > > Key: AVRO-2051 > URL: https://issues.apache.org/jira/browse/AVRO-2051 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.2 >Reporter: Daniel Kulp > > See > https://lists.apache.org/thread.html/dd34ab8439137a81a9de29ad4161f37b17638394cea0806765689976@%3Cuser.avro.apache.org%3E > Basically, the getJsonProp method, being synchronized, is causing thread > contention issues when trying to share schemas between threads.My > proposal (pull request forthcoming) is to treat "props" as an immutable map > and do a copy+add+swap for the addProp method. This will make the addProp > call slower (particularly for large maps of props), but would make the reads > significantly faster as no locking will be needed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (AVRO-2251) Modify Perf.java to better support automation scripts
[ https://issues.apache.org/jira/browse/AVRO-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2251. --- Resolution: Fixed Fix Version/s: 1.9.0 Merged. Thanks! > Modify Perf.java to better support automation scripts > - > > Key: AVRO-2251 > URL: https://issues.apache.org/jira/browse/AVRO-2251 > Project: Avro > Issue Type: Test >Reporter: Raymie Stata >Assignee: Raymie Stata >Priority: Major > Fix For: 1.9.0 > > > To better support automated performance-test suites, this patch proposes two > new arguments to the 'Perf.java' command-line tool: > The `-o' argument gives 'Perf.java' the name of a file that should get the > results of the run. Currently, Perf.java sends output to System.out – but if > 'Perf.java' is invoked using Maven, which is the easiest way to invoke it, > then System.out is polluted with a bunch of other output. Redirecting > 'Perf.java' output metrics to a file makes it easier for automation scripts > to process those metrics. > The `-c [spec]` argument tells 'Perf.java' to generate a comma-separated > output. By default, all benchmark metrics are output, but the optional > `spec` argument can be used to indicate exactly which metrics should be > included in the CSV output. The default output of 'Perf.java' is optimized > for human inspection – for example, it includes the text "ms" in the > running-time column so humans will understand the units of the running-time > metric. The `-c` option will tell 'Perf.java' to generate machine-optimized > output that is easier to consume by automation scripts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2208) Unneeded OSGi import generated for Guava dependency
[ https://issues.apache.org/jira/browse/AVRO-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2208. --- Resolution: Fixed > Unneeded OSGi import generated for Guava dependency > --- > > Key: AVRO-2208 > URL: https://issues.apache.org/jira/browse/AVRO-2208 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.2 >Reporter: Arnoud Glimmerveen >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.9.0 > > > The classes from Guava used by Avro are shaded into the final jar, however > the OSGi imports are calculated before shading, resulting in a > com.google.common* dependency declared in Import-Package manifest header. As > a consequence in an OSGi environment one has to install a version of Guava > even though that version will not be used by Avro. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2207) avro-maven-plugin don't specify encoding for files generated from avdls and avprs
[
https://issues.apache.org/jira/browse/AVRO-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2207.
---
Resolution: Fixed
Fix Version/s: 1.9.0
> avro-maven-plugin don't specify encoding for files generated from avdls and
> avprs
> -
>
> Key: AVRO-2207
> URL: https://issues.apache.org/jira/browse/AVRO-2207
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.2
>Reporter: Arek Burdach
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 1.9.0
>
>
> When someone generates source file from avdl/avpr, compiler uses system
> default encoding for output files. It can be different from this expected by
> project's compiler. It is quite usual thing that developers specify source
> encoding on project level using {{project.build.sourceEncoding}} property and
> expects that it will solve the problem.
> The problem doesn't exists when sources are generated from avsc. That
> implementation sets output encoding to this taken from
> {{project.build.sourceEncoding}}.
> Proposed solutions is to use the same behaviour in other two cases (avdl and
> avpr).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AVRO-2208) Unneeded OSGi import generated for Guava dependency
[ https://issues.apache.org/jira/browse/AVRO-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp reassigned AVRO-2208: - Assignee: Daniel Kulp > Unneeded OSGi import generated for Guava dependency > --- > > Key: AVRO-2208 > URL: https://issues.apache.org/jira/browse/AVRO-2208 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.2 >Reporter: Arnoud Glimmerveen >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.9.0 > > > The classes from Guava used by Avro are shaded into the final jar, however > the OSGi imports are calculated before shading, resulting in a > com.google.common* dependency declared in Import-Package manifest header. As > a consequence in an OSGi environment one has to install a version of Guava > even though that version will not be used by Avro. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2208) Unneeded OSGi import generated for Guava dependency
[ https://issues.apache.org/jira/browse/AVRO-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2208: -- Fix Version/s: 1.9.0 > Unneeded OSGi import generated for Guava dependency > --- > > Key: AVRO-2208 > URL: https://issues.apache.org/jira/browse/AVRO-2208 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.2 >Reporter: Arnoud Glimmerveen >Priority: Major > Fix For: 1.9.0 > > > The classes from Guava used by Avro are shaded into the final jar, however > the OSGi imports are calculated before shading, resulting in a > com.google.common* dependency declared in Import-Package manifest header. As > a consequence in an OSGi environment one has to install a version of Guava > even though that version will not be used by Avro. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AVRO-2207) avro-maven-plugin don't specify encoding for files generated from avdls and avprs
[
https://issues.apache.org/jira/browse/AVRO-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp reassigned AVRO-2207:
-
Assignee: Daniel Kulp
> avro-maven-plugin don't specify encoding for files generated from avdls and
> avprs
> -
>
> Key: AVRO-2207
> URL: https://issues.apache.org/jira/browse/AVRO-2207
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.2
>Reporter: Arek Burdach
>Assignee: Daniel Kulp
>Priority: Major
>
> When someone generates source file from avdl/avpr, compiler uses system
> default encoding for output files. It can be different from this expected by
> project's compiler. It is quite usual thing that developers specify source
> encoding on project level using {{project.build.sourceEncoding}} property and
> expects that it will solve the problem.
> The problem doesn't exists when sources are generated from avsc. That
> implementation sets output encoding to this taken from
> {{project.build.sourceEncoding}}.
> Proposed solutions is to use the same behaviour in other two cases (avdl and
> avpr).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-1126) Upgrade to Jackson 2+
[
https://issues.apache.org/jira/browse/AVRO-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673233#comment-16673233
]
Daniel Kulp commented on AVRO-1126:
---
I'm going to take Romain's work on flipping from Jackson to javax.json and
hopefully finish it up and get it in for 1.9. This will change a bunch of
public API's, but moving from Jackson 1.9 to anything will break things and
moving to the standard API's is probably a better option moving forward.
Which javax.json provider to default to can be decided later.
> Upgrade to Jackson 2+
> -
>
> Key: AVRO-1126
> URL: https://issues.apache.org/jira/browse/AVRO-1126
> Project: Avro
> Issue Type: Task
> Components: java
>Reporter: James Tyrrell
>Assignee: Charles Honton
>Priority: Critical
> Fix For: 1.9.0
>
>
> Quite annoyingly with Jackson 2+ the base package name has changed from
> org.codehaus.jackson to com.fasterxml.jackson so in addition to changing the
> dependencies from:
> {code:xml}
>
> org.codehaus.jackson
> jackson-core-asl
> ${jackson.version}
>
>
> org.codehaus.jackson
> jackson-mapper-asl
> ${jackson.version}
>
> {code}
> to:
> {code:xml}
>
> com.fasterxml.jackson.core
> jackson-core
> ${jackson.version}
>
>
> com.fasterxml.jackson.core
> jackson-databind
> ${jackson.version}
>
> {code}
> the base package in the code needs to be updated. More info can be found
> [here|http://wiki.fasterxml.com/JacksonUpgradeFrom19To20], I am happy to do
> the work just let me know what is preferable i.e. should I just attach a
> patch to this issue?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-1126) Upgrade to Jackson 2+
[
https://issues.apache.org/jira/browse/AVRO-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673644#comment-16673644
]
Daniel Kulp commented on AVRO-1126:
---
Yea I'm working on getting the patch for
https://issues.apache.org/jira/browse/AVRO-1605 updated and applyable. Hoping
to have that committed on master shortly. It's a start.
> Upgrade to Jackson 2+
> -
>
> Key: AVRO-1126
> URL: https://issues.apache.org/jira/browse/AVRO-1126
> Project: Avro
> Issue Type: Task
> Components: java
>Reporter: James Tyrrell
>Assignee: Charles Honton
>Priority: Critical
> Fix For: 1.9.0
>
>
> Quite annoyingly with Jackson 2+ the base package name has changed from
> org.codehaus.jackson to com.fasterxml.jackson so in addition to changing the
> dependencies from:
> {code:xml}
>
> org.codehaus.jackson
> jackson-core-asl
> ${jackson.version}
>
>
> org.codehaus.jackson
> jackson-mapper-asl
> ${jackson.version}
>
> {code}
> to:
> {code:xml}
>
> com.fasterxml.jackson.core
> jackson-core
> ${jackson.version}
>
>
> com.fasterxml.jackson.core
> jackson-databind
> ${jackson.version}
>
> {code}
> the base package in the code needs to be updated. More info can be found
> [here|http://wiki.fasterxml.com/JacksonUpgradeFrom19To20], I am happy to do
> the work just let me know what is preferable i.e. should I just attach a
> patch to this issue?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2051) Thread contention accessing JsonProperties props
[ https://issues.apache.org/jira/browse/AVRO-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2051. --- Resolution: Fixed Assignee: Daniel Kulp Fix Version/s: 1.9.0 > Thread contention accessing JsonProperties props > > > Key: AVRO-2051 > URL: https://issues.apache.org/jira/browse/AVRO-2051 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.2 >Reporter: Daniel Kulp >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.9.0 > > > See > https://lists.apache.org/thread.html/dd34ab8439137a81a9de29ad4161f37b17638394cea0806765689976@%3Cuser.avro.apache.org%3E > Basically, the getJsonProp method, being synchronized, is causing thread > contention issues when trying to share schemas between threads.My > proposal (pull request forthcoming) is to treat "props" as an immutable map > and do a copy+add+swap for the addProp method. This will make the addProp > call slower (particularly for large maps of props), but would make the reads > significantly faster as no locking will be needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2216) GenericData toString generates invalid JSON when map keys are not Strings
[
https://issues.apache.org/jira/browse/AVRO-2216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2216.
---
Resolution: Fixed
Assignee: Daniel Kulp
> GenericData toString generates invalid JSON when map keys are not Strings
> -
>
> Key: AVRO-2216
> URL: https://issues.apache.org/jira/browse/AVRO-2216
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Alberto Lago
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 1.9.0
>
>
> toString() method for records with maps that have non String Keys generates
> wrong Json:
> This code:
> {code:java}
> Schema intMapSchema = new Schema.Parser().parse("{\"type\": \"map\",
> \"values\": \"string\", \"java-key-class\" : \"java.lang.Integer\"}");
> Field intMapField = new Field("intMap", Schema.createMap(intMapSchema), null,
> null);
> Schema decMapSchema = new Schema.Parser().parse("{\"type\": \"map\",
> \"values\": \"string\", \"java-key-class\" : \"java.math.BigDecimal\"}");
> Field decMapField = new Field("decMap", Schema.createMap(decMapSchema), null,
> null);
> Schema boolMapSchema = new Schema.Parser().parse("{\"type\": \"map\",
> \"values\": \"string\", \"java-key-class\" : \"java.lang.Boolean\"}");
> Field boolMapField = new Field("boolMap", Schema.createMap(decMapSchema),
> null, null);
> Schema fileMapSchema = new Schema.Parser().parse("{\"type\": \"map\",
> \"values\": \"string\", \"java-key-class\" : \"java.io.File\"}");
> Field fileMapField = new Field("fileMap", Schema.createMap(decMapSchema),
> null, null);
> Schema schema = Schema.createRecord("my_record", "doc", "mytest", false);
> schema.setFields(Arrays.asList(intMapField,decMapField,boolMapField,fileMapField));
> HashMap intPair = new HashMap<>();
> intPair.put(1, "one");
> intPair.put(2, "two");
> HashMap decPair = new HashMap<>();
> decPair.put(java.math.BigDecimal.valueOf(1), "one");
> decPair.put(java.math.BigDecimal.valueOf(2), "two");
> HashMap boolPair = new HashMap<>();
> boolPair.put(true, "isTrue");
> boolPair.put(false, "isFalse");
> boolPair.put(null, null);
> HashMap filePair = new HashMap<>();
> java.io.File f = new java.io.File(
> getClass().getResource("/mapTestFile.txt").toURI() );
> filePair.put(f, "File");
> GenericRecord r = new GenericData.Record(schema);
> r.put(intMapField.name(), intPair);
> r.put(decMapField.name(), decPair);
> r.put(boolMapField.name(), boolPair);
> r.put(fileMapField.name(), filePair);
> String json = r.toString();
> {code}
> Would generate the following json:
> {code:java}
> {"intMap": {1: "one", 2: "two"}, "decMap": {2: "two", 1: "one"}, "boolMap":
> {null: null, false: "isFalse", true: "isTrue"}, "fileMap":
> {/workspace/avro/lang/java/avro/target/test-classes/mapTestFile.txt:
> "File"}}{code}
> It's missing double quotes for all keys.
> Note that I used classes that are considered as Stringable by ReflectData
> (plus Integer.class ) and thus, would be able to be serializable.
> [This change
> |https://github.com/apache/avro/compare/master...trompa:master?diff=split=master#diff-5a41450f3008ee0da59dec14ada2356aL543]to
> force the use of Strings always for Map Keys should be enough to fix this
> error.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2179) Malformed data, the BinaryDecoder allocates large arrays
[ https://issues.apache.org/jira/browse/AVRO-2179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2179. --- Resolution: Fixed Assignee: Daniel Kulp Fix Version/s: 1.9.0 For 1.9.0, I've added a system property of "org.apache.avro.limits.string.maxLength" that can be used to set a maximum string length to read for Utf8 strings. > Malformed data, the BinaryDecoder allocates large arrays > > > Key: AVRO-2179 > URL: https://issues.apache.org/jira/browse/AVRO-2179 > Project: Apache Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.2 >Reporter: Jonny >Assignee: Daniel Kulp >Priority: Critical > Fix For: 1.9.0 > > Attachments: image2018-5-10_11-7-40.png, image2018-5-11_21-27-56.png > > > Recentlly I use avro find it has this problem:The memory suddenly explode and > trigger the fullgc, So I dump the memory file. And then I use jvisualvm tool > to analysis I found a org.apache.avro.util.Utf8 object and the bytes array > length was over 1610470037. > The method of readInt() shouldn't over limit, So I think it should be > restricted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2127) Throwing more specific exception if an avro file has currupted magic
[ https://issues.apache.org/jira/browse/AVRO-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2127: -- Resolution: Fixed Assignee: Daniel Kulp (was: Dmitrii Bundin) Fix Version/s: 1.9.0 Status: Resolved (was: Patch Available) > Throwing more specific exception if an avro file has currupted magic > > > Key: AVRO-2127 > URL: https://issues.apache.org/jira/browse/AVRO-2127 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.8.2 >Reporter: Dmitrii Bundin >Assignee: Daniel Kulp >Priority: Minor > Fix For: 1.9.0 > > > Curently we have IOException thrown if an avro file has incorrect magic. > It seems reasonable to throw a subclass of IOException to be able to handle > incorrect magic (in case length are match) in user code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1126) Upgrade to Jackson 2+
[
https://issues.apache.org/jira/browse/AVRO-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673544#comment-16673544
]
Daniel Kulp commented on AVRO-1126:
---
Actually, flipping to javax.json will be a HUGE disruption to pretty much
everyone. Just updating to Jackson 2.9.7 is mostly a search/replace exercise
for package names and "getIntValue()" -> "intValue()". Thus, I think just
updating to 2.9.7 is likely a better option for users. It's a significantly
easier update.
> Upgrade to Jackson 2+
> -
>
> Key: AVRO-1126
> URL: https://issues.apache.org/jira/browse/AVRO-1126
> Project: Avro
> Issue Type: Task
> Components: java
>Reporter: James Tyrrell
>Assignee: Charles Honton
>Priority: Critical
> Fix For: 1.9.0
>
>
> Quite annoyingly with Jackson 2+ the base package name has changed from
> org.codehaus.jackson to com.fasterxml.jackson so in addition to changing the
> dependencies from:
> {code:xml}
>
> org.codehaus.jackson
> jackson-core-asl
> ${jackson.version}
>
>
> org.codehaus.jackson
> jackson-mapper-asl
> ${jackson.version}
>
> {code}
> to:
> {code:xml}
>
> com.fasterxml.jackson.core
> jackson-core
> ${jackson.version}
>
>
> com.fasterxml.jackson.core
> jackson-databind
> ${jackson.version}
>
> {code}
> the base package in the code needs to be updated. More info can be found
> [here|http://wiki.fasterxml.com/JacksonUpgradeFrom19To20], I am happy to do
> the work just let me know what is preferable i.e. should I just attach a
> patch to this issue?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2079) Add ability to use Java 8 date/time types instead of Joda time.
[
https://issues.apache.org/jira/browse/AVRO-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634562#comment-16634562
]
Daniel Kulp commented on AVRO-2079:
---
Note: I'm getting a random failure in one of the tests:
{code:java}
Failed tests:
TestSpecificLogicalTypes.testAbilityToReadJsr310RecordWrittenAsJodaRecord:148
Expected: is "16:10:08.420"
but: was "16:10:08.42"
{code}
> Add ability to use Java 8 date/time types instead of Joda time.
> ---
>
> Key: AVRO-2079
> URL: https://issues.apache.org/jira/browse/AVRO-2079
> Project: Avro
> Issue Type: Improvement
> Components: java, logical types
>Affects Versions: 1.8.2
>Reporter: Auke van Leeuwen
>Priority: Major
> Labels: patch-available
> Fix For: 1.9.0
>
>
> Currently, for the date/time related logical types, we are generating Joda
> date/time objects. Since we've moved to Java-8 (AVRO-2043) it seems logical
> to also provide the possibility to generate {{java.time.*}} date/time objects
> instead of the Joda time variants.
> I propose to make this is a switch in {{SpecificCompiler.java}} which will
> default to Joda (I think), but can be set to generate the Java 8 versions.
> (I'm currently trying to run through the code to see if I can make it work.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-1214) Generated protocol's method should return void instead of Void like one-way message
[
https://issues.apache.org/jira/browse/AVRO-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-1214:
--
Resolution: Fixed
Assignee: Daniel Kulp
Status: Resolved (was: Patch Available)
> Generated protocol's method should return void instead of Void like one-way
> message
> ---
>
> Key: AVRO-1214
> URL: https://issues.apache.org/jira/browse/AVRO-1214
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.7.2
>Reporter: Sébastien Launay
>Assignee: Daniel Kulp
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: AVRO-1214-unbox-void-2012-12-05.patch
>
>
> Using the following IDL:
> {noformat}
> @namespace("org.apache.avro.test")
> protocol Simple {
> error TestError {
> string message;
> }
> string hello(string greeting);
> void `error`() throws TestError;
> void ping() oneway;
> }
> {noformat}
> Will produce the interface:
> {code:java}
> package org.apache.avro.test;
> public interface Simple {
> public static final org.apache.avro.Protocol PROTOCOL =
> org.apache.avro.Protocol.parse("...");
> java.lang.CharSequence hello(java.lang.CharSequence greeting) throws
> org.apache.avro.AvroRemoteException;
> java.lang.Void error() throws org.apache.avro.AvroRemoteException,
> org.apache.avro.test.TestError;
> void ping();
> public interface Callback extends Simple {
> public static final org.apache.avro.Protocol PROTOCOL =
> org.apache.avro.test.Simple.PROTOCOL;
> void hello(java.lang.CharSequence greeting,
> org.apache.avro.ipc.Callback callback) throws
> java.io.IOException;
> void error(org.apache.avro.ipc.Callback callback) throws
> java.io.IOException;
> }
> }
> {code}
> Then one is forced to add {{return null;}} statement(s) in the interface
> implementation for the {{error()}} method which can be a bit cumbersome and
> that's not the case for the oneway {{ping()}} method.
> This is fine on the Callback though because the developer can just ignore the
> callback argument.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-1946) If user exists by default in docker, there will be an error, and the setup fails.
[ https://issues.apache.org/jira/browse/AVRO-1946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-1946: -- Resolution: Fixed Status: Resolved (was: Patch Available) patch applied > If user exists by default in docker, there will be an error, and the setup > fails. > - > > Key: AVRO-1946 > URL: https://issues.apache.org/jira/browse/AVRO-1946 > Project: Apache Avro > Issue Type: Bug > Components: docker >Affects Versions: 1.9.0 >Reporter: Suraj Acharya >Assignee: Suraj Acharya >Priority: Minor > Fix For: 1.9.0 > > Attachments: AVRO-1946.patch > > > Currently, when a user already exists in a docker, it errors and stops > running. > However, for a group it continues after throwing a warning. This is > inconsistent. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-1577) TestSpecificCompiler is not closing resources
[
https://issues.apache.org/jira/browse/AVRO-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-1577:
--
Resolution: Fixed
Assignee: Daniel Kulp
Status: Resolved (was: Patch Available)
Most of this was done already when we moved to Java 8 and switched to
try-with-resources. A few methods were missed which I just updated.
> TestSpecificCompiler is not closing resources
> -
>
> Key: AVRO-1577
> URL: https://issues.apache.org/jira/browse/AVRO-1577
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.7.7
> Environment: Windows
>Reporter: Stevo Slavic
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: AVRO-1577.patch
>
>
> Test methods in {{TestSpecificCompiler}} are opening various {{Closable}}
> resources, but they are not closing them. Because of this, file deletion in
> {{tearDown}} silently fails (on platforms like Windows which are locking
> files that are being used). This causes few test methods to fail since they
> are using same temp file as output file - they generate new file content but
> only if file is not already present, and then assertions comparing actual and
> expected output content fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-1843) Clarify importance of writer's schema in documentation
[ https://issues.apache.org/jira/browse/AVRO-1843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-1843: -- Resolution: Fixed Status: Resolved (was: Patch Available) PR was merged > Clarify importance of writer's schema in documentation > -- > > Key: AVRO-1843 > URL: https://issues.apache.org/jira/browse/AVRO-1843 > Project: Apache Avro > Issue Type: Improvement > Components: doc >Reporter: Shannon Carey >Priority: Critical > Fix For: 1.9.0 > > > I'll be submitting a PR with some improvements to the Java Getting Started > page as well as the Specification which make it clearer that Avro must read > all data with the writer's schema before converting it into the reader's > schema and why, and explaining that's why the schema should be available next > to serialized data. Currently, it's arguably too easy to misinterpret Avro as > only requiring a single, reader's schema in order to read data while still > following the resolution rules which make Avro seem similar to JSON > (resolution by field name). For example, the Java API examples only appear to > involve one schema, hiding the fact that it reads in the writer's schema > implicitly. Also, the ability to serialize to JSON (where field names and > some type info is present) makes this misconception easy to believe. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-1955) avrogen.exe supports only windows platform
[
https://issues.apache.org/jira/browse/AVRO-1955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-1955.
---
Resolution: Fixed
Assignee: Daniel Kulp
Fix Version/s: 1.9.0
> avrogen.exe supports only windows platform
> --
>
> Key: AVRO-1955
> URL: https://issues.apache.org/jira/browse/AVRO-1955
> Project: Apache Avro
> Issue Type: Bug
> Components: csharp
>Reporter: Naruto TAKAHASHI
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 1.9.0
>
>
> avrogen.exe output directory path separator is '\' .
> When executing avrogen.exe on macOS or Linux, output directory path is not
> expected.
> e.g. sample.avsc
> {code:javascript}
> {
> "type": "record",
> "name": "ExampleModel",
> "namespace": "com.example",
> "fields": [
> {
> "name": "num",
> "type": "long"
> },
> {
> "name": "str",
> "type": "string"
> }
> ]
> }
> {code}
> {code}
> mono ./build/codegen/Release/avrogen.exe -s sample.avsc output
> {code}
> avrogen.exe outputs below two regular files.
> - "output\com\example"
> - "output\com\example\ExampleModel.cs"
> avrogen.exe should use Path API of .NET when creating output directory path.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-1952) Prevent JIT compilation in C#
[ https://issues.apache.org/jira/browse/AVRO-1952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-1952. --- Resolution: Fixed Assignee: Daniel Kulp Fix Version/s: 1.9.0 > Prevent JIT compilation in C# > - > > Key: AVRO-1952 > URL: https://issues.apache.org/jira/browse/AVRO-1952 > Project: Apache Avro > Issue Type: Improvement > Components: csharp > Environment: Typical Unity (C#) development for mobile, iOS > compilation. >Reporter: Yuki Kuwabara >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.9.0 > > > C# code includes JIT API (Reflection.Emit) and it occurs error in iOS or > other platform that does not support JIT compile. > Reflection.Emit is used in very specific point and it can be replaced with > different safe code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-1956) C sharp build fails in docker image
[
https://issues.apache.org/jira/browse/AVRO-1956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-1956.
---
Resolution: Cannot Reproduce
Marking as not reproducible as with the latest Docker installs/versions and
latest "master" code, it seems to work fine.
> C sharp build fails in docker image
> ---
>
> Key: AVRO-1956
> URL: https://issues.apache.org/jira/browse/AVRO-1956
> Project: Apache Avro
> Issue Type: Bug
>Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Major
>
> When you run the build in the centos docker image you get
> {quote}
> Unhandled Exception:
> System.ArgumentException: NUnit components for version 2.0.50727 of the CLR
> are not installed
> {quote}
> As noted by [~Simon24601] in the mailing list:
> {quote}
> From discussion with [~jcustenborder], it seems like the C# tests won't run
> on a Mac if we keep the --framework=4.0 in this line; removing it (as has
> been done for 1.8.2) makes the tests run. But removing it makes the tests
> fail on my system (CentOS 7).
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2063) C: VERSION, SOVERSION not set correctly for shared library
[ https://issues.apache.org/jira/browse/AVRO-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2063. --- Resolution: Fixed Fix Version/s: 1.9.0 > C: VERSION, SOVERSION not set correctly for shared library > -- > > Key: AVRO-2063 > URL: https://issues.apache.org/jira/browse/AVRO-2063 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.8.2 >Reporter: Robert Edmonds >Priority: Minor > Fix For: 1.9.0 > > > Hi, > I'm attempting to package avro-c for Debian. While doing so, I noticed that > the libavro.so symlink that gets installed points directly to the > libavro.so.23.0.0 library file, without an intermediate "libavro.so.23" > symlink, and that the SONAME embedded in the library file is set to > "libavro.so.23.0.0", rather than "libavro.so.23", which is what I would > expect for a library trying to follow the libtool-style current/revision/age > scheme on Linux systems. > It looks like the recommended way to get CMake to produce the typical symlink > chain and SONAME is to set the VERSION property to "23.0.0" and the SOVERSION > to 23 on the shared library target. See this article for details: > http://pusling.com/blog/?p=352. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2008) change the error module to be thread-safe on Windows as well as on UNIX
[ https://issues.apache.org/jira/browse/AVRO-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2008: -- Resolution: Fixed Fix Version/s: (was: 1.8.3) 1.9.0 Status: Resolved (was: Patch Available) > change the error module to be thread-safe on Windows as well as on UNIX > --- > > Key: AVRO-2008 > URL: https://issues.apache.org/jira/browse/AVRO-2008 > Project: Apache Avro > Issue Type: Improvement > Components: c >Affects Versions: 1.8.1 >Reporter: Eliyahu Machluf >Assignee: Eliyahu Machluf >Priority: Major > Fix For: 1.9.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2026) Incorporate valgrind memory testing for C
[ https://issues.apache.org/jira/browse/AVRO-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2026: -- Resolution: Fixed Fix Version/s: (was: 1.8.3) 1.9.0 Status: Resolved (was: Patch Available) > Incorporate valgrind memory testing for C > - > > Key: AVRO-2026 > URL: https://issues.apache.org/jira/browse/AVRO-2026 > Project: Apache Avro > Issue Type: Improvement > Components: c >Reporter: John Gill >Priority: Major > Fix For: 1.9.0 > > Attachments: AVRO-2026.patch > > > With the work done for AVRO-1167 & AVRO-766 in identifying memory leaks > decided to make it a standard part of the testing framework. This issue > requires https://github.com/apache/avro/pull/218 otherwise test_avro_schema > and test_avro_values fail due to to avro_schema_copy calls. > Adding valgrind tool to the docker container > Adding valgrind to CMakeLists.txt to optionally do valgrind testing if the > tool is found > Cleaning up existing tests that were failing due to not properly cleaning up > memory > Lastly, as part of ./build.sh test for lang/c do not remove the build/c > directory. The memcheck output is kept so that memory leak issues can be > reviewed and addressed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2286) DataFileReader leaks file descriptor on exception during construction
[
https://issues.apache.org/jira/browse/AVRO-2286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2286.
---
Resolution: Fixed
Fix Version/s: 1.9.0
> DataFileReader leaks file descriptor on exception during construction
> -
>
> Key: AVRO-2286
> URL: https://issues.apache.org/jira/browse/AVRO-2286
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.2
>Reporter: Holger Fleischmann
>Assignee: Martin Jubelgas
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: DataFileReaderCloseBug.java
>
>
> The constructor {{DataFileReader(File, DatumReader)}} does not close the
> already opened file again when it throws an exception. This leads to a leaked
> file descriptor and especially to not being able to delete the still open
> invalid file then (at least on Windows).
> This case is important when Avro files may be truncated on write, e.g. due to
> a hard shutdown, and the application must be able to recover from such a
> situation on the next restart.
> Please see the attached file {{DataFileReaderCloseBug.java}} to reproduce the
> problem.
> I did not find that problem in Jira/mailing lists. Thanks for your help in
> advance :)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-1940) avrogencpp, scoped enums or c++11 enum class
[ https://issues.apache.org/jira/browse/AVRO-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-1940: -- Resolution: Fixed Fix Version/s: 1.9.0 Status: Resolved (was: Patch Available) 1.9 is using c++11 and now uses enum classes. > avrogencpp, scoped enums or c++11 enum class > > > Key: AVRO-1940 > URL: https://issues.apache.org/jira/browse/AVRO-1940 > Project: Apache Avro > Issue Type: Improvement > Components: c++ >Reporter: Alexander Moriarty >Assignee: Alexander Moriarty >Priority: Major > Fix For: 1.9.0 > > > It would be nice if avrogencpp could generated scoped enums. > Either with C++11 enum class or just adding an additional namespace. > This sort of relates to the issue I opened a few days ago: #AVRO-1936 in that > one way this has been gotten around in the existing AvrogencppTests.cc is by > generating one namespace per schema. > This of course could be a breaking change depending how it is implemented. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2058) ReflectData#isNonStringMap returns true for Utf8 keys
[
https://issues.apache.org/jira/browse/AVRO-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-2058:
--
Resolution: Fixed
Fix Version/s: 1.9.0
Status: Resolved (was: Patch Available)
> ReflectData#isNonStringMap returns true for Utf8 keys
> -
>
> Key: AVRO-2058
> URL: https://issues.apache.org/jira/browse/AVRO-2058
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.2
>Reporter: Sam Schlegel
>Assignee: Sam Schlegel
>Priority: Critical
> Fix For: 1.9.0
>
> Attachments: AVRO-2058.patch
>
>
> Since {{Utf8}} does not have an {{Stringable}} notation, and is not in
> {{SpecificData#stringableClasses}}, {{ReflectData#isNonStringMap}} returns
> true. This also causes {{ReflectData#isArray}} to return true for maps with
> Utf8 keys, and thus {{GenericData#resolveUnion}} fails as well. This
> ultimately causes {{ReflectData#write}} to fail for schemas that contain a
> union that contains a map, where the data uses Utf8 for strings.
> This following test case reproduces the issue:
> {code:java}
> @Test public void testUnionWithMapWithUtf8Keys() {
> Schema s = new Schema.Parser().parse
> ("[\"null\", {\"type\":\"map\",\"values\":\"float\"}]");
> GenericData data = ReflectData.get();
> HashMap map = new HashMap();
> map.put(new Utf8("foo"), 1.0f);
> assertEquals(1, data.resolveUnion(s, map));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-2069) Use primitive fields in generated getters & setters in Java code
[ https://issues.apache.org/jira/browse/AVRO-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2069: -- Resolution: Fixed Fix Version/s: 1.9.0 Status: Resolved (was: Patch Available) > Use primitive fields in generated getters & setters in Java code > > > Key: AVRO-2069 > URL: https://issues.apache.org/jira/browse/AVRO-2069 > Project: Apache Avro > Issue Type: Improvement >Affects Versions: 1.8.2 >Reporter: Daniil Gitelson >Assignee: Daniil Gitelson >Priority: Major > Fix For: 1.9.0 > > > Currently, for primitive types (such as int, long, etc) generated getters and > setters return and accept java.lang.* boxed (while fields actually holds > primitive values). This is inefeccient and produces code boilerplate. > Changed this behaviour in pull request: > https://github.com/apache/avro/pull/243 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-1844) Avro-C build procedure doesn't set include paths for Jansson
[ https://issues.apache.org/jira/browse/AVRO-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-1844. --- Resolution: Fixed Fix Version/s: 1.9.0 > Avro-C build procedure doesn't set include paths for Jansson > > > Key: AVRO-1844 > URL: https://issues.apache.org/jira/browse/AVRO-1844 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.8.0 > Environment: gcc 4.9.3, Scientific Linux Fermi release 6.7 >Reporter: Jim Pivarski >Priority: Major > Fix For: 1.9.0 > > > I `configure` Jansson with a user-space `PREFIX` because I don't have root > access, and when I `make install`, the Jansson includes and libs appear in > new `include` and `lib` directories, as expected. > I `cmake` with `-DCMAKE_INSTALL_PREFIX:PATH=` the same path as `PREFIX` and > CMake is successful: it finds Jansson and makes the Makefiles. > BUT, when I `make`, it fails because it can't find `jansson.h`. The same is > true if I explicitly set `PKG_CONFIG_PATH` to point to > `$PREFIX/lib/pkgconfig` and `pkg-config jansson --cflags --libs` works. > If I explicitly add full paths to Avro's and Jansson's `#include ` > lines in the source code, Avro compiles without trouble. > (`avro-c-1.8.0/src/schema.c`, `avro-c-1.8.0/src/value-json.c`, > `include/jansson.h`) > Presumably, the Jansson part of the build process is only partly implemented, > such that the CMake script searches for the appropriate version but `-I` > flags are not generated for the compiler. I don't know enough CMake to know > where to insert these. > Fixing this issue is the only way I know of to make Avro-C installable on a > system without root access, apart from the hack I described above. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2004) Stats View is not working
[ https://issues.apache.org/jira/browse/AVRO-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2004: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Stats View is not working > - > > Key: AVRO-2004 > URL: https://issues.apache.org/jira/browse/AVRO-2004 > Project: Apache Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.1 >Reporter: Ken Dombeck >Assignee: Ken Dombeck >Priority: Minor > Fix For: 1.9.0 > > Attachments: WorkingStatsPage.png > > > When trying to view the IPC stats page the charts are not being displayed. > The issue is that the CSS and JS resources are not being included in the jar > artifact. > There are 2 possible solutions that I see to this. > # Move the CSS and JS resources to be with the Velocity template that > generates the HTML. > # Change Maven to include the resources where they currently are. > Github [pull request|https://github.com/apache/avro/pull/202] has been > created for option 1. > This relates to the feature created in > [AVRO-587|https://issues.apache.org/jira/browse/AVRO-587] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2007) for avro-c enable getting enum size (i.e. number of symbols at enum schema)
[ https://issues.apache.org/jira/browse/AVRO-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2007: -- Resolution: Duplicate Fix Version/s: (was: 1.8.1) 1.9.0 Status: Resolved (was: Patch Available) Duplicate of AVRO-1528 > for avro-c enable getting enum size (i.e. number of symbols at enum schema) > --- > > Key: AVRO-2007 > URL: https://issues.apache.org/jira/browse/AVRO-2007 > Project: Apache Avro > Issue Type: Improvement > Components: c >Affects Versions: 1.8.1 >Reporter: Eliyahu Machluf >Priority: Minor > Fix For: 1.9.0 > > > Avro-c library stores internaly enum as int value, there is no validation, > when setting an enum value, it match the schema. > Therefore, I want to enable getting enum size (i.e. number of symbols at enum > schema), in order to enable validating (when using the instance) that the > value is at enum range. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2065) Avro java code generation for Unions doesn't set converters for unions
[
https://issues.apache.org/jira/browse/AVRO-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2065.
---
Resolution: Fixed
Fix Version/s: (was: 1.8.3)
1.9.0
> Avro java code generation for Unions doesn't set converters for unions
> --
>
> Key: AVRO-2065
> URL: https://issues.apache.org/jira/browse/AVRO-2065
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.2
>Reporter: Stephane Maarek
>Priority: Blocker
> Fix For: 1.9.0
>
>
> The full issue is here:
> https://stackoverflow.com/questions/45581437/how-to-specify-converter-for-default-value-in-avro-union-logical-type-fields
> {code}
> {
> "name":"my_optional_date",
> "type":[
> {
>"type":"long",
>"logicalType":"timestamp-millis"
> },
> "null"
> ],
> "default":1502250227187
> }
> {code}
> Doesn't generate the right conversions
> {code}
> private static final org.apache.avro.Conversion[] conversions =
> new org.apache.avro.Conversion[] {
> null, // <-- THIS ONE IS NOT SET PROPERLY, should be
> TIMESTAMP_CONVERSION
> null
> };
> {code}
> The code fails on
> {code}
> BuggyRecord.Builder buggyRecordBuilder = BuggyRecord.newBuilder();
> buggyRecordBuilder.build();
> {code}
> with
> {code}
> org.apache.avro.AvroRuntimeException: java.lang.ClassCastException:
> java.lang.Long cannot be cast to org.joda.time.DateTime
> at com.example.BuggyRecord$Builder.build(BuggyRecord.java:301)
> at BuggyRecordTest.Foo(BuggyRecordTest.java:10)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to
> org.joda.time.DateTime
> at com.example.BuggyRecord$Builder.build(BuggyRecord.java:298)
> ... 23 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-1749) Maven plugin goal: automatic schemas (.avsc) generation from Java classes (.java)
[ https://issues.apache.org/jira/browse/AVRO-1749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-1749. --- Resolution: Fixed Assignee: Daniel Kulp Fix Version/s: 1.9.0 > Maven plugin goal: automatic schemas (.avsc) generation from Java classes > (.java) > - > > Key: AVRO-1749 > URL: https://issues.apache.org/jira/browse/AVRO-1749 > Project: Apache Avro > Issue Type: New Feature > Components: java >Reporter: Matheus Santana >Assignee: Daniel Kulp >Priority: Minor > Fix For: 1.9.0 > > > Current maven plugin includes goals for generating Java code (classes and > interfaces) from IDL (.avdl files) and Avro protocol / schemas definitions > (.avpr / .avsc). > It would be nice to provide a goal for automatic [induced schemas from Java > code|https://avro.apache.org/docs/current/api/java/org/apache/avro/tool/InduceSchemaTool.html]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-1777) Select best matching record when writing a union in python
[
https://issues.apache.org/jira/browse/AVRO-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-1777.
---
Resolution: Fixed
Assignee: Daniel Kulp
Fix Version/s: 1.9.0
> Select best matching record when writing a union in python
> --
>
> Key: AVRO-1777
> URL: https://issues.apache.org/jira/browse/AVRO-1777
> Project: Apache Avro
> Issue Type: Improvement
> Components: python
>Affects Versions: 1.7.7
>Reporter: Steven Aerts
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 1.9.0
>
>
> Unlike javascript, python is not using wrapped types.
> So when writing a union it needs to guess find out which type it will output.
> At the moment it takes the last validating type.
> I propose to take the type with the most matching fields.
> So I propose to change in {{io.py}}:
> {code}
> # resolve union
> index_of_schema = -1
> for i, candidate_schema in enumerate(writers_schema.schemas):
> if validate(candidate_schema, datum):
> index_of_schema = i
> if index_of_schema < 0: raise AvroTypeException(writers_schema, datum)
> {code}
> into
> {code}
> # resolve union
> index_of_schema = -1
> found_fields = -1
> for i, candidate_schema in enumerate(writers_schema.schemas):
> if validate(candidate_schema, datum):
> nr_fields = candidate_schema.type in ['record', 'error', 'request'] and
> len(candidate_schema.fields) or 1
> if nr_fields > found_fields:
> index_of_schema = i
> found_fields = nr_fields
> if index_of_schema < 0: raise AvroTypeException(writers_schema, datum)
> {code}
> If you want, I can create a pull request for this. And apply it both on py3
> as py.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2034) Nested schema types with unexpected fields causes json parse failure
[
https://issues.apache.org/jira/browse/AVRO-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2034.
---
Resolution: Fixed
Assignee: Daniel Kulp
Fix Version/s: 1.9.0
> Nested schema types with unexpected fields causes json parse failure
>
>
> Key: AVRO-2034
> URL: https://issues.apache.org/jira/browse/AVRO-2034
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.1
>Reporter: Todd Nine
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 1.9.0
>
>
> When parsing a nested type with an unexpected field using the JSON parser,
> this results in an error. To reproduce, see the class {{TestNestedRecords}}
> in the referenced PR.
> https://github.com/apache/avro/pull/224
> Note that this only occurs when the following pattern exists in the schema.
> # regular field
> # nested record with additional field
> # Any subsequent field following the nested record with an unexpected field
> appears to reproduce the problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2131) Records with unions with self references fail to parse.
[
https://issues.apache.org/jira/browse/AVRO-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2131.
---
Resolution: Fixed
Assignee: Daniel Kulp
Fix Version/s: 1.9.0
> Records with unions with self references fail to parse.
> ---
>
> Key: AVRO-2131
> URL: https://issues.apache.org/jira/browse/AVRO-2131
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.1
> Environment: Java 1.8.0_152-b1
>Reporter: Jeff Maxwell
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 1.9.0
>
>
> Records with unions with self references fail to parse.
> The example below fails to parse with {{"Type not supported: Node"}}
> {code:javascript}
> [
> {
> "namespace": "tree",
> "type": "record",
> "name": "Node",
> "fields": [
> {
> "name": "left",
> "type": [
> "null",
> {
> "type": "Node"
> }
> ],
> "default": null
> },
> {
> "name": "right",
> "type": [
> "null",
> {
> "type": "Node"
> }
> ],
> "default": null
> }
> ]
> }
> ]
> {code}
> When we don't allow nullability it parses successfully:
> {code:javascript}
> [
> {
> "namespace": "tree",
> "type": "record",
> "name": "Node",
> "fields": [
> {
> "name": "left",
> "type": "Node"
> },
> {
> "name": "right",
> "type": "Node"
> }
> ]
> }
> ]
> {code}
> The root cause: When the second element of the union, {{\{"type":"Node"}\}},
> is parsed there is no path that can successfully handle the {{JsonNode}}.
> The solution is to add this logic to the {{Schema.parse(JsonNode schema,
> Names names)}} method:
> {code:java}
> } else { //For unions with self reference
> Name nameFromType = new Name(type, names.space);
> if (names.containsKey(nameFromType)) {
> return names.get(nameFromType);
> }
> throw new SchemaParseException("Type not supported: "+type);
> }
> {code}
> Pull request attached
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2129) Add getEncoder method to generated Java class
[ https://issues.apache.org/jira/browse/AVRO-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2129. --- Resolution: Fixed Assignee: Daniel Kulp Fix Version/s: 1.9.0 > Add getEncoder method to generated Java class > - > > Key: AVRO-2129 > URL: https://issues.apache.org/jira/browse/AVRO-2129 > Project: Apache Avro > Issue Type: Improvement >Reporter: Michael Hewitt >Assignee: Daniel Kulp >Priority: Minor > Fix For: 1.9.0 > > > Add the getEncoder method to to access encoder instance > Unless the encoder should be protected not sure why there is access to the > decoder and not the encoder. If there is a better way to get a > BinaryMessageEncoder for this schemaStore let me know. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-1904) C: record with no fields should be allowed
[ https://issues.apache.org/jira/browse/AVRO-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-1904: -- Resolution: Fixed Fix Version/s: 1.9.0 Status: Resolved (was: Patch Available) > C: record with no fields should be allowed > -- > > Key: AVRO-1904 > URL: https://issues.apache.org/jira/browse/AVRO-1904 > Project: Apache Avro > Issue Type: Bug > Components: c >Reporter: Ben Walsh >Priority: Minor > Fix For: 1.9.0 > > Attachments: AVRO-1904.patch > > > Avro-C should allow record with no fields. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-1906) C: allow file with no records
[ https://issues.apache.org/jira/browse/AVRO-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-1906: -- Resolution: Fixed Fix Version/s: 1.9.0 Status: Resolved (was: Patch Available) > C: allow file with no records > - > > Key: AVRO-1906 > URL: https://issues.apache.org/jira/browse/AVRO-1906 > Project: Apache Avro > Issue Type: Bug > Components: c >Reporter: Ben Walsh >Priority: Minor > Fix For: 1.9.0 > > Attachments: AVRO-1906.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-1528) avro_schema_enum_get returns invalid pointer for unknown indices
[ https://issues.apache.org/jira/browse/AVRO-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-1528. --- Resolution: Fixed Fix Version/s: 1.9.0 > avro_schema_enum_get returns invalid pointer for unknown indices > > > Key: AVRO-1528 > URL: https://issues.apache.org/jira/browse/AVRO-1528 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.7.6 >Reporter: Jeno I. Hajdu >Priority: Trivial > Fix For: 1.9.0 > > Attachments: AVRO-1528.patch > > > When avro_schema_enum_get() is called with invalid indices it returns an > invalid pointer instead of NULL, this leads to segfaults. Similar problem is > already handled in avro_schema_enum_get_by_name() when converting the other > way round, avro_schema_enum_get() could handle it likewise. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-1542) std::auto_ptr
[ https://issues.apache.org/jira/browse/AVRO-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-1542. --- Resolution: Fixed Assignee: Daniel Kulp > std::auto_ptr > - > > Key: AVRO-1542 > URL: https://issues.apache.org/jira/browse/AVRO-1542 > Project: Apache Avro > Issue Type: Improvement > Components: c++ >Affects Versions: 1.7.6 >Reporter: Sean Middleditch >Assignee: Daniel Kulp >Priority: Trivial > Fix For: 1.9.0 > > > std::auto_ptr is deprecated, meaning that it may be removed from a future > version of the C++ standard (though vendors would not likely remove it... > probably). > Avro should at its next API-breaking opportunity replace its use of auto_ptr > with std::unique_ptr or shared_ptr or a custom smart pointer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2155) Generate documentation for C# classes and enums
[ https://issues.apache.org/jira/browse/AVRO-2155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2155: -- Resolution: Fixed Assignee: Daniel Kulp Fix Version/s: 1.9.0 Status: Resolved (was: Patch Available) > Generate documentation for C# classes and enums > --- > > Key: AVRO-2155 > URL: https://issues.apache.org/jira/browse/AVRO-2155 > Project: Apache Avro > Issue Type: New Feature > Components: csharp >Affects Versions: 1.8.2 >Reporter: Brian Lachniet >Assignee: Daniel Kulp >Priority: Minor > Fix For: 1.9.0 > > > Update the C# "avrogen" code generation to populate documentation for code > entities (classes and enums) that are generated from named schemas (record, > enum and fixed). At this time, it only generates documentation for Avro > fields (C# fields and properties). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2156) Allow custom namespace for C# generated code
[ https://issues.apache.org/jira/browse/AVRO-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2156: -- Resolution: Fixed Assignee: Daniel Kulp Fix Version/s: 1.9.0 Status: Resolved (was: Patch Available) > Allow custom namespace for C# generated code > > > Key: AVRO-2156 > URL: https://issues.apache.org/jira/browse/AVRO-2156 > Project: Apache Avro > Issue Type: New Feature > Components: csharp >Affects Versions: 1.8.2 >Reporter: Brian Lachniet >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.9.0 > > > The namespace used in Avro schemas/protocols does not always map nicely to a > standard C# namespace. Avrogen (the C# code generator) currently uses the > Avro namespace as the C# namespace. I propose that we update the avrogen > utility to allow the user to override the namespace used in the generated > code via a command line parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-1363) C# UnionSchema fails to parse the the unions with same type names with different namespaces
[
https://issues.apache.org/jira/browse/AVRO-1363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-1363.
---
Resolution: Fixed
Assignee: Daniel Kulp
Fix Version/s: 1.9.0
> C# UnionSchema fails to parse the the unions with same type names with
> different namespaces
> ---
>
> Key: AVRO-1363
> URL: https://issues.apache.org/jira/browse/AVRO-1363
> Project: Apache Avro
> Issue Type: Bug
> Components: csharp
>Affects Versions: 1.7.4
>Reporter: Ramana Suvarapu
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: reader.avpr
>
>
> C# Union schema is unable to handle same type names with different
> namespaces. It's throwing "SchemaParseException("Duplicate type in union:
> "...) exception.
> In the below code, key of uniqueSchemas should be FullName of the type which
> is similar to Java implementation.
> internal static UnionSchema NewInstance(JArray jarr, PropertyMap props,
> SchemaNames names, string encspace)
> {
> List schemas = new List();
> IDictionary uniqueSchemas = new
> Dictionary();
> foreach (JToken jvalue in jarr)
> {
> Schema unionType = Schema.ParseJson(jvalue, names, encspace);
> if (null == unionType)
> throw new SchemaParseException("Invalid JSON in union" +
> jvalue.ToString());
> string name = unionType.Name;
> if (uniqueSchemas.ContainsKey(name))
> throw new SchemaParseException("Duplicate type in union:
> " + name);
> uniqueSchemas.Add(name, name);
> schemas.Add(unionType);
> }
> return new UnionSchema(schemas, props);
> }
> See the attachement to recreate the issue
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-2281) Performance improvement in ruby binary decoder and encoder
[ https://issues.apache.org/jira/browse/AVRO-2281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2281: -- Resolution: Fixed Assignee: Daniel Kulp Fix Version/s: 1.9.0 Status: Resolved (was: Patch Available) > Performance improvement in ruby binary decoder and encoder > -- > > Key: AVRO-2281 > URL: https://issues.apache.org/jira/browse/AVRO-2281 > Project: Apache Avro > Issue Type: Improvement > Components: ruby >Affects Versions: 1.9.0 >Reporter: Kyle Phelps >Assignee: Daniel Kulp >Priority: Minor > Fix For: 1.9.0 > > > The ruby binary decoder has some inefficient memory usage patterns. The > decoding process relies on `unpack` quite heavily, but it allocates an array > that is not used as we just access the first element of the result. In ruby > 2.4 we can use the optimized version of this, `unpack1` which avoids > allocating the unused array. In `byte!`, we can go one step further and just > use the `readbyte` method provided by the IO module - this improves the > performance of `byte!` by about 50%. Additionally, there are a few strings in > the encoder and decoder that should be frozen to reduce unnecessary string > allocations. > > With these changes, I've seen about a 20% performance improvement when > decoding. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-1702) Add LogicalType support to c++ library
[ https://issues.apache.org/jira/browse/AVRO-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-1702: -- Fix Version/s: 1.9.0 > Add LogicalType support to c++ library > -- > > Key: AVRO-1702 > URL: https://issues.apache.org/jira/browse/AVRO-1702 > Project: Apache Avro > Issue Type: New Feature > Components: c++ >Reporter: peter liu >Assignee: Aniket Mokashi >Priority: Major > Fix For: 1.9.0 > > > I'd like to port the logicaltype support to c++ library -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-1939) Add command line tool for generating canonical form of schema
[ https://issues.apache.org/jira/browse/AVRO-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-1939. --- Resolution: Fixed Assignee: Daniel Kulp Fix Version/s: 1.9.0 > Add command line tool for generating canonical form of schema > - > > Key: AVRO-1939 > URL: https://issues.apache.org/jira/browse/AVRO-1939 > Project: Apache Avro > Issue Type: Improvement >Reporter: Erik Forsberg >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.9.0 > > > Being able to generate canonical form of schemas from avro-tools would be > useful, especially when writing canonical schema support in other languages > (i.e AVRO-1938). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-1743) BlockingBinaryEncoder should override writeFixed(ByteBuffer bytes)
[ https://issues.apache.org/jira/browse/AVRO-1743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-1743. --- Resolution: Fixed Assignee: Daniel Kulp Fix Version/s: 1.9.0 > BlockingBinaryEncoder should override writeFixed(ByteBuffer bytes) > -- > > Key: AVRO-1743 > URL: https://issues.apache.org/jira/browse/AVRO-1743 > Project: Apache Avro > Issue Type: Bug > Components: java >Affects Versions: 1.7.6, 1.7.7 > Environment: All environments >Reporter: Dmitry Spikhalskiy >Assignee: Daniel Kulp >Priority: Major > Labels: starter > Fix For: 1.9.0 > > > BlockingBinaryEncoder which extends BufferedBinaryEncoder should override > "public void writeFixed(ByteBuffer bytes) throws IOException" method. > Now if we use BlockingBinaryEncoder - all writeFixed(ByteBuffer bytes) are > addressed by BufferedBinaryEncoder. As a result, if "!bytes.hasArray() && > bytes.remaining() > bulkLimit", then in flushBuffer() we flush empty buffer > from BufferedBinaryEncoder and don't flush actual buffer from > BlockingBinaryEncoder. > I prepared localized unit tests to replicate bugs here: > https://github.com/Spikhalskiy/avro-blockingbinaryencoder-error > Bug could appears in silently incorrect serialization (We will read another > object) or in deserialization errors. Both replicated in provided tests. > Looks like BlockingBinaryEncoder which extends BufferedBinaryEncoder is > error-prone approach and mistake in class hierarchy. We mostly override > everything from BufferedBinaryEncoder, creating unused buffers and fields > (like double pos, buf, etc), and it's already not first bug relating to > "somebody forget to override method in BlockingBinaryEncoder from > BufferedBinaryEncoder" (ex. https://issues.apache.org/jira/browse/AVRO-88). > So, this classes should be separated at all or have common interface, or at > least work with same buffer and pos instances. But BlockingBinaryEncoder > shouldn't inherit method implementations, which work with another buffer > object. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2250) Release 1.9.0
[ https://issues.apache.org/jira/browse/AVRO-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698183#comment-16698183 ] Daniel Kulp commented on AVRO-2250: --- Fokko: in 1.8.x guava was used in several places, not just tests. For 1.9, I removed the guava uses in the non-tests and you beat me to the tests. :) > Release 1.9.0 > - > > Key: AVRO-2250 > URL: https://issues.apache.org/jira/browse/AVRO-2250 > Project: Apache Avro > Issue Type: Task >Reporter: Nandor Kollar >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-1658) Add avroDoc on reflect
[
https://issues.apache.org/jira/browse/AVRO-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-1658:
--
Resolution: Fixed
Fix Version/s: 1.9.0
Status: Resolved (was: Patch Available)
> Add avroDoc on reflect
> --
>
> Key: AVRO-1658
> URL: https://issues.apache.org/jira/browse/AVRO-1658
> Project: Apache Avro
> Issue Type: New Feature
> Components: java
>Affects Versions: 1.7.7
>Reporter: Zhaonan Sun
>Assignee: Zhaonan Sun
>Priority: Major
> Labels: reflection
> Fix For: 1.9.0
>
> Attachments:
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch,
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch,
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch
>
>
> Looks like @AvroMeta can't add reserved fields, like @AvroMeta("doc", "some
> doc") will have exceptions.
> I would be greate if we have a @AvroDoc("some documentations") in
> org.apache.avro.reflect
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-1961) [JAVA] Generate getters that return an Optional
[ https://issues.apache.org/jira/browse/AVRO-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-1961. --- Resolution: Fixed Fix Version/s: 1.9.0 > [JAVA] Generate getters that return an Optional > --- > > Key: AVRO-1961 > URL: https://issues.apache.org/jira/browse/AVRO-1961 > Project: Apache Avro > Issue Type: New Feature >Reporter: Niels Basjes >Assignee: Niels Basjes >Priority: Minor > Fix For: 1.9.0 > > > A colleague of mine indicated that having getters that return an Optional > (java 8 thingy) would be very useful for him. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2250) Release 1.9.0
[ https://issues.apache.org/jira/browse/AVRO-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693971#comment-16693971 ] Daniel Kulp commented on AVRO-2250: --- Not sure yet... we're slowly going through almost 2 years of open pull requests. It's taking time, but at least we're moving forward. > Release 1.9.0 > - > > Key: AVRO-2250 > URL: https://issues.apache.org/jira/browse/AVRO-2250 > Project: Apache Avro > Issue Type: Task >Reporter: Nandor Kollar >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-1902) C: namespace "" should mean namespace = NULL
[
https://issues.apache.org/jira/browse/AVRO-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-1902:
--
Resolution: Fixed
Assignee: Daniel Kulp
Fix Version/s: 1.9.0
Status: Resolved (was: Patch Available)
> C: namespace "" should mean namespace = NULL
>
>
> Key: AVRO-1902
> URL: https://issues.apache.org/jira/browse/AVRO-1902
> Project: Apache Avro
> Issue Type: Bug
> Components: c
>Reporter: Ben Walsh
>Assignee: Daniel Kulp
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: AVRO-1902.patch
>
>
> Similar to AVRO-1295 but for the C library. Namespace "" should be the NULL
> namespace when the schema is parsed. For example this breaks:
> ```
> {"type": "record", "name": "R", "fields": [
> {"name": "s", "type": {"type": "record", "namespace": "x", "name": "Y",
> "fields": [
> {"name": "e", "type": {"type": "record", "namespace": "", "name": "Z",
> "fields": [
> {"name": "f", "type": "Z"}
> ]}}
> ]}},
> {"name": "t", "type": "Z"}
> ]}
> ```
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2290) TestSpecificLogicalTypes.testRecordWithJsr310LogicalTypes breaks on Java 11
[
https://issues.apache.org/jira/browse/AVRO-2290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2290.
---
Resolution: Fixed
Assignee: Daniel Kulp
Fix Version/s: 1.9.0
> TestSpecificLogicalTypes.testRecordWithJsr310LogicalTypes breaks on Java 11
> ---
>
> Key: AVRO-2290
> URL: https://issues.apache.org/jira/browse/AVRO-2290
> Project: Apache Avro
> Issue Type: Sub-task
> Components: java
>Reporter: Ismaël Mejía
>Assignee: Daniel Kulp
>Priority: Minor
> Fix For: 1.9.0
>
>
> Java 11 has nano-second precision so the test breaks, the test should now
> detect this and adapt accordingly.
> {code}
> [ERROR]
> testRecordWithJsr310LogicalTypes(org.apache.avro.specific.TestSpecificLo
> gicalTypes) Time elapsed: 0.24 s <<< FAILURE!
> java.lang.AssertionError: Should match written record expected:<{"b": true,
> "i32
> ": 34, "i64": 35, "f32": 3.14, "f64": 3019.34, "s": null, "d": 2018-12-20,
> "t":
> 15:02:53.535103, "ts": 2018-12-20T14:02:53.535127Z, "dec": 123.45}> but
> was:<{"b
> ": true, "i32": 34, "i64": 35, "f32": 3.14, "f64": 3019.34, "s": null, "d":
> 2018
> -12-20, "t": 15:02:53.535, "ts": 2018-12-20T14:02:53.535Z, "dec": 123.45}>
> at
> org.apache.avro.specific.TestSpecificLogicalTypes.testRecordWithJsr31
> 0LogicalTypes(TestSpecificLogicalTypes.java:132)
> [ERROR]
> testAbilityToReadJodaRecordWrittenAsJsr310Record(org.apache.avro.specifi
> c.TestSpecificLogicalTypes) Time elapsed: 0.005 s <<< FAILURE!
> java.lang.AssertionError:
> Expected: is "15:02:53.639158"
> but: was "15:02:53.639"
> at
> org.apache.avro.specific.TestSpecificLogicalTypes.testAbilityToReadJo
> daRecordWrittenAsJsr310Record(TestSpecificLogicalTypes.java:204)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2335) Remove Joda Time Library
[
https://issues.apache.org/jira/browse/AVRO-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805121#comment-16805121
]
Daniel Kulp commented on AVRO-2335:
---
I'm not sure if this is something that we want for 1.9... As of right now,
for 1.9, we've made JSR-310 the default. However, Joda is there as an option
(and the dependency is marked optional) to aid in migrating code from 1.8 to
1.9. That said, we'd like to fully remove it at some point, I'm just
questioning whether it should be for 1.9 or 1.10.
> Remove Joda Time Library
>
>
> Key: AVRO-2335
> URL: https://issues.apache.org/jira/browse/AVRO-2335
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: David Mollitor
>Assignee: Fokko Driesprong
>Priority: Major
>
> Since [AVRO-2079] was completed, Avro supports Java's standard {{java.time}}
> libraries. Please remove the Joda Time implementations and the Joda Time
> dependency from Avro. Avro 2.0 perhaps?
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. *Users are now asked to migrate to java.time (JSR-310).*
> https://www.joda.org/joda-time/
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-2077) Avro tools should have an option to check reader-writer compatibility
[ https://issues.apache.org/jira/browse/AVRO-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2077: -- Fix Version/s: (was: 1.9.0) > Avro tools should have an option to check reader-writer compatibility > - > > Key: AVRO-2077 > URL: https://issues.apache.org/jira/browse/AVRO-2077 > Project: Apache Avro > Issue Type: New Feature > Components: java, tools >Reporter: Nandor Kollar >Assignee: Nandor Kollar >Priority: Minor > > It seems that avro-tools doesn't have any option to check for a given Avro > file and a given reader schema (JSON file or URL) that the reader's schema is > compatible with the writer's schema. This tool should report every > compatibility problem (if there's any). > According my knowledge, there's no such option for avro-tools as of now. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-732) Generated protocol's method should not throw AvroRemoteException
[ https://issues.apache.org/jira/browse/AVRO-732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-732. -- Resolution: Fixed Assignee: Daniel Kulp Hadoop Flags: Incompatible change > Generated protocol's method should not throw AvroRemoteException > > > Key: AVRO-732 > URL: https://issues.apache.org/jira/browse/AVRO-732 > Project: Apache Avro > Issue Type: Bug > Components: java >Reporter: Sharad Agarwal >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.9.0 > > Attachments: > AVRO-732-remove-generated-remote-exception-2012-12-11.patch, > AVRO-732-remove-generated-remote-exception-2012-12-17.patch > > > If user does NOT define the throws clause in the idl, the code is generated > with "throws AvroRemoteException" clause. However on throwing the > AvroRemoteException from the implementation, the serialization fails. This is > not intuitive to users. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2368) Provide way to distinguish whether a Schema.Field has a default value or its default is null
[
https://issues.apache.org/jira/browse/AVRO-2368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2368.
---
Resolution: Fixed
Assignee: Daniel Kulp
Field.hasDefaultValue() method added.
> Provide way to distinguish whether a Schema.Field has a default value or its
> default is null
>
>
> Key: AVRO-2368
> URL: https://issues.apache.org/jira/browse/AVRO-2368
> Project: Apache Avro
> Issue Type: Task
> Components: java
>Reporter: Ivan Greene
>Assignee: Daniel Kulp
>Priority: Blocker
> Fix For: 1.9.0
>
>
> After making {{Schema.Field#defaultValue}} package private, there is no way
> to distinguish whether a Field has a default value of 'null' or has no
> default at all. This is a breaking change, notably affecting Hortonworks
> Schema Registry. See discussion here:
> [https://github.com/hortonworks/registry/pull/547]
> I propose we should make that method public and still deprecated. Otherwise I
> don't see a clear way to release a version of the Hortonworks registry that
> will be compatible with both Avro 1.8.x and 1.9.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-1275) GenericData's toString() method generates the wrong JSON encoding for the bytes type
[
https://issues.apache.org/jira/browse/AVRO-1275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-1275.
---
Resolution: Fixed
Assignee: Daniel Kulp
Hadoop Flags: Incompatible change
> GenericData's toString() method generates the wrong JSON encoding for the
> bytes type
>
>
> Key: AVRO-1275
> URL: https://issues.apache.org/jira/browse/AVRO-1275
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Reporter: Tom White
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 1.9.0
>
>
> We discovered in AVRO-1274 that for bytes GenericData.toString(Object datum)
> incorrectly generates
> {noformat}
> { "bytes" : "A" }
> {noformat}
> rather than just {{"A"}} for the JSON encoding defined in the spec
> (http://avro.apache.org/docs/current/spec.html#json_encoding).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-1777) Select best matching record when writing a union in python
[
https://issues.apache.org/jira/browse/AVRO-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16810955#comment-16810955
]
Daniel Kulp commented on AVRO-1777:
---
I'm re-opening the issue and removing it from 1.9.0.The patch causes
interop test failures which is obviously a major problem. I don't know enough
python to figure it out.
> Select best matching record when writing a union in python
> --
>
> Key: AVRO-1777
> URL: https://issues.apache.org/jira/browse/AVRO-1777
> Project: Apache Avro
> Issue Type: Improvement
> Components: python
>Affects Versions: 1.7.7
>Reporter: Steven Aerts
>Assignee: Daniel Kulp
>Priority: Major
>
> Unlike javascript, python is not using wrapped types.
> So when writing a union it needs to guess find out which type it will output.
> At the moment it takes the last validating type.
> I propose to take the type with the most matching fields.
> So I propose to change in {{io.py}}:
> {code}
> # resolve union
> index_of_schema = -1
> for i, candidate_schema in enumerate(writers_schema.schemas):
> if validate(candidate_schema, datum):
> index_of_schema = i
> if index_of_schema < 0: raise AvroTypeException(writers_schema, datum)
> {code}
> into
> {code}
> # resolve union
> index_of_schema = -1
> found_fields = -1
> for i, candidate_schema in enumerate(writers_schema.schemas):
> if validate(candidate_schema, datum):
> nr_fields = candidate_schema.type in ['record', 'error', 'request'] and
> len(candidate_schema.fields) or 1
> if nr_fields > found_fields:
> index_of_schema = i
> found_fields = nr_fields
> if index_of_schema < 0: raise AvroTypeException(writers_schema, datum)
> {code}
> If you want, I can create a pull request for this. And apply it both on py3
> as py.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-1289) Python: Schema objects should polymorphically interact with data-walker interface
[
https://issues.apache.org/jira/browse/AVRO-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-1289:
--
Fix Version/s: (was: 1.9.0)
> Python: Schema objects should polymorphically interact with data-walker
> interface
> -
>
> Key: AVRO-1289
> URL: https://issues.apache.org/jira/browse/AVRO-1289
> Project: Apache Avro
> Issue Type: Improvement
> Components: python
>Affects Versions: 1.7.5
>Reporter: Jeremy Kahn
>Assignee: Jeremy Kahn
>Priority: Minor
> Labels: features
>
> Python {{avro.schema}} objects should be able to call back to a general
> data-and-schema parallel-walker ("validate" would be one of those, but so
> could be "default-filler").
> There should be an {{avro.walker}} interface that owns the parallel state (a
> datum-reader/deserializer, a datum-writer/serializer, a validator, or a
> default-filler -- see AVRO-1265). Schema polymorphism would allow us to
> eliminate the large (and highly redundant) function-dispatch methods in
> {{avro.io}} by making the {{avro.schema.Schema}} subclass responsible for
> calling back to the {{avro.walker}} object.
> Assigning this to v1.8.0 because it may be difficult to duplicate *every*
> behavior of 1.7.* with the same function signatures, especially where this
> refactor may be eliminate entire classes.
> This factoring ought to make it easier to improve or extend objects that meet
> this {{walker}} interface -- validators & serializers might be able to store
> more state about their position within a record, for example, to yield more
> informative error messages upon mismatch (as requested by Jonathan Coveney on
> the user mailing list).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-1777) Select best matching record when writing a union in python
[
https://issues.apache.org/jira/browse/AVRO-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-1777:
--
Fix Version/s: (was: 1.9.0)
> Select best matching record when writing a union in python
> --
>
> Key: AVRO-1777
> URL: https://issues.apache.org/jira/browse/AVRO-1777
> Project: Apache Avro
> Issue Type: Improvement
> Components: python
>Affects Versions: 1.7.7
>Reporter: Steven Aerts
>Assignee: Daniel Kulp
>Priority: Major
>
> Unlike javascript, python is not using wrapped types.
> So when writing a union it needs to guess find out which type it will output.
> At the moment it takes the last validating type.
> I propose to take the type with the most matching fields.
> So I propose to change in {{io.py}}:
> {code}
> # resolve union
> index_of_schema = -1
> for i, candidate_schema in enumerate(writers_schema.schemas):
> if validate(candidate_schema, datum):
> index_of_schema = i
> if index_of_schema < 0: raise AvroTypeException(writers_schema, datum)
> {code}
> into
> {code}
> # resolve union
> index_of_schema = -1
> found_fields = -1
> for i, candidate_schema in enumerate(writers_schema.schemas):
> if validate(candidate_schema, datum):
> nr_fields = candidate_schema.type in ['record', 'error', 'request'] and
> len(candidate_schema.fields) or 1
> if nr_fields > found_fields:
> index_of_schema = i
> found_fields = nr_fields
> if index_of_schema < 0: raise AvroTypeException(writers_schema, datum)
> {code}
> If you want, I can create a pull request for this. And apply it both on py3
> as py.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-1738) add java tool for outputting schema fingerprints
[ https://issues.apache.org/jira/browse/AVRO-1738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-1738. --- Resolution: Fixed > add java tool for outputting schema fingerprints > > > Key: AVRO-1738 > URL: https://issues.apache.org/jira/browse/AVRO-1738 > Project: Apache Avro > Issue Type: New Feature > Components: java >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.9.0 > > Attachments: AVRO-1738.1.patch > > > over in AVRO-1694 I wanted to quickly check that the Java library came up > with the same md5/sha fingerprint for some shcemas that the proposed Ruby > implementation does. > I noticed we don't have a tool that exposes the functionality yet, which > seems like a commonly useful thing to do. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-1777) Select best matching record when writing a union in python
[
https://issues.apache.org/jira/browse/AVRO-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-1777.
---
Resolution: Fixed
Assignee: Daniel Kulp
> Select best matching record when writing a union in python
> --
>
> Key: AVRO-1777
> URL: https://issues.apache.org/jira/browse/AVRO-1777
> Project: Apache Avro
> Issue Type: Improvement
> Components: python
>Affects Versions: 1.7.7
>Reporter: Steven Aerts
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 1.9.0
>
>
> Unlike javascript, python is not using wrapped types.
> So when writing a union it needs to guess find out which type it will output.
> At the moment it takes the last validating type.
> I propose to take the type with the most matching fields.
> So I propose to change in {{io.py}}:
> {code}
> # resolve union
> index_of_schema = -1
> for i, candidate_schema in enumerate(writers_schema.schemas):
> if validate(candidate_schema, datum):
> index_of_schema = i
> if index_of_schema < 0: raise AvroTypeException(writers_schema, datum)
> {code}
> into
> {code}
> # resolve union
> index_of_schema = -1
> found_fields = -1
> for i, candidate_schema in enumerate(writers_schema.schemas):
> if validate(candidate_schema, datum):
> nr_fields = candidate_schema.type in ['record', 'error', 'request'] and
> len(candidate_schema.fields) or 1
> if nr_fields > found_fields:
> index_of_schema = i
> found_fields = nr_fields
> if index_of_schema < 0: raise AvroTypeException(writers_schema, datum)
> {code}
> If you want, I can create a pull request for this. And apply it both on py3
> as py.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2368) Provide way to distinguish whether a Schema.Field has a default value or its default is null
[
https://issues.apache.org/jira/browse/AVRO-2368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16810855#comment-16810855
]
Daniel Kulp commented on AVRO-2368:
---
Yes. The goal is to not expose the JSON implementation in the public API's.
> Provide way to distinguish whether a Schema.Field has a default value or its
> default is null
>
>
> Key: AVRO-2368
> URL: https://issues.apache.org/jira/browse/AVRO-2368
> Project: Apache Avro
> Issue Type: Task
> Components: java
>Reporter: Ivan Greene
>Assignee: Daniel Kulp
>Priority: Blocker
> Fix For: 1.9.0
>
>
> After making {{Schema.Field#defaultValue}} package private, there is no way
> to distinguish whether a Field has a default value of 'null' or has no
> default at all. This is a breaking change, notably affecting Hortonworks
> Schema Registry. See discussion here:
> [https://github.com/hortonworks/registry/pull/547]
> I propose we should make that method public and still deprecated. Otherwise I
> don't see a clear way to release a version of the Hortonworks registry that
> will be compatible with both Avro 1.8.x and 1.9.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (AVRO-1777) Select best matching record when writing a union in python
[
https://issues.apache.org/jira/browse/AVRO-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp reopened AVRO-1777:
---
> Select best matching record when writing a union in python
> --
>
> Key: AVRO-1777
> URL: https://issues.apache.org/jira/browse/AVRO-1777
> Project: Apache Avro
> Issue Type: Improvement
> Components: python
>Affects Versions: 1.7.7
>Reporter: Steven Aerts
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 1.9.0
>
>
> Unlike javascript, python is not using wrapped types.
> So when writing a union it needs to guess find out which type it will output.
> At the moment it takes the last validating type.
> I propose to take the type with the most matching fields.
> So I propose to change in {{io.py}}:
> {code}
> # resolve union
> index_of_schema = -1
> for i, candidate_schema in enumerate(writers_schema.schemas):
> if validate(candidate_schema, datum):
> index_of_schema = i
> if index_of_schema < 0: raise AvroTypeException(writers_schema, datum)
> {code}
> into
> {code}
> # resolve union
> index_of_schema = -1
> found_fields = -1
> for i, candidate_schema in enumerate(writers_schema.schemas):
> if validate(candidate_schema, datum):
> nr_fields = candidate_schema.type in ['record', 'error', 'request'] and
> len(candidate_schema.fields) or 1
> if nr_fields > found_fields:
> index_of_schema = i
> found_fields = nr_fields
> if index_of_schema < 0: raise AvroTypeException(writers_schema, datum)
> {code}
> If you want, I can create a pull request for this. And apply it both on py3
> as py.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2202) Use of -fstack-protector or -D_GLIBCXX_DEBUG causes memory corruption when combined with musl libc static builds
[ https://issues.apache.org/jira/browse/AVRO-2202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2202. --- Resolution: Fixed Assignee: Daniel Kulp (was: Thiruvalluvan M. G.) I'm not able to reproduce any crash with "normal" libc setups. However, I did update the CMakeLists.txt to do two things: 1) Make sure we preserve existing CMAKE_CXX_FLAGS and CMAKE_CXX_FLAGS_DEBUG so things like the -g are not removed. This is definitely important in trying to debug things. 2) Only add the above problematic flags if AVRO_ADD_PROTECTOR_FLAGS is set which should only be set for our "build.sh test" runs. It's not set for all Debug builds so you should be able to do your own build without those flags and hopefully then not have your crash. > Use of -fstack-protector or -D_GLIBCXX_DEBUG causes memory corruption when > combined with musl libc static builds > > > Key: AVRO-2202 > URL: https://issues.apache.org/jira/browse/AVRO-2202 > Project: Apache Avro > Issue Type: Bug > Components: c++ >Affects Versions: 1.8.2 > Environment: OS: Gentoo Linux > Compiler: Gcc 6.4.0 targeting musl libc 1.19 with stdlibc++ in a static build >Reporter: Josh Scoggins >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.9.0 > > > Observed with master and version 1.8.2. Inside of the C++ implementation's > CMakeLists.txt the lines 56-60 were added sometime in 2016 which activate > features inside of libstdc++ to help find errors. This change has two side > effects: > # -g is eliminated from the build so debugging gdb becomes harder as source > position data has been removed > # The safety containers that implicitly wrap all standard containers somehow > (I can't figure out exactly how) cause memory corruption when setMetadata is > called (in version 1.8.2) or when a Writer object is destroyed (on master). > Since our software uses musl libc with libstdc++ I believe that it is an > interaction with these debugging features which cause memory corruption (as I > said, I have no clue _why_ it is happening, only that these features _cause_ > it :() > Previously, we were using 1.7.7 which did not have these lines in its > CMakeLists.txt and thus did not exhibit any sort of memory corruption. I was > able to work around this by removing lines 56-60 from CMakeLists.txt so our > static libs could be built and our unit tests not crash by writing to memory > it did not own. There should be a toggle or something similar to turn off > these features for debug builds as they are for developers of avro only. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2369) Provide external way to construct Schema.Field with default value of 'null'
[
https://issues.apache.org/jira/browse/AVRO-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2369.
---
Resolution: Fixed
Assignee: Daniel Kulp
I added a Field.NULL_DEFAULT_VALUE object that can be used to specify that a
null should be set as the default for the union. Not the greatest solution,
but I was struggling to find any alternative that wouldn't cause a gigantic
migration issue.
> Provide external way to construct Schema.Field with default value of 'null'
> ---
>
> Key: AVRO-2369
> URL: https://issues.apache.org/jira/browse/AVRO-2369
> Project: Apache Avro
> Issue Type: Task
> Components: java
>Reporter: Ivan Greene
>Assignee: Daniel Kulp
>Priority: Blocker
> Fix For: 1.9.0
>
>
> After making the Schema.Field constructor which takes the default value as a
> JsonNode was made package private, there is no external way to construct a
> field that has a default value of 'null'. Internally that constructor will
> call {{JacksonUtils.toJsonNode(defaultValue)}}, which will return 'null' when
> passed null, and the resulting Field will not have a default value (the json
> node would need to be NullNode instead of simply null itself). This will
> affect projects that need a way to dynamically build schemas and their fields.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2360) Java 8 timestamp-millis / timestamp-micros type inconsistency
[ https://issues.apache.org/jira/browse/AVRO-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2360. --- Resolution: Fixed Assignee: Daniel Kulp Hadoop Flags: Incompatible change Fix Version/s: 1.9.0 > Java 8 timestamp-millis / timestamp-micros type inconsistency > - > > Key: AVRO-2360 > URL: https://issues.apache.org/jira/browse/AVRO-2360 > Project: Apache Avro > Issue Type: Bug > Components: java >Affects Versions: 1.9.0 > Environment: JDK 11.0.2 on Linux >Reporter: Raman Gupta >Assignee: Daniel Kulp >Priority: Blocker > Fix For: 1.9.0 > > > I am trying the new JSR 310 date/time types with a snapshot of 1.9.0. > I see what seems to be an inconsistency. If I generate my code with a logical > type of "timestamp-millis", then the code is generated with `Instant`, as > expected. However, on JDK11 on Linux, if I do `Instant.now()` the Instant > value created contains microseconds. When setting this Instant on an instance > of the generated Avro SpecificRecord, I am unable to round-trip the data: > Before serialization to bytes: > System.out.println(mySpecificRecord.tsMillis()) // 2019-03-27T23:16:09.665308Z > After serializing to bytes, and then back into a specific record the > microseconds are now truncated: > System.out.println(mySpecificRecord.tsMillis()) // 2019-03-27T23:16:09.665Z > I believe the compiler should generate a class that truncates the > microseconds at `setter` time for "timestamp-millis", so that the value > before serialization, and after deserialization, are the same. This can be > done with a call to the method `truncatedTo(ChronoUnit.MILLIS)`. > Another, possibly related, oddity is that the "timestamp-micros" type > generates a class with `long` as the type of the field. Since Instant can > support both milli and micro-second precision, I don't see the reason for > this behavior. > Followup on AVRO-2079. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2355) Add compressionLevel to ZStandard compression
[ https://issues.apache.org/jira/browse/AVRO-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2355. --- Resolution: Fixed Assignee: Daniel Kulp > Add compressionLevel to ZStandard compression > - > > Key: AVRO-2355 > URL: https://issues.apache.org/jira/browse/AVRO-2355 > Project: Apache Avro > Issue Type: New Feature > Components: java >Reporter: Scott Carey >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.9.0 > > > ZStandard compression should not be released without support for compression > level selection. > Its biggest advantage is the massive range by which you can select the > compression level, all while keeping decompression throughput very high. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2354) Add CombineAvroKeyValueFileInputFormat in avro-mapred to combine small avro keyvalue files into combineSplit
[ https://issues.apache.org/jira/browse/AVRO-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2354. --- Resolution: Fixed Assignee: Daniel Kulp Fix Version/s: 1.9.0 > Add CombineAvroKeyValueFileInputFormat in avro-mapred to combine small avro > keyvalue files into combineSplit > > > Key: AVRO-2354 > URL: https://issues.apache.org/jira/browse/AVRO-2354 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Wang, Xinglong >Assignee: Daniel Kulp >Priority: Minor > Fix For: 1.9.0 > > > In our production env, we generate avro files to track some user behavior > events. Every hour, we will have several avro files created. And daily, we > will run MR to do analysis, when using AvroKeyValueInputFormat, a lot of > small mappers started due to we have small avro files. > A combine file inputformat will be very helpful for such case. > Hadoop already provided some implementation for sequencefile and text file. > This Jira is propose a CombineAvroKeyValueFileInputFormat class to implement > the same for avro keyvalue files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2357) (ReflectData) Support for generic types in protocol definitions
[
https://issues.apache.org/jira/browse/AVRO-2357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2357.
---
Resolution: Fixed
> (ReflectData) Support for generic types in protocol definitions
> ---
>
> Key: AVRO-2357
> URL: https://issues.apache.org/jira/browse/AVRO-2357
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: Ivan Greene
>Priority: Minor
> Fix For: 1.9.0
>
>
> For a Java interface extending another interface with type parameters, we may
> resolve actual type parameters and build the protocol definition based upon
> those.
> For example, let's say we have a generic protocol defined by a Java interface:
> {code:java}
> public interface CrudProto {
> void persist(T record);
> T fetchById(I id);
> }{code}
> It would be natural to define a set of interfaces that extend this, such as:
> {code:java}
> public interface FooBarRecordProto extends CrudProto {}
> public interface OtherRecordProto extends CrudProto {}
> {code}
> Calling ReflectData.get().getProtocol(FooBarRecordProto.class) should be able
> to resolve that this protocol deals in FooBarRecords and Strings, and build a
> protocol accordingly.
> Currently, this call will produce an exception stating that a schema for 'T'
> cannot be resolved.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2020) Update BUILD instructions with correct versions for language-specific libraries
[
https://issues.apache.org/jira/browse/AVRO-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2020.
---
Resolution: Fixed
Fix Version/s: (was: 1.7.9)
(was: 1.8.3)
> Update BUILD instructions with correct versions for language-specific
> libraries
> ---
>
> Key: AVRO-2020
> URL: https://issues.apache.org/jira/browse/AVRO-2020
> Project: Apache Avro
> Issue Type: Task
> Components: community, doc
>Affects Versions: 1.7.7, 1.8.1, 1.9.0
>Reporter: Sean Busbey
>Priority: Critical
> Fix For: 1.9.0
>
>
> our BUILD instructions current list a few things I know are wrong (Ruby 1.8
> support) and some things I'm pretty sure are wrong (Java 1.6 support).
> we should do a pass checking all the languages.
> {code}
> REQUIREMENTS
> The following packages must be installed before Avro can be built:
> - Java: JDK 1.6, Maven 2 or better, protobuf-compile
> - PHP: php5, phpunit, php5-gmp
> - Python: 2.5 or greater, python-setuptools for dist target
> - C: gcc, cmake, asciidoc, source-highlight
> - C++: cmake 2.8.4 or greater, g++, flex, bison, libboost-dev
> - C#: mono-devel mono-gmcs nunit
> - JavaScript: nodejs, npm
> - Ruby: ruby 1.86 or greater, ruby-dev, gem, rake, echoe, yajl-ruby
> - Perl: perl 5.8.1 or greater, gmake, Module::Install,
>Module::Install::ReadmeFromPod, Module::Install::Repository,
>Math::BigInt, JSON::XS, Try::Tiny, Regexp::Common, Encode,
>IO::String, Object::Tiny, Compress::ZLib, Test::More,
>Test::Exception, Test::Pod
> - Apache Ant 1.7
> - Apache Forrest 0.8 (for documentation)
> - md5sum, sha1sum, used by top-level dist target
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2182) avro-maven-plugin: IDLProtocolMojo should regard property "project.build.sourceEncoding" for file generation
[ https://issues.apache.org/jira/browse/AVRO-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2182. --- Resolution: Fixed > avro-maven-plugin: IDLProtocolMojo should regard property > "project.build.sourceEncoding" for file generation > > > Key: AVRO-2182 > URL: https://issues.apache.org/jira/browse/AVRO-2182 > Project: Apache Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.2 > Environment: Tested on windows with Java 8. >Reporter: Hans-Peter Werner >Priority: Major > Fix For: 1.9.0 > > Attachments: JIRA-AVRO-2182.patch > > > Currently it's immpossible to use e.g. german Umlauts in neither documention > nor fieldnames. > Linked to [AVRO-1471|http://issues.apache.org/jira/browse/AVRO-1471] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2251) Modify Perf.java to better support automation scripts
[ https://issues.apache.org/jira/browse/AVRO-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2251. --- Resolution: Fixed > Modify Perf.java to better support automation scripts > - > > Key: AVRO-2251 > URL: https://issues.apache.org/jira/browse/AVRO-2251 > Project: Apache Avro > Issue Type: Test > Components: java >Reporter: Raymie Stata >Assignee: Raymie Stata >Priority: Major > Fix For: 1.9.0 > > > To better support automated performance-test suites, this patch proposes two > new arguments to the 'Perf.java' command-line tool: > The `-o' argument gives 'Perf.java' the name of a file that should get the > results of the run. Currently, Perf.java sends output to System.out – but if > 'Perf.java' is invoked using Maven, which is the easiest way to invoke it, > then System.out is polluted with a bunch of other output. Redirecting > 'Perf.java' output metrics to a file makes it easier for automation scripts > to process those metrics. > The `-c [spec]` argument tells 'Perf.java' to generate a comma-separated > output. By default, all benchmark metrics are output, but the optional > `spec` argument can be used to indicate exactly which metrics should be > included in the CSV output. The default output of 'Perf.java' is optimized > for human inspection – for example, it includes the text "ms" in the > running-time column so humans will understand the units of the running-time > metric. The `-c` option will tell 'Perf.java' to generate machine-optimized > output that is easier to consume by automation scripts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-1626) Missing lang/csharp/src/apache/perf/app.config
[
https://issues.apache.org/jira/browse/AVRO-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-1626.
---
Resolution: Fixed
> Missing lang/csharp/src/apache/perf/app.config
> --
>
> Key: AVRO-1626
> URL: https://issues.apache.org/jira/browse/AVRO-1626
> Project: Apache Avro
> Issue Type: Bug
> Components: csharp
>Reporter: Niels Basjes
>Priority: Major
> Fix For: 1.9.0
>
>
> This error is output during the build
> {code}
> Target _CopyAppConfigFile:
> /usr/lib/mono/4.5/Microsoft.Common.targets: error : Cannot copy
> /home/nbasjes/avro/lang/csharp/src/apache/perf/app.config to
> /home/nbasjes/avro/lang/csharp/build/perf/Release/Avro.perf.exe.config, as
> the source file doesn't exist.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-1559) Drop support for Ruby 1.8
[ https://issues.apache.org/jira/browse/AVRO-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-1559. --- Resolution: Fixed 1.9 will be Ruby 2.3.3+ > Drop support for Ruby 1.8 > - > > Key: AVRO-1559 > URL: https://issues.apache.org/jira/browse/AVRO-1559 > Project: Apache Avro > Issue Type: Wish > Components: ruby >Affects Versions: 1.7.7 >Reporter: Willem van Bergen >Assignee: Willem van Bergen >Priority: Major > Fix For: 1.9.0 > > Attachments: AVRO-1559.patch > > > - Ruby 1.8 is EOL, and is even security issues aren't addressed anymore. > - It is also getting hard to set up Ruby 1.8 to run the tests (e.g. on a > recent OSX, it won't compile without manual fiddling). > - Handling character encodings in Ruby 1.9 is very different than Ruby 1.8. > Supporting both at the same time adds a lot of overhead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2365) Enhancements to induce maven plugin goal
[ https://issues.apache.org/jira/browse/AVRO-2365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2365: -- Fix Version/s: (was: 1.8.3) > Enhancements to induce maven plugin goal > > > Key: AVRO-2365 > URL: https://issues.apache.org/jira/browse/AVRO-2365 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Ivan Greene >Priority: Major > Fix For: 1.9.0 > > > Before a release of 1.8.3 or 1.9.0 which will introduce the new 'induce' > maven plugin goal (see AVRO-1749), there are a few issues with it which in my > mind are critical to enable real-world use of the goal: > * Allow excluding classes based on a pattern, and/or the ability to specify > multiple sourceDirectories > * Option to specify whether to use ReflectData.AllowNull or ReflectData > * Throw MojoExecutionException with an informative message when appropriate, > instead of just printing stacktraces. This is very important, we definitely > need a build to fail when we hit a problem. > * Use the current project defined encoding rather than hard coded UTF-8. > * Use try-with-resources to ensure the PrintWriter gets closed in the event > of exception > Nice to haves: > * Specify custom ReflectData implementation (i.e user declared extensions) > to be used > * Specify separate output directories for schemata and protocols (these > could both default to the single output directory if not specified > individually) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2365) Enhancements to induce maven plugin goal
[ https://issues.apache.org/jira/browse/AVRO-2365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2365. --- Resolution: Fixed Assignee: Daniel Kulp > Enhancements to induce maven plugin goal > > > Key: AVRO-2365 > URL: https://issues.apache.org/jira/browse/AVRO-2365 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Ivan Greene >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.9.0 > > > Before a release of 1.8.3 or 1.9.0 which will introduce the new 'induce' > maven plugin goal (see AVRO-1749), there are a few issues with it which in my > mind are critical to enable real-world use of the goal: > * Allow excluding classes based on a pattern, and/or the ability to specify > multiple sourceDirectories > * Option to specify whether to use ReflectData.AllowNull or ReflectData > * Throw MojoExecutionException with an informative message when appropriate, > instead of just printing stacktraces. This is very important, we definitely > need a build to fail when we hit a problem. > * Use the current project defined encoding rather than hard coded UTF-8. > * Use try-with-resources to ensure the PrintWriter gets closed in the event > of exception > Nice to haves: > * Specify custom ReflectData implementation (i.e user declared extensions) > to be used > * Specify separate output directories for schemata and protocols (these > could both default to the single output directory if not specified > individually) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2364) Use Jackson TokenBuffer in JsonDecoder
[ https://issues.apache.org/jira/browse/AVRO-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2364. --- Resolution: Fixed Assignee: Daniel Kulp Fix Version/s: 1.9.0 > Use Jackson TokenBuffer in JsonDecoder > -- > > Key: AVRO-2364 > URL: https://issues.apache.org/jira/browse/AVRO-2364 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.8.2 >Reporter: Anthony Miyaguchi >Assignee: Daniel Kulp >Priority: Trivial > Fix For: 1.9.0 > > > The JsonDecoder implements a class for replaying parts of the JSON stream > when accessing fields out of order. > This module can take advantage of > [TokenBuffers|http://static.javadoc.io/com.fasterxml.jackson.core/jackson-databind/2.9.8/com/fasterxml/jackson/databind/util/TokenBuffer.html] > to reduce code size. This buffer can be read back as a parser. > > [https://github.com/apache/avro/blob/db992cf8686ffa681b0c592696b37ad519c33ee6/lang/java/avro/src/main/java/org/apache/avro/io/JsonDecoder.java#L514-L754] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-1922) Fixed dimension for array
[
https://issues.apache.org/jira/browse/AVRO-1922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-1922:
--
Fix Version/s: (was: 1.9.0)
> Fixed dimension for array
> -
>
> Key: AVRO-1922
> URL: https://issues.apache.org/jira/browse/AVRO-1922
> Project: Apache Avro
> Issue Type: New Feature
> Components: spec
>Affects Versions: 1.8.1
>Reporter: Jim Pivarski
>Priority: Major
>
> This is a feature request for future versions of the Avro specification.
> We have found one kind of data structure that is hard to express in Avro:
> tensors. Although we can (and do) build matrices as {"type": "array",
> "items": {"type": "array", "items": "double"}}, this type does not specify
> that the grid of numbers is rectangular. We believe that rectangular arrays
> of numbers (or other nested types) would be a strong addition to Avro, both
> as a type system and as a serialization format. With the total size of all
> dimensions fixed in the schema, they would not need to be repeated in each
> serialized datum.
> For instance, suppose there was an extension of type "array" to specify
> dimensions:
> {"type": "array", "dimensions": [3, 3, 3, 3], "items": "double"}
> This 3-by-3-by-3-by-3 tensor (representing, for instance, the Riemann
> curvature tensor in 3-space) specifies that 81 double-precision numbers
> (3*3*3*3) are expected for each datum. With nested arrays, the size, "3,"
> would have to be separately encoded 40 times (1 + 3*(1 + 3*(1 + 3))) for each
> datum, even though they would never change in a dataset of Riemann tensors.
> With a "dimensions" attribute in the schema, only the content needs to be
> serialized. Moreover, this extension can clearly be used with any other
> "items" type, to make dense tables of strings, for instance.
> Avro has been extended in a similar way in the past. The "fixed" type is a
> "bytes" without the need to specify the number of bytes for each datum. Our
> proposal provides a similar packing for structured objects that can be
> significant for large numbers of dimensions, as shown above. The advantage to
> consumers of Avro data is that we can write functions that do not need to
> check all array sizes at runtime (for operations like tensor contractions and
> products).
> We have searched the web and the Avro JIRA site for similar proposals and
> found none, so we're adding this proposal to JIRA in addition to this e-mail.
> Please let us know if you have any comments, or if we can provide any more
> information.
> Thank you!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-1547) AvroApp Schema Tool
[ https://issues.apache.org/jira/browse/AVRO-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-1547: -- Fix Version/s: (was: 1.9.0) > AvroApp Schema Tool > --- > > Key: AVRO-1547 > URL: https://issues.apache.org/jira/browse/AVRO-1547 > Project: Apache Avro > Issue Type: New Feature > Components: java >Reporter: Lewis John McGibbney >Assignee: Lewis John McGibbney >Priority: Minor > > Over in Gora, I have been thinking for a while that the process of writing > JSON data beans is rather time consuming when beans are LARGE. > I wanted to open this ticket for a while and now only get around to. I > proposed to have the following > A simple HTML webpage that defines a form of sorts, the form will enable > users to create JSON schemas and will be driven by enabling users to enter > Object values based on the current Avro specification document e.g. ti will > be restrictive in scope. > On top of this I propose to then use simple JQuery to send a request to the > JSONBlob API [0], obtain a JSON representation of the data and then pretty > print write this information to a file within the browser. The users can then > save this file focally and do with it what they wish. > I think that this page can easily be hosted alongside the current static Avro > website and that there is no need to write a web application for this yet. > I'll try to work on it sooner rather than later as this would also lower the > barrier for users of Gora (as I am sure it would Users of other technologies > requiring definition of Objects via JSOn schemas). > I've not assigned this against any component as there is none which I feel > appropriate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-813) EOFException is thrown during normal operation
[
https://issues.apache.org/jira/browse/AVRO-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-813.
--
Resolution: Fixed
> EOFException is thrown during normal operation
> --
>
> Key: AVRO-813
> URL: https://issues.apache.org/jira/browse/AVRO-813
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.5.0
>Reporter: Bruno Dumon
>Assignee: Bruno Dumon
>Priority: Major
> Labels: memex
> Fix For: 1.9.0
>
> Attachments: avro-813-patch.txt
>
>
> In an application that uses Avro as RPC mechanism (with the NettyTransceiver,
> but that's irrelevant), I've noticed in jprofiler that during normal
> operation quite some time was spent creating EOFExceptions:
> {noformat}
> 5.4% - 2,004 ms org.apache.avro.ipc.generic.GenericResponder.readRequest
> 5.0% - 1,871 ms org.apache.avro.generic.GenericDatumReader.read
> 4.9% - 1,832 ms org.apache.avro.generic.GenericDatumReader.read
> 4.9% - 1,832 ms org.apache.avro.generic.GenericDatumReader.readRecord
> 4.5% - 1,670 ms org.apache.avro.generic.GenericDatumReader.read
> 4.5% - 1,670 ms org.apache.avro.generic.GenericDatumReader.readRecord
> 4.3% - 1,596 ms org.apache.avro.generic.GenericDatumReader.read
> 2.8% - 1,048 ms org.apache.avro.generic.GenericDatumReader.readArray
> 1.3% - 477 ms org.apache.avro.io.ValidatingDecoder.arrayNext
> 1.3% - 471 ms org.apache.avro.io.BinaryDecoder.arrayNext
> 1.3% - 466 ms org.apache.avro.io.BinaryDecoder.doReadItemCount
> 1.3% - 466 ms org.apache.avro.io.BinaryDecoder.readLong
> 1.3% - 466 ms org.apache.avro.io.BinaryDecoder.ensureBounds
> 1.3% - 466 ms org.apache.avro.io.BinaryDecoder$ByteSource.compactAndFill
> 1.3% - 466 ms
> org.apache.avro.io.BinaryDecoder$InputStreamByteSource.tryReadRaw
> 1.3% - 466 ms org.apache.avro.util.ByteBufferInputStream.read
> 1.3% - 466 ms org.apache.avro.util.ByteBufferInputStream.getBuffer
> 1.3% - 466 ms java.io.EOFException.
> 1.3% - 466 ms java.io.IOException.
> 1.2% - 460 ms java.lang.Exception.
> 1.2% - 460 ms java.lang.Throwable.
> 1.2% - 460 ms java.lang.Throwable.fillInStackTrace
> {noformat}
> These exceptions are produced by the ByteBufferInputStream (which modifies
> InputStream's contract: return -1 at eof), but are catched higher up by the
> tryReadRaw method.
> What happens is this:
> The message in question has an (empty) array at the end of its message, thus
> the reader tries to read the size of this array in BinaryDecoder.readLong.
> This calls ensureBounds(10), whose contract is that it should read 10 bytes
> if they are available, and otherwise be quiet. ensureBounds calls via
> compactAndFill the tryReadRaw method. It is this method which catches the
> EOFException, because it only 'tries' to read so many bytes.
> Note that InputStreamByteSource.readRaw (without the 'try' part) does itself
> check if read < 0 in order to throw EOFException, making the throwing of
> EOFException in ByteBufferInputStream unnecessary (for this particular usage).
> There was some talk about EOFException in AVRO-392 too, though it seems this
> particular common case was not mentioned there. When using Avro RPC, or more
> in general, when using Avro to read small messages rather than large files,
> it seems like one can very easily run into this EOFException situation, which
> hurts performance.
> I'll attach a patch which simply removes the throwing of EOFException in
> ByteBufferInputStream, but this will likely break other cases which rely on
> the EOFException being thrown (haven't researched this to the bottom).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-1658) Add avroDoc on reflect
[
https://issues.apache.org/jira/browse/AVRO-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-1658.
---
Resolution: Fixed
> Add avroDoc on reflect
> --
>
> Key: AVRO-1658
> URL: https://issues.apache.org/jira/browse/AVRO-1658
> Project: Apache Avro
> Issue Type: New Feature
> Components: java
>Affects Versions: 1.7.7
>Reporter: Zhaonan Sun
>Assignee: Raymie Stata
>Priority: Major
> Labels: reflection
> Fix For: 1.9.0
>
> Attachments:
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch,
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch,
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch
>
>
> Looks like @AvroMeta can't add reserved fields, like @AvroMeta("doc", "some
> doc") will have exceptions.
> I would be greate if we have a @AvroDoc("some documentations") in
> org.apache.avro.reflect
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-1981) ReflectionTypeLoadException
[
https://issues.apache.org/jira/browse/AVRO-1981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-1981:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> ReflectionTypeLoadException
> ---
>
> Key: AVRO-1981
> URL: https://issues.apache.org/jira/browse/AVRO-1981
> Project: Apache Avro
> Issue Type: Bug
> Components: csharp
>Affects Versions: 1.8.2
>Reporter: Josh Steward
>Assignee: Brian Lachniet
>Priority: Minor
> Fix For: 1.9.0
>
>
> In the ObjectCreator the call to FindType(name, throwError) can throw a
> ReflectionTypeLoadException during the call to assembly.GetTypes() if any of
> the types are not able to load, MSDN link below for ref. This effectively
> kills the entire deserialization process. I put in a simple workaround to
> just continue if this happens. In my particular case it was trying to load
> types from a devexpress test runner dll.
> https://msdn.microsoft.com/en-us/library/system.reflection.assembly.gettypes(v=vs.110).aspx
> private Type FindType(string name,bool throwError)
> {
> //other code
> foreach (Assembly assembly in assemblies)
> {
> // Fix for Mono 3.0.10
> if (assembly.FullName.StartsWith("MonoDevelop.NUnit"))
> continue;
> /*Added catch in case assembly contains unloadable
> types*/
> try
> {
> types = assembly.GetTypes();
> }
> catch (ReflectionTypeLoadException ex)
> {
> continue;
> }
> //other code
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-2329) Avro C# hides inner exceptions
[
https://issues.apache.org/jira/browse/AVRO-2329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-2329:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Avro C# hides inner exceptions
> --
>
> Key: AVRO-2329
> URL: https://issues.apache.org/jira/browse/AVRO-2329
> Project: Apache Avro
> Issue Type: Improvement
> Components: csharp
>Affects Versions: 1.8.2
>Reporter: Brian Lachniet
>Assignee: Brian Lachniet
>Priority: Major
> Fix For: 1.9.0
>
>
> The Avro C# library hides the inner exception in many places where it throws
> {{AvroException}}. This makes it difficult to track down the root cause of
> some problems.
> Update places that throw {{AvroException}} to include the inner exception
> when available/applicable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2162) Add Zstandard compression to avro file format
[ https://issues.apache.org/jira/browse/AVRO-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2162. --- Resolution: Fixed Fix Version/s: 1.9.0 Initial code merged for 1.9. Further updates/enhancements can be tracked in additional JIRA's > Add Zstandard compression to avro file format > - > > Key: AVRO-2162 > URL: https://issues.apache.org/jira/browse/AVRO-2162 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Scott Carey >Priority: Major > Fix For: 1.9.0 > > > I'd like to add Zstandard compression for Avro. > At compression level 1 It is almost as fast as Snappy at compression, with > compression ratios more like gzip. At higher levels of compression, it is > more compact than gzip -9 with much lower CPU when compressing and roughly 3x > faster decompression. > > Adding it to Java is fairly easy. We'll need to say something about it in > the spec however, as an 'optinal' codec. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2106) Revisit dependencies on Jetty, servlet-api, and Netty
[ https://issues.apache.org/jira/browse/AVRO-2106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2106: -- Hadoop Flags: Incompatible change > Revisit dependencies on Jetty, servlet-api, and Netty > - > > Key: AVRO-2106 > URL: https://issues.apache.org/jira/browse/AVRO-2106 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.2, 1.8.0 >Reporter: Nandor Kollar >Priority: Major > > The compile scoped dependency on jetty servlet-api in the IPC pom file can be > problematic if using Avro in a webapp environment. Would it be possible to > make this dependency either optional or provided? Or maybe Avro modularize > into sub-modules in such a way that desired features can be assembled > piecemeal? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2106) Revisit dependencies on Jetty, servlet-api, and Netty
[ https://issues.apache.org/jira/browse/AVRO-2106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2106. --- Resolution: Fixed Assignee: Daniel Kulp Fix Version/s: 1.9.0 Split netty and jetty out of avro-ipc into separate modules. If you just need netty transports, you can avoid dealing with jetty entirely. > Revisit dependencies on Jetty, servlet-api, and Netty > - > > Key: AVRO-2106 > URL: https://issues.apache.org/jira/browse/AVRO-2106 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.2, 1.8.0 >Reporter: Nandor Kollar >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.9.0 > > > The compile scoped dependency on jetty servlet-api in the IPC pom file can be > problematic if using Avro in a webapp environment. Would it be possible to > make this dependency either optional or provided? Or maybe Avro modularize > into sub-modules in such a way that desired features can be assembled > piecemeal? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2302) Invalid namespace importing Avro files generated from Protobuf
[
https://issues.apache.org/jira/browse/AVRO-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2302.
---
Resolution: Fixed
Assignee: Daniel Kulp
> Invalid namespace importing Avro files generated from Protobuf
> --
>
> Key: AVRO-2302
> URL: https://issues.apache.org/jira/browse/AVRO-2302
> Project: Apache Avro
> Issue Type: Bug
>Affects Versions: 1.8.2
>Reporter: Pedro Carneiro
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 1.9.0
>
>
> An error occurs when importing Avro files into BigQuery.
> The files were generated using a ProtobufData schema.
> The error shown is similar to:
> {code}
> BigQuery error in load operation: Error processing job
> '': Error while reading data, error message: The
> Apache Avro library failed to parse the header with the following error:
> Invalid namespace:
> some.package.SomeClassProto$
> {code}
> When using a modified version of the ProtobufData class, that yields a
> namespace like
> {noformat}
> some.package.SomeClassProto
> {noformat}
> (without the dollar), the import worked without errors.
> Potentially related to AVRO-2143?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2373) Remove Empty Package 'ipc'
[ https://issues.apache.org/jira/browse/AVRO-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2373. --- Resolution: Fixed Assignee: Daniel Kulp > Remove Empty Package 'ipc' > -- > > Key: AVRO-2373 > URL: https://issues.apache.org/jira/browse/AVRO-2373 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.9.0 >Reporter: David Mollitor >Assignee: Daniel Kulp >Priority: Minor > Fix For: 1.9.0 > > > https://github.com/apache/avro/tree/master/lang/java/avro/src/main/java/org/apache/avro/ipc -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2497) Avro-grpc failed when spotless:check alone
[ https://issues.apache.org/jira/browse/AVRO-2497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2497: -- Resolution: Fixed Fix Version/s: 1.9.1 Status: Resolved (was: Patch Available) > Avro-grpc failed when spotless:check alone > -- > > Key: AVRO-2497 > URL: https://issues.apache.org/jira/browse/AVRO-2497 > Project: Apache Avro > Issue Type: Test > Components: java >Affects Versions: 1.9.0 > Environment: jkd-8 > maven-3.3.9 >Reporter: Smart Yang >Assignee: Kengo Seki >Priority: Minor > Fix For: 1.9.1 > > > Enter avro-release-1.9.0/lang/java/grpc > mvn spotless:check > Result: BUILD FAILURE > Log: Could not find goal 'check' in plugin > org.apache.avro:avro-maven-plugin:1.9.0 among available goals help, > idl-protocol, induce, protocol, schema -> [Help 1] > But the lang/java/ipc other directories don't have this problem. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (AVRO-2479) Add m2e lifecycle mapping file
[ https://issues.apache.org/jira/browse/AVRO-2479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2479. --- Resolution: Fixed Assignee: Daniel Kulp Fix Version/s: 1.9.1 > Add m2e lifecycle mapping file > -- > > Key: AVRO-2479 > URL: https://issues.apache.org/jira/browse/AVRO-2479 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Pino Silvaggio >Assignee: Daniel Kulp >Priority: Trivial > Fix For: 1.9.1 > > Original Estimate: 1h > Remaining Estimate: 1h > > Instead of including the lifecycle plugin in the pom with the correct > mappings we could just add the file by default. > I suggest enabling build on configuration. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (AVRO-2465) Fix a wrong description in the avro-tools random's usage
[
https://issues.apache.org/jira/browse/AVRO-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-2465:
--
Fix Version/s: 1.9.1
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Fix a wrong description in the avro-tools random's usage
>
>
> Key: AVRO-2465
> URL: https://issues.apache.org/jira/browse/AVRO-2465
> Project: Apache Avro
> Issue Type: Bug
> Components: java, tools
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Minor
> Fix For: 1.9.1
>
>
> {{avro-tools random}}'s usage says the default compression codec is null:
> {code}
> $ java -jar lang/java/tools/target/avro-tools-1.10.0-SNAPSHOT.jar random
> Usage: outFile (filename or '-' for stdout)
> Option Description
> -- ---
> --codec Compression codec (default: null)
> {code}
> But actually the deflate codec is used.
> {code}
> $ java -jar lang/java/tools/target/avro-tools-1.10.0-SNAPSHOT.jar random
> --count 0 --schema-file share/test/schemas/weather.avsc /tmp/outFile.avro
> $ java -jar lang/java/tools/target/avro-tools-1.10.0-SNAPSHOT.jar getmeta
> /tmp/outFile.avro
> avro.schema {"type":"record","name":"Weather","namespace":"test","doc":"A
> weather
> reading.","fields":[{"name":"station","type":"string","order":"ignore"},{"name":"time","type":"long"},{"name":"temp","type":"int"}]}
> avro.codecdeflate
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
