[jira] [Commented] (AVRO-1831) Take advantage of JSR 3.5 annotations in the generated java classes.
[ https://issues.apache.org/jira/browse/AVRO-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17497397#comment-17497397 ] Daniel Kulp commented on AVRO-1831: --- I really don't remember either. That said, definitely make sure it's an optional and not required dependency if it's added back in. > Take advantage of JSR 3.5 annotations in the generated java classes. > > > Key: AVRO-1831 > URL: https://issues.apache.org/jira/browse/AVRO-1831 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Zoltan Farkas >Priority: Minor > > it would be nice if the generated records would take advantage of: > @javax.annotation.Nullable > @javax.annotation.Nonnull > to annotate the fields that can be null... > This would have a documenting role, and more importantly allow findbugs to > detect incorrect use of the fields. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (AVRO-3209) many threads blocked by forName0()
[
https://issues.apache.org/jira/browse/AVRO-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17416912#comment-17416912
]
Daniel Kulp commented on AVRO-3209:
---
Is this reproducible with 1.10? 1.7.7 is ancient.The latest code uses a
cache of classes so I suspect that this is already fixed.
> many threads blocked by forName0()
> --
>
> Key: AVRO-3209
> URL: https://issues.apache.org/jira/browse/AVRO-3209
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.7.7
>Reporter: yang
>Priority: Major
>
> When use hundred of theads read avro files, those avro files has same schema,
> most theads would blocked at forName0 such as below:
> {code:java}
> "tomcat-http--1051" #5078 daemon prio=5 os_prio=0 tid=0x7f4ba0339000
> nid=0x4755f waiting for monitor entry [0x7f45232ef000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.avro.util.ClassUtils.forName(ClassUtils.java:102)
> at org.apache.avro.util.ClassUtils.forName(ClassUtils.java:82)
> at
> org.apache.avro.specific.SpecificData.getClass(SpecificData.java:132)
> at
> org.apache.avro.specific.SpecificData.newRecord(SpecificData.java:330)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:173)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-3206) Provide more information in serialization error messages in SpecificDatumWriter
[
https://issues.apache.org/jira/browse/AVRO-3206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-3206.
---
Fix Version/s: 1.11.0
Resolution: Fixed
> Provide more information in serialization error messages in
> SpecificDatumWriter
> ---
>
> Key: AVRO-3206
> URL: https://issues.apache.org/jira/browse/AVRO-3206
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.10.2
>Reporter: Gyula Komlossi
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> In certain cases, when there is an error (like NPE, ClassCastException) in
> the "*writeField*" method of the "*SpecificDatumWriter*" class, the thrown
> exception doesn't contain the specific field causing the problem.
> Similarly as implemented in GenericDatumWriter, the same exceptions could be
> caught and their message improved by adding the related field name causing
> the problem.
> Currently, the message is like this:
> {code:java}
> java.lang.NullPointerException: null of string of
> org.apache.avro.test.TestRecord
> {code}
> But with the improvement it would be:
> {code:java}
> java.lang.NullPointerException: null of string in field 'name' of
> org.apache.avro.test.TestRecord
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-3204) Rust: Update dependencies
[ https://issues.apache.org/jira/browse/AVRO-3204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-3204. --- Fix Version/s: 1.11.0 Resolution: Fixed > Rust: Update dependencies > - > > Key: AVRO-3204 > URL: https://issues.apache.org/jira/browse/AVRO-3204 > Project: Apache Avro > Issue Type: Improvement > Components: rust >Affects Versions: 1.11.0 >Reporter: Martin Tzvetanov Grigorov >Assignee: Martin Tzvetanov Grigorov >Priority: Minor > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 40m > Remaining Estimate: 0h > > It seems Dependabot does not cover Rust module yet. > We should keep the (dev-)dependencies up to date manually for the time being. > > *Update*: I've also added an entry to dependabot.yml for Rust! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AVRO-3075) Avro.AvroException namespaces with reserved words cause deserialisation issue when inside an array.
[ https://issues.apache.org/jira/browse/AVRO-3075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-3075. --- Fix Version/s: 1.11.0 Resolution: Fixed > Avro.AvroException namespaces with reserved words cause deserialisation issue > when inside an array. > --- > > Key: AVRO-3075 > URL: https://issues.apache.org/jira/browse/AVRO-3075 > Project: Apache Avro > Issue Type: Bug > Components: csharp >Affects Versions: 1.10.1 >Reporter: Dave Craft >Priority: Minor > Labels: pull-request-available > Fix For: 1.11.0 > > Attachments: ComplexTypeWithReservedWords.avsc > > > Given the attached avro file.. when using avrogen and the specific > deserializer I'm getting an exception: > > Avro.AvroException : Unable to find type > 'Avro.Test.Specific.@return.ArrayItem' in all loaded assemblies in field > ArrayItems > > I have looked into it and it seems to be that when FInding types it's not > considering mangled type names.. I have a fix available here, mainly though > it's to help you understand the issue, as it might not be the best solution. > [https://github.com/apache/avro/pull/1131] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AVRO-3082) Improve interop test traceability for C# and Perl on CI
[ https://issues.apache.org/jira/browse/AVRO-3082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-3082. --- Fix Version/s: 1.11.0 Resolution: Fixed > Improve interop test traceability for C# and Perl on CI > --- > > Key: AVRO-3082 > URL: https://issues.apache.org/jira/browse/AVRO-3082 > Project: Apache Avro > Issue Type: Improvement > Components: build, csharp, interop, perl >Reporter: Kengo Seki >Assignee: Kengo Seki >Priority: Minor > Fix For: 1.11.0 > > > Currently, interop tests for C# and Perl don't output logs about which files > were checked and which were skipped. > Adding such messages is useful for ensuring if these tests correctly work. > https://github.com/apache/avro/runs/2133175520?check_suite_focus=true#step:8:52 > https://github.com/apache/avro/runs/2133175833?check_suite_focus=true#step:10:10 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AVRO-3140) Node 14 tests are timing out
[ https://issues.apache.org/jira/browse/AVRO-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-3140. --- Resolution: Fixed > Node 14 tests are timing out > > > Key: AVRO-3140 > URL: https://issues.apache.org/jira/browse/AVRO-3140 > Project: Apache Avro > Issue Type: Test > Components: js >Affects Versions: 1.10.2 >Reporter: Michael A. Smith >Priority: Major > Fix For: 1.11.0 > > > Node 14 tests have been running and failing due to a time out [since we > started testing Node > 14|https://github.com/apache/avro/commit/4fd00e0f3f7d925422293f7b64eb9827e69f9c6a]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AVRO-3110) Infinite Recursion
[
https://issues.apache.org/jira/browse/AVRO-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-3110:
--
Fix Version/s: 1.11.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Infinite Recursion
> --
>
> Key: AVRO-3110
> URL: https://issues.apache.org/jira/browse/AVRO-3110
> Project: Apache Avro
> Issue Type: Bug
> Components: csharp
>Affects Versions: 1.10.2
> Environment: Windows
>Reporter: Ebere Abanonu
>Assignee: Ebere Abanonu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> Test Name:AvroSchemaGenerator.Tests.GetSchemaTest.TestRecursiveSchema
> Test Outcome: Passed
> Result StandardOutput:
> {"type":"record","namespace":"AvroSchemaGenerator.Tests","name":"Recursive","fields":[{"name":"Fo","type":["null",{"type":"record","namespace":"AvroSchemaGenerator.Tests","name":"SimpleFoo","fields":[{"name":"Age","type":"int"},{"name":"Name","type":"string"},{"name":"FactTime","type":"long"},{"name":"Point","type":"double"},{"name":"Precision","type":"float"},{"name":"Attending","type":"boolean"},{"name":"Id","type":["null","bytes"],"default":null}]}],"default":null},{"name":"Recurse","type":["null","Recursive"],"default":null}]}
> //ClassCache
> AddClassNameMapItem(rs, objType);
> var c = GetClass(rs);
> foreach (var f in rs.Fields)
> {
> var t = c.GetPropertyType(f);
> LoadClassCache(t, f.Schema);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-3134) C#: Method ToString of AvroDecimal causes ArgumentOutOfRangeException
[ https://issues.apache.org/jira/browse/AVRO-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-3134. --- Fix Version/s: 1.11.0 Assignee: Daniel Kulp Resolution: Fixed > C#: Method ToString of AvroDecimal causes ArgumentOutOfRangeException > - > > Key: AVRO-3134 > URL: https://issues.apache.org/jira/browse/AVRO-3134 > Project: Apache Avro > Issue Type: Bug > Components: csharp >Affects Versions: 1.10.2 >Reporter: Thiago Froes >Assignee: Daniel Kulp >Priority: Minor > Fix For: 1.11.0 > > > Method ToString of class AvroDecimal causes ArgumentOutOfRangeException when > Scale is greater than Unscaled number size. Samples: > * 0.001 > * 0.01 > When Scale as equal as Unscaled number size, the result é wrong, because is > missing a "0", Sample: > * 0.1: in this case the result is ".1" > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AVRO-3150) Checkstyle requires LF line endings: add .gitattributes hint
[
https://issues.apache.org/jira/browse/AVRO-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-3150.
---
Fix Version/s: 1.11.0
Assignee: Daniel Kulp
Resolution: Fixed
I changed the checkstyle rule instead to allow the various cr/lf endings. Not
sure why it was restricted.
> Checkstyle requires LF line endings: add .gitattributes hint
>
>
> Key: AVRO-3150
> URL: https://issues.apache.org/jira/browse/AVRO-3150
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.10.2
>Reporter: Ryan
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 1.11.0
>
>
> Running the Java unit tests on a Windows machine with the commonly configured
> git line ending configuration:
> {quote}{{git config --global core.autocrlf true}}
> {quote}
> Results in test failure of:
> {quote}{{NewlineAtEndOfFile: Expected line ending for file is LF(\n), but
> CRLF(\r\n) is detected.}}
> {quote}
> This is due to a checkstyle rule. If the unit tests require LF line endings
> then it would be helpful to add a .gitattributes file to automatically ensure
> the correct line endings are used on git clone.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-3171) JavaScript: test failures for BlockEncoder#empty
[
https://issues.apache.org/jira/browse/AVRO-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-3171.
---
Fix Version/s: 1.11.0
Resolution: Fixed
> JavaScript: test failures for BlockEncoder#empty
>
>
> Key: AVRO-3171
> URL: https://issues.apache.org/jira/browse/AVRO-3171
> Project: Apache Avro
> Issue Type: Bug
> Components: javascript
>Reporter: Martin Tzvetanov Grigorov
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Running `./build.sh test` in lang/js fails with:
>
> {code:java}
> 380 passing (3s)
> 1 failing 1) files
>BlockEncoder
> empty:
> Error: Timeout of 2000ms exceeded. For async tests and hooks, ensure
> "done()" is called; if returning a Promise, ensure it resolves.
> (/home/martin/git/apache/avro/lang/js/test/test_files.js)
> at listOnTimeout (internal/timers.js:554:17)
> at processTimers (internal/timers.js:497:7)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-3081) Fix Java interop test on CI to read files generated by other languages
[ https://issues.apache.org/jira/browse/AVRO-3081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-3081. --- Fix Version/s: 1.11.0 Resolution: Fixed > Fix Java interop test on CI to read files generated by other languages > -- > > Key: AVRO-3081 > URL: https://issues.apache.org/jira/browse/AVRO-3081 > Project: Apache Avro > Issue Type: Improvement > Components: build, interop, java >Reporter: Kengo Seki >Assignee: Kengo Seki >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, Java interop test reads only files generated by itself on CI, so > it's not an interop test actually. > It should read files generated by another language. Python may be the first > candidate, since it supports all codecs except for xz. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AVRO-2596) Upgrade ant to version 1.10.8
[ https://issues.apache.org/jira/browse/AVRO-2596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2596. --- Assignee: Ismaël Mejía (was: Fokko Driesprong) Resolution: Fixed > Upgrade ant to version 1.10.8 > - > > Key: AVRO-2596 > URL: https://issues.apache.org/jira/browse/AVRO-2596 > Project: Apache Avro > Issue Type: Improvement >Affects Versions: 1.9.1 >Reporter: Fokko Driesprong >Assignee: Ismaël Mejía >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AVRO-2493) Unable to register Logical Type for custom Conversion class
[
https://issues.apache.org/jira/browse/AVRO-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2493.
---
Fix Version/s: 1.10.0
Resolution: Fixed
> Unable to register Logical Type for custom Conversion class
> ---
>
> Key: AVRO-2493
> URL: https://issues.apache.org/jira/browse/AVRO-2493
> Project: Apache Avro
> Issue Type: Bug
> Components: java, logical types
>Affects Versions: 1.9.0
>Reporter: Travis Yocum
>Priority: Major
> Fix For: 1.10.0
>
>
> I have created a custom conversion class for Java's (1.8)
> java.time.OffsetDateTime.
> {code:java}
> public class OffsetDateTimeConversion extends Conversion {
> private static final DateTimeFormatter DATE_TIME_FORMATTER =
> DateTimeFormatter.ofPattern("-MM-dd'T'HH:mm:ss.nnZ");
> @Override
> public Class getConvertedType() {
> return OffsetDateTime.class;
> }
> @Override
> public String getLogicalTypeName() {
> return "offset-date-time";
> }
> @Override
> public OffsetDateTime fromCharSequence(CharSequence value, Schema schema,
> LogicalType type) {
> return OffsetDateTime.parse(value, DATE_TIME_FORMATTER);
> }
> @Override
> public CharSequence toCharSequence(OffsetDateTime value, Schema schema,
> LogicalType type) {
> return value.format(DATE_TIME_FORMATTER);
> }
> }
> {code}
> And my simple schema to test (including a field that uses the built-in
> "time-millis" converter in Avro 1.9 for testing purposes):
> TimeTest.avsc
>
> {code:java}
> {
>"type": "record",
>"name": "TimeTest",
>"namespace": "time.test",
>"fields": [
> {
>"name": "createTime",
>"type": {
>"type": "int",
>"logicalType": "time-millis"
>}
> },
> {
>"name": "createDateTime",
>"type": {
>"type": "string",
>"logicalType": "offset-date-time"
>}
> }
>]
> }
> {code}
>
> I've also created a LogicalType for my conversion field that I need to
> register:
>
> {code:java}
> public class OffsetDateTimeLogicalType extends LogicalType {
> public static final String LOGICAL_DATE_TIME_NAME = "offset-date-time";
> public OffsetDateTimeLogicalType() {
> super(LOGICAL_DATE_TIME_NAME);
> }
> @Override
> public void validate(Schema schema) {
> super.validate(schema);
> if (schema.getType() != Schema.Type.STRING) {
> throw new IllegalArgumentException("Logical type
> 'offset-date-time' must be of type string");
> }
> }
> }{code}
>
> I've been debugging the avro-maven-plugin and have been able to pinpoint
> where my issue is occurring while parsing the schema:
>
> {code:java}
> // parse logical type if present
> result.logicalType = LogicalTypes.fromSchemaIgnoreInvalid(result);
> {code}
>
> The schema's "logicalType" property is being set to null because it's not a
> registered logical type.
> The code I need to execute is:
>
> {code:java}
> LogicalTypes.register(getLogicalTypeName(), schema -> new
> OffsetDateTimeLogicalType());
> {code}
> Without modifying the plugin code, I see no way to execute this register so
> that the Schema.parse method (which is called before the conversion class is
> executed) will recognize this LogicalType.
>
> The plugin has a config property to allow for
> registering the Decimal logical types but nothing to enable other logical
> types.
> Am I missing something or is this a real issue?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-2579) Test can fail with a different order of fields reflected
[
https://issues.apache.org/jira/browse/AVRO-2579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2579.
---
Fix Version/s: 1.10.0
Resolution: Fixed
> Test can fail with a different order of fields reflected
>
>
> Key: AVRO-2579
> URL: https://issues.apache.org/jira/browse/AVRO-2579
> Project: Apache Avro
> Issue Type: Bug
>Reporter: contextshuffling
>Priority: Minor
> Fix For: 1.10.0
>
>
> {{org.apache.avro.reflect.TestReflect#testAvroDoc compares the toString()
> version of the DocTest.class. It uses java.lang.Class.getDeclaredFields() to
> get the fields. However, this API does not guarantee the order of fields that
> gets returned. So test may fails due to a different order.}}
> {{https://github.com/contextshuffling/avro/blob/f9399780bfafd46bce1292da314f15b90f869572/lang/java/avro/src/test/java/org/apache/avro/reflect/TestReflect.java#L1319}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-2649) [Java] Argument order enforced in avro-tools cli.
[
https://issues.apache.org/jira/browse/AVRO-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2649.
---
Fix Version/s: 1.10.0
Resolution: Fixed
> [Java] Argument order enforced in avro-tools cli.
> -
>
> Key: AVRO-2649
> URL: https://issues.apache.org/jira/browse/AVRO-2649
> Project: Apache Avro
> Issue Type: Improvement
>Affects Versions: 1.9.1
>Reporter: Ryan Skraba
>Priority: Minor
> Fix For: 1.10.0
>
>
> The following command line works:
> {code}
> $ avrotool compile -string -bigDecimal schema
> ./lang/java/tools/src/test/compiler/input/fieldtest.avsc /tmp/output
> Input files to compile:
> ./lang/java/tools/src/test/compiler/input/fieldtest.avsc
> {code}
> Switching the first two flags causes an error:
> {code}
> $ avrotool compile -bigDecimal -string schema
> ./lang/java/tools/src/test/compiler/input/fieldtest.avsc /tmp/output
> Expected "schema" or "protocol".
> {code}
> There's really no need to enforce a command line option order, especially for
> flags, and it's not user friendly. Most CLI tools don't. There's several
> good command line parsers to help.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-2438) SpecificData.deepCopy() cannot be used with URI fields
[
https://issues.apache.org/jira/browse/AVRO-2438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2438.
---
Fix Version/s: 1.10.0
Resolution: Fixed
> SpecificData.deepCopy() cannot be used with URI fields
> --
>
> Key: AVRO-2438
> URL: https://issues.apache.org/jira/browse/AVRO-2438
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0, 1.8.2
>Reporter: Sebastian J.
>Assignee: Zezeng Wang
>Priority: Major
> Fix For: 1.10.0
>
>
> Having a schema fragment like this:
> {code:java}
> {
> "name": "ownerId",
> "type": [
> "null",
> {
> "type": "string",
> "java-class": "java.net.URI"
> }
> ],
> "default": null
> }{code}
> can be perfectly deserialized in a generated POJO with
> {code:java}
> @org.apache.avro.specific.AvroGenerated
> public class MyAvroDataObject extends
> org.apache.avro.specific.SpecificRecordBase implements
> org.apache.avro.specific.SpecificRecord {
> ...
> @Deprecated public java.net.URI ownerId;{code}
> as
> {{GenericDatumReader.readString(Object, Schema, Decoder)}} uses via the
> {{stringClassCache}} with
> {code:java}
> {"type":"string","java-class":"java.net.URI"}=class java.net.URI{code}
> The {{URI}} class itself to rehydrate the value via {{newInstanceFromString}}.
>
> On the other hand, {{deepCopy}} only considers the schema-type of the field
> and turns in {{org.apache.avro.generic.GenericData.deepCopy(Schema, T)}}
> the {{URI}} value into an {{org.apache.avro.util.Utf8}} via the {{String}}
> case which then causes a {{ClassCastException}}:
> {noformat}
> java.lang.ClassCastException: org.apache.avro.util.Utf8 cannot be cast to
> java.net.URI
> at com.example.MyAvroDataObject.put(MyAvroDataObject.java:104)
> at org.apache.avro.generic.GenericData.setField(GenericData.java:660)
> at org.apache.avro.generic.GenericData.setField(GenericData.java:677)
> at org.apache.avro.generic.GenericData.deepCopy(GenericData.java:1082)
> at org.apache.avro.generic.GenericData.deepCopy(GenericData.java:1102)
> at
> org.apache.avro.generic.GenericData.deepCopy(GenericData.java:1080){noformat}
>
> The following dirty hack seems to avoid the issue - but is not in sync with
> the {{stringClassCache}} which should be consulted, too:
> {code:java}
> case STRING:
> // Strings are immutable
> if (value instanceof String) {
> return (T)value;
> }
> // Dirty Harry 9 3/4 start
> // URIs are immutable and are probably modeled as an URI itself
> // TODO: Check with stringClassCache & the schema
> else if ((value instanceof URI)
> && URI.class.getName().equals(schema.getProp("java-class"))
> ) {
> return (T)value;
> }
> // Dirt Harry 9 3/4 end
> // Some CharSequence subclasses are mutable, so we still need to make
> // a copy
> else if (value instanceof Utf8) {
> // Utf8 copy constructor is more efficient than converting
> // to string and then back to Utf8
> return (T)new Utf8((Utf8)value);
> }
> return (T)new Utf8(value.toString());
> {code}
>
> Also tried with Avro {{1.10-SNAPSHOT}} of 2019-06-20 /
> {{2d3b1fe7efd865639663ba785877182e7e038c45}} due to
> [https://github.com/apache/avro/pull/329] - but the issue remains.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-2806) Upgrade Netty to 4.x
[ https://issues.apache.org/jira/browse/AVRO-2806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2806. --- Fix Version/s: 1.10.0 Assignee: Daniel Kulp Resolution: Fixed > Upgrade Netty to 4.x > > > Key: AVRO-2806 > URL: https://issues.apache.org/jira/browse/AVRO-2806 > Project: Apache Avro > Issue Type: Task > Components: java >Affects Versions: 1.9.2 >Reporter: Gabor Szadovszky >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.10.0 > > > ipc-netty still uses the 3.x line which is ended almost 4 years ago. We have > a couple of CVEs which won't be fixed in 3.x so it is important to migrate to > the 4.x. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AVRO-2842) PHP Add phpcs and fix all violations against PSR12
[ https://issues.apache.org/jira/browse/AVRO-2842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2842. --- Fix Version/s: 1.10.0 Resolution: Fixed > PHP Add phpcs and fix all violations against PSR12 > -- > > Key: AVRO-2842 > URL: https://issues.apache.org/jira/browse/AVRO-2842 > Project: Apache Avro > Issue Type: Improvement > Components: php >Reporter: Siad Ardroumli >Priority: Major > Fix For: 1.10.0 > > > To improve maintability and readability, Avro should follow the PSR12 > standard of phpfig. > * Add static code analyzer (phpcs) > * Fix all violations using PSR12 sniff -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AVRO-2703) Use KMP Algorithm For Sync Marker Search
[ https://issues.apache.org/jira/browse/AVRO-2703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2703. --- Fix Version/s: 1.10.0 Resolution: Fixed > Use KMP Algorithm For Sync Marker Search > > > Key: AVRO-2703 > URL: https://issues.apache.org/jira/browse/AVRO-2703 > Project: Apache Avro > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Fix For: 1.10.0 > > > Use the [Knuth-Morris-Pratt > algorithm|https://en.wikipedia.org/wiki/Knuth%E2%80%93Morris%E2%80%93Pratt_algorithm] > to search an Avro file for the sequence boundary instead of the current > naive approach based on a circular buffer. > bq. The KMP algorithm has a better worst-case performance than the > straightforward algorithm -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AVRO-2427) Add undocumented codecs to the specification
[ https://issues.apache.org/jira/browse/AVRO-2427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2427: -- Fix Version/s: 1.10.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Add undocumented codecs to the specification > > > Key: AVRO-2427 > URL: https://issues.apache.org/jira/browse/AVRO-2427 > Project: Apache Avro > Issue Type: Improvement > Components: doc, spec >Reporter: Kengo Seki >Assignee: Kengo Seki >Priority: Major > Fix For: 1.10.0 > > > It seems that there's no reference to bzip2, xz, and zstd as an optional > codec in [the > specification|http://avro.apache.org/docs/1.9.0/spec.html#Optional+Codecs]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AVRO-2555) Update IDL docs to remove mention of avroj.jar
[ https://issues.apache.org/jira/browse/AVRO-2555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2555: -- Fix Version/s: 1.10.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Update IDL docs to remove mention of avroj.jar > -- > > Key: AVRO-2555 > URL: https://issues.apache.org/jira/browse/AVRO-2555 > Project: Apache Avro > Issue Type: Improvement > Components: doc, java >Reporter: Erik Tank >Assignee: Erik Tank >Priority: Major > Fix For: 1.10.0 > > Attachments: 0001-AVRO-2555-doc_update.patch > > > AVRO-320 renamed the avroj-tools to avro-tools.jar, however, the IDL > documentation wasn't updated. The attached patch updates the Usage section > demonstrating how to convert an .avdl to .avpr. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AVRO-2624) Avoid ByteBuffer incompatibility when compiling with JDK9+
[
https://issues.apache.org/jira/browse/AVRO-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2624.
---
Fix Version/s: 1.10.0
Assignee: Daniel Kulp
Resolution: Fixed
> Avoid ByteBuffer incompatibility when compiling with JDK9+
> --
>
> Key: AVRO-2624
> URL: https://issues.apache.org/jira/browse/AVRO-2624
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Reporter: Michael A. Smith
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 1.10.0
>
>
> Like MRESOLVER-85 and similar, the java implementation suffers from a
> compatibility break since java 9. The problem can be seen in the PR for
> AVRO-2603, https://github.com/apache/avro/pull/706, which fails for JAVA 11,
> but not for 8.
> The error is:
> {noformat}
> Caused by: java.lang.NoSuchMethodError:
> java.nio.ByteBuffer.limit(I)Ljava/nio/ByteBuffer;
> [py-test] at
> org.apache.avro.io.BinaryDecoder.readBytes(BinaryDecoder.java:317)
> {noformat}
> * This should not occur when artifacts are compiled with JDK8, even if run in
> Java 11 runtime. i.e., This shouldn't be a big issue while maven artifacts
> are being published with JDK8 (the lowest Java runtime version we support).
> * Likewise, this should not occur when artifacts are compiled with JDK11
> (with {{-target 1.8}}) and run in a Java 11 runtime, as with the JAVA=11
> build targets.
> * This *will* occur when the artifacts are compiled with JDK11 (with
> {{-target 1.8}}) and run in a Java 8 runtime.
> * It will be important to fix when Avro publishes artifacts built with JDK11
> and JDK8 is still meant to be supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AVRO-1720) Add an avro-tool to count records in an avro file
[
https://issues.apache.org/jira/browse/AVRO-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-1720:
--
Fix Version/s: 1.10.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Add an avro-tool to count records in an avro file
> -
>
> Key: AVRO-1720
> URL: https://issues.apache.org/jira/browse/AVRO-1720
> Project: Apache Avro
> Issue Type: New Feature
> Components: java
>Reporter: Janosch Woschitz
>Priority: Minor
> Labels: starter
> Fix For: 1.10.0
>
> Attachments: AVRO-1720-with-extended-unittests.patch, AVRO-1720.patch
>
>
> If you're dealing with bigger avro files (>100MB) it would be nice to have a
> way to quickly count the amount of records contained within that file.
> With the current state of avro-tools the only way to achieve this (to my
> current knowledge) is to dump the data to json and count the amount of
> records. For bigger files this might take a while due to the serialization
> overhead and since every record needs to be looked at.
> I added a new tool which is optimized for counting records, it does not
> serialize the records and reads only the block count for each block.
> {panel:title=Naive benchmark}
> {noformat}
> # the input file had a size of ~300MB
> $ du -sh sample.avro
> 323Msample.avro
> # using the new count tool
> $ time java -jar avro-tools.jar count sample.avro
> 331439
> real0m4.670s
> user0m6.167s
> sys 0m0.513s
> # the current way of counting records
> $ time java -jar avro-tools.jar tojson sample.avro | wc
> 331439 54904484 1838231743
> real0m52.760s
> user1m42.317s
> sys 0m3.209s
> # the overhead of wc is rather minor
> $ time java -jar avro-tools.jar tojson sample.avro > /dev/null
> real0m47.834s
> user0m53.317s
> sys 0m1.194s
> {noformat}
> {panel}
> This tool uses the HDFS API to handle files from any supported filesystem. I
> added the unit tests to the already existing TestDataFileTools since it
> provided convenient utility functions which I could reuse for my test
> scenarios.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-2769) After upgrade to 1.9.2, generated code for "put" uses value and value$
[
https://issues.apache.org/jira/browse/AVRO-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2769.
---
Resolution: Cannot Reproduce
No reproducible test case provided after 2 months.
> After upgrade to 1.9.2, generated code for "put" uses value and value$
> --
>
> Key: AVRO-2769
> URL: https://issues.apache.org/jira/browse/AVRO-2769
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.2
> Environment: macOS Catalina
> JDK 8 (oracle)
>Reporter: Alexander Ubillus
>Priority: Blocker
>
> After upgrade to 1.9.2, generated code for "put" uses value and value$,
> causing compilation exceptions:
> {code:java}
> // Used by DatumReader. Applications should not call.
> @SuppressWarnings(value="unchecked")
> public void put(int field$, java.lang.Object value$) {
> switch (field$) {
> case 0: name = (java.lang.CharSequence)value; break;
> case 1: length = (java.math.BigDecimal)value; break;
> case 2: metadata =
> (java.util.Map)value; break;
> default: throw new org.apache.avro.AvroRuntimeException("Bad index");
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-2359) Support Logical Types in C#
[ https://issues.apache.org/jira/browse/AVRO-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2359. --- Fix Version/s: 1.9.2 Resolution: Fixed This was merged into 1.9.2 > Support Logical Types in C# > --- > > Key: AVRO-2359 > URL: https://issues.apache.org/jira/browse/AVRO-2359 > Project: Apache Avro > Issue Type: New Feature > Components: csharp >Affects Versions: 1.8.2 >Reporter: Tim Roberts >Priority: Critical > Fix For: 1.9.2 > > > By not supporting Logical Types, the C# Avro implementation is severely > limited in terms of its interoperability in heterogeneous environments. While > the implementation will safely ignore logical type declarations in processed > schemas; at runtime the semantics of a "date" for example, will be lost when > receiving an Avro payload that was encoded by the Java platform using the > same schema. The C# implementation will never be able to retrieve the date > value from the encoded int. > I propose that the C# Avro implementation be extended to support the Logical > Types as defined by the current specification. I have also explored the > lang/csharp codebase and believe that I could produce a PR to support this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AVRO-2782) C# implmenetation of GenericRecord Doesn't have
[ https://issues.apache.org/jira/browse/AVRO-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2782. --- Fix Version/s: 1.10.0 Resolution: Fixed > C# implmenetation of GenericRecord Doesn't have > > > Key: AVRO-2782 > URL: https://issues.apache.org/jira/browse/AVRO-2782 > Project: Apache Avro > Issue Type: Improvement > Components: csharp >Reporter: Yaniv Rubinpur >Priority: Major > Fix For: 1.10.0 > > > GenericRecord in C# can only be accessed by field name and not by field index > (Java version support access by index). Accessing a dictionary for each field > is costly. > In Java There is: Record.put(int i, Object v). In C# GenericRecord there is > only: Add(string fieldName, object fieldValue) (and string indexer). > > And there is no way to extend GenericRecord to support indexed access. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AVRO-2792) Fix C# logical type tests to work in other timezones than UTC
[
https://issues.apache.org/jira/browse/AVRO-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-2792:
--
Fix Version/s: 1.10.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Fix C# logical type tests to work in other timezones than UTC
> -
>
> Key: AVRO-2792
> URL: https://issues.apache.org/jira/browse/AVRO-2792
> Project: Apache Avro
> Issue Type: Bug
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Major
> Fix For: 1.10.0
>
>
> I ran the C# unit tests outside of a container and got the following error.
> This is because my machine's timezone is JST (UTC+9).
> {code}
> $ cd lang/csharp
> $ ./build.sh test
> (snip)
> X TestLogical_TimestampMicrosecond() [22ms]
> Error Message:
> Expected: 1990-01-01 14:15:30
> But was: 1990-01-01 05:15:30
> Stack Trace:
> at Avro.Test.Generic.GenericTests.test[T](String s, T value) in
> /home/sekikn/repos/avro/lang/csharp/src/apache/test/Generic/GenericTests.cs:line
> 38
>at Avro.Test.Generic.GenericTests.TestLogical_TimestampMicrosecond() in
> /home/sekikn/repos/avro/lang/csharp/src/apache/test/Generic/GenericTests.cs:line
> 139
> X TestLogical_TimestampMillisecond() [< 1ms]
> Error Message:
> Expected: 1990-01-01 14:15:30
> But was: 1990-01-01 05:15:30
> Stack Trace:
> at Avro.Test.Generic.GenericTests.test[T](String s, T value) in
> /home/sekikn/repos/avro/lang/csharp/src/apache/test/Generic/GenericTests.cs:line
> 38
>at Avro.Test.Generic.GenericTests.TestLogical_TimestampMillisecond() in
> /home/sekikn/repos/avro/lang/csharp/src/apache/test/Generic/GenericTests.cs:line
> 133
> X TestTimestampMicrosecond("01/01/2019 14:20:00","01/01/2019 14:20:00Z") [<
> 1ms]
> Error Message:
> Expected: 2019-01-01 14:20:00
> But was: 2019-01-01 05:20:00
> Stack Trace:
> at Avro.Test.LogicalTypeTests.TestTimestampMicrosecond(String s, String
> e) in
> /home/sekikn/repos/avro/lang/csharp/src/apache/test/Util/LogicalTypeTests.cs:line
> 143
> X TestTimestampMillisecond("01/01/2019 14:20:00","01/01/2019 14:20:00Z") [<
> 1ms]
> Error Message:
> Expected: 2019-01-01 14:20:00
> But was: 2019-01-01 05:20:00
> Stack Trace:
> at Avro.Test.LogicalTypeTests.TestTimestampMillisecond(String s, String
> e) in
> /home/sekikn/repos/avro/lang/csharp/src/apache/test/Util/LogicalTypeTests.cs:line
> 119
> Test Run Failed.
> Total tests: 584
> Passed: 580
> Failed: 4
> Total time: 2.3598 Seconds
> {code}
> [As the specification
> says|https://avro.apache.org/docs/current/spec.html#Timestamp+%28millisecond+precision%29],
> Avro's timestamp logical type has a long value that represents the
> difference from the unix epoch (1 Jan 1970 00:00:00 UTC) and it doesn't have
> timezone information. So when deserializing it, it's returned as a C#
> DateTime object with UTC.
> Therefore, if the input string lacks timezone information, we should assume
> it represents UTC datetime in these test cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-2784) Append to existing file in C#
[ https://issues.apache.org/jira/browse/AVRO-2784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2784. --- Resolution: Fixed > Append to existing file in C# > - > > Key: AVRO-2784 > URL: https://issues.apache.org/jira/browse/AVRO-2784 > Project: Apache Avro > Issue Type: New Feature > Components: csharp >Affects Versions: 1.9.2 >Reporter: Vladimir Aleksandrov >Priority: Minor > Labels: ready-to-commit > Fix For: 1.10.0 > > > Propose a possibility to append to existing files in C# in a similar way how > it is implemented in Java. > > My proposal is to add two new public static methods OpenAppendWriter() (one > receiving string path and other - streams) to DataFileWriter class to be > consistent with C# implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AVRO-2753) Add leaveOpen option to DataFileReader and Writer to keep Stream open
[ https://issues.apache.org/jira/browse/AVRO-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2753. --- Fix Version/s: 1.10.0 Resolution: Fixed > Add leaveOpen option to DataFileReader and Writer to keep Stream open > - > > Key: AVRO-2753 > URL: https://issues.apache.org/jira/browse/AVRO-2753 > Project: Apache Avro > Issue Type: Improvement >Reporter: Zoltan Csizmadia >Assignee: Zoltan Csizmadia >Priority: Minor > Fix For: 1.10.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > # If a stream is used to open DataFileReader or DataFileWriter, an optional > leaveOpen (default=false) should exist as an additional argument to flag the > object not to close and dispose the stream when the reader/writer object is > disposed. > # The stream objects are not disposed (only closed) as of now, leaving the > reader and writer objects not disposed. > > leaveOpen follows the Stream pattern which used in other Stream related > areas. This can be useful if streams are chained together, e.g. > Compressor/decompressor streams chained together with Avro reader/writer > objects. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AVRO-2714) [C#] Use new .NET Core APIs for better performance
[ https://issues.apache.org/jira/browse/AVRO-2714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2714. --- Fix Version/s: 1.10.0 Resolution: Fixed > [C#] Use new .NET Core APIs for better performance > -- > > Key: AVRO-2714 > URL: https://issues.apache.org/jira/browse/AVRO-2714 > Project: Apache Avro > Issue Type: Bug > Components: csharp >Reporter: Eric Erhardt >Priority: Major > Fix For: 1.10.0 > > > The BinaryDecoder will create temporary buffers to read strings and floats, > and then throw the buffers away. These unnecessary allocations add objects to > the GC unnecessarily. > > See > [https://github.com/apache/avro/blob/ca38fb98f83e0345ca6c49b0660e9fbd03ea3cd5/lang/csharp/src/apache/main/IO/BinaryDecoder.cs#L144] > [https://github.com/apache/avro/blob/ca38fb98f83e0345ca6c49b0660e9fbd03ea3cd5/lang/csharp/src/apache/main/IO/BinaryDecoder.cs#L95] > > We can multi-target the library and use new APIs on .NET Core 2.1 and .NET > Standard 2.1 to eliminate these allocations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AVRO-2278) GenericData.Record field getter not correct
[
https://issues.apache.org/jira/browse/AVRO-2278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2278.
---
Fix Version/s: 1.10.0
Resolution: Fixed
> GenericData.Record field getter not correct
> ---
>
> Key: AVRO-2278
> URL: https://issues.apache.org/jira/browse/AVRO-2278
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.2, 1.9.2
>Reporter: Zoltan Farkas
>Assignee: Zoltan Farkas
>Priority: Major
> Fix For: 1.10.0
>
>
> Currently the get field implementation is not correct in GenericData.Record:
> at:
> https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/generic/GenericData.java#L209
> {code}
>@Override public Object get(String key) {
> Field field = schema.getField(key);
> if (field == null) return null;
> return values[field.pos()];
> }
> {code}
> The method returns null when a field is not present, making it impossible to
> distinguish between:
> field value = null
> and
> field does not exist.
> A more "correct" implementation would be:
> {code}
> @Override public Object get(String key) {
> Field field = schema.getField(key);
> if (field == null) {
> throw new IllegalArgumentException("Invalid field " + key);
> }
> return values[field.pos()];
> }
> {code}
> this will make the behavior consistent with put which will throw a exception
> when setting a non existent field.
> when I make this change in my fork, some bugs in unit tests showed up
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-2841) PHP: Add support for bzip2 codec
[ https://issues.apache.org/jira/browse/AVRO-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2841. --- Fix Version/s: 1.10.0 Resolution: Fixed > PHP: Add support for bzip2 codec > > > Key: AVRO-2841 > URL: https://issues.apache.org/jira/browse/AVRO-2841 > Project: Apache Avro > Issue Type: Improvement > Components: php >Reporter: Siad Ardroumli >Priority: Major > Fix For: 1.10.0 > > > Add support for bzip2 codec like the java implementation has. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AVRO-2639) SpecificCompiler option to generate Optionals only when nullable
[ https://issues.apache.org/jira/browse/AVRO-2639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2639. --- Fix Version/s: 1.10.0 Resolution: Fixed > SpecificCompiler option to generate Optionals only when nullable > > > Key: AVRO-2639 > URL: https://issues.apache.org/jira/browse/AVRO-2639 > Project: Apache Avro > Issue Type: New Feature > Components: java >Affects Versions: 1.9.1 >Reporter: Mike Roberts >Priority: Minor > Fix For: 1.10.0 > > > When `gettersReturnOptional` all getters return `Optional`, not just the > getters for nullable fields. > It would be nice to add a configuration option > (`gettersReturnOptionalOnlyForNullable`?) that made `SpecificCompiler` do > this. It would make the Java code more accurately reflect the underlying > schema. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AVRO-2763) Resource leak: '' is never closed
[ https://issues.apache.org/jira/browse/AVRO-2763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp reassigned AVRO-2763: - Assignee: Zezeng Wang > Resource leak: '' is never closed > - > > Key: AVRO-2763 > URL: https://issues.apache.org/jira/browse/AVRO-2763 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.9.0, 1.8.2, 1.9.1, 1.9.2 >Reporter: Zezeng Wang >Assignee: Zezeng Wang >Priority: Major > Fix For: 1.10.0 > > > E.g > https://github.com/apache/avro/blob/8199b9626e4badd34a85946cd223a91863f44cee/lang/java/avro/src/test/java/org/apache/avro/TestDataFileCorruption.java#L79 > https://github.com/apache/avro/blob/8199b9626e4badd34a85946cd223a91863f44cee/lang/java/avro/src/test/java/org/apache/avro/TestSchemaBuilder.java#L779 > https://github.com/apache/avro/blob/8199b9626e4badd34a85946cd223a91863f44cee/lang/java/mapred/src/test/java/org/apache/avro/mapred/TestAvroTextOutputFormat.java#L73 > .. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AVRO-2763) Resource leak: '' is never closed
[ https://issues.apache.org/jira/browse/AVRO-2763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2763. --- Fix Version/s: 1.10.0 Resolution: Fixed > Resource leak: '' is never closed > - > > Key: AVRO-2763 > URL: https://issues.apache.org/jira/browse/AVRO-2763 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.9.0, 1.8.2, 1.9.1, 1.9.2 >Reporter: Zezeng Wang >Priority: Major > Fix For: 1.10.0 > > > E.g > https://github.com/apache/avro/blob/8199b9626e4badd34a85946cd223a91863f44cee/lang/java/avro/src/test/java/org/apache/avro/TestDataFileCorruption.java#L79 > https://github.com/apache/avro/blob/8199b9626e4badd34a85946cd223a91863f44cee/lang/java/avro/src/test/java/org/apache/avro/TestSchemaBuilder.java#L779 > https://github.com/apache/avro/blob/8199b9626e4badd34a85946cd223a91863f44cee/lang/java/mapred/src/test/java/org/apache/avro/mapred/TestAvroTextOutputFormat.java#L73 > .. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AVRO-2558) Update java/mr-example pom to load into m2e
[ https://issues.apache.org/jira/browse/AVRO-2558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2558. --- Fix Version/s: 1.10.0 Resolution: Fixed > Update java/mr-example pom to load into m2e > --- > > Key: AVRO-2558 > URL: https://issues.apache.org/jira/browse/AVRO-2558 > Project: Apache Avro > Issue Type: Bug > Components: doc, java >Reporter: Zezeng Wang >Priority: Minor > Fix For: 1.10.0 > > > In the pom of java-example and mr-example, the following problems exist: > Plugin execution not covered by lifecycle configuration: > org.apache.avro:avro-maven-plugin:1.7.5:schema (execution: default, phase: > generate-sources) > Although it works fine, it is a bit unfriendly as an example for newcomers to > learn. > Thanks. > Best regards. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AVRO-2786) Initialize the buffer area for StringBuilder/List
[
https://issues.apache.org/jira/browse/AVRO-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2786.
---
Fix Version/s: 1.10.0
Resolution: Fixed
> Initialize the buffer area for StringBuilder/List
> -
>
> Key: AVRO-2786
> URL: https://issues.apache.org/jira/browse/AVRO-2786
> Project: Apache Avro
> Issue Type: Wish
> Components: java
>Reporter: Zezeng Wang
>Assignee: Zezeng Wang
>Priority: Minor
> Fix For: 1.10.0
>
>
> Under the StringBuilder/List is a char array to store. If the internal buffer
> overflows (the default length is 16/10), it will automatically become larger.
> A new internal buffer will be allocated to the array copy. Although it cannot
> be completely avoided, it can reduce the expansion It's not bad.
> E.g:
>
> [https://github.com/apache/avro/blob/87aefa28ad6cb11ab95010cdeac2223cfb701c0d/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java#L964]
>
> [https://github.com/apache/avro/blob/87aefa28ad6cb11ab95010cdeac2223cfb701c0d/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java#L944]
> {code:java}
> final List list = (List) value;
> final List annots = new ArrayList<>(); //-init
> for(Object o : list){
> annots.add(o.toString());
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-2770) Javadoc for Optional methods generated via record.vm is invalid
[
https://issues.apache.org/jira/browse/AVRO-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2770.
---
Fix Version/s: 1.10.0
Resolution: Fixed
> Javadoc for Optional methods generated via record.vm is invalid
> ---
>
> Key: AVRO-2770
> URL: https://issues.apache.org/jira/browse/AVRO-2770
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.2
>Reporter: Jeff Maxwell
>Priority: Major
> Fix For: 1.10.0
>
>
> Javadoc for Optional methods generated via record.vm is invalid
> Replace <> with {{}} and {{}}
> PR here: [https://github.com/apache/avro/pull/839]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-2070) Tolerate any Number when writing primitive values in Java in GenericDatumWriter
[
https://issues.apache.org/jira/browse/AVRO-2070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2070.
---
Fix Version/s: 1.10.0
Resolution: Fixed
> Tolerate any Number when writing primitive values in Java in
> GenericDatumWriter
> ---
>
> Key: AVRO-2070
> URL: https://issues.apache.org/jira/browse/AVRO-2070
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: Daniil Gitelson
>Assignee: Rabi Kumar K C
>Priority: Major
> Fix For: 1.10.0
>
>
> Tolerating any Number (instead of concrete Long, Double, Float) makes
> possible to use mutable Number implmentation for performance reasons
> (specially for primitive collection iterations)
> Currently, this only works for int only:
> {code:java}
> // Here it works
> case INT: out.writeInt(((Number)datum).intValue()); break;
> // This should be replaced with ((Number)datum).longValue() etc
> case LONG:out.writeLong((Long)datum); break;
> case FLOAT: out.writeFloat((Float)datum); break;
> case DOUBLE: out.writeDouble((Double)datum); break;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-2648) Incorrect validation of numeric default values
[
https://issues.apache.org/jira/browse/AVRO-2648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2648.
---
Fix Version/s: 1.10.0
Resolution: Fixed
> Incorrect validation of numeric default values
> --
>
> Key: AVRO-2648
> URL: https://issues.apache.org/jira/browse/AVRO-2648
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Reporter: Jeffrey Mullins
>Assignee: Jeffrey Mullins
>Priority: Major
> Fix For: 1.10.0
>
>
> Validation of numeric default values is incorrect and results in API
> inconsistencies. Below are a few examples.
> Double value as int default value:
> {code:java}
> public void testDoubleAsIntDefaultValue() {
> Schema.Field field = new Schema.Field("myField",
> Schema.create(Schema.Type.INT), "doc", 1.1);
> field.hasDefaultValue(); // true
> field.defaultValue(); // internal DoubleNode (1.1)
> field.defaultVal(); // null
> GenericData.get().getDefaultValue(field); // Integer (1)
>
> field = new Schema.Field("myField", Schema.create(Schema.Type.INT),
> "doc", 1.0);
> field.hasDefaultValue(); // true
> field.defaultValue(); // internal DoubleNode (1.0)
> field.defaultVal(); // null
> GenericData.get().getDefaultValue(field); // Integer (1)
> }{code}
>
> {color:#172b4d}Invalid long as int default value:{color}
> {code:java}
> public void testInvalidLongAsIntDefault() {
> Schema.Field field = new Schema.Field("myField",
> Schema.create(Schema.Type.INT), "doc", Integer.MAX_VALUE + 1L);
> field.hasDefaultValue(); // true
> field.defaultValue(); // internal LongNode (2147483648)
> field.defaultVal(); // Long (2147483648)
> GenericData.get().getDefaultValue(field); // Integer (-2147483648)
> }{code}
> Additionally, since the underlying Schema.FIeld.defaultValue() is no longer
> public it's not possible for users to disable default value validation and
> reliably validate schemas independently.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-2762) Unknown resource closed
[
https://issues.apache.org/jira/browse/AVRO-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2762.
---
Fix Version/s: 1.10.0
Resolution: Fixed
> Unknown resource closed
> ---
>
> Key: AVRO-2762
> URL: https://issues.apache.org/jira/browse/AVRO-2762
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.9.2
>Reporter: Zezeng Wang
>Assignee: Zezeng Wang
>Priority: Minor
> Fix For: 1.10.0
>
>
> DataFileReader will open the resource file, but it only deals with the
> resource closing in abnormal situations, and it does not inform the user that
> he needs to actively release the resource.
> {code}
> /** Construct a reader for a file. */
> protected DataFileReader(SeekableInput sin, DatumReader reader, boolean
> closeOnError) throws IOException {
> super(reader);
> try {
> this.sin = new SeekableInputStream(sin);
> initialize(this.sin);
> blockFinished();
> } catch (final Throwable e) {
> if (closeOnError) {
> IOUtils.closeQuietly(sin);
> }
> throw e;
> }
> }
> {code}
> I think we should add a description in the comment to help the user read it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-2829) PHP: Add php linter
[ https://issues.apache.org/jira/browse/AVRO-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2829. --- Fix Version/s: 1.10.0 Resolution: Fixed > PHP: Add php linter > --- > > Key: AVRO-2829 > URL: https://issues.apache.org/jira/browse/AVRO-2829 > Project: Apache Avro > Issue Type: Bug > Components: php >Reporter: Siad Ardroumli >Priority: Major > Fix For: 1.10.0 > > > As seen in the build.sh there is no syntax check. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AVRO-2608) The generated java-class attribute URI has an unhandled exception type of URISyntaxException
[
https://issues.apache.org/jira/browse/AVRO-2608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-2608:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> The generated java-class attribute URI has an unhandled exception type of
> URISyntaxException
>
>
> Key: AVRO-2608
> URL: https://issues.apache.org/jira/browse/AVRO-2608
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Reporter: Zezeng Wang
>Assignee: Zezeng Wang
>Priority: Major
> Fix For: 1.10.0
>
>
> When creating a java.net.URI Object,there is no handling of a
> URISyntaxException.
> Given a specific record generated from the following avsc:
> {code:java}
> {"namespace": "example.avro",
> "type": "record",
> "name": "User",
> "fields": [
> {"name": "name", "type": "string"},
> {"name": "favorite_number", "type": ["int", "null"]},
> {"name": "favorite_color", "type": ["string", "null"]},
>{
> "name": "ownerAddress",
> "type": [
> "null",
> {
> "type": "string",
> "java-class": "java.net.URI"
> }
> ],
> "default": null
> }
> ]
> }{code}
> and class
> @org.apache.avro.specific.AvroGenerated
> public class User extends org.apache.avro.specific.
> {
> ..
> private java.net.URI ownerAddress;
> ..
> {color:red}this.ownerAddress = new java.net.URI(in.readString());{color}
> *{color:red}//There is Unhandled exception type URISyntaxException{color}*
> }{
> If you want to use the customDecode method, there will be an error.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-2419) Generate private attribute in Java
[
https://issues.apache.org/jira/browse/AVRO-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2419.
---
Fix Version/s: 1.10.0
Resolution: Fixed
> Generate private attribute in Java
> --
>
> Key: AVRO-2419
> URL: https://issues.apache.org/jira/browse/AVRO-2419
> Project: Apache Avro
> Issue Type: Wish
> Components: java
>Affects Versions: 1.7.7
>Reporter: khuynhthanhchiluyen
>Assignee: Zezeng Wang
>Priority: Major
> Fix For: 1.10.0
>
>
> I used file user.avsc like below:
> {
> "namespace":"example.avro",
> "type":"record",
> "name":"User",
> "fields":[
>
> { "name":"name", "type":"string" }
> ,
>
> { "name":"favorite_number", "type":[ "int",
> "null" ] }
> ,
>
> { "name":"favorite_color", "type":[
> "string", "null" ] }
> ]
> }
>
> And I used avro-1.7.7.jar to generate my Java code:
> java -jar java -jar avro-tools-1.7.7.jar compile -string schema user.avsc
> D:\result
>
> But all attributes of User class (name, favorite_number, favorite_color)
> appear to be "public". Where did I miss to make them private (for
> encapsulation). Thank you very much.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-2573) Javadoc Warning in java compilation
[ https://issues.apache.org/jira/browse/AVRO-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2573. --- Fix Version/s: 1.10.0 Resolution: Fixed > Javadoc Warning in java compilation > --- > > Key: AVRO-2573 > URL: https://issues.apache.org/jira/browse/AVRO-2573 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Zezeng Wang >Assignee: Zezeng Wang >Priority: Minor > Fix For: 1.10.0 > > > In ./build.sh dist, there are many tags @link: reference not found > @return null and more... > So, I just have a little time to do something for the community. > Thanks. > Best regards. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AVRO-2569) @Deprecated annotation usage does not conform to the Java specification
[ https://issues.apache.org/jira/browse/AVRO-2569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2569. --- Fix Version/s: 1.10.0 Resolution: Fixed > @Deprecated annotation usage does not conform to the Java specification > --- > > Key: AVRO-2569 > URL: https://issues.apache.org/jira/browse/AVRO-2569 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Zezeng Wang >Assignee: Zezeng Wang >Priority: Minor > Fix For: 1.10.0 > > > In lang/java/compiler/../SpecificCompiler#javaUnbox(Schema, boolean), > Utility for template use. Returns the unboxed java type for a Schema > including the void type > Used for preventing unnecessary returns for RPC methods without response but > with error(s). > This is great. > But in the javaUnbox (Schema) comment added @deprecated, but did not add > @Deprecated on the method definition, JDK does not want this, they should > exist at the same time, otherwise the compiler will have a warning -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AVRO-2589) Java (trevni) codec testing improvements
[ https://issues.apache.org/jira/browse/AVRO-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2589. --- Fix Version/s: 1.10.0 Resolution: Fixed > Java (trevni) codec testing improvements > > > Key: AVRO-2589 > URL: https://issues.apache.org/jira/browse/AVRO-2589 > Project: Apache Avro > Issue Type: Improvement >Reporter: Jacob Tolar >Assignee: Jacob Tolar >Priority: Major > Fix For: 1.10.0 > > > In https://issues.apache.org/jira/browse/AVRO-2245 I made some improvements > to compression codec tests and it was suggested there to make the same > changes for trevni. > > The same fixes for improving behavior of sliced ByteBuffers can also be made. > > The trevni code is mostly duplicated from what is in avro-core; in the future > perhaps these could be combined into a single interface and codebase. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AVRO-2554) Fix Ruby interop test to read all supported codecs
[
https://issues.apache.org/jira/browse/AVRO-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-2554:
--
Fix Version/s: 1.10.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Fix Ruby interop test to read all supported codecs
> --
>
> Key: AVRO-2554
> URL: https://issues.apache.org/jira/browse/AVRO-2554
> Project: Apache Avro
> Issue Type: Bug
> Components: interop, ruby
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Major
> Fix For: 1.10.0
>
>
> I've just noticed AVRO-2456 had a bug on the Ruby data interop test. It
> checks only the null codec and doesn't others.
> {code}
> sekikn@4e7deb5781b2:~/avro$ git diff
>
>
> diff --git a/lang/ruby/interop/test_interop.rb
> b/lang/ruby/interop/test_interop.rb
>
> index a63c158c..f4528f00 100755
>
>
> --- a/lang/ruby/interop/test_interop.rb
>
>
> +++ b/lang/ruby/interop/test_interop.rb
>
>
> @@ -30,6 +30,7 @@ class TestInterop < Test::Unit::TestCase
>
>
> sep, codec = File.basename(fn, 'avro').rpartition('_')[1, 2]
>
>
> sep.empty? || CODECS_TO_VALIDATE.include?(codec)
>
>
>end
>
>
> + puts files
>
>
>
>
>
>files.each do |fn|
>
>
> define_method("test_read_#{File.basename(fn, 'avro')}") do
>
>
> sekikn@4e7deb5781b2:~/avro$ ls -l build/interop/data
> total 220
> -rw-r--r-- 1 sekikn sekikn 1279 Sep 9 01:53 c.avro
> -rw-r--r-- 1 sekikn sekikn 1258 Sep 9 01:53 csharp.avro
> -rw-r--r-- 1 sekikn sekikn 1240 Sep 9 01:53 csharp_deflate.avro
> -rw-r--r-- 1 sekikn sekikn 38453 Sep 9 01:43 java.avro
> -rw-r--r-- 1 sekikn sekikn 29436 Sep 9 01:43 java_deflate.avro
> -rw-r--r-- 1 sekikn sekikn 35412 Sep 9 01:43 java_snappy.avro
> -rw-r--r-- 1 sekikn sekikn 31929 Sep 9 01:43 java_zstandard.avro
> -rw-r--r-- 1 sekikn sekikn 1204 Sep 9 01:53 perl.avro
> -rw-r--r-- 1 sekikn sekikn 1186 Sep 9 01:53 perl_deflate.avro
> -rw-r--r-- 1 sekikn sekikn 1202 Sep 9 01:53 perl_zstandard.avro
> -rw-r--r-- 1 sekikn sekikn 1138 Sep 9 01:53 php.avro
> -rw-r--r-- 1 sekikn sekikn 1124 Sep 9 01:53 php_deflate.avro
> -rw-r--r-- 1 sekikn sekikn 1390 Sep 9 01:53 py.avro
> -rw-r--r-- 1 sekikn sekikn 1390 Sep 9 01:53 py3.avro
> -rw-r--r-- 1 sekikn sekikn 1375 Sep 9 01:53 py3_deflate.avro
> -rw-r--r-- 1 sekikn sekikn 1384 Sep 9 01:53 py3_snappy.avro
> -rw-r--r-- 1 sekikn sekikn 1388 Sep 9 01:53 py3_zstandard.avro
> -rw-r--r-- 1 sekikn sekikn 1375 Sep 9 01:53 py_deflate.avro
> -rw-r--r-- 1 sekikn sekikn 1384 Sep 9 01:53 py_snappy.avro
> -rw-r--r-- 1 sekikn sekikn 1388 Sep 9 01:53 py_zstandard.avro
> -rw-r--r-- 1 sekikn sekikn 1398 Sep 9 01:53 ruby.avro
> -rw-r--r-- 1 sekikn sekikn 1448 Sep 9 01:53 ruby_deflate.avro
> -rw-r--r-- 1 sekikn sekikn 1411 Sep 9 01:53 ruby_snappy.avro
> -rw-r--r-- 1 sekikn sekikn 1354 Sep 9 01:53 ruby_zstandard.avro
> sekikn@4e7deb5781b2:~/avro$ cd lang/ruby; rake interop
>
>
> /home/sekikn/avro/lang/ruby/interop/../../../build/interop/data/ruby.avro
>
>
>
[jira] [Updated] (AVRO-2547) Add bzip2 support to the Perl bindings
[ https://issues.apache.org/jira/browse/AVRO-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2547: -- Fix Version/s: 1.10.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Add bzip2 support to the Perl bindings > -- > > Key: AVRO-2547 > URL: https://issues.apache.org/jira/browse/AVRO-2547 > Project: Apache Avro > Issue Type: New Feature > Components: perl >Reporter: Kengo Seki >Assignee: Kengo Seki >Priority: Major > Fix For: 1.10.0 > > > For the same reason as AVRO-2546, it'd be nice if the Perl bindings support > bzip2 codec. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AVRO-2523) Perf's usage doesn't show the option for specific record test
[
https://issues.apache.org/jira/browse/AVRO-2523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-2523:
--
Fix Version/s: 1.10.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Perf's usage doesn't show the option for specific record test
> -
>
> Key: AVRO-2523
> URL: https://issues.apache.org/jira/browse/AVRO-2523
> Project: Apache Avro
> Issue Type: Bug
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Trivial
> Fix For: 1.10.0
>
>
> I tried to run the (now deprecated) Perf tool to measure the performance for
> specific records, but couldn't find that option in the usage.
> {code:java}
> ~/avro/lang/java/ipc$ mvn exec:java -Dexec.classpathScope=test
> -Dexec.mainClass=org.apache.avro.io.Perf -Dexec.args="-h"
> (snip)
> Usage: Perf [-o ] [-c ] { -nowrite | -noread }-basic | -i | -ls |
> -l | -f | -d | -b | -by | -s | -a | -m | -ee | -uu | -record | -R | -Rv | -Rr
> | -Rd | -Ro | -Rp | -generic | -G | -Gs | -Gn | -Gf | -Gd | -Go | -Gp |
> -generic-onetime | -Gotd | -Gotr | -Got | -reflect | -REFr | -REFbr | -REFf |
> -REFd | -REFia | -REFla | -REFda | -REFfa | -REFnf | -REFno | -REFnlf |
> -REFnlfb }
> -o file (send output to a file)
> -c [n][t][e][b][c][m] (format as no-header CSV; include Name, Time,
> Entries/sec, Bytes/sec, bytes/Cycle, and/or min time/op; no spec=all fields)
> -nowrite (do not execute write tests)
> -noread (do not execute write tests)
> -basic (executes all basic tests):
> -i (IntTest)
> -ls (SmallLongTest)
> -l (LongTest)
> -f (FloatTest)
> -d (DoubleTest)
> -b (BoolTest)
> -by (BytesTest)
> -s (StringTest)
> -a (ArrayTest)
> -m (MapTest)
> -ee (ExtendedEnumResolveTest)
> -uu (UnchangedUnionResolveTest)
> -record (executes all record tests):
> -R (RecordTest)
> -Rv (ValidatingRecord)
> -Rr (ResolvingRecord)
> -Rd (RecordWithDefault)
> -Ro (RecordWithOutOfOrder)
> -Rp (RecordWithPromotion)
> -generic (executes all generic tests):
> -G (GenericTest)
> -Gs (GenericStrings)
> -Gn (GenericNested)
> -Gf (GenericNestedFake)
> -Gd (GenericWithDefault)
> -Go (GenericWithOutOfOrder)
> -Gp (GenericWithPromotion)
> -generic-onetime (executes all generic-onetime tests):
> -Gotd (GenericOneTimeDecoderUse)
> -Gotr (GenericOneTimeReaderUse)
> -Got (GenericOneTimeUse)
> -reflect (executes all reflect tests):
> -REFr (ReflectRecordTest)
> -REFbr (ReflectBigRecordTest)
> -REFf (ReflectFloatTest)
> -REFd (ReflectDoubleTest)
> -REFia (ReflectIntArrayTest)
> -REFla (ReflectLongArrayTest)
> -REFda (ReflectDoubleArrayTest)
> -REFfa (ReflectFloatArrayTest)
> -REFnf (ReflectNestedFloatArrayTest)
> -REFno (ReflectNestedObjectArrayTest)
> -REFnlf (ReflectNestedLargeFloatArrayTest)
> -REFnlfb (ReflectNestedLargeFloatArrayBlockedTest)
> {code}
> But there's a "-Sf" option actually, though it's not displayed in the above
> messages.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-2718) Avro with Snappy enabled can't be compiled in Windows
[ https://issues.apache.org/jira/browse/AVRO-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2718. --- Fix Version/s: 1.10.0 Assignee: Daniel Kulp Resolution: Fixed > Avro with Snappy enabled can't be compiled in Windows > - > > Key: AVRO-2718 > URL: https://issues.apache.org/jira/browse/AVRO-2718 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.9.1 >Reporter: Michael Spector >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.10.0 > > > Undefined macro: __bswap_32 > Numerous casting issues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AVRO-2486) fwrite error ignored
[ https://issues.apache.org/jira/browse/AVRO-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2486. --- Assignee: Daniel Kulp Resolution: Fixed > fwrite error ignored > > > Key: AVRO-2486 > URL: https://issues.apache.org/jira/browse/AVRO-2486 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.8.0, 1.8.1, 1.9.0, 1.8.2 >Reporter: mathew boorman >Assignee: Daniel Kulp >Priority: Critical > Fix For: 1.10.0 > > > C library: Errors from fwrite are ignored, resulting in corrupt files. > > BugFix pull request: https://github.com/apache/avro/pull/594 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AVRO-2572) [avrocpp] Too restrictive SONAME
[
https://issues.apache.org/jira/browse/AVRO-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2572.
---
Fix Version/s: (was: 1.10.0)
Resolution: Won't Do
As mentioned on the PR, 1.8 and 1.9 are specifically NOT binary compatible with
each other. Thus, renaming the .so to just libavrocpp.so.1 would cause all
kinds of issues with upgrades and library versions. At the very least, the
name would have to contain the minor version, like libavrocpp.so.1.9
> [avrocpp] Too restrictive SONAME
>
>
> Key: AVRO-2572
> URL: https://issues.apache.org/jira/browse/AVRO-2572
> Project: Apache Avro
> Issue Type: Bug
> Components: c++
>Affects Versions: 1.8.1, 1.9.0, 1.8.2, 1.9.1
>Reporter: Laurent Stacul
>Priority: Major
>
> Hello,
> While I was upgrading Apache Avro from 1.8.2 to 1.9.1, I checked the binary
> compatibility using RedHat libabigail. If the code was OK which is what I
> expected from the Semantic Versioning of the project, I did not expect that
> the binary compatibility check failed due to the soname.
> I checked and found out this point was raised for the C component and fixed
> there: [https://github.com/apache/avro/pull/239.]
> If the binary compatibility is respected across the major versions of
> libavrocpp, I would expect
> {code:java}
> SONAME Library soname: [libavrocpp.so.1]{code}
> instead of:
> {code:java}
> SONAME Library soname: [libavrocpp.so.1.10.0-SNAPSHOT.0]{code}
> Do you agree on this ?
> Regards,
> Laurent
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-2551) Changed datum API to value API in examples\quickstop
[
https://issues.apache.org/jira/browse/AVRO-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2551.
---
Fix Version/s: 1.10.0
Resolution: Fixed
> Changed datum API to value API in examples\quickstop
>
>
> Key: AVRO-2551
> URL: https://issues.apache.org/jira/browse/AVRO-2551
> Project: Apache Avro
> Issue Type: Improvement
> Components: c
>Reporter: Zezeng Wang
>Assignee: Zezeng Wang
>Priority: Minor
> Fix For: 1.10.0
>
>
> Starting with version 1.6.0, the Avro C library has a new API for handling
> Avro data. To help distinguish between the two APIs, we refer to the old one
> as the _legacy_ or _datum_ API, and the new one as the _value_ API.
> The legacy API is still present, but it’s deprecated — we shouldn’t use the
> {{avro_datum_t}} type or the {{avro_datum_*}} functions in new code.
> But the quickstop as an example for newcomers still used the legacy API.
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AVRO-2453) Implement interop test for the C bindings
[
https://issues.apache.org/jira/browse/AVRO-2453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-2453:
--
Fix Version/s: 1.10.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Implement interop test for the C bindings
> -
>
> Key: AVRO-2453
> URL: https://issues.apache.org/jira/browse/AVRO-2453
> Project: Apache Avro
> Issue Type: Test
> Components: c
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Major
> Fix For: 1.10.0
>
>
> A file exists but its implementation is empty.
> {code:title=lang/c/tests/test_interop_data.c}
> int main(void)
> {
> return 0;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AVRO-2505) `./build.sh clean` in the c++ directory doesn't remove all temporary files
[
https://issues.apache.org/jira/browse/AVRO-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-2505:
--
Fix Version/s: 1.9.1
Resolution: Fixed
Status: Resolved (was: Patch Available)
> `./build.sh clean` in the c++ directory doesn't remove all temporary files
> --
>
> Key: AVRO-2505
> URL: https://issues.apache.org/jira/browse/AVRO-2505
> Project: Apache Avro
> Issue Type: Improvement
> Components: build
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Trivial
> Fix For: 1.9.1
>
>
> {code}
> sekikn@a80af4ec301b:~/avro$ cd lang/c++
> sekikn@a80af4ec301b:~/avro/lang/c++$ ./build.sh test
> (snip)
> sekikn@a80af4ec301b:~/avro/lang/c++$ ./build.sh clean
> -- Boost version: 1.62.0
> -- Found the following Boost libraries:
> -- filesystem
> -- iostreams
> -- program_options
> -- regex
> -- system
> Enabled snappy codec
> -- Configuring done
> -- Generating done
> -- Build files have been written to: /home/sekikn/avro/lang/c++/build
> sekikn@a80af4ec301b:~/avro/lang/c++$ ls test*.df
> test8.df test9.df test_skip.df
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (AVRO-2465) Fix a wrong description in the avro-tools random's usage
[
https://issues.apache.org/jira/browse/AVRO-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-2465:
--
Fix Version/s: 1.9.1
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Fix a wrong description in the avro-tools random's usage
>
>
> Key: AVRO-2465
> URL: https://issues.apache.org/jira/browse/AVRO-2465
> Project: Apache Avro
> Issue Type: Bug
> Components: java, tools
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Minor
> Fix For: 1.9.1
>
>
> {{avro-tools random}}'s usage says the default compression codec is null:
> {code}
> $ java -jar lang/java/tools/target/avro-tools-1.10.0-SNAPSHOT.jar random
> Usage: outFile (filename or '-' for stdout)
> Option Description
> -- ---
> --codec Compression codec (default: null)
> {code}
> But actually the deflate codec is used.
> {code}
> $ java -jar lang/java/tools/target/avro-tools-1.10.0-SNAPSHOT.jar random
> --count 0 --schema-file share/test/schemas/weather.avsc /tmp/outFile.avro
> $ java -jar lang/java/tools/target/avro-tools-1.10.0-SNAPSHOT.jar getmeta
> /tmp/outFile.avro
> avro.schema {"type":"record","name":"Weather","namespace":"test","doc":"A
> weather
> reading.","fields":[{"name":"station","type":"string","order":"ignore"},{"name":"time","type":"long"},{"name":"temp","type":"int"}]}
> avro.codecdeflate
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (AVRO-2497) Avro-grpc failed when spotless:check alone
[ https://issues.apache.org/jira/browse/AVRO-2497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2497: -- Resolution: Fixed Fix Version/s: 1.9.1 Status: Resolved (was: Patch Available) > Avro-grpc failed when spotless:check alone > -- > > Key: AVRO-2497 > URL: https://issues.apache.org/jira/browse/AVRO-2497 > Project: Apache Avro > Issue Type: Test > Components: java >Affects Versions: 1.9.0 > Environment: jkd-8 > maven-3.3.9 >Reporter: Smart Yang >Assignee: Kengo Seki >Priority: Minor > Fix For: 1.9.1 > > > Enter avro-release-1.9.0/lang/java/grpc > mvn spotless:check > Result: BUILD FAILURE > Log: Could not find goal 'check' in plugin > org.apache.avro:avro-maven-plugin:1.9.0 among available goals help, > idl-protocol, induce, protocol, schema -> [Help 1] > But the lang/java/ipc other directories don't have this problem. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (AVRO-2479) Add m2e lifecycle mapping file
[ https://issues.apache.org/jira/browse/AVRO-2479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2479. --- Resolution: Fixed Assignee: Daniel Kulp Fix Version/s: 1.9.1 > Add m2e lifecycle mapping file > -- > > Key: AVRO-2479 > URL: https://issues.apache.org/jira/browse/AVRO-2479 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Pino Silvaggio >Assignee: Daniel Kulp >Priority: Trivial > Fix For: 1.9.1 > > Original Estimate: 1h > Remaining Estimate: 1h > > Instead of including the lifecycle plugin in the pom with the correct > mappings we could just add the file by default. > I suggest enabling build on configuration. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (AVRO-2489) Build error C3861: 'sleep': identifier not found
[
https://issues.apache.org/jira/browse/AVRO-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2489.
---
Resolution: Fixed
Assignee: Daniel Kulp
Fix Version/s: 1.9.1
> Build error C3861: 'sleep': identifier not found
> -
>
> Key: AVRO-2489
> URL: https://issues.apache.org/jira/browse/AVRO-2489
> Project: Apache Avro
> Issue Type: Bug
> Components: c++
>Affects Versions: 1.9.0
> Environment: Windows Server 2016,
> VS2019,
> cmake 3.15,
> vcpkg 2019.06.26
> Jenkins 2.186
>Reporter: Andrew Ripa
>Assignee: Daniel Kulp
>Priority: Minor
> Labels: build
> Fix For: 1.9.1
>
>
> Steps to reproduce:
> * Fetched 1.9.0 repo
> [https://github.com/apache/avro/releases/tag/release-1.9.0] from git and cd
> to 'lang/c++' folder
> * Project generated with
> {code:java}
> mkdir _build
> cd _build
> cmake -G "Visual Studio 16 2019" -A x64 .. -DCMAKE_BUILD_TYPE=Release
> -DCMAKE_TOOLCHAIN_FILE="PATH_TO_vcpkg.cmake"
> -DCMAKE_INSTALL_PREFIX="PATH_TO_INSTALL"{code}
> * Built with
> {code:java}
> cmake --build . --config Release
> {code}
> * Installed with
> {code:java}
> cmake --build . --config Release --target Install
> {code}
> During the last (install) step received an error
> {noformat}
> PATH_TO_WORKSPACE\workspace\avro\avro\lang\c++\test\DataFileTests.cc(592,13):
> error C3861: 'sleep': identifier not found
> [PATH_TO_WORKSPACE\workspace\avro\avro\lang\c++\_build\DataFileTests.vcxproj]{noformat}
>
> Possible hotfix found
> [https://social.msdn.microsoft.com/Forums/vstudio/en-US/fdc9d1c3-1082-4c0d-a717-a39313562d4f/sleep?forum=vclanguage]
> Need to change 'sleep(1)' to '{color:#ff}*S*{color}leep(1)' in
> lang/c++/testDataFileTests.cc
> After that build (and install) completed without errors
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (AVRO-2489) Build error C3861: 'sleep': identifier not found
[
https://issues.apache.org/jira/browse/AVRO-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896498#comment-16896498
]
Daniel Kulp commented on AVRO-2489:
---
Changing from sleep to Sleep is definitely not the right solution.
It should be one of:
a) Add {code:c++}#include {code} to import the sleep definition
or
b) Likely more "C++":
{code:c++}
#include
#include
std::this_thread::sleep_for(std::chrono::seconds(1));
{code}
Any chance you could double check one (or both) of those?
> Build error C3861: 'sleep': identifier not found
> -
>
> Key: AVRO-2489
> URL: https://issues.apache.org/jira/browse/AVRO-2489
> Project: Apache Avro
> Issue Type: Bug
> Components: c++
>Affects Versions: 1.9.0
> Environment: Windows Server 2016,
> VS2019,
> cmake 3.15,
> vcpkg 2019.06.26
> Jenkins 2.186
>Reporter: Andrew Ripa
>Priority: Minor
> Labels: build
>
> Steps to reproduce:
> * Fetched 1.9.0 repo
> [https://github.com/apache/avro/releases/tag/release-1.9.0] from git and cd
> to 'lang/c++' folder
> * Project generated with
> {code:java}
> mkdir _build
> cd _build
> cmake -G "Visual Studio 16 2019" -A x64 .. -DCMAKE_BUILD_TYPE=Release
> -DCMAKE_TOOLCHAIN_FILE="PATH_TO_vcpkg.cmake"
> -DCMAKE_INSTALL_PREFIX="PATH_TO_INSTALL"{code}
> * Built with
> {code:java}
> cmake --build . --config Release
> {code}
> * Installed with
> {code:java}
> cmake --build . --config Release --target Install
> {code}
> During the last (install) step received an error
> {noformat}
> PATH_TO_WORKSPACE\workspace\avro\avro\lang\c++\test\DataFileTests.cc(592,13):
> error C3861: 'sleep': identifier not found
> [PATH_TO_WORKSPACE\workspace\avro\avro\lang\c++\_build\DataFileTests.vcxproj]{noformat}
>
> Possible hotfix found
> [https://social.msdn.microsoft.com/Forums/vstudio/en-US/fdc9d1c3-1082-4c0d-a717-a39313562d4f/sleep?forum=vclanguage]
> Need to change 'sleep(1)' to '{color:#ff}*S*{color}leep(1)' in
> lang/c++/testDataFileTests.cc
> After that build (and install) completed without errors
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Resolved] (AVRO-2373) Remove Empty Package 'ipc'
[ https://issues.apache.org/jira/browse/AVRO-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2373. --- Resolution: Fixed Assignee: Daniel Kulp > Remove Empty Package 'ipc' > -- > > Key: AVRO-2373 > URL: https://issues.apache.org/jira/browse/AVRO-2373 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.9.0 >Reporter: David Mollitor >Assignee: Daniel Kulp >Priority: Minor > Fix For: 1.9.0 > > > https://github.com/apache/avro/tree/master/lang/java/avro/src/main/java/org/apache/avro/ipc -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2369) Provide external way to construct Schema.Field with default value of 'null'
[
https://issues.apache.org/jira/browse/AVRO-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2369.
---
Resolution: Fixed
Assignee: Daniel Kulp
I added a Field.NULL_DEFAULT_VALUE object that can be used to specify that a
null should be set as the default for the union. Not the greatest solution,
but I was struggling to find any alternative that wouldn't cause a gigantic
migration issue.
> Provide external way to construct Schema.Field with default value of 'null'
> ---
>
> Key: AVRO-2369
> URL: https://issues.apache.org/jira/browse/AVRO-2369
> Project: Apache Avro
> Issue Type: Task
> Components: java
>Reporter: Ivan Greene
>Assignee: Daniel Kulp
>Priority: Blocker
> Fix For: 1.9.0
>
>
> After making the Schema.Field constructor which takes the default value as a
> JsonNode was made package private, there is no external way to construct a
> field that has a default value of 'null'. Internally that constructor will
> call {{JacksonUtils.toJsonNode(defaultValue)}}, which will return 'null' when
> passed null, and the resulting Field will not have a default value (the json
> node would need to be NullNode instead of simply null itself). This will
> affect projects that need a way to dynamically build schemas and their fields.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2202) Use of -fstack-protector or -D_GLIBCXX_DEBUG causes memory corruption when combined with musl libc static builds
[ https://issues.apache.org/jira/browse/AVRO-2202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2202. --- Resolution: Fixed Assignee: Daniel Kulp (was: Thiruvalluvan M. G.) I'm not able to reproduce any crash with "normal" libc setups. However, I did update the CMakeLists.txt to do two things: 1) Make sure we preserve existing CMAKE_CXX_FLAGS and CMAKE_CXX_FLAGS_DEBUG so things like the -g are not removed. This is definitely important in trying to debug things. 2) Only add the above problematic flags if AVRO_ADD_PROTECTOR_FLAGS is set which should only be set for our "build.sh test" runs. It's not set for all Debug builds so you should be able to do your own build without those flags and hopefully then not have your crash. > Use of -fstack-protector or -D_GLIBCXX_DEBUG causes memory corruption when > combined with musl libc static builds > > > Key: AVRO-2202 > URL: https://issues.apache.org/jira/browse/AVRO-2202 > Project: Apache Avro > Issue Type: Bug > Components: c++ >Affects Versions: 1.8.2 > Environment: OS: Gentoo Linux > Compiler: Gcc 6.4.0 targeting musl libc 1.19 with stdlibc++ in a static build >Reporter: Josh Scoggins >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.9.0 > > > Observed with master and version 1.8.2. Inside of the C++ implementation's > CMakeLists.txt the lines 56-60 were added sometime in 2016 which activate > features inside of libstdc++ to help find errors. This change has two side > effects: > # -g is eliminated from the build so debugging gdb becomes harder as source > position data has been removed > # The safety containers that implicitly wrap all standard containers somehow > (I can't figure out exactly how) cause memory corruption when setMetadata is > called (in version 1.8.2) or when a Writer object is destroyed (on master). > Since our software uses musl libc with libstdc++ I believe that it is an > interaction with these debugging features which cause memory corruption (as I > said, I have no clue _why_ it is happening, only that these features _cause_ > it :() > Previously, we were using 1.7.7 which did not have these lines in its > CMakeLists.txt and thus did not exhibit any sort of memory corruption. I was > able to work around this by removing lines 56-60 from CMakeLists.txt so our > static libs could be built and our unit tests not crash by writing to memory > it did not own. There should be a toggle or something similar to turn off > these features for debug builds as they are for developers of avro only. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1777) Select best matching record when writing a union in python
[
https://issues.apache.org/jira/browse/AVRO-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16810955#comment-16810955
]
Daniel Kulp commented on AVRO-1777:
---
I'm re-opening the issue and removing it from 1.9.0.The patch causes
interop test failures which is obviously a major problem. I don't know enough
python to figure it out.
> Select best matching record when writing a union in python
> --
>
> Key: AVRO-1777
> URL: https://issues.apache.org/jira/browse/AVRO-1777
> Project: Apache Avro
> Issue Type: Improvement
> Components: python
>Affects Versions: 1.7.7
>Reporter: Steven Aerts
>Assignee: Daniel Kulp
>Priority: Major
>
> Unlike javascript, python is not using wrapped types.
> So when writing a union it needs to guess find out which type it will output.
> At the moment it takes the last validating type.
> I propose to take the type with the most matching fields.
> So I propose to change in {{io.py}}:
> {code}
> # resolve union
> index_of_schema = -1
> for i, candidate_schema in enumerate(writers_schema.schemas):
> if validate(candidate_schema, datum):
> index_of_schema = i
> if index_of_schema < 0: raise AvroTypeException(writers_schema, datum)
> {code}
> into
> {code}
> # resolve union
> index_of_schema = -1
> found_fields = -1
> for i, candidate_schema in enumerate(writers_schema.schemas):
> if validate(candidate_schema, datum):
> nr_fields = candidate_schema.type in ['record', 'error', 'request'] and
> len(candidate_schema.fields) or 1
> if nr_fields > found_fields:
> index_of_schema = i
> found_fields = nr_fields
> if index_of_schema < 0: raise AvroTypeException(writers_schema, datum)
> {code}
> If you want, I can create a pull request for this. And apply it both on py3
> as py.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-1777) Select best matching record when writing a union in python
[
https://issues.apache.org/jira/browse/AVRO-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-1777:
--
Fix Version/s: (was: 1.9.0)
> Select best matching record when writing a union in python
> --
>
> Key: AVRO-1777
> URL: https://issues.apache.org/jira/browse/AVRO-1777
> Project: Apache Avro
> Issue Type: Improvement
> Components: python
>Affects Versions: 1.7.7
>Reporter: Steven Aerts
>Assignee: Daniel Kulp
>Priority: Major
>
> Unlike javascript, python is not using wrapped types.
> So when writing a union it needs to guess find out which type it will output.
> At the moment it takes the last validating type.
> I propose to take the type with the most matching fields.
> So I propose to change in {{io.py}}:
> {code}
> # resolve union
> index_of_schema = -1
> for i, candidate_schema in enumerate(writers_schema.schemas):
> if validate(candidate_schema, datum):
> index_of_schema = i
> if index_of_schema < 0: raise AvroTypeException(writers_schema, datum)
> {code}
> into
> {code}
> # resolve union
> index_of_schema = -1
> found_fields = -1
> for i, candidate_schema in enumerate(writers_schema.schemas):
> if validate(candidate_schema, datum):
> nr_fields = candidate_schema.type in ['record', 'error', 'request'] and
> len(candidate_schema.fields) or 1
> if nr_fields > found_fields:
> index_of_schema = i
> found_fields = nr_fields
> if index_of_schema < 0: raise AvroTypeException(writers_schema, datum)
> {code}
> If you want, I can create a pull request for this. And apply it both on py3
> as py.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (AVRO-1777) Select best matching record when writing a union in python
[
https://issues.apache.org/jira/browse/AVRO-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp reopened AVRO-1777:
---
> Select best matching record when writing a union in python
> --
>
> Key: AVRO-1777
> URL: https://issues.apache.org/jira/browse/AVRO-1777
> Project: Apache Avro
> Issue Type: Improvement
> Components: python
>Affects Versions: 1.7.7
>Reporter: Steven Aerts
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 1.9.0
>
>
> Unlike javascript, python is not using wrapped types.
> So when writing a union it needs to guess find out which type it will output.
> At the moment it takes the last validating type.
> I propose to take the type with the most matching fields.
> So I propose to change in {{io.py}}:
> {code}
> # resolve union
> index_of_schema = -1
> for i, candidate_schema in enumerate(writers_schema.schemas):
> if validate(candidate_schema, datum):
> index_of_schema = i
> if index_of_schema < 0: raise AvroTypeException(writers_schema, datum)
> {code}
> into
> {code}
> # resolve union
> index_of_schema = -1
> found_fields = -1
> for i, candidate_schema in enumerate(writers_schema.schemas):
> if validate(candidate_schema, datum):
> nr_fields = candidate_schema.type in ['record', 'error', 'request'] and
> len(candidate_schema.fields) or 1
> if nr_fields > found_fields:
> index_of_schema = i
> found_fields = nr_fields
> if index_of_schema < 0: raise AvroTypeException(writers_schema, datum)
> {code}
> If you want, I can create a pull request for this. And apply it both on py3
> as py.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-1738) add java tool for outputting schema fingerprints
[ https://issues.apache.org/jira/browse/AVRO-1738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-1738. --- Resolution: Fixed > add java tool for outputting schema fingerprints > > > Key: AVRO-1738 > URL: https://issues.apache.org/jira/browse/AVRO-1738 > Project: Apache Avro > Issue Type: New Feature > Components: java >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.9.0 > > Attachments: AVRO-1738.1.patch > > > over in AVRO-1694 I wanted to quickly check that the Java library came up > with the same md5/sha fingerprint for some shcemas that the proposed Ruby > implementation does. > I noticed we don't have a tool that exposes the functionality yet, which > seems like a commonly useful thing to do. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-1777) Select best matching record when writing a union in python
[
https://issues.apache.org/jira/browse/AVRO-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-1777.
---
Resolution: Fixed
Assignee: Daniel Kulp
> Select best matching record when writing a union in python
> --
>
> Key: AVRO-1777
> URL: https://issues.apache.org/jira/browse/AVRO-1777
> Project: Apache Avro
> Issue Type: Improvement
> Components: python
>Affects Versions: 1.7.7
>Reporter: Steven Aerts
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 1.9.0
>
>
> Unlike javascript, python is not using wrapped types.
> So when writing a union it needs to guess find out which type it will output.
> At the moment it takes the last validating type.
> I propose to take the type with the most matching fields.
> So I propose to change in {{io.py}}:
> {code}
> # resolve union
> index_of_schema = -1
> for i, candidate_schema in enumerate(writers_schema.schemas):
> if validate(candidate_schema, datum):
> index_of_schema = i
> if index_of_schema < 0: raise AvroTypeException(writers_schema, datum)
> {code}
> into
> {code}
> # resolve union
> index_of_schema = -1
> found_fields = -1
> for i, candidate_schema in enumerate(writers_schema.schemas):
> if validate(candidate_schema, datum):
> nr_fields = candidate_schema.type in ['record', 'error', 'request'] and
> len(candidate_schema.fields) or 1
> if nr_fields > found_fields:
> index_of_schema = i
> found_fields = nr_fields
> if index_of_schema < 0: raise AvroTypeException(writers_schema, datum)
> {code}
> If you want, I can create a pull request for this. And apply it both on py3
> as py.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2368) Provide way to distinguish whether a Schema.Field has a default value or its default is null
[
https://issues.apache.org/jira/browse/AVRO-2368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16810855#comment-16810855
]
Daniel Kulp commented on AVRO-2368:
---
Yes. The goal is to not expose the JSON implementation in the public API's.
> Provide way to distinguish whether a Schema.Field has a default value or its
> default is null
>
>
> Key: AVRO-2368
> URL: https://issues.apache.org/jira/browse/AVRO-2368
> Project: Apache Avro
> Issue Type: Task
> Components: java
>Reporter: Ivan Greene
>Assignee: Daniel Kulp
>Priority: Blocker
> Fix For: 1.9.0
>
>
> After making {{Schema.Field#defaultValue}} package private, there is no way
> to distinguish whether a Field has a default value of 'null' or has no
> default at all. This is a breaking change, notably affecting Hortonworks
> Schema Registry. See discussion here:
> [https://github.com/hortonworks/registry/pull/547]
> I propose we should make that method public and still deprecated. Otherwise I
> don't see a clear way to release a version of the Hortonworks registry that
> will be compatible with both Avro 1.8.x and 1.9.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-1289) Python: Schema objects should polymorphically interact with data-walker interface
[
https://issues.apache.org/jira/browse/AVRO-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-1289:
--
Fix Version/s: (was: 1.9.0)
> Python: Schema objects should polymorphically interact with data-walker
> interface
> -
>
> Key: AVRO-1289
> URL: https://issues.apache.org/jira/browse/AVRO-1289
> Project: Apache Avro
> Issue Type: Improvement
> Components: python
>Affects Versions: 1.7.5
>Reporter: Jeremy Kahn
>Assignee: Jeremy Kahn
>Priority: Minor
> Labels: features
>
> Python {{avro.schema}} objects should be able to call back to a general
> data-and-schema parallel-walker ("validate" would be one of those, but so
> could be "default-filler").
> There should be an {{avro.walker}} interface that owns the parallel state (a
> datum-reader/deserializer, a datum-writer/serializer, a validator, or a
> default-filler -- see AVRO-1265). Schema polymorphism would allow us to
> eliminate the large (and highly redundant) function-dispatch methods in
> {{avro.io}} by making the {{avro.schema.Schema}} subclass responsible for
> calling back to the {{avro.walker}} object.
> Assigning this to v1.8.0 because it may be difficult to duplicate *every*
> behavior of 1.7.* with the same function signatures, especially where this
> refactor may be eliminate entire classes.
> This factoring ought to make it easier to improve or extend objects that meet
> this {{walker}} interface -- validators & serializers might be able to store
> more state about their position within a record, for example, to yield more
> informative error messages upon mismatch (as requested by Jonathan Coveney on
> the user mailing list).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-1275) GenericData's toString() method generates the wrong JSON encoding for the bytes type
[
https://issues.apache.org/jira/browse/AVRO-1275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-1275.
---
Resolution: Fixed
Assignee: Daniel Kulp
Hadoop Flags: Incompatible change
> GenericData's toString() method generates the wrong JSON encoding for the
> bytes type
>
>
> Key: AVRO-1275
> URL: https://issues.apache.org/jira/browse/AVRO-1275
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Reporter: Tom White
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 1.9.0
>
>
> We discovered in AVRO-1274 that for bytes GenericData.toString(Object datum)
> incorrectly generates
> {noformat}
> { "bytes" : "A" }
> {noformat}
> rather than just {{"A"}} for the JSON encoding defined in the spec
> (http://avro.apache.org/docs/current/spec.html#json_encoding).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2368) Provide way to distinguish whether a Schema.Field has a default value or its default is null
[
https://issues.apache.org/jira/browse/AVRO-2368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2368.
---
Resolution: Fixed
Assignee: Daniel Kulp
Field.hasDefaultValue() method added.
> Provide way to distinguish whether a Schema.Field has a default value or its
> default is null
>
>
> Key: AVRO-2368
> URL: https://issues.apache.org/jira/browse/AVRO-2368
> Project: Apache Avro
> Issue Type: Task
> Components: java
>Reporter: Ivan Greene
>Assignee: Daniel Kulp
>Priority: Blocker
> Fix For: 1.9.0
>
>
> After making {{Schema.Field#defaultValue}} package private, there is no way
> to distinguish whether a Field has a default value of 'null' or has no
> default at all. This is a breaking change, notably affecting Hortonworks
> Schema Registry. See discussion here:
> [https://github.com/hortonworks/registry/pull/547]
> I propose we should make that method public and still deprecated. Otherwise I
> don't see a clear way to release a version of the Hortonworks registry that
> will be compatible with both Avro 1.8.x and 1.9.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-732) Generated protocol's method should not throw AvroRemoteException
[ https://issues.apache.org/jira/browse/AVRO-732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-732. -- Resolution: Fixed Assignee: Daniel Kulp Hadoop Flags: Incompatible change > Generated protocol's method should not throw AvroRemoteException > > > Key: AVRO-732 > URL: https://issues.apache.org/jira/browse/AVRO-732 > Project: Apache Avro > Issue Type: Bug > Components: java >Reporter: Sharad Agarwal >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.9.0 > > Attachments: > AVRO-732-remove-generated-remote-exception-2012-12-11.patch, > AVRO-732-remove-generated-remote-exception-2012-12-17.patch > > > If user does NOT define the throws clause in the idl, the code is generated > with "throws AvroRemoteException" clause. However on throwing the > AvroRemoteException from the implementation, the serialization fails. This is > not intuitive to users. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2077) Avro tools should have an option to check reader-writer compatibility
[ https://issues.apache.org/jira/browse/AVRO-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2077: -- Fix Version/s: (was: 1.9.0) > Avro tools should have an option to check reader-writer compatibility > - > > Key: AVRO-2077 > URL: https://issues.apache.org/jira/browse/AVRO-2077 > Project: Apache Avro > Issue Type: New Feature > Components: java, tools >Reporter: Nandor Kollar >Assignee: Nandor Kollar >Priority: Minor > > It seems that avro-tools doesn't have any option to check for a given Avro > file and a given reader schema (JSON file or URL) that the reader's schema is > compatible with the writer's schema. This tool should report every > compatibility problem (if there's any). > According my knowledge, there's no such option for avro-tools as of now. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-1658) Add avroDoc on reflect
[
https://issues.apache.org/jira/browse/AVRO-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-1658.
---
Resolution: Fixed
> Add avroDoc on reflect
> --
>
> Key: AVRO-1658
> URL: https://issues.apache.org/jira/browse/AVRO-1658
> Project: Apache Avro
> Issue Type: New Feature
> Components: java
>Affects Versions: 1.7.7
>Reporter: Zhaonan Sun
>Assignee: Raymie Stata
>Priority: Major
> Labels: reflection
> Fix For: 1.9.0
>
> Attachments:
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch,
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch,
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch
>
>
> Looks like @AvroMeta can't add reserved fields, like @AvroMeta("doc", "some
> doc") will have exceptions.
> I would be greate if we have a @AvroDoc("some documentations") in
> org.apache.avro.reflect
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-813) EOFException is thrown during normal operation
[
https://issues.apache.org/jira/browse/AVRO-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-813.
--
Resolution: Fixed
> EOFException is thrown during normal operation
> --
>
> Key: AVRO-813
> URL: https://issues.apache.org/jira/browse/AVRO-813
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.5.0
>Reporter: Bruno Dumon
>Assignee: Bruno Dumon
>Priority: Major
> Labels: memex
> Fix For: 1.9.0
>
> Attachments: avro-813-patch.txt
>
>
> In an application that uses Avro as RPC mechanism (with the NettyTransceiver,
> but that's irrelevant), I've noticed in jprofiler that during normal
> operation quite some time was spent creating EOFExceptions:
> {noformat}
> 5.4% - 2,004 ms org.apache.avro.ipc.generic.GenericResponder.readRequest
> 5.0% - 1,871 ms org.apache.avro.generic.GenericDatumReader.read
> 4.9% - 1,832 ms org.apache.avro.generic.GenericDatumReader.read
> 4.9% - 1,832 ms org.apache.avro.generic.GenericDatumReader.readRecord
> 4.5% - 1,670 ms org.apache.avro.generic.GenericDatumReader.read
> 4.5% - 1,670 ms org.apache.avro.generic.GenericDatumReader.readRecord
> 4.3% - 1,596 ms org.apache.avro.generic.GenericDatumReader.read
> 2.8% - 1,048 ms org.apache.avro.generic.GenericDatumReader.readArray
> 1.3% - 477 ms org.apache.avro.io.ValidatingDecoder.arrayNext
> 1.3% - 471 ms org.apache.avro.io.BinaryDecoder.arrayNext
> 1.3% - 466 ms org.apache.avro.io.BinaryDecoder.doReadItemCount
> 1.3% - 466 ms org.apache.avro.io.BinaryDecoder.readLong
> 1.3% - 466 ms org.apache.avro.io.BinaryDecoder.ensureBounds
> 1.3% - 466 ms org.apache.avro.io.BinaryDecoder$ByteSource.compactAndFill
> 1.3% - 466 ms
> org.apache.avro.io.BinaryDecoder$InputStreamByteSource.tryReadRaw
> 1.3% - 466 ms org.apache.avro.util.ByteBufferInputStream.read
> 1.3% - 466 ms org.apache.avro.util.ByteBufferInputStream.getBuffer
> 1.3% - 466 ms java.io.EOFException.
> 1.3% - 466 ms java.io.IOException.
> 1.2% - 460 ms java.lang.Exception.
> 1.2% - 460 ms java.lang.Throwable.
> 1.2% - 460 ms java.lang.Throwable.fillInStackTrace
> {noformat}
> These exceptions are produced by the ByteBufferInputStream (which modifies
> InputStream's contract: return -1 at eof), but are catched higher up by the
> tryReadRaw method.
> What happens is this:
> The message in question has an (empty) array at the end of its message, thus
> the reader tries to read the size of this array in BinaryDecoder.readLong.
> This calls ensureBounds(10), whose contract is that it should read 10 bytes
> if they are available, and otherwise be quiet. ensureBounds calls via
> compactAndFill the tryReadRaw method. It is this method which catches the
> EOFException, because it only 'tries' to read so many bytes.
> Note that InputStreamByteSource.readRaw (without the 'try' part) does itself
> check if read < 0 in order to throw EOFException, making the throwing of
> EOFException in ByteBufferInputStream unnecessary (for this particular usage).
> There was some talk about EOFException in AVRO-392 too, though it seems this
> particular common case was not mentioned there. When using Avro RPC, or more
> in general, when using Avro to read small messages rather than large files,
> it seems like one can very easily run into this EOFException situation, which
> hurts performance.
> I'll attach a patch which simply removes the throwing of EOFException in
> ByteBufferInputStream, but this will likely break other cases which rely on
> the EOFException being thrown (haven't researched this to the bottom).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-1547) AvroApp Schema Tool
[ https://issues.apache.org/jira/browse/AVRO-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-1547: -- Fix Version/s: (was: 1.9.0) > AvroApp Schema Tool > --- > > Key: AVRO-1547 > URL: https://issues.apache.org/jira/browse/AVRO-1547 > Project: Apache Avro > Issue Type: New Feature > Components: java >Reporter: Lewis John McGibbney >Assignee: Lewis John McGibbney >Priority: Minor > > Over in Gora, I have been thinking for a while that the process of writing > JSON data beans is rather time consuming when beans are LARGE. > I wanted to open this ticket for a while and now only get around to. I > proposed to have the following > A simple HTML webpage that defines a form of sorts, the form will enable > users to create JSON schemas and will be driven by enabling users to enter > Object values based on the current Avro specification document e.g. ti will > be restrictive in scope. > On top of this I propose to then use simple JQuery to send a request to the > JSONBlob API [0], obtain a JSON representation of the data and then pretty > print write this information to a file within the browser. The users can then > save this file focally and do with it what they wish. > I think that this page can easily be hosted alongside the current static Avro > website and that there is no need to write a web application for this yet. > I'll try to work on it sooner rather than later as this would also lower the > barrier for users of Gora (as I am sure it would Users of other technologies > requiring definition of Objects via JSOn schemas). > I've not assigned this against any component as there is none which I feel > appropriate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-1922) Fixed dimension for array
[
https://issues.apache.org/jira/browse/AVRO-1922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-1922:
--
Fix Version/s: (was: 1.9.0)
> Fixed dimension for array
> -
>
> Key: AVRO-1922
> URL: https://issues.apache.org/jira/browse/AVRO-1922
> Project: Apache Avro
> Issue Type: New Feature
> Components: spec
>Affects Versions: 1.8.1
>Reporter: Jim Pivarski
>Priority: Major
>
> This is a feature request for future versions of the Avro specification.
> We have found one kind of data structure that is hard to express in Avro:
> tensors. Although we can (and do) build matrices as {"type": "array",
> "items": {"type": "array", "items": "double"}}, this type does not specify
> that the grid of numbers is rectangular. We believe that rectangular arrays
> of numbers (or other nested types) would be a strong addition to Avro, both
> as a type system and as a serialization format. With the total size of all
> dimensions fixed in the schema, they would not need to be repeated in each
> serialized datum.
> For instance, suppose there was an extension of type "array" to specify
> dimensions:
> {"type": "array", "dimensions": [3, 3, 3, 3], "items": "double"}
> This 3-by-3-by-3-by-3 tensor (representing, for instance, the Riemann
> curvature tensor in 3-space) specifies that 81 double-precision numbers
> (3*3*3*3) are expected for each datum. With nested arrays, the size, "3,"
> would have to be separately encoded 40 times (1 + 3*(1 + 3*(1 + 3))) for each
> datum, even though they would never change in a dataset of Riemann tensors.
> With a "dimensions" attribute in the schema, only the content needs to be
> serialized. Moreover, this extension can clearly be used with any other
> "items" type, to make dense tables of strings, for instance.
> Avro has been extended in a similar way in the past. The "fixed" type is a
> "bytes" without the need to specify the number of bytes for each datum. Our
> proposal provides a similar packing for structured objects that can be
> significant for large numbers of dimensions, as shown above. The advantage to
> consumers of Avro data is that we can write functions that do not need to
> check all array sizes at runtime (for operations like tensor contractions and
> products).
> We have searched the web and the Avro JIRA site for similar proposals and
> found none, so we're adding this proposal to JIRA in addition to this e-mail.
> Please let us know if you have any comments, or if we can provide any more
> information.
> Thank you!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2182) avro-maven-plugin: IDLProtocolMojo should regard property "project.build.sourceEncoding" for file generation
[ https://issues.apache.org/jira/browse/AVRO-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2182. --- Resolution: Fixed > avro-maven-plugin: IDLProtocolMojo should regard property > "project.build.sourceEncoding" for file generation > > > Key: AVRO-2182 > URL: https://issues.apache.org/jira/browse/AVRO-2182 > Project: Apache Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.2 > Environment: Tested on windows with Java 8. >Reporter: Hans-Peter Werner >Priority: Major > Fix For: 1.9.0 > > Attachments: JIRA-AVRO-2182.patch > > > Currently it's immpossible to use e.g. german Umlauts in neither documention > nor fieldnames. > Linked to [AVRO-1471|http://issues.apache.org/jira/browse/AVRO-1471] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-1559) Drop support for Ruby 1.8
[ https://issues.apache.org/jira/browse/AVRO-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-1559. --- Resolution: Fixed 1.9 will be Ruby 2.3.3+ > Drop support for Ruby 1.8 > - > > Key: AVRO-1559 > URL: https://issues.apache.org/jira/browse/AVRO-1559 > Project: Apache Avro > Issue Type: Wish > Components: ruby >Affects Versions: 1.7.7 >Reporter: Willem van Bergen >Assignee: Willem van Bergen >Priority: Major > Fix For: 1.9.0 > > Attachments: AVRO-1559.patch > > > - Ruby 1.8 is EOL, and is even security issues aren't addressed anymore. > - It is also getting hard to set up Ruby 1.8 to run the tests (e.g. on a > recent OSX, it won't compile without manual fiddling). > - Handling character encodings in Ruby 1.9 is very different than Ruby 1.8. > Supporting both at the same time adds a lot of overhead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2020) Update BUILD instructions with correct versions for language-specific libraries
[
https://issues.apache.org/jira/browse/AVRO-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2020.
---
Resolution: Fixed
Fix Version/s: (was: 1.7.9)
(was: 1.8.3)
> Update BUILD instructions with correct versions for language-specific
> libraries
> ---
>
> Key: AVRO-2020
> URL: https://issues.apache.org/jira/browse/AVRO-2020
> Project: Apache Avro
> Issue Type: Task
> Components: community, doc
>Affects Versions: 1.7.7, 1.8.1, 1.9.0
>Reporter: Sean Busbey
>Priority: Critical
> Fix For: 1.9.0
>
>
> our BUILD instructions current list a few things I know are wrong (Ruby 1.8
> support) and some things I'm pretty sure are wrong (Java 1.6 support).
> we should do a pass checking all the languages.
> {code}
> REQUIREMENTS
> The following packages must be installed before Avro can be built:
> - Java: JDK 1.6, Maven 2 or better, protobuf-compile
> - PHP: php5, phpunit, php5-gmp
> - Python: 2.5 or greater, python-setuptools for dist target
> - C: gcc, cmake, asciidoc, source-highlight
> - C++: cmake 2.8.4 or greater, g++, flex, bison, libboost-dev
> - C#: mono-devel mono-gmcs nunit
> - JavaScript: nodejs, npm
> - Ruby: ruby 1.86 or greater, ruby-dev, gem, rake, echoe, yajl-ruby
> - Perl: perl 5.8.1 or greater, gmake, Module::Install,
>Module::Install::ReadmeFromPod, Module::Install::Repository,
>Math::BigInt, JSON::XS, Try::Tiny, Regexp::Common, Encode,
>IO::String, Object::Tiny, Compress::ZLib, Test::More,
>Test::Exception, Test::Pod
> - Apache Ant 1.7
> - Apache Forrest 0.8 (for documentation)
> - md5sum, sha1sum, used by top-level dist target
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-1626) Missing lang/csharp/src/apache/perf/app.config
[
https://issues.apache.org/jira/browse/AVRO-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-1626.
---
Resolution: Fixed
> Missing lang/csharp/src/apache/perf/app.config
> --
>
> Key: AVRO-1626
> URL: https://issues.apache.org/jira/browse/AVRO-1626
> Project: Apache Avro
> Issue Type: Bug
> Components: csharp
>Reporter: Niels Basjes
>Priority: Major
> Fix For: 1.9.0
>
>
> This error is output during the build
> {code}
> Target _CopyAppConfigFile:
> /usr/lib/mono/4.5/Microsoft.Common.targets: error : Cannot copy
> /home/nbasjes/avro/lang/csharp/src/apache/perf/app.config to
> /home/nbasjes/avro/lang/csharp/build/perf/Release/Avro.perf.exe.config, as
> the source file doesn't exist.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2251) Modify Perf.java to better support automation scripts
[ https://issues.apache.org/jira/browse/AVRO-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2251. --- Resolution: Fixed > Modify Perf.java to better support automation scripts > - > > Key: AVRO-2251 > URL: https://issues.apache.org/jira/browse/AVRO-2251 > Project: Apache Avro > Issue Type: Test > Components: java >Reporter: Raymie Stata >Assignee: Raymie Stata >Priority: Major > Fix For: 1.9.0 > > > To better support automated performance-test suites, this patch proposes two > new arguments to the 'Perf.java' command-line tool: > The `-o' argument gives 'Perf.java' the name of a file that should get the > results of the run. Currently, Perf.java sends output to System.out – but if > 'Perf.java' is invoked using Maven, which is the easiest way to invoke it, > then System.out is polluted with a bunch of other output. Redirecting > 'Perf.java' output metrics to a file makes it easier for automation scripts > to process those metrics. > The `-c [spec]` argument tells 'Perf.java' to generate a comma-separated > output. By default, all benchmark metrics are output, but the optional > `spec` argument can be used to indicate exactly which metrics should be > included in the CSV output. The default output of 'Perf.java' is optimized > for human inspection – for example, it includes the text "ms" in the > running-time column so humans will understand the units of the running-time > metric. The `-c` option will tell 'Perf.java' to generate machine-optimized > output that is easier to consume by automation scripts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2357) (ReflectData) Support for generic types in protocol definitions
[
https://issues.apache.org/jira/browse/AVRO-2357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2357.
---
Resolution: Fixed
> (ReflectData) Support for generic types in protocol definitions
> ---
>
> Key: AVRO-2357
> URL: https://issues.apache.org/jira/browse/AVRO-2357
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: Ivan Greene
>Priority: Minor
> Fix For: 1.9.0
>
>
> For a Java interface extending another interface with type parameters, we may
> resolve actual type parameters and build the protocol definition based upon
> those.
> For example, let's say we have a generic protocol defined by a Java interface:
> {code:java}
> public interface CrudProto {
> void persist(T record);
> T fetchById(I id);
> }{code}
> It would be natural to define a set of interfaces that extend this, such as:
> {code:java}
> public interface FooBarRecordProto extends CrudProto {}
> public interface OtherRecordProto extends CrudProto {}
> {code}
> Calling ReflectData.get().getProtocol(FooBarRecordProto.class) should be able
> to resolve that this protocol deals in FooBarRecords and Strings, and build a
> protocol accordingly.
> Currently, this call will produce an exception stating that a schema for 'T'
> cannot be resolved.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2354) Add CombineAvroKeyValueFileInputFormat in avro-mapred to combine small avro keyvalue files into combineSplit
[ https://issues.apache.org/jira/browse/AVRO-2354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2354. --- Resolution: Fixed Assignee: Daniel Kulp Fix Version/s: 1.9.0 > Add CombineAvroKeyValueFileInputFormat in avro-mapred to combine small avro > keyvalue files into combineSplit > > > Key: AVRO-2354 > URL: https://issues.apache.org/jira/browse/AVRO-2354 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Wang, Xinglong >Assignee: Daniel Kulp >Priority: Minor > Fix For: 1.9.0 > > > In our production env, we generate avro files to track some user behavior > events. Every hour, we will have several avro files created. And daily, we > will run MR to do analysis, when using AvroKeyValueInputFormat, a lot of > small mappers started due to we have small avro files. > A combine file inputformat will be very helpful for such case. > Hadoop already provided some implementation for sequencefile and text file. > This Jira is propose a CombineAvroKeyValueFileInputFormat class to implement > the same for avro keyvalue files. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2364) Use Jackson TokenBuffer in JsonDecoder
[ https://issues.apache.org/jira/browse/AVRO-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2364. --- Resolution: Fixed Assignee: Daniel Kulp Fix Version/s: 1.9.0 > Use Jackson TokenBuffer in JsonDecoder > -- > > Key: AVRO-2364 > URL: https://issues.apache.org/jira/browse/AVRO-2364 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.8.2 >Reporter: Anthony Miyaguchi >Assignee: Daniel Kulp >Priority: Trivial > Fix For: 1.9.0 > > > The JsonDecoder implements a class for replaying parts of the JSON stream > when accessing fields out of order. > This module can take advantage of > [TokenBuffers|http://static.javadoc.io/com.fasterxml.jackson.core/jackson-databind/2.9.8/com/fasterxml/jackson/databind/util/TokenBuffer.html] > to reduce code size. This buffer can be read back as a parser. > > [https://github.com/apache/avro/blob/db992cf8686ffa681b0c592696b37ad519c33ee6/lang/java/avro/src/main/java/org/apache/avro/io/JsonDecoder.java#L514-L754] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2365) Enhancements to induce maven plugin goal
[ https://issues.apache.org/jira/browse/AVRO-2365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2365: -- Fix Version/s: (was: 1.8.3) > Enhancements to induce maven plugin goal > > > Key: AVRO-2365 > URL: https://issues.apache.org/jira/browse/AVRO-2365 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Ivan Greene >Priority: Major > Fix For: 1.9.0 > > > Before a release of 1.8.3 or 1.9.0 which will introduce the new 'induce' > maven plugin goal (see AVRO-1749), there are a few issues with it which in my > mind are critical to enable real-world use of the goal: > * Allow excluding classes based on a pattern, and/or the ability to specify > multiple sourceDirectories > * Option to specify whether to use ReflectData.AllowNull or ReflectData > * Throw MojoExecutionException with an informative message when appropriate, > instead of just printing stacktraces. This is very important, we definitely > need a build to fail when we hit a problem. > * Use the current project defined encoding rather than hard coded UTF-8. > * Use try-with-resources to ensure the PrintWriter gets closed in the event > of exception > Nice to haves: > * Specify custom ReflectData implementation (i.e user declared extensions) > to be used > * Specify separate output directories for schemata and protocols (these > could both default to the single output directory if not specified > individually) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2365) Enhancements to induce maven plugin goal
[ https://issues.apache.org/jira/browse/AVRO-2365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2365. --- Resolution: Fixed Assignee: Daniel Kulp > Enhancements to induce maven plugin goal > > > Key: AVRO-2365 > URL: https://issues.apache.org/jira/browse/AVRO-2365 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Ivan Greene >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.9.0 > > > Before a release of 1.8.3 or 1.9.0 which will introduce the new 'induce' > maven plugin goal (see AVRO-1749), there are a few issues with it which in my > mind are critical to enable real-world use of the goal: > * Allow excluding classes based on a pattern, and/or the ability to specify > multiple sourceDirectories > * Option to specify whether to use ReflectData.AllowNull or ReflectData > * Throw MojoExecutionException with an informative message when appropriate, > instead of just printing stacktraces. This is very important, we definitely > need a build to fail when we hit a problem. > * Use the current project defined encoding rather than hard coded UTF-8. > * Use try-with-resources to ensure the PrintWriter gets closed in the event > of exception > Nice to haves: > * Specify custom ReflectData implementation (i.e user declared extensions) > to be used > * Specify separate output directories for schemata and protocols (these > could both default to the single output directory if not specified > individually) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2335) Remove Joda Time Library
[
https://issues.apache.org/jira/browse/AVRO-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805121#comment-16805121
]
Daniel Kulp commented on AVRO-2335:
---
I'm not sure if this is something that we want for 1.9... As of right now,
for 1.9, we've made JSR-310 the default. However, Joda is there as an option
(and the dependency is marked optional) to aid in migrating code from 1.8 to
1.9. That said, we'd like to fully remove it at some point, I'm just
questioning whether it should be for 1.9 or 1.10.
> Remove Joda Time Library
>
>
> Key: AVRO-2335
> URL: https://issues.apache.org/jira/browse/AVRO-2335
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Reporter: David Mollitor
>Assignee: Fokko Driesprong
>Priority: Major
>
> Since [AVRO-2079] was completed, Avro supports Java's standard {{java.time}}
> libraries. Please remove the Joda Time implementations and the Joda Time
> dependency from Avro. Avro 2.0 perhaps?
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. *Users are now asked to migrate to java.time (JSR-310).*
> https://www.joda.org/joda-time/
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2360) Java 8 timestamp-millis / timestamp-micros type inconsistency
[ https://issues.apache.org/jira/browse/AVRO-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2360. --- Resolution: Fixed Assignee: Daniel Kulp Hadoop Flags: Incompatible change Fix Version/s: 1.9.0 > Java 8 timestamp-millis / timestamp-micros type inconsistency > - > > Key: AVRO-2360 > URL: https://issues.apache.org/jira/browse/AVRO-2360 > Project: Apache Avro > Issue Type: Bug > Components: java >Affects Versions: 1.9.0 > Environment: JDK 11.0.2 on Linux >Reporter: Raman Gupta >Assignee: Daniel Kulp >Priority: Blocker > Fix For: 1.9.0 > > > I am trying the new JSR 310 date/time types with a snapshot of 1.9.0. > I see what seems to be an inconsistency. If I generate my code with a logical > type of "timestamp-millis", then the code is generated with `Instant`, as > expected. However, on JDK11 on Linux, if I do `Instant.now()` the Instant > value created contains microseconds. When setting this Instant on an instance > of the generated Avro SpecificRecord, I am unable to round-trip the data: > Before serialization to bytes: > System.out.println(mySpecificRecord.tsMillis()) // 2019-03-27T23:16:09.665308Z > After serializing to bytes, and then back into a specific record the > microseconds are now truncated: > System.out.println(mySpecificRecord.tsMillis()) // 2019-03-27T23:16:09.665Z > I believe the compiler should generate a class that truncates the > microseconds at `setter` time for "timestamp-millis", so that the value > before serialization, and after deserialization, are the same. This can be > done with a call to the method `truncatedTo(ChronoUnit.MILLIS)`. > Another, possibly related, oddity is that the "timestamp-micros" type > generates a class with `long` as the type of the field. Since Instant can > support both milli and micro-second precision, I don't see the reason for > this behavior. > Followup on AVRO-2079. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2355) Add compressionLevel to ZStandard compression
[ https://issues.apache.org/jira/browse/AVRO-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2355. --- Resolution: Fixed Assignee: Daniel Kulp > Add compressionLevel to ZStandard compression > - > > Key: AVRO-2355 > URL: https://issues.apache.org/jira/browse/AVRO-2355 > Project: Apache Avro > Issue Type: New Feature > Components: java >Reporter: Scott Carey >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.9.0 > > > ZStandard compression should not be released without support for compression > level selection. > Its biggest advantage is the massive range by which you can select the > compression level, all while keeping decompression throughput very high. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AVRO-2162) Add Zstandard compression to avro file format
[ https://issues.apache.org/jira/browse/AVRO-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2162. --- Resolution: Fixed Fix Version/s: 1.9.0 Initial code merged for 1.9. Further updates/enhancements can be tracked in additional JIRA's > Add Zstandard compression to avro file format > - > > Key: AVRO-2162 > URL: https://issues.apache.org/jira/browse/AVRO-2162 > Project: Apache Avro > Issue Type: Improvement > Components: java >Reporter: Scott Carey >Priority: Major > Fix For: 1.9.0 > > > I'd like to add Zstandard compression for Avro. > At compression level 1 It is almost as fast as Snappy at compression, with > compression ratios more like gzip. At higher levels of compression, it is > more compact than gzip -9 with much lower CPU when compressing and roughly 3x > faster decompression. > > Adding it to Java is fairly easy. We'll need to say something about it in > the spec however, as an 'optinal' codec. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-1981) ReflectionTypeLoadException
[
https://issues.apache.org/jira/browse/AVRO-1981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-1981:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> ReflectionTypeLoadException
> ---
>
> Key: AVRO-1981
> URL: https://issues.apache.org/jira/browse/AVRO-1981
> Project: Apache Avro
> Issue Type: Bug
> Components: csharp
>Affects Versions: 1.8.2
>Reporter: Josh Steward
>Assignee: Brian Lachniet
>Priority: Minor
> Fix For: 1.9.0
>
>
> In the ObjectCreator the call to FindType(name, throwError) can throw a
> ReflectionTypeLoadException during the call to assembly.GetTypes() if any of
> the types are not able to load, MSDN link below for ref. This effectively
> kills the entire deserialization process. I put in a simple workaround to
> just continue if this happens. In my particular case it was trying to load
> types from a devexpress test runner dll.
> https://msdn.microsoft.com/en-us/library/system.reflection.assembly.gettypes(v=vs.110).aspx
> private Type FindType(string name,bool throwError)
> {
> //other code
> foreach (Assembly assembly in assemblies)
> {
> // Fix for Mono 3.0.10
> if (assembly.FullName.StartsWith("MonoDevelop.NUnit"))
> continue;
> /*Added catch in case assembly contains unloadable
> types*/
> try
> {
> types = assembly.GetTypes();
> }
> catch (ReflectionTypeLoadException ex)
> {
> continue;
> }
> //other code
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AVRO-2329) Avro C# hides inner exceptions
[
https://issues.apache.org/jira/browse/AVRO-2329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-2329:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Avro C# hides inner exceptions
> --
>
> Key: AVRO-2329
> URL: https://issues.apache.org/jira/browse/AVRO-2329
> Project: Apache Avro
> Issue Type: Improvement
> Components: csharp
>Affects Versions: 1.8.2
>Reporter: Brian Lachniet
>Assignee: Brian Lachniet
>Priority: Major
> Fix For: 1.9.0
>
>
> The Avro C# library hides the inner exception in many places where it throws
> {{AvroException}}. This makes it difficult to track down the root cause of
> some problems.
> Update places that throw {{AvroException}} to include the inner exception
> when available/applicable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2302) Invalid namespace importing Avro files generated from Protobuf
[
https://issues.apache.org/jira/browse/AVRO-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp resolved AVRO-2302.
---
Resolution: Fixed
Assignee: Daniel Kulp
> Invalid namespace importing Avro files generated from Protobuf
> --
>
> Key: AVRO-2302
> URL: https://issues.apache.org/jira/browse/AVRO-2302
> Project: Apache Avro
> Issue Type: Bug
>Affects Versions: 1.8.2
>Reporter: Pedro Carneiro
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 1.9.0
>
>
> An error occurs when importing Avro files into BigQuery.
> The files were generated using a ProtobufData schema.
> The error shown is similar to:
> {code}
> BigQuery error in load operation: Error processing job
> '': Error while reading data, error message: The
> Apache Avro library failed to parse the header with the following error:
> Invalid namespace:
> some.package.SomeClassProto$
> {code}
> When using a modified version of the ProtobufData class, that yields a
> namespace like
> {noformat}
> some.package.SomeClassProto
> {noformat}
> (without the dollar), the import worked without errors.
> Potentially related to AVRO-2143?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2106) Revisit dependencies on Jetty, servlet-api, and Netty
[ https://issues.apache.org/jira/browse/AVRO-2106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp resolved AVRO-2106. --- Resolution: Fixed Assignee: Daniel Kulp Fix Version/s: 1.9.0 Split netty and jetty out of avro-ipc into separate modules. If you just need netty transports, you can avoid dealing with jetty entirely. > Revisit dependencies on Jetty, servlet-api, and Netty > - > > Key: AVRO-2106 > URL: https://issues.apache.org/jira/browse/AVRO-2106 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.2, 1.8.0 >Reporter: Nandor Kollar >Assignee: Daniel Kulp >Priority: Major > Fix For: 1.9.0 > > > The compile scoped dependency on jetty servlet-api in the IPC pom file can be > problematic if using Avro in a webapp environment. Would it be possible to > make this dependency either optional or provided? Or maybe Avro modularize > into sub-modules in such a way that desired features can be assembled > piecemeal? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-2106) Revisit dependencies on Jetty, servlet-api, and Netty
[ https://issues.apache.org/jira/browse/AVRO-2106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Kulp updated AVRO-2106: -- Hadoop Flags: Incompatible change > Revisit dependencies on Jetty, servlet-api, and Netty > - > > Key: AVRO-2106 > URL: https://issues.apache.org/jira/browse/AVRO-2106 > Project: Apache Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.2, 1.8.0 >Reporter: Nandor Kollar >Priority: Major > > The compile scoped dependency on jetty servlet-api in the IPC pom file can be > problematic if using Avro in a webapp environment. Would it be possible to > make this dependency either optional or provided? Or maybe Avro modularize > into sub-modules in such a way that desired features can be assembled > piecemeal? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AVRO-1577) TestSpecificCompiler is not closing resources
[
https://issues.apache.org/jira/browse/AVRO-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Kulp updated AVRO-1577:
--
Resolution: Fixed
Assignee: Daniel Kulp
Status: Resolved (was: Patch Available)
Most of this was done already when we moved to Java 8 and switched to
try-with-resources. A few methods were missed which I just updated.
> TestSpecificCompiler is not closing resources
> -
>
> Key: AVRO-1577
> URL: https://issues.apache.org/jira/browse/AVRO-1577
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.7.7
> Environment: Windows
>Reporter: Stevo Slavic
>Assignee: Daniel Kulp
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: AVRO-1577.patch
>
>
> Test methods in {{TestSpecificCompiler}} are opening various {{Closable}}
> resources, but they are not closing them. Because of this, file deletion in
> {{tearDown}} silently fails (on platforms like Windows which are locking
> files that are being used). This causes few test methods to fail since they
> are using same temp file as output file - they generate new file content but
> only if file is not already present, and then assertions comparing actual and
> expected output content fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
