[jira] [Commented] (AVRO-2364) Use Jackson TokenBuffer in JsonDecoder

[Anthony Miyaguchi (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805608#comment-16805608
 ] 

Anthony Miyaguchi commented on AVRO-2364:
-

TokenBuffers are mentioned on the [Jackson wiki 
|https://github.com/FasterXML/jackson-docs/wiki/Presentation:-Jackson-Performance]
 for being the most efficient representation for replaying parts of the token 
stream. I think it would be interesting to see if there is a measurable change 
in decoding JSON documents. Are there benchmarks for the encoders and decoders?

> Use Jackson TokenBuffer in JsonDecoder
> --
>
> Key: AVRO-2364
> URL: https://issues.apache.org/jira/browse/AVRO-2364
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Anthony Miyaguchi
>Priority: Trivial
>
> The JsonDecoder implements a class for replaying parts of the JSON stream 
> when accessing fields out of order.
> This module can take advantage of 
> [TokenBuffers|http://static.javadoc.io/com.fasterxml.jackson.core/jackson-databind/2.9.8/com/fasterxml/jackson/databind/util/TokenBuffer.html]
>  to reduce code size. This buffer can be read back as a parser.
>  
> [https://github.com/apache/avro/blob/db992cf8686ffa681b0c592696b37ad519c33ee6/lang/java/avro/src/main/java/org/apache/avro/io/JsonDecoder.java#L514-L754]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Feature freeze Apache Avro 1.9

[Ivan Greene]

Hi Fokko, just now seeing your original email about features for 1.9. I have a 
small enhancement to ReflectData's protocol building ready for review here: 
https://github.com/apache/avro/pull/490 


I also opened a Jira earlier today about enhancements to the new ‘induce’ maven 
goal in the plugin: https://issues.apache.org/jira/browse/AVRO-2365 


I would love to address at least a few of the items listed in that Jira before 
1.9 is ultimately released, as the induce goal is an entirely new feature and 
we should make it as usable as possible out of the gate. I will try to get a PR 
opened up this weekend for those.

—Ivan

> On Mar 29, 2019, at 4:26 PM, Driesprong, Fokko  wrote:
> 
> Thanks all for the feedback.
> 
> Regarding the discussion Joda time, I've made a PR to completely remove the
> Joda dependency: https://github.com/apache/avro/pull/495
> 
> What do you guys think, could we push this for 1.9? This would obviously
> break backward compatibility, but then again, people should not use Joda
> time anymore.
> 
> Cheers, Fokko
> 
> Op vr 29 mrt. 2019 om 10:37 schreef Nandor Kollar
> :
> 
>> I totally support making JSR310 as the default instead of Joda
>> time. Should we consider removing the JSR310 from the class names, for
>> example rename Jsr310TimeConversions to TimeConversions and the existing
>> TimeConversions to JodaTimeConversions, and deprecate the latter?
>> 
>> On Thu, Mar 28, 2019 at 6:53 PM Daniel Kulp  wrote:
>> 
>>> I made few commits today that change the generated code from the Schema
>>> compiler a bit….  The changes make Avro 1.9 a bit more incompatible with
>>> 1.8, but since they are going to have to migrate anyway, I thought it
>> would
>>> be important to make the changes now rather then forcing everyone to do
>> so
>>> in 1.10.   The changes:
>>> 
>>> 1) Default for “time” classes change from Joda to JSR310.  There is a
>> flag
>>> to specify joda if they need/want it, but I think the “default” should be
>>> what we plan on supporting going forward, and I don’t think joda should
>> be
>>> it. At this point, joda should go away.  :)
>>> 
>>> 2) Private fields  - we were, by default, generating “@Deprecated public”
>>> fields.   The idea of generating deprecated code by default annoys me.
>>> Thus, I changed the default to “private”.  Again, there is a setting to
>>> make them public (or deprecated/public) if they want it, but for the
>>> default, I believe private is what we want.
>>> 
>>> While those will increase the effort to migrate from 1.8 to 1.9, I think
>>> it’s better to do that now than waiting any longer.
>>> 
>>> Thoughts?
>>> Dan
>>> 
>>> 
 On Mar 26, 2019, at 9:31 AM, Driesprong, Fokko 
>>> wrote:
 
 Hi all,
 
 I'd like to cut the branch for Apache Avro 1.9 release this Friday, and
 start moving to a release candidate so we can test. If there are any
 features that you would like to get in, please let me know.
 
 Cheers, Fokko
>>> 
>>> --
>>> Daniel Kulp
>>> dk...@apache.org  - http://dankulp.com/blog <
>>> http://dankulp.com/blog>
>>> Talend Community Coder - http://talend.com 
>>> 
>> 

Re: Feature freeze Apache Avro 1.9

[Driesprong, Fokko]

Thanks all for the feedback.

Regarding the discussion Joda time, I've made a PR to completely remove the
Joda dependency: https://github.com/apache/avro/pull/495

What do you guys think, could we push this for 1.9? This would obviously
break backward compatibility, but then again, people should not use Joda
time anymore.

Cheers, Fokko

Op vr 29 mrt. 2019 om 10:37 schreef Nandor Kollar
:

> I totally support making JSR310 as the default instead of Joda
> time. Should we consider removing the JSR310 from the class names, for
> example rename Jsr310TimeConversions to TimeConversions and the existing
> TimeConversions to JodaTimeConversions, and deprecate the latter?
>
> On Thu, Mar 28, 2019 at 6:53 PM Daniel Kulp  wrote:
>
> > I made few commits today that change the generated code from the Schema
> > compiler a bit….  The changes make Avro 1.9 a bit more incompatible with
> > 1.8, but since they are going to have to migrate anyway, I thought it
> would
> > be important to make the changes now rather then forcing everyone to do
> so
> > in 1.10.   The changes:
> >
> > 1) Default for “time” classes change from Joda to JSR310.  There is a
> flag
> > to specify joda if they need/want it, but I think the “default” should be
> > what we plan on supporting going forward, and I don’t think joda should
> be
> > it. At this point, joda should go away.  :)
> >
> > 2) Private fields  - we were, by default, generating “@Deprecated public”
> > fields.   The idea of generating deprecated code by default annoys me.
> >  Thus, I changed the default to “private”.  Again, there is a setting to
> > make them public (or deprecated/public) if they want it, but for the
> > default, I believe private is what we want.
> >
> > While those will increase the effort to migrate from 1.8 to 1.9, I think
> > it’s better to do that now than waiting any longer.
> >
> > Thoughts?
> > Dan
> >
> >
> > > On Mar 26, 2019, at 9:31 AM, Driesprong, Fokko 
> > wrote:
> > >
> > > Hi all,
> > >
> > > I'd like to cut the branch for Apache Avro 1.9 release this Friday, and
> > > start moving to a release candidate so we can test. If there are any
> > > features that you would like to get in, please let me know.
> > >
> > > Cheers, Fokko
> >
> > --
> > Daniel Kulp
> > dk...@apache.org  - http://dankulp.com/blog <
> > http://dankulp.com/blog>
> > Talend Community Coder - http://talend.com 
> >
>

[jira] [Created] (AVRO-2365) Enhancements to induce maven plugin goal

[Ivan Greene (JIRA)]

Ivan Greene created AVRO-2365:
-

 Summary: Enhancements to induce maven plugin goal
 Key: AVRO-2365
 URL: https://issues.apache.org/jira/browse/AVRO-2365
 Project: Apache Avro
  Issue Type: Improvement
  Components: java
Reporter: Ivan Greene
 Fix For: 1.9.0, 1.8.3


Before a release of 1.8.3 or 1.9.0 which will introduce the new 'induce' maven 
plugin goal (see AVRO-1749), there are a few issues with it which in my mind 
are critical to enable real-world use of the goal:
 * Allow excluding classes based on a pattern, and/or the ability to specify 
multiple sourceDirectories
 * Option to specify whether to use ReflectData.AllowNull or ReflectData
 * Throw MojoExecutionException with an informative message when appropriate, 
instead of just printing stacktraces. This is very important, we definitely 
need a build to fail when we hit a problem.
 * Use the current project defined encoding rather than hard coded UTF-8.
 * Use try-with-resources to ensure the PrintWriter gets closed in the event of 
exception

Nice to haves:
 * Specify custom ReflectData implementation (i.e user declared extensions) to 
be used
 * Specify separate output directories for schemata and protocols (these could 
both default to the single output directory if not specified individually)

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AVRO-2364) Use Jackson TokenBuffer in JsonDecoder

[Anthony Miyaguchi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anthony Miyaguchi updated AVRO-2364:

Description: 
The JsonDecoder implements a class for replaying parts of the JSON stream when 
accessing fields out of order.

This module can take advantage of 
[TokenBuffers|http://static.javadoc.io/com.fasterxml.jackson.core/jackson-databind/2.9.8/com/fasterxml/jackson/databind/util/TokenBuffer.html]
 to reduce code size. This buffer can be read back as a parser.

 

[https://github.com/apache/avro/blob/db992cf8686ffa681b0c592696b37ad519c33ee6/lang/java/avro/src/main/java/org/apache/avro/io/JsonDecoder.java#L514-L754]

 

> Use Jackson TokenBuffer in JsonDecoder
> --
>
> Key: AVRO-2364
> URL: https://issues.apache.org/jira/browse/AVRO-2364
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Anthony Miyaguchi
>Priority: Trivial
>
> The JsonDecoder implements a class for replaying parts of the JSON stream 
> when accessing fields out of order.
> This module can take advantage of 
> [TokenBuffers|http://static.javadoc.io/com.fasterxml.jackson.core/jackson-databind/2.9.8/com/fasterxml/jackson/databind/util/TokenBuffer.html]
>  to reduce code size. This buffer can be read back as a parser.
>  
> [https://github.com/apache/avro/blob/db992cf8686ffa681b0c592696b37ad519c33ee6/lang/java/avro/src/main/java/org/apache/avro/io/JsonDecoder.java#L514-L754]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AVRO-2364) Use Jackson TokenBuffer in JsonDecoder

[Anthony Miyaguchi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anthony Miyaguchi updated AVRO-2364:

Affects Version/s: 1.8.2
 Priority: Trivial  (was: Major)
  Component/s: java
   Issue Type: Improvement  (was: Bug)

> Use Jackson TokenBuffer in JsonDecoder
> --
>
> Key: AVRO-2364
> URL: https://issues.apache.org/jira/browse/AVRO-2364
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Anthony Miyaguchi
>Priority: Trivial
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AVRO-2364) Use Jackson TokenBuffer in JsonDecoder

[Anthony Miyaguchi (JIRA)]

Anthony Miyaguchi created AVRO-2364:
---

 Summary: Use Jackson TokenBuffer in JsonDecoder
 Key: AVRO-2364
 URL: https://issues.apache.org/jira/browse/AVRO-2364
 Project: Apache Avro
  Issue Type: Bug
Reporter: Anthony Miyaguchi






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (AVRO-2361) Deprecate java-mapred Module

[David Mollitor (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805174#comment-16805174
 ] 

David Mollitor edited comment on AVRO-2361 at 3/29/19 4:25 PM:
---

So, to your point, no docs on it officially on the github repo, there are still 
bug fixes, but I've talked to several folks about ideas for improving MR and 
they all tell me the same thing: do not waste our yours/our time.  All new 
workloads are on Spark.


was (Author: belugabehr):
So, to your point, no docs on it officially on the github repo, but I've talked 
to several folks about ideas for improving MR and they all tell me the same 
thing: do not waste our yours/our time.  All new workloads are on Spark.

> Deprecate java-mapred Module
> 
>
> Key: AVRO-2361
> URL: https://issues.apache.org/jira/browse/AVRO-2361
> Project: Apache Avro
>  Issue Type: Task
>  Components: java
>Affects Versions: 1.9.0
>Reporter: David Mollitor
>Priority: Major
> Fix For: 1.10.0
>
>
> MapReduce is deprecated in Hadoop 3.  Avro should do the same.  The future is 
> Spark (and a splash of Tez/Impala).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2361) Deprecate java-mapred Module

[David Mollitor (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805174#comment-16805174
 ] 

David Mollitor commented on AVRO-2361:
--

So, to your point, no docs on it officially on the github repo, but I've talked 
to several folks about ideas for improving MR and they all tell me the same 
thing: do not waste our yours/our time.  All new workloads are on Spark.

> Deprecate java-mapred Module
> 
>
> Key: AVRO-2361
> URL: https://issues.apache.org/jira/browse/AVRO-2361
> Project: Apache Avro
>  Issue Type: Task
>  Components: java
>Affects Versions: 1.9.0
>Reporter: David Mollitor
>Priority: Major
> Fix For: 1.10.0
>
>
> MapReduce is deprecated in Hadoop 3.  Avro should do the same.  The future is 
> Spark (and a splash of Tez/Impala).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2335) Remove Joda Time Library

[Nandor Kollar (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805172#comment-16805172
 ] 

Nandor Kollar commented on AVRO-2335:
-

In my opinion we should let Avro users be able to use Joda time in 1.9 and 
defer the removal to 1.10, I vote for marking Joda as deprecated and making 
JSR-310 as default for 1.9.

> Remove Joda Time Library
> 
>
> Key: AVRO-2335
> URL: https://issues.apache.org/jira/browse/AVRO-2335
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: David Mollitor
>Assignee: Fokko Driesprong
>Priority: Major
>
> Since [AVRO-2079] was completed, Avro supports Java's standard {{java.time}} 
> libraries.  Please remove the Joda Time implementations and the Joda Time 
> dependency from Avro.  Avro 2.0 perhaps?
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to 
> Java SE 8. *Users are now asked to migrate to java.time (JSR-310).*
> https://www.joda.org/joda-time/
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2335) Remove Joda Time Library

[Daniel Kulp (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805121#comment-16805121
 ] 

Daniel Kulp commented on AVRO-2335:
---

I'm not sure if this is something that we want for 1.9...   As of right now, 
for 1.9, we've made JSR-310 the default.  However, Joda is there as an option 
(and the dependency is marked optional) to aid in migrating code from 1.8 to 
1.9.   That said, we'd like to fully remove it at some point, I'm just 
questioning whether it should be for 1.9 or 1.10.

> Remove Joda Time Library
> 
>
> Key: AVRO-2335
> URL: https://issues.apache.org/jira/browse/AVRO-2335
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: David Mollitor
>Assignee: Fokko Driesprong
>Priority: Major
>
> Since [AVRO-2079] was completed, Avro supports Java's standard {{java.time}} 
> libraries.  Please remove the Joda Time implementations and the Joda Time 
> dependency from Avro.  Avro 2.0 perhaps?
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to 
> Java SE 8. *Users are now asked to migrate to java.time (JSR-310).*
> https://www.joda.org/joda-time/
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2362) Cannot Run spotless:apply From Root Pom

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804998#comment-16804998
 ] 

Hudson commented on AVRO-2362:
--

FAILURE: Integrated in Jenkins build AvroJava #631 (See 
[https://builds.apache.org/job/AvroJava/631/])
AVRO-2362: Enable calling mvn spotless:apply to auto format code (dan: 
[https://github.com/apache/avro/commit/db992cf8686ffa681b0c592696b37ad519c33ee6])
* (edit) pom.xml
* (edit) lang/java/pom.xml


> Cannot Run spotless:apply From Root Pom
> ---
>
> Key: AVRO-2362
> URL: https://issues.apache.org/jira/browse/AVRO-2362
> Project: Apache Avro
>  Issue Type: Improvement
>Affects Versions: 1.9.0
>Reporter: David Mollitor
>Assignee: Ismaël Mejía
>Priority: Major
> Fix For: 1.9.0
>
>
> I was running {{mvn install}} from the root directory and it failed because 
> some of my code was not 'spotless'.  In response, I ran {{mvn 
> spotless:apply}} but it fails to run from the root avro project directory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: AvroJava #631

[Apache Jenkins Server]

See 

Changes:

[dan] Rename JSR310 conversion class to TimeConversions, deprecate Joda

[dan] AVRO-2362: Enable calling mvn spotless:apply to auto format code

--
[...truncated 408.51 KB...]
Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-grouper/3.0.0-M3/surefire-grouper-3.0.0-M3.pom
Progress (1): 2.2/2.6 kBProgress (1): 2.6 kBDownloaded 
from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-grouper/3.0.0-M3/surefire-grouper-3.0.0-M3.pom
 (2.6 kB at 176 kB/s)
Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit47/3.0.0-M3/surefire-junit47-3.0.0-M3.jar
Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/common-junit48/3.0.0-M3/common-junit48-3.0.0-M3.jar
Downloading from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-grouper/3.0.0-M3/surefire-grouper-3.0.0-M3.jar
Progress (1): 2.2/93 kBProgress (2): 2.2/93 kB | 2.2/22 kBProgress (2): 5.0/93 
kB | 2.2/22 kBProgress (3): 5.0/93 kB | 2.2/22 kB | 2.2/40 kBProgress (3): 
5.0/93 kB | 5.0/22 kB | 2.2/40 kBProgress (3): 7.7/93 kB | 5.0/22 kB | 2.2/40 
kBProgress (3): 7.7/93 kB | 7.7/22 kB | 2.2/40 kBProgress (3): 7.7/93 kB | 
7.7/22 kB | 5.0/40 kBProgress (3): 10/93 kB | 7.7/22 kB | 5.0/40 kB Progress 
(3): 10/93 kB | 10/22 kB | 5.0/40 kB Progress (3): 13/93 kB | 10/22 kB | 5.0/40 
kBProgress (3): 13/93 kB | 10/22 kB | 7.7/40 kBProgress (3): 16/93 kB | 10/22 
kB | 7.7/40 kBProgress (3): 16/93 kB | 13/22 kB | 7.7/40 kBProgress (3): 19/93 
kB | 13/22 kB | 7.7/40 kBProgress (3): 19/93 kB | 13/22 kB | 10/40 kB Progress 
(3): 21/93 kB | 13/22 kB | 10/40 kBProgress (3): 21/93 kB | 16/22 kB | 10/40 
kBProgress (3): 24/93 kB | 16/22 kB | 10/40 kBProgress (3): 24/93 kB | 16/22 kB 
| 13/40 kBProgress (3): 27/93 kB | 16/22 kB | 13/40 kBProgress (3): 27/93 kB | 
19/22 kB | 13/40 kBProgress (3): 30/93 kB | 19/22 kB | 13/40 kBProgress (3): 
30/93 kB | 19/22 kB | 16/40 kBProgress (3): 32/93 kB | 19/22 kB | 16/40 
kBProgress (3): 32/93 kB | 21/22 kB | 16/40 kBProgress (3): 32/93 kB | 21/22 kB 
| 19/40 kBProgress (3): 32/93 kB | 22 kB | 19/40 kB   Progress (3): 32/93 kB | 
22 kB | 21/40 kBProgress (3): 32/93 kB | 22 kB | 24/40 kBProgress (3): 36/93 kB 
| 22 kB | 24/40 kBProgress (3): 36/93 kB | 22 kB | 27/40 kBProgress (3): 40/93 
kB | 22 kB | 27/40 kBProgress (3): 45/93 kB | 22 kB | 27/40 kBProgress (3): 
45/93 kB | 22 kB | 30/40 kBProgress (3): 49/93 kB | 22 kB | 30/40 kBProgress 
(3): 49/93 kB | 22 kB | 32/40 kBProgress (3): 49/93 kB | 22 kB | 35/40 
kBProgress (3): 49/93 kB | 22 kB | 38/40 kBProgress (3): 53/93 kB | 22 kB | 
38/40 kBProgress (3): 57/93 kB | 22 kB | 38/40 kBProgress (3): 57/93 kB | 22 kB 
| 40 kB   Progress (3): 61/93 kB | 22 kB | 40 kBProgress (3): 65/93 kB | 22 kB 
| 40 kBProgress (3): 69/93 kB | 22 kB | 40 kBProgress (3): 73/93 kB | 22 kB | 
40 kBProgress (3): 77/93 kB | 22 kB | 40 kBProgress (3): 81/93 kB | 22 kB | 40 
kBProgress (3): 85/93 kB | 22 kB | 40 kBProgress (3): 90/93 kB | 22 kB | 40 
kBProgress (3): 93 kB | 22 kB | 40 kB  
Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-grouper/3.0.0-M3/surefire-grouper-3.0.0-M3.jar
 (40 kB at 1.9 MB/s)
Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/common-junit48/3.0.0-M3/common-junit48-3.0.0-M3.jar
 (22 kB at 939 kB/s)
Downloaded from central: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit47/3.0.0-M3/surefire-junit47-3.0.0-M3.jar
 (93 kB at 3.6 MB/s)
[INFO] 
[INFO] ---
[INFO]  T E S T S
[INFO] ---
[INFO] Running org.apache.avro.reflect.TestReflectData
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.829 s 
- in org.apache.avro.reflect.TestReflectData
[INFO] Running org.apache.avro.TestProtocol
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in 
org.apache.avro.TestProtocol
[INFO] Running org.apache.avro.TestSchemaCompatibilityMissingEnumSymbols
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s 
- in org.apache.avro.TestSchemaCompatibilityMissingEnumSymbols
[INFO] Running org.apache.avro.TestSchemaCompatibilityTypeMismatch
[INFO] Tests run: 29, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 s 
- in org.apache.avro.TestSchemaCompatibilityTypeMismatch
[INFO] Running org.apache.avro.io.TestBlockingIO
[INFO] Tests run: 376, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.103 
s - in org.apache.avro.io.TestBlockingIO
[INFO] Running org.apache.avro.TestSchemaCompatibilityMultiple
[INFO] Tests run: 1, Failures: 0, Errors: 0, 

[jira] [Commented] (AVRO-2361) Deprecate java-mapred Module

[JIRA]

[ 
https://issues.apache.org/jira/browse/AVRO-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804995#comment-16804995
 ] 

Ismaël Mejía commented on AVRO-2361:


Wait what MapReduce es deprecated from Hadoop 3? Do you have a link on that? 
Because I had only seen some properties and API being deprecated but not the 
whole thing.

> Deprecate java-mapred Module
> 
>
> Key: AVRO-2361
> URL: https://issues.apache.org/jira/browse/AVRO-2361
> Project: Apache Avro
>  Issue Type: Task
>  Components: java
>Affects Versions: 1.9.0
>Reporter: David Mollitor
>Priority: Major
> Fix For: 1.10.0
>
>
> MapReduce is deprecated in Hadoop 3.  Avro should do the same.  The future is 
> Spark (and a splash of Tez/Impala).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AVRO-2335) Remove Joda Time Library

[Fokko Driesprong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong reassigned AVRO-2335:
--

Assignee: Fokko Driesprong

> Remove Joda Time Library
> 
>
> Key: AVRO-2335
> URL: https://issues.apache.org/jira/browse/AVRO-2335
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: David Mollitor
>Assignee: Fokko Driesprong
>Priority: Major
>
> Since [AVRO-2079] was completed, Avro supports Java's standard {{java.time}} 
> libraries.  Please remove the Joda Time implementations and the Joda Time 
> dependency from Avro.  Avro 2.0 perhaps?
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to 
> Java SE 8. *Users are now asked to migrate to java.time (JSR-310).*
> https://www.joda.org/joda-time/
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2362) Cannot Run spotless:apply From Root Pom

[JIRA]

[ 
https://issues.apache.org/jira/browse/AVRO-2362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804991#comment-16804991
 ] 

Ismaël Mejía commented on AVRO-2362:


How to contribute page updated with the shorter version of the command. So we 
should be ok now.

> Cannot Run spotless:apply From Root Pom
> ---
>
> Key: AVRO-2362
> URL: https://issues.apache.org/jira/browse/AVRO-2362
> Project: Apache Avro
>  Issue Type: Improvement
>Affects Versions: 1.9.0
>Reporter: David Mollitor
>Assignee: Ismaël Mejía
>Priority: Major
> Fix For: 1.9.0
>
>
> I was running {{mvn install}} from the root directory and it failed because 
> some of my code was not 'spotless'.  In response, I ran {{mvn 
> spotless:apply}} but it fails to run from the root avro project directory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AVRO-2362) Cannot Run spotless:apply From Root Pom

[Nandor Kollar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nandor Kollar resolved AVRO-2362.
-
   Resolution: Fixed
Fix Version/s: 1.9.0

> Cannot Run spotless:apply From Root Pom
> ---
>
> Key: AVRO-2362
> URL: https://issues.apache.org/jira/browse/AVRO-2362
> Project: Apache Avro
>  Issue Type: Improvement
>Affects Versions: 1.9.0
>Reporter: David Mollitor
>Assignee: Ismaël Mejía
>Priority: Major
> Fix For: 1.9.0
>
>
> I was running {{mvn install}} from the root directory and it failed because 
> some of my code was not 'spotless'.  In response, I ran {{mvn 
> spotless:apply}} but it fails to run from the root avro project directory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AVRO-2363) avro.schema.Schema in py3 prevents same record type from being used multiple times in same schema

[Gabriel Andersson (JIRA)]

Gabriel Andersson created AVRO-2363:
---

 Summary: avro.schema.Schema in py3 prevents same record type from 
being used multiple times in same schema
 Key: AVRO-2363
 URL: https://issues.apache.org/jira/browse/AVRO-2363
 Project: Apache Avro
  Issue Type: Bug
  Components: python
Affects Versions: 1.9.0
Reporter: Gabriel Andersson


After mostly worked in Java with avro, schemas have been used in an hierachial 
way. I.e. Schema A imports Schema B and Schema C, Schema B imports schema C. 

 

This works in a confluent setting. But when loading the schema, the application 
crashes due to reusing record types. 

The following relaxes this relation.

avro.schema.py

 
{code:java}
diff --git a/lang/py3/avro/schema.py b/lang/py3/avro/schema.py
index 7ce29731..63af3581 100644
--- a/lang/py3/avro/schema.py
+++ b/lang/py3/avro/schema.py
@@ -397,9 +397,6 @@ class Names(object):
 if schema.fullname in VALID_TYPES:
   raise SchemaParseException(
   '%s is a reserved type name.' % schema.fullname)
-if schema.fullname in self.names:
-  raise SchemaParseException(
-  'Avro name %r already exists.' % schema.fullname)
 
 logger.log(DEBUG_VERBOSE, 'Register new name for %r', schema.fullname)
 self._names[schema.fullname] = schema
{code}
 

 

Example schema:
{code:java}
{
  "doc": "",
  "fields": [
{
  "default": null,
  "doc": "Optional. The context used for tracing of flow.",
  "name": "b",
  "type": [
"null",
{
  "doc": "",
  "fields": [
{
  "doc": "",
  "name": "c",
  "type": {
"doc": "",
"fields": [
  {
"doc": "",
"name": "value",
"type":  "string"
  
  }
],
"name": "C",
"type": "record"
  }
}
  ],
  "name": "B",
  "type": "record"
}
  ]
},
{
  "doc": "",
  "name": "c",
  "type": {
"doc": "",
"fields": [
  {
"doc": "",
"name": "value",
"type":  "string"

  }
],
"name": "C",
"type": "record"
  }
}
  ],
  "name": "A",
  "namespace": "namespace",
  "type": "record"
}
{code}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2362) Cannot Run spotless:apply From Root Pom

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804950#comment-16804950
 ] 

ASF subversion and git services commented on AVRO-2362:
---

Commit db992cf8686ffa681b0c592696b37ad519c33ee6 in avro's branch 
refs/heads/master from Ismaël Mejía
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=db992cf ]

AVRO-2362: Enable calling mvn spotless:apply to auto format code


> Cannot Run spotless:apply From Root Pom
> ---
>
> Key: AVRO-2362
> URL: https://issues.apache.org/jira/browse/AVRO-2362
> Project: Apache Avro
>  Issue Type: Improvement
>Affects Versions: 1.9.0
>Reporter: David Mollitor
>Assignee: Ismaël Mejía
>Priority: Major
>
> I was running {{mvn install}} from the root directory and it failed because 
> some of my code was not 'spotless'.  In response, I ran {{mvn 
> spotless:apply}} but it fails to run from the root avro project directory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2360) Java 8 timestamp-millis / timestamp-micros type inconsistency

[Fokko Driesprong (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804921#comment-16804921
 ] 

Fokko Driesprong commented on AVRO-2360:


Thanks for giving 1.9-SNAPSHOT a try. We need to get this fixed before we can 
ship it.

> Java 8 timestamp-millis / timestamp-micros type inconsistency
> -
>
> Key: AVRO-2360
> URL: https://issues.apache.org/jira/browse/AVRO-2360
> Project: Apache Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0
> Environment: JDK 11.0.2 on Linux
>Reporter: Raman Gupta
>Assignee: Daniel Kulp
>Priority: Blocker
> Fix For: 1.9.0
>
>
> I am trying the new JSR 310 date/time types with a snapshot of 1.9.0.
> I see what seems to be an inconsistency. If I generate my code with a logical 
> type of "timestamp-millis", then the code is generated with `Instant`, as 
> expected. However, on JDK11 on Linux, if I do `Instant.now()` the Instant 
> value created contains microseconds. When setting this Instant on an instance 
> of the generated Avro SpecificRecord, I am unable to round-trip the data:
> Before serialization to bytes:
> System.out.println(mySpecificRecord.tsMillis()) // 2019-03-27T23:16:09.665308Z
> After serializing to bytes, and then back into a specific record the 
> microseconds are now truncated:
> System.out.println(mySpecificRecord.tsMillis()) // 2019-03-27T23:16:09.665Z
> I believe the compiler should generate a class that truncates the 
> microseconds at `setter` time for "timestamp-millis", so that the value 
> before serialization, and after deserialization, are the same. This can be 
> done with a call to the method `truncatedTo(ChronoUnit.MILLIS)`.
> Another, possibly related, oddity is that the "timestamp-micros" type 
> generates a class with `long` as the type of the field. Since Instant can 
> support both milli and micro-second precision, I don't see the reason for 
> this behavior.
> Followup on AVRO-2079.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AVRO-1886) Improve error reporting for schema validation

[Tim Perkins (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Perkins updated AVRO-1886:
--
   Resolution: Fixed
Fix Version/s: (was: 1.10.0)
   1.9.0
   Status: Resolved  (was: Patch Available)

> Improve error reporting for schema validation
> -
>
> Key: AVRO-1886
> URL: https://issues.apache.org/jira/browse/AVRO-1886
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: ruby
>Reporter: Miroslav Csonka
>Assignee: Miroslav Csonka
>Priority: Major
> Fix For: 1.9.0
>
>
> In Funding Circle we use avro schema to validate some financial data and 
> we've found ourselves wondering what is invalid. We've noticed that avro 
> [does already 
> validation|https://github.com/apache/avro/blob/master/lang/ruby/lib/avro/schema.rb#L94],
>  but does not report what fields are invalid, so we've decided to change it 
> and we use Avro as a validation library.
> h5. What error messages to expect?
> We communicate errors for a given payload in format "at   
> ".
> For  we've decided to use syntax similar to json path:
> * . refers to root
> * \[0\] first index in array
> * .age would match 42 from {"age": 42}
> Error within object *person* and his optional array *houses* on *second item* 
> with key *number_of_rooms*:
> {quote}
> at .person.houses\[1\].number_of_rooms expected type long, got string 
> with value "not valid at all"
> {quote}
> Example of multiple errors for a single payload:
> {quote}
> at .\[0\] expected type int, got null
> at .\[1\] expected type int, got string with value \"e\"
> {quote}
> More examples can be seen at 
> https://github.com/apache/avro/pull/111/files#diff-830ac80276854fceefa3f92c346d165f



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Automatic formatting of Java code

[Driesprong, Fokko]

Thanks Ismaël for taking the effort to get this in. This will add some
additional work now, but it will make it easier in the long run because we
will have less merge conflicts.

Op di 26 mrt. 2019 om 15:57 schreef Ismaël Mejía :

> Hello,
>
> As part of the changes for the upcoming 1.9.0 version. We introduced
> automatic code formatting for the Java codebase by relying on a maven
> plugin called Spotless that we can use to enforce and auto format new
> contributions by relying in an Eclipse Java code formatter [2].
>
> Contributors can configure the formatter in their favorite IDE, but
> the easiest way is to rely in the maven plugin from the command line
> tool. You can find more details in how to use it to prepare your
> PRs/patches in the "How to contribution" webpage [3].
>
> As a consequence of this change being merged today various PRs need to
> be updated / rebased. So if you have something pending or if you are
> working on a new contribution please pay attention to this.
>
> Sorry for the inconvenience and hope you find this new approch to code
> formatting useful.
>
> [1] https://issues.apache.org/jira/browse/AVRO-2353
> [2]
> https://github.com/apache/avro/blob/master/lang/java/eclipse-java-formatter.xml
> [3] https://cwiki.apache.org/confluence/display/AVRO/How+To+Contribute
>

Re: Feature freeze Apache Avro 1.9

[Nandor Kollar]

I totally support making JSR310 as the default instead of Joda
time. Should we consider removing the JSR310 from the class names, for
example rename Jsr310TimeConversions to TimeConversions and the existing
TimeConversions to JodaTimeConversions, and deprecate the latter?

On Thu, Mar 28, 2019 at 6:53 PM Daniel Kulp  wrote:

> I made few commits today that change the generated code from the Schema
> compiler a bit….  The changes make Avro 1.9 a bit more incompatible with
> 1.8, but since they are going to have to migrate anyway, I thought it would
> be important to make the changes now rather then forcing everyone to do so
> in 1.10.   The changes:
>
> 1) Default for “time” classes change from Joda to JSR310.  There is a flag
> to specify joda if they need/want it, but I think the “default” should be
> what we plan on supporting going forward, and I don’t think joda should be
> it. At this point, joda should go away.  :)
>
> 2) Private fields  - we were, by default, generating “@Deprecated public”
> fields.   The idea of generating deprecated code by default annoys me.
>  Thus, I changed the default to “private”.  Again, there is a setting to
> make them public (or deprecated/public) if they want it, but for the
> default, I believe private is what we want.
>
> While those will increase the effort to migrate from 1.8 to 1.9, I think
> it’s better to do that now than waiting any longer.
>
> Thoughts?
> Dan
>
>
> > On Mar 26, 2019, at 9:31 AM, Driesprong, Fokko 
> wrote:
> >
> > Hi all,
> >
> > I'd like to cut the branch for Apache Avro 1.9 release this Friday, and
> > start moving to a release candidate so we can test. If there are any
> > features that you would like to get in, please let me know.
> >
> > Cheers, Fokko
>
> --
> Daniel Kulp
> dk...@apache.org  - http://dankulp.com/blog <
> http://dankulp.com/blog>
> Talend Community Coder - http://talend.com 
>

[jira] [Updated] (AVRO-2359) Support Logical Types in C#

[Tim Roberts (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Roberts updated AVRO-2359:
--
Affects Version/s: 1.8.2

> Support Logical Types in C#
> ---
>
> Key: AVRO-2359
> URL: https://issues.apache.org/jira/browse/AVRO-2359
> Project: Apache Avro
>  Issue Type: New Feature
>  Components: csharp
>Affects Versions: 1.8.2
>Reporter: Tim Roberts
>Priority: Critical
>
> By not supporting Logical Types, the C# Avro implementation is severely 
> limited in terms of its interoperability in heterogeneous environments. While 
> the implementation will safely ignore logical type declarations in processed 
> schemas; at runtime the semantics of a "date" for example, will be lost when 
> receiving an Avro payload that was encoded by the Java platform using the 
> same schema. The C# implementation will never be able to retrieve the date 
> value from the encoded int.
> I propose that the C# Avro implementation be extended to support the Logical 
> Types as defined by the current specification. I have also explored the 
> lang/csharp codebase and believe that I could produce a PR to support this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)