[jira] [Commented] (AVRO-1783) Gracefully handle strings with wrong character encoding

[Ryan Blue (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15095272#comment-15095272
 ] 

Ryan Blue commented on AVRO-1783:
-

The patch looks good to me. I attempted to test it, but everything works in 
jruby 1.7.6 without the patch and with 1.7.3 I get a NoSuchMethodException 
before I can hit the problem. Can someone that has reproduced the bug verify 
that the patch fixes it? Then I'll commit this.

> Gracefully handle strings with wrong character encoding
> ---
>
> Key: AVRO-1783
> URL: https://issues.apache.org/jira/browse/AVRO-1783
> Project: Avro
>  Issue Type: Bug
>  Components: ruby
>Affects Versions: 1.7.7
>Reporter: Martin Kleppmann
> Attachments: AVRO-1783.patch
>
>
> In the [vote thread for Avro 
> 1.8.0-rc2|http://mail-archives.apache.org/mod_mbox/avro-dev/201601.mbox/%3CCAGHyZ6K-oe35%2BOYROK6MSwrHxfPHvjmqhJAfRJL2dzexYw6YSw%40mail.gmail.com%3E],
>  [~busbey] noticed that [phunt's 
> avro-rpc-quickstart|https://github.com/phunt/avro-rpc-quickstart] fails:
> {code}
> busbey$ ruby sample_ipc_client.rb avro_user pat Hello_World
> Avro::IO::AvroTypeError: The datum
> "\x89\xA9\xD1\xFF@NUm\xEA\x9A\xFB\xDAx\xF5Zq"
> is not an example of schema
> {"type":"fixed","name":"MD5","namespace":"org.apache.avro.ipc","size":16}
>   write_data at
> /Users/busbey/.rvm/gems/jruby-1.7.3/gems/avro-1.8.0/lib/avro/io.rb:543
> write_record at
> /Users/busbey/.rvm/gems/jruby-1.7.3/gems/avro-1.8.0/lib/avro/io.rb:610
> each at org/jruby/RubyArray.java:1613
> write_record at
> /Users/busbey/.rvm/gems/jruby-1.7.3/gems/avro-1.8.0/lib/avro/io.rb:609
>   write_data at
> /Users/busbey/.rvm/gems/jruby-1.7.3/gems/avro-1.8.0/lib/avro/io.rb:561
>write at
> /Users/busbey/.rvm/gems/jruby-1.7.3/gems/avro-1.8.0/lib/avro/io.rb:538
>  write_handshake_request at
> /Users/busbey/.rvm/gems/jruby-1.7.3/gems/avro-1.8.0/lib/avro/ipc.rb:136
>  request at
> /Users/busbey/.rvm/gems/jruby-1.7.3/gems/avro-1.8.0/lib/avro/ipc.rb:105
>  request at
> /Users/busbey/.rvm/gems/jruby-1.7.3/gems/avro-1.8.0/lib/avro/ipc.rb:117
>   (root) at sample_ipc_client.rb:49
> {code}
> I tried reproducing the error, and it is quite strange. avro-rpc-quickstart 
> works fine for me in Ruby (MRI) 2.2 and 2.1, and in JRuby 1.7.23. However, 
> [~busbey] was using JRuby 1.7.3 (as visible from the path names above), and 
> in this particular version of JRuby I was able to reproduce the issue.
> It seems that in some circumstances (but not always, bizarrely), JRuby 1.7.3 
> returns a UTF-8 encoded string from {{Digest::MD5.digest}}, rather than a 
> binary-encoded string. {{Schema.validate}} checks that the string is suitable 
> for writing as datum for a {{fixed}} type by calling {{#size}}. In this case, 
> although the MD5 digest of the schema is a 16-byte string, if you interpret 
> it as a UTF-8 encoded string, it consists of only 13 characters (i.e. some 
> sequences are interpreted as multibyte characters).
> Rather than trying to divine why JRuby is being weird here, I think this is 
> an opportunity to fix Avro's handling of strings to make it robust against 
> unexpected encodings.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1773) Infinite loop caused by race condition

[vincent ye (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15094564#comment-15094564
 ] 

vincent ye commented on AVRO-1773:
--

Thank you for taking time to take care of it.

> Infinite loop caused by race condition
> --
>
> Key: AVRO-1773
> URL: https://issues.apache.org/jira/browse/AVRO-1773
> Project: Avro
>  Issue Type: Sub-task
>  Components: java
>Affects Versions: 1.7.7
>Reporter: vincent ye
>Priority: Critical
> Fix For: 1.7.8, 1.8.0
>
>
> org.apache.avro.LogicalTypes#fromSchemaIgnoreInvalid lookups and update  
> CACHE. CACHE is backed by hashmap without synchronization. In a multithreaded 
> environment, it causes a infinite loop in hashmap lookup. The race condition 
> problem is described in the following blog 
> http://mailinator.blogspot.com/2009/06/beautiful-race-condition.html.
> I experience this infinite loop in Spark jobs with high concurrency.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1773) Infinite loop caused by race condition

[Ryan Blue (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15094547#comment-15094547
 ] 

Ryan Blue commented on AVRO-1773:
-

I'm marking this a duplicate of AVRO-1781 since we've been having more 
discussion over there (even though this was filed first).

> Infinite loop caused by race condition
> --
>
> Key: AVRO-1773
> URL: https://issues.apache.org/jira/browse/AVRO-1773
> Project: Avro
>  Issue Type: Sub-task
>  Components: java
>Affects Versions: 1.7.7
>Reporter: vincent ye
>Priority: Critical
> Fix For: 1.7.8, 1.8.0
>
>
> org.apache.avro.LogicalTypes#fromSchemaIgnoreInvalid lookups and update  
> CACHE. CACHE is backed by hashmap without synchronization. In a multithreaded 
> environment, it causes a infinite loop in hashmap lookup. The race condition 
> problem is described in the following blog 
> http://mailinator.blogspot.com/2009/06/beautiful-race-condition.html.
> I experience this infinite loop in Spark jobs with high concurrency.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1781) Schema.parse is not thread safe

[Ryan Blue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryan Blue updated AVRO-1781:

Attachment: AVRO-1781.2.patch

> Schema.parse is not thread safe
> ---
>
> Key: AVRO-1781
> URL: https://issues.apache.org/jira/browse/AVRO-1781
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Sean Busbey
>Priority: Blocker
> Fix For: 1.8.0
>
> Attachments: AVRO-1781.1.patch, AVRO-1781.2.patch
>
>
> Post AVRO-1497, Schema.parse calls {{LogicalTypes.fromSchemaIgnoreInvalid}} 
> on any schema that is expressed as a JSON object (anything except bare 
> primitives).
> That static method relies on a static cache based on WeakIdentityHashMap 
> (WIHM).
> WIHM clearly states that it isn't threadsafe 
> [ref|https://github.com/apache/avro/blob/branch-1.8/lang/java/avro/src/main/java/org/apache/avro/util/WeakIdentityHashMap.java#L42]
> {code}
>  * 
>  * Note that this implementation is not synchronized.
>  * 
>  */
> public class WeakIdentityHashMap implements Map {
> {code}
> All of the Schema.Parser instances use that same static Schema.parse method.
> The end result is that as-is it's only safe to have a single thread parsing 
> schemas in a given JVM.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1781) Schema.parse is not thread safe

[Ryan Blue (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15094503#comment-15094503
 ] 

Ryan Blue commented on AVRO-1781:
-

The problem is that Guava's MapMaker maps don't allow Null. I've updated the 
patch to use Optional and tests are passing.

I also ran into trouble with the mapred module because the Guava dependency, 
despite being shaded, was overriding Hadoop's guava dependency. I've added a 
ban for all versions of Guava other than 11.0.2 and for avro-guava-dependencies 
(to make sure it doesn't leak Guava classes into the classpath). Unfortunately, 
the only way to avoid hitting the Guava ban is to use version 11.0.2. If I use 
19.0 and build/test from the mapred directory it correctly uses 11.0.2, but if 
I build from the lang/java directory the dependencies are all resolved at once 
and 19.0 overrides the transitive dependency's version. Luckily, everything 
works with 11.0.2 and the jar is even a little smaller.

I also had to update this to exclude Google's JSR301 jar, which might be GPL 
and is banned by avro-tools. I'm attaching a new patch.

> Schema.parse is not thread safe
> ---
>
> Key: AVRO-1781
> URL: https://issues.apache.org/jira/browse/AVRO-1781
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Sean Busbey
>Priority: Blocker
> Fix For: 1.8.0
>
> Attachments: AVRO-1781.1.patch
>
>
> Post AVRO-1497, Schema.parse calls {{LogicalTypes.fromSchemaIgnoreInvalid}} 
> on any schema that is expressed as a JSON object (anything except bare 
> primitives).
> That static method relies on a static cache based on WeakIdentityHashMap 
> (WIHM).
> WIHM clearly states that it isn't threadsafe 
> [ref|https://github.com/apache/avro/blob/branch-1.8/lang/java/avro/src/main/java/org/apache/avro/util/WeakIdentityHashMap.java#L42]
> {code}
>  * 
>  * Note that this implementation is not synchronized.
>  * 
>  */
> public class WeakIdentityHashMap implements Map {
> {code}
> All of the Schema.Parser instances use that same static Schema.parse method.
> The end result is that as-is it's only safe to have a single thread parsing 
> schemas in a given JVM.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1728) Update LICENSE and NOTICE files included in Java binaries

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15094357#comment-15094357
 ] 

Hudson commented on AVRO-1728:
--

SUCCESS: Integrated in AvroJava #567 (See 
[https://builds.apache.org/job/AvroJava/567/])
AVRO-1780: Java: Fix NPE in tools. Contributed by Tom White.

Changes for AVRO-1728 moved NOTICE, which was used to create the
no-argument output for Java's avro-tools. Using the new location fixed
the NPE and this also includes an update to show just the top of NOTICE
since it is now much longer. (blue: rev 1724287)
* trunk/lang/java/tools/src/main/java/org/apache/avro/tool/Main.java


> Update LICENSE and NOTICE files included in Java binaries
> -
>
> Key: AVRO-1728
> URL: https://issues.apache.org/jira/browse/AVRO-1728
> Project: Avro
>  Issue Type: Sub-task
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Ryan Blue
>Assignee: Ryan Blue
> Fix For: 1.8.0
>
> Attachments: AVRO-1728.1.patch, AVRO-1728.2.patch, AVRO-1728.3.patch, 
> AVRO-1728.4.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1780) Avro tools jar fails with NPE

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15094358#comment-15094358
 ] 

Hudson commented on AVRO-1780:
--

SUCCESS: Integrated in AvroJava #567 (See 
[https://builds.apache.org/job/AvroJava/567/])
AVRO-1780: Java: Fix NPE in tools. Contributed by Tom White.

Changes for AVRO-1728 moved NOTICE, which was used to create the
no-argument output for Java's avro-tools. Using the new location fixed
the NPE and this also includes an update to show just the top of NOTICE
since it is now much longer. (blue: rev 1724287)
* trunk/lang/java/tools/src/main/java/org/apache/avro/tool/Main.java


> Avro tools jar fails with NPE
> -
>
> Key: AVRO-1780
> URL: https://issues.apache.org/jira/browse/AVRO-1780
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Sean Busbey
>Assignee: Tom White
>Priority: Blocker
> Fix For: 1.8.0
>
> Attachments: AVRO-1780.patch
>
>
> following our license/notice updates, teh avro-tools jar fails with a NPE 
> because it wants to print out a NOTICE.txt in the root of the jar.
> {code}
> busbey$ java -jar avro-tools-1.8.0.jar
> Version 1.8.0 of Exception in thread "main" java.lang.NullPointerException
>   at org.apache.avro.tool.Main.printStream(Main.java:105)
>   at org.apache.avro.tool.Main.run(Main.java:92)
>   at org.apache.avro.tool.Main.main(Main.java:74)
> busbey$ java -jar avro-tools-1.8.0.jar --help
> Version 1.8.0 of Exception in thread "main" java.lang.NullPointerException
>   at org.apache.avro.tool.Main.printStream(Main.java:105)
>   at org.apache.avro.tool.Main.run(Main.java:92)
>   at org.apache.avro.tool.Main.main(Main.java:74)
> {code}
> We should probably not print the entire NOTICE unless a cli arg is given, 
> since it is much bigger now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: AVRO-python #588

[Apache Jenkins Server]

See 

Changes:

[blue] AVRO-1780: Java: Fix NPE in tools. Contributed by Tom White.

Changes for AVRO-1728 moved NOTICE, which was used to create the
no-argument output for Java's avro-tools. Using the new location fixed
the NPE and this also includes an update to show just the top of NOTICE
since it is now much longer.

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-6 (docker Ubuntu ubuntu) in workspace 

Reverting  to depth 
infinity with ignoreExternals: false
Updating https://svn.apache.org/repos/asf/avro/trunk at revision 
'2016-01-12T17:29:15.772 +'
U lang/java/tools/src/main/java/org/apache/avro/tool/Main.java
At revision 1724289
[py] $ /home/jenkins/tools/ant/latest/bin/ant -file build.xml test
Buildfile: 


init:

ivy-download:

build:
 [copy] Copying 1 file to 

 [copy] Copying 1 file to 

 [copy] Copying 1 file to 

 [copy] Copying 1 file to 

[ivy:retrieve] :: Ivy 2.2.0 - 20100923230623 :: http://ant.apache.org/ivy/ ::
[ivy:retrieve] :: loading settings :: file = 

[ivy:retrieve] :: resolving dependencies :: 
org.apache.avro#python;work...@priapus.apache.org
[ivy:retrieve]  confs: [default]
[ivy:retrieve] :: resolution report :: resolve 1199ms :: artifacts dl 1ms
-
|  |modules||   artifacts   |
|   conf   | number| search|dwnlded|evicted|| number|dwnlded|
-
|  default |   1   |   0   |   0   |   0   ||   0   |   0   |
-
[ivy:retrieve] 
[ivy:retrieve] :: problems summary ::
[ivy:retrieve]  WARNINGS
[ivy:retrieve]  module not found: 
org.apache.avro#avro-tools;1.9.0-SNAPSHOT
[ivy:retrieve]   central: tried
[ivy:retrieve]
http://repo1.maven.org/maven2/org/apache/avro/avro-tools/1.9.0-SNAPSHOT/avro-tools-1.9.0-SNAPSHOT.pom
[ivy:retrieve]-- artifact 
org.apache.avro#avro-tools;1.9.0-SNAPSHOT!avro-tools.jar:
[ivy:retrieve]
http://repo1.maven.org/maven2/org/apache/avro/avro-tools/1.9.0-SNAPSHOT/avro-tools-1.9.0-SNAPSHOT.jar
[ivy:retrieve]   apache-snapshots: tried
[ivy:retrieve]
https://repository.apache.org/content/groups/snapshots/org/apache/avro/avro-tools/1.9.0-SNAPSHOT/avro-tools-1.9.0-SNAPSHOT.pom
[ivy:retrieve]-- artifact 
org.apache.avro#avro-tools;1.9.0-SNAPSHOT!avro-tools.jar:
[ivy:retrieve]
https://repository.apache.org/content/groups/snapshots/org/apache/avro/avro-tools/1.9.0-SNAPSHOT/avro-tools-1.9.0-SNAPSHOT.jar
[ivy:retrieve]   local-maven2: tried
[ivy:retrieve]
/home/jenkins/.m2/repository/org/apache/avro/avro-tools/1.9.0-SNAPSHOT/avro-tools-1.9.0-SNAPSHOT.xml
[ivy:retrieve]-- artifact 
org.apache.avro#avro-tools;1.9.0-SNAPSHOT!avro-tools.jar:
[ivy:retrieve]
/home/jenkins/.m2/repository/org/apache/avro/avro-tools/1.9.0-SNAPSHOT/avro-tools-1.9.0-SNAPSHOT.jar
[ivy:retrieve]  ::
[ivy:retrieve]  ::  UNRESOLVED DEPENDENCIES ::
[ivy:retrieve]  ::
[ivy:retrieve]  :: org.apache.avro#avro-tools;1.9.0-SNAPSHOT: not found
[ivy:retrieve]  ::
[ivy:retrieve]  ERRORS
[ivy:retrieve]  Server access Error: java.lang.RuntimeException: Could not 
generate DH keypair 
url=https://repository.apache.org/content/groups/snapshots/org/apache/avro/avro-tools/1.9.0-SNAPSHOT/maven-metadata.xml
[ivy:retrieve]  Server access Error: java.lang.RuntimeException: Could not 
generate DH keypair 
url=https://repository.apache.org/content/groups/snapshots/org/apache/avro/avro-tools/1.9.0-SNAPSHOT/avro-tools-1.9.0-SNAPSHOT.pom
[ivy:retrieve]  Server access Error: java.lang.RuntimeException: Could not 
generate DH keypair 
url=https://repository.apache.org/content/groups/snapshots/org/apache/avro/avro-tools/1.9.0-SNAPSHOT/avro-tools-1.9.0-SNAPSHOT.jar
[ivy:retrieve] 
[ivy:retrieve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS

BUILD FAILED
:146: 
impossible to resolve dependencies:
resolve 

[jira] [Commented] (AVRO-1780) Avro tools jar fails with NPE

[Ryan Blue (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15094309#comment-15094309
 ] 

Ryan Blue commented on AVRO-1780:
-

Thanks for fixing this, Tom! I committed your patch.

> Avro tools jar fails with NPE
> -
>
> Key: AVRO-1780
> URL: https://issues.apache.org/jira/browse/AVRO-1780
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Sean Busbey
>Assignee: Tom White
>Priority: Blocker
> Fix For: 1.8.0
>
> Attachments: AVRO-1780.patch
>
>
> following our license/notice updates, teh avro-tools jar fails with a NPE 
> because it wants to print out a NOTICE.txt in the root of the jar.
> {code}
> busbey$ java -jar avro-tools-1.8.0.jar
> Version 1.8.0 of Exception in thread "main" java.lang.NullPointerException
>   at org.apache.avro.tool.Main.printStream(Main.java:105)
>   at org.apache.avro.tool.Main.run(Main.java:92)
>   at org.apache.avro.tool.Main.main(Main.java:74)
> busbey$ java -jar avro-tools-1.8.0.jar --help
> Version 1.8.0 of Exception in thread "main" java.lang.NullPointerException
>   at org.apache.avro.tool.Main.printStream(Main.java:105)
>   at org.apache.avro.tool.Main.run(Main.java:92)
>   at org.apache.avro.tool.Main.main(Main.java:74)
> {code}
> We should probably not print the entire NOTICE unless a cli arg is given, 
> since it is much bigger now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (AVRO-1780) Avro tools jar fails with NPE

[Ryan Blue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryan Blue resolved AVRO-1780.
-
Resolution: Fixed

> Avro tools jar fails with NPE
> -
>
> Key: AVRO-1780
> URL: https://issues.apache.org/jira/browse/AVRO-1780
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Sean Busbey
>Assignee: Tom White
>Priority: Blocker
> Fix For: 1.8.0
>
> Attachments: AVRO-1780.patch
>
>
> following our license/notice updates, teh avro-tools jar fails with a NPE 
> because it wants to print out a NOTICE.txt in the root of the jar.
> {code}
> busbey$ java -jar avro-tools-1.8.0.jar
> Version 1.8.0 of Exception in thread "main" java.lang.NullPointerException
>   at org.apache.avro.tool.Main.printStream(Main.java:105)
>   at org.apache.avro.tool.Main.run(Main.java:92)
>   at org.apache.avro.tool.Main.main(Main.java:74)
> busbey$ java -jar avro-tools-1.8.0.jar --help
> Version 1.8.0 of Exception in thread "main" java.lang.NullPointerException
>   at org.apache.avro.tool.Main.printStream(Main.java:105)
>   at org.apache.avro.tool.Main.run(Main.java:92)
>   at org.apache.avro.tool.Main.main(Main.java:74)
> {code}
> We should probably not print the entire NOTICE unless a cli arg is given, 
> since it is much bigger now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1780) Avro tools jar fails with NPE

[Ryan Blue (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryan Blue updated AVRO-1780:

Assignee: Tom White

> Avro tools jar fails with NPE
> -
>
> Key: AVRO-1780
> URL: https://issues.apache.org/jira/browse/AVRO-1780
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Sean Busbey
>Assignee: Tom White
>Priority: Blocker
> Fix For: 1.8.0
>
> Attachments: AVRO-1780.patch
>
>
> following our license/notice updates, teh avro-tools jar fails with a NPE 
> because it wants to print out a NOTICE.txt in the root of the jar.
> {code}
> busbey$ java -jar avro-tools-1.8.0.jar
> Version 1.8.0 of Exception in thread "main" java.lang.NullPointerException
>   at org.apache.avro.tool.Main.printStream(Main.java:105)
>   at org.apache.avro.tool.Main.run(Main.java:92)
>   at org.apache.avro.tool.Main.main(Main.java:74)
> busbey$ java -jar avro-tools-1.8.0.jar --help
> Version 1.8.0 of Exception in thread "main" java.lang.NullPointerException
>   at org.apache.avro.tool.Main.printStream(Main.java:105)
>   at org.apache.avro.tool.Main.run(Main.java:92)
>   at org.apache.avro.tool.Main.main(Main.java:74)
> {code}
> We should probably not print the entire NOTICE unless a cli arg is given, 
> since it is much bigger now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1780) Avro tools jar fails with NPE

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15094307#comment-15094307
 ] 

ASF subversion and git services commented on AVRO-1780:
---

Commit 1724287 from [~b...@cloudera.com] in branch 'avro/trunk'
[ https://svn.apache.org/r1724287 ]

AVRO-1780: Java: Fix NPE in tools. Contributed by Tom White.

Changes for AVRO-1728 moved NOTICE, which was used to create the
no-argument output for Java's avro-tools. Using the new location fixed
the NPE and this also includes an update to show just the top of NOTICE
since it is now much longer.

> Avro tools jar fails with NPE
> -
>
> Key: AVRO-1780
> URL: https://issues.apache.org/jira/browse/AVRO-1780
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Sean Busbey
>Priority: Blocker
> Fix For: 1.8.0
>
> Attachments: AVRO-1780.patch
>
>
> following our license/notice updates, teh avro-tools jar fails with a NPE 
> because it wants to print out a NOTICE.txt in the root of the jar.
> {code}
> busbey$ java -jar avro-tools-1.8.0.jar
> Version 1.8.0 of Exception in thread "main" java.lang.NullPointerException
>   at org.apache.avro.tool.Main.printStream(Main.java:105)
>   at org.apache.avro.tool.Main.run(Main.java:92)
>   at org.apache.avro.tool.Main.main(Main.java:74)
> busbey$ java -jar avro-tools-1.8.0.jar --help
> Version 1.8.0 of Exception in thread "main" java.lang.NullPointerException
>   at org.apache.avro.tool.Main.printStream(Main.java:105)
>   at org.apache.avro.tool.Main.run(Main.java:92)
>   at org.apache.avro.tool.Main.main(Main.java:74)
> {code}
> We should probably not print the entire NOTICE unless a cli arg is given, 
> since it is much bigger now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1728) Update LICENSE and NOTICE files included in Java binaries

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15094308#comment-15094308
 ] 

ASF subversion and git services commented on AVRO-1728:
---

Commit 1724287 from [~b...@cloudera.com] in branch 'avro/trunk'
[ https://svn.apache.org/r1724287 ]

AVRO-1780: Java: Fix NPE in tools. Contributed by Tom White.

Changes for AVRO-1728 moved NOTICE, which was used to create the
no-argument output for Java's avro-tools. Using the new location fixed
the NPE and this also includes an update to show just the top of NOTICE
since it is now much longer.

> Update LICENSE and NOTICE files included in Java binaries
> -
>
> Key: AVRO-1728
> URL: https://issues.apache.org/jira/browse/AVRO-1728
> Project: Avro
>  Issue Type: Sub-task
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Ryan Blue
>Assignee: Ryan Blue
> Fix For: 1.8.0
>
> Attachments: AVRO-1728.1.patch, AVRO-1728.2.patch, AVRO-1728.3.patch, 
> AVRO-1728.4.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (AVRO-1780) Avro tools jar fails with NPE

[Tom White (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-1780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom White updated AVRO-1780:

Attachment: AVRO-1780.patch

Here's a fix that prints the top part of the notice file.

> Avro tools jar fails with NPE
> -
>
> Key: AVRO-1780
> URL: https://issues.apache.org/jira/browse/AVRO-1780
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Sean Busbey
>Priority: Blocker
> Fix For: 1.8.0
>
> Attachments: AVRO-1780.patch
>
>
> following our license/notice updates, teh avro-tools jar fails with a NPE 
> because it wants to print out a NOTICE.txt in the root of the jar.
> {code}
> busbey$ java -jar avro-tools-1.8.0.jar
> Version 1.8.0 of Exception in thread "main" java.lang.NullPointerException
>   at org.apache.avro.tool.Main.printStream(Main.java:105)
>   at org.apache.avro.tool.Main.run(Main.java:92)
>   at org.apache.avro.tool.Main.main(Main.java:74)
> busbey$ java -jar avro-tools-1.8.0.jar --help
> Version 1.8.0 of Exception in thread "main" java.lang.NullPointerException
>   at org.apache.avro.tool.Main.printStream(Main.java:105)
>   at org.apache.avro.tool.Main.run(Main.java:92)
>   at org.apache.avro.tool.Main.main(Main.java:74)
> {code}
> We should probably not print the entire NOTICE unless a cli arg is given, 
> since it is much bigger now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1779) Avro docs convenience artifact missing LICENSE/NOTICE

[Tom White (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093774#comment-15093774
 ] 

Tom White commented on AVRO-1779:
-

+1. I ran a docker build and confirmed that the doc artifact contained LICENSE 
and NOTICE.

> Avro docs convenience artifact missing LICENSE/NOTICE
> -
>
> Key: AVRO-1779
> URL: https://issues.apache.org/jira/browse/AVRO-1779
> Project: Avro
>  Issue Type: Bug
>  Components: doc
>Affects Versions: 1.8.0
>Reporter: Sean Busbey
>Assignee: Ryan Blue
>Priority: Blocker
> Fix For: 1.8.0
>
> Attachments: AVRO-1779.1.diff
>
>
> for releases we generate a convenience artifact with our docs. at present, 
> this tarball is missing our needed LICENSE/NOTICE files



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1779) Avro docs convenience artifact missing LICENSE/NOTICE

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093770#comment-15093770
 ] 

Hudson commented on AVRO-1779:
--

SUCCESS: Integrated in AvroJava #566 (See 
[https://builds.apache.org/job/AvroJava/566/])
AVRO-1779. Avro docs convenience artifact missing LICENSE/NOTICE. Contributed 
by blue. (tomwhite: rev 1724203)
* trunk/CHANGES.txt
* trunk/build.sh
* trunk/lang/c++/Doxyfile


> Avro docs convenience artifact missing LICENSE/NOTICE
> -
>
> Key: AVRO-1779
> URL: https://issues.apache.org/jira/browse/AVRO-1779
> Project: Avro
>  Issue Type: Bug
>  Components: doc
>Affects Versions: 1.8.0
>Reporter: Sean Busbey
>Assignee: Ryan Blue
>Priority: Blocker
> Fix For: 1.8.0
>
> Attachments: AVRO-1779.1.diff
>
>
> for releases we generate a convenience artifact with our docs. at present, 
> this tarball is missing our needed LICENSE/NOTICE files



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: AVRO-python #587

[Apache Jenkins Server]

See 

Changes:

[tomwhite] AVRO-1779. Avro docs convenience artifact missing LICENSE/NOTICE. 
Contributed by blue.

--
[...truncated 2025 lines...]
A lang/c/jansson/configure.ac
A lang/c/jansson/README.rst
A lang/c/NEWS
A lang/c/CMakeLists.txt
A lang/c/examples
A lang/c/examples/CMakeLists.txt
A lang/c/examples/quickstop.c
A lang/c/examples/.gitignore
A lang/c/NOTICE
A lang/c/LICENSE
A lang/c/AUTHORS
A lang/c/CPackConfig.txt
A lang/c/ChangeLog
A lang/c/src
A lang/c/src/avroappend.c
A lang/c/src/avrocat.c
A lang/c/src/schema.c
A lang/c/src/datum_equal.c
A lang/c/src/resolver.c
A lang/c/src/datum_size.c
A lang/c/src/memoize.c
A lang/c/src/schema.h
A lang/c/src/schema_specific.c
A lang/c/src/datum_value.c
A lang/c/src/string.c
A lang/c/src/avromod.c
A lang/c/src/io.c
A lang/c/src/wrapped-buffer.c
A lang/c/src/datafile.c
A lang/c/src/value-sizeof.c
A lang/c/src/generic.c
A lang/c/src/avro-c.pc.in
A lang/c/src/errors.c
A lang/c/src/codec.c
A lang/c/src/array.c
A lang/c/src/codec.h
A lang/c/src/value-json.c
A lang/c/src/resolved-reader.c
A lang/c/src/datum_write.c
A lang/c/src/avropipe.c
A lang/c/src/allocation.c
A lang/c/src/st.c
A lang/c/src/avro
A lang/c/src/avro/consumer.h
A lang/c/src/avro/errors.h
A lang/c/src/avro/generic.h
A lang/c/src/avro/value.h
A lang/c/src/avro/msinttypes.h
A lang/c/src/avro/schema.h
A lang/c/src/avro/resolver.h
A lang/c/src/avro/basics.h
A lang/c/src/avro/legacy.h
A lang/c/src/avro/platform.h
A lang/c/src/avro/allocation.h
A lang/c/src/avro/msstdint.h
A lang/c/src/avro/refcount.h
A lang/c/src/avro/io.h
A lang/c/src/avro/data.h
A lang/c/src/schema_equal.c
A lang/c/src/encoding_binary.c
A lang/c/src/encoding.h
A lang/c/src/consumer.c
A lang/c/src/consume-binary.c
A lang/c/src/st.h
A lang/c/src/CMakeLists.txt
A lang/c/src/resolved-writer.c
A lang/c/src/datum_skip.c
A lang/c/src/avro_generic_internal.h
A lang/c/src/value-write.c
A lang/c/src/dump.c
A lang/c/src/datum_read.c
A lang/c/src/avro_private.h
A lang/c/src/dump.h
A lang/c/src/datum.c
A lang/c/src/avro.h
A lang/c/src/value.c
A lang/c/src/map.c
A lang/c/src/datum.h
A lang/c/src/datum_validate.c
A lang/c/src/.gitignore
A lang/c/src/value-read.c
A lang/c/src/value-hash.c
A lang/c/docs
A lang/c/docs/index.txt
A lang/c/docs/CMakeLists.txt
A lang/c/README.maintaining_win32.txt
A lang/c/FindSnappy.cmake
A lang/c/README
AUlang/c/version.sh
A lang/perl
A lang/perl/t
A lang/perl/t/01_schema.t
A lang/perl/t/01_names.t
A lang/perl/t/05_protocol.t
A lang/perl/t/02_bin_encode.t
A lang/perl/t/00_compile.t
A lang/perl/t/04_datafile.t
A lang/perl/t/03_bin_decode.t
A lang/perl/.gitignore
A lang/perl/MANIFEST
A lang/perl/lib
AUlang/perl/lib/Avro.pm
A lang/perl/lib/Avro
AUlang/perl/lib/Avro/DataFile.pm
AUlang/perl/lib/Avro/BinaryDecoder.pm
AUlang/perl/lib/Avro/DataFileReader.pm
AUlang/perl/lib/Avro/Schema.pm
AUlang/perl/lib/Avro/Protocol.pm
AUlang/perl/lib/Avro/BinaryEncoder.pm
AUlang/perl/lib/Avro/DataFileWriter.pm
A lang/perl/lib/Avro/Protocol
AUlang/perl/lib/Avro/Protocol/Message.pm
AUlang/perl/Makefile.PL
A lang/perl/Changes
A lang/perl/bin
A lang/perl/bin/avro-to-json
A lang/perl/MANIFEST.SKIP
A lang/perl/xt
A lang/perl/xt/pod.t
AUlang/perl/README
A lang/perl/.shipit
A lang/perl/NOTICE
A lang/perl/LICENSE
A share
A share/docker
A share/docker/Dockerfile
A share/editors
A share/editors/avro-idl.vim
A share/editors/README.txt
A share/test
A share/test/interop
A share/test/interop/rpc
A share/test/interop/rpc/echo
A share/test/interop/rpc/echo/foo
AUshare/test/interop/rpc/echo/foo/response.avro
AUshare/test/interop/rpc/echo/foo/request.avro
A share/test/interop/rpc/hello
A share/test/interop/rpc/hello/world
AUshare/test/interop/rpc/hello/world/request.avro
AUshare/test/interop/rpc/hello/world/response.avro
A share/test/interop/rpc/add
A sh

[jira] [Commented] (AVRO-1779) Avro docs convenience artifact missing LICENSE/NOTICE

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093748#comment-15093748
 ] 

ASF subversion and git services commented on AVRO-1779:
---

Commit 1724205 from tomwh...@apache.org in branch 'avro/branches/branch-1.8'
[ https://svn.apache.org/r1724205 ]

AVRO-1779. Avro docs convenience artifact missing LICENSE/NOTICE. Contributed 
by blue.

> Avro docs convenience artifact missing LICENSE/NOTICE
> -
>
> Key: AVRO-1779
> URL: https://issues.apache.org/jira/browse/AVRO-1779
> Project: Avro
>  Issue Type: Bug
>  Components: doc
>Affects Versions: 1.8.0
>Reporter: Sean Busbey
>Assignee: Ryan Blue
>Priority: Blocker
> Fix For: 1.8.0
>
> Attachments: AVRO-1779.1.diff
>
>
> for releases we generate a convenience artifact with our docs. at present, 
> this tarball is missing our needed LICENSE/NOTICE files



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1779) Avro docs convenience artifact missing LICENSE/NOTICE

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093746#comment-15093746
 ] 

ASF subversion and git services commented on AVRO-1779:
---

Commit 1724203 from tomwh...@apache.org in branch 'avro/trunk'
[ https://svn.apache.org/r1724203 ]

AVRO-1779. Avro docs convenience artifact missing LICENSE/NOTICE. Contributed 
by blue.

> Avro docs convenience artifact missing LICENSE/NOTICE
> -
>
> Key: AVRO-1779
> URL: https://issues.apache.org/jira/browse/AVRO-1779
> Project: Avro
>  Issue Type: Bug
>  Components: doc
>Affects Versions: 1.8.0
>Reporter: Sean Busbey
>Assignee: Ryan Blue
>Priority: Blocker
> Fix For: 1.8.0
>
> Attachments: AVRO-1779.1.diff
>
>
> for releases we generate a convenience artifact with our docs. at present, 
> this tarball is missing our needed LICENSE/NOTICE files



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1781) Schema.parse is not thread safe

[Tom White (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093689#comment-15093689
 ] 

Tom White commented on AVRO-1781:
-

This is probably the most pragmatic approach to take. Do you want to 
incorporate the fix for AVRO-1760 at the same time?

I get a NPE when I run {{mvn clean install -DskipTests -e}}:

{noformat}
[ERROR] Failed to execute goal 
org.apache.avro:avro-maven-plugin:1.9.0-SNAPSHOT:schema (schemas) on project 
avro-ipc: Execution schemas of goal 
org.apache.avro:avro-maven-plugin:1.9.0-SNAPSHOT:schema failed. 
NullPointerException -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.apache.avro:avro-maven-plugin:1.9.0-SNAPSHOT:schema (schemas) on project 
avro-ipc: Execution schemas of goal 
org.apache.avro:avro-maven-plugin:1.9.0-SNAPSHOT:schema failed.
{noformat}

> Schema.parse is not thread safe
> ---
>
> Key: AVRO-1781
> URL: https://issues.apache.org/jira/browse/AVRO-1781
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Sean Busbey
>Priority: Blocker
> Fix For: 1.8.0
>
> Attachments: AVRO-1781.1.patch
>
>
> Post AVRO-1497, Schema.parse calls {{LogicalTypes.fromSchemaIgnoreInvalid}} 
> on any schema that is expressed as a JSON object (anything except bare 
> primitives).
> That static method relies on a static cache based on WeakIdentityHashMap 
> (WIHM).
> WIHM clearly states that it isn't threadsafe 
> [ref|https://github.com/apache/avro/blob/branch-1.8/lang/java/avro/src/main/java/org/apache/avro/util/WeakIdentityHashMap.java#L42]
> {code}
>  * 
>  * Note that this implementation is not synchronized.
>  * 
>  */
> public class WeakIdentityHashMap implements Map {
> {code}
> All of the Schema.Parser instances use that same static Schema.parse method.
> The end result is that as-is it's only safe to have a single thread parsing 
> schemas in a given JVM.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Avro release 1.8.0 (rc2)

[Tom White]

I'm closing the vote.

Thanks for all your testing Sean!

Cheers,
Tom

On Mon, Jan 11, 2016 at 5:09 PM, Sean Busbey  wrote:

> * docs LICENSE : AVRO-1779
> * NPE in avro-tools : AVRO-1780
>
> I haven't gotten to verify the avro-rpc-quickstart problem yet, but there
> is a concurrency bug in Schema.parse for the java library. Filed as blocker
> under AVRO-1781.
>
> On Sat, Jan 9, 2016 at 12:51 AM, Sean Busbey  wrote:
>
> > -1 (non-binding)
> >
> > bad:
> > * the artifact avro-doc-1.8.0.tar.gz has no LICENSE/NOTICE files
> > * the avro-tools jar fails with NPE[1]
> >
> > I'll file issues.
> >
> > mixed:
> >
> > * ruby successfully loads and I can run some simple IO, but phunt's
> > avro-rpc-quickstart fails[2]. I'm not sure yet if the failure is expected
> > given the changes in 1.8.0.
> >
> > good:
> > * other LICENSE/NOTICE spot-check looks fine
> > * signatures match
> > * checksums match
> > * tag matches src tarball, ignoring one nit
> >
> > Only in avro-src/avro-src-1.8.0/lang/java/mapred/src/test/
> > resources/org/apache/avro/mapreduce/mapreduce-test-input.avro:
> SUCCESS.crc
> >
> > [1]:
> >
> > busbey$ java -jar avro-tools-1.8.0.jar
> > Version 1.8.0 of Exception in thread "main"
> java.lang.NullPointerException
> > at org.apache.avro.tool.Main.printStream(Main.java:105)
> > at org.apache.avro.tool.Main.run(Main.java:92)
> > at org.apache.avro.tool.Main.main(Main.java:74)
> > busbey$ java -jar avro-tools-1.8.0.jar --help
> > Version 1.8.0 of Exception in thread "main"
> java.lang.NullPointerException
> > at org.apache.avro.tool.Main.printStream(Main.java:105)
> > at org.apache.avro.tool.Main.run(Main.java:92)
> > at org.apache.avro.tool.Main.main(Main.java:74)
> >
> >   looks like the tooling looks for NOTICE.txt at the top level and we
> > don't have one there anymore :(
> >
> >
> >
> https://github.com/apache/avro/blob/trunk/lang/java/tools/src/main/java/org/apache/avro/tool/Main.java#L92
> >
> >
> > [2]: https://github.com/phunt/avro-rpc-quickstart
> >
> > busbey$ ruby sample_ipc_client.rb avro_user pat Hello_World
> > Avro::IO::AvroTypeError: The datum "\x89\xA9\xD1\xFF@NUm
> \xEA\x9A\xFB\xDAx\xF5Zq"
> > is not an example of schema
> > {"type":"fixed","name":"MD5","namespace":"org.apache.avro.ipc","size":16}
> >write_data at
> > /Users/busbey/.rvm/gems/jruby-1.7.3/gems/avro-1.8.0/lib/avro/io.rb:543
> >  write_record at
> > /Users/busbey/.rvm/gems/jruby-1.7.3/gems/avro-1.8.0/lib/avro/io.rb:610
> >  each at org/jruby/RubyArray.java:1613
> >  write_record at
> > /Users/busbey/.rvm/gems/jruby-1.7.3/gems/avro-1.8.0/lib/avro/io.rb:609
> >write_data at
> > /Users/busbey/.rvm/gems/jruby-1.7.3/gems/avro-1.8.0/lib/avro/io.rb:561
> > write at
> > /Users/busbey/.rvm/gems/jruby-1.7.3/gems/avro-1.8.0/lib/avro/io.rb:538
> >   write_handshake_request at
> > /Users/busbey/.rvm/gems/jruby-1.7.3/gems/avro-1.8.0/lib/avro/ipc.rb:136
> >   request at
> > /Users/busbey/.rvm/gems/jruby-1.7.3/gems/avro-1.8.0/lib/avro/ipc.rb:105
> >   request at
> > /Users/busbey/.rvm/gems/jruby-1.7.3/gems/avro-1.8.0/lib/avro/ipc.rb:117
> >(root) at sample_ipc_client.rb:49
> >
> >
> > On Wed, Jan 6, 2016 at 9:16 AM, Tom White  wrote:
> >
> >> I've created another release candidate for Avro 1.8.0 that fixes the
> >> problems found in rc1. (Thanks to Ryan and Martin for the patches to fix
> >> them.)
> >>
> >> The changes are listed at:
> >> http://s.apache.org/avro180
> >>
> >> The release artifacts can be found here:
> >> https://dist.apache.org/repos/dist/dev/avro/avro-1.8.0-rc2/
> >>
> >> The tag corresponding to this release candidate is:
> >> http://svn.apache.org/repos/asf/avro/tags/release-1.8.0-rc2/
> >>
> >> You can find the KEYS file here:
> >> https://dist.apache.org/repos/dist/release/avro/KEYS
> >>
> >> The Maven staging repository is at:
> >> https://repository.apache.org/content/repositories/orgapacheavro-1004/
> >>
> >> Please download, verify, and test. The vote will remain open for at
> least
> >> 72 hours. Again, this is the first release that has been built using
> >> Docker, so please pay extra attention to the languages you are
> interested
> >> in. Thanks in advance for voting!
> >>
> >> Cheers,
> >> Tom
> >>
> >
> >
> >
> > --
> > Sean
> >
>
>
>
> --
> Sean
>