[jira] [Commented] (AVRO-1493) Avoid the "Turkish Locale Problem"

2015-12-09 Thread Kevin Schultz (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048863#comment-15048863
 ] 

Kevin Schultz commented on AVRO-1493:
-

We are seeing the same error using Avro on Android. I dug around a bit and at 
least for the fingerprint generation problem my guess is that the problem is 
caused by a few places where the Locale is not explicitly set. 

org.apache.avro.generic.GenericData.java:  
builder.append(hex.toUpperCase());
org.apache.avro.Schema.java:  order 
Field.Order.valueOf(orderNode.getTextValue().toUpperCase());
org.apache.avro.Schema.java:  private Type() { 
this.name = this.name().toLowerCase(); }
org.apache.avro.Schema.java:  private Order() { 
this.name = this.name().toLowerCase(); }



> Avoid the "Turkish Locale Problem"
> --
>
> Key: AVRO-1493
> URL: https://issues.apache.org/jira/browse/AVRO-1493
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.7.6
> Environment: Hadoop trunk build error on mac-os with turkish locale.
>Reporter: Serkan Taş
> Fix For: 1.7.8
>
>
> Locale dependent String.toUpperCase(), String.toLowerCase() causes unexpected 
> behavior if the the locale is Turkish
> Not sure about String.equalsIgnoreCase(..).
> Here is the error :
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile 
> (default-testCompile) on project hadoop-common: Compilation failure
> [ERROR] 
> /Users/serkan/programlar/dev/hadooptest/hadoop-trunk/hadoop-common-project/hadoop-common/target/generated-test-sources/java/org/apache/hadoop/io/serializer/avro/AvroRecord.java:[10,244]
>  unmappable character for encoding UTF-8
> [ERROR] -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile 
> (default-testCompile) on project hadoop-common: Compilation failure
> /Users/serkan/programlar/dev/hadooptest/hadoop-trunk/hadoop-common-project/hadoop-common/target/generated-test-sources/java/org/apache/hadoop/io/serializer/avro/AvroRecord.java:[10,244]
>  unmappable character for encoding UTF-8
> I f i check the code i discovered the reason for error :
>  public static final org.apache.avro.Schema SCHEMA$ = new 
> org.apache.avro.Schema.Parser().parse("{\"type\":\"record\",\"name\":\"AvroRecord\",\"namespace\":\"org.apache.hadoop.io.serializer.avro\",\"fields\":[{\"name\":\"intField\",\"type\":\"Ýnt\"}]}");
> For the code generated from schema, locale dependent capitalization of letter 
> "i" turns in to "Ý" should be the same for "I" to "ı".
> Same bug exist in OPENEJB-1071, OAK-260, IBATIS-218.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1584) Json output doesn't generate base64 for byte arrays

2015-12-09 Thread Ryan Blue (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049388#comment-15049388
 ] 

Ryan Blue commented on AVRO-1584:
-

I think using a helper library would be good. My main concern here is correctly 
representing the data and not performance, but there's a base64 helper lib from 
Jackson you can use that allows you to add a character or byte at a time 
(Base64Variants.defaultVariant()) that would work for ByteBuffer. I think a 
helper method would be fine.

For fixed, I'm referring to the GenericData.Fixed class for generic. That 
corresponds to the "fixed" type in the spec that is a fixed-length byte array. 
Right now, those become a JSON list of integers. Thanks, David!

> Json output doesn't generate base64 for byte arrays
> ---
>
> Key: AVRO-1584
> URL: https://issues.apache.org/jira/browse/AVRO-1584
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.7.7
> Environment: Pure java.
>Reporter: Christophe Lorenz
> Attachments: AVRO-1584.patch
>
>
> The Json output of java generated code doesn't correctly encode byte arrays.
> Using this simple schema : 
> {"namespace": "example.avro",
>  "type": "record",
>  "name": "ByteArrayEncoding",
>  "fields": [ {"name": "data", "type": "bytes"} ]
> }
> The toString()  
>   System.out.println(new ByteArrayEncoding(ByteBuffer.wrap(new 
> byte[]{0,31,65,66,67,(byte)255,(byte)182})));
> Returns raw bytes to string in the json :
> {"data": {"bytes": "  ABC??"}}
> As a byte array is not tied to be a valid string, it should be converted back 
> and forth to Base64 like other Json implementations : 
> {"data": {"bytes": "AB9BQkP/tg=="}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Can somebody look at: https://issues.apache.org/jira/browse/AVRO-1723

2015-12-09 Thread Ryan Blue 
Zoltan, thanks for taking the time to contribute this patch. I've been 
watching this one and IIRC it is pretty close.


One thing that makes it more difficult is that it depends on AVRO-1667 
and the linked branch includes another uncommitted fix. Sorting that out 
actually prevented me from looking at it a couple weeks ago and it would 
be awesome if you had the time to determine what is really blocking the 
patch and work with me on the other issues first.


Lets focus on getting AVRO-1667 in and then we'll get this done. Does 
that work for you?


rb

On 12/09/2015 12:10 PM, Zoltan Farkas wrote:

This is quite a nice improvement to the avro IDL, and the amount of code is 
small.

thank you

—Z




--
Ryan Blue
Software Engineer
Cloudera, Inc.

[jira] [Commented] (AVRO-1493) Avoid the "Turkish Locale Problem"

2015-12-09 Thread Kevin Schultz (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049384#comment-15049384
 ] 

Kevin Schultz commented on AVRO-1493:
-

I have a test case that illustrates the issue & a patch to fix it, I just need 
to find my way through the Avro contribution process (and remember how to use 
SVN)

https://github.com/krschultz/avro/commit/39d6b9db492b0f5f4d0ed1a32f0cb5c7be0fa11f

> Avoid the "Turkish Locale Problem"
> --
>
> Key: AVRO-1493
> URL: https://issues.apache.org/jira/browse/AVRO-1493
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.7.6
> Environment: Hadoop trunk build error on mac-os with turkish locale.
>Reporter: Serkan Taş
> Fix For: 1.7.8
>
>
> Locale dependent String.toUpperCase(), String.toLowerCase() causes unexpected 
> behavior if the the locale is Turkish
> Not sure about String.equalsIgnoreCase(..).
> Here is the error :
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile 
> (default-testCompile) on project hadoop-common: Compilation failure
> [ERROR] 
> /Users/serkan/programlar/dev/hadooptest/hadoop-trunk/hadoop-common-project/hadoop-common/target/generated-test-sources/java/org/apache/hadoop/io/serializer/avro/AvroRecord.java:[10,244]
>  unmappable character for encoding UTF-8
> [ERROR] -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile 
> (default-testCompile) on project hadoop-common: Compilation failure
> /Users/serkan/programlar/dev/hadooptest/hadoop-trunk/hadoop-common-project/hadoop-common/target/generated-test-sources/java/org/apache/hadoop/io/serializer/avro/AvroRecord.java:[10,244]
>  unmappable character for encoding UTF-8
> I f i check the code i discovered the reason for error :
>  public static final org.apache.avro.Schema SCHEMA$ = new 
> org.apache.avro.Schema.Parser().parse("{\"type\":\"record\",\"name\":\"AvroRecord\",\"namespace\":\"org.apache.hadoop.io.serializer.avro\",\"fields\":[{\"name\":\"intField\",\"type\":\"Ýnt\"}]}");
> For the code generated from schema, locale dependent capitalization of letter 
> "i" turns in to "Ý" should be the same for "I" to "ı".
> Same bug exist in OPENEJB-1071, OAK-260, IBATIS-218.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (AVRO-1493) Avoid the "Turkish Locale Problem"

2015-12-09 Thread Ryan Blue (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049427#comment-15049427
 ] 

Ryan Blue commented on AVRO-1493:
-

Thanks, [~krschultz]! Feel free to open a pull request against the Avro github 
project (and post a link) if that's easier. Otherwise, svn diff is your friend. 
I haven't used SVN in years -- I do everything through git-svn and development 
in git.

> Avoid the "Turkish Locale Problem"
> --
>
> Key: AVRO-1493
> URL: https://issues.apache.org/jira/browse/AVRO-1493
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.7.6
> Environment: Hadoop trunk build error on mac-os with turkish locale.
>Reporter: Serkan Taş
> Fix For: 1.7.8
>
>
> Locale dependent String.toUpperCase(), String.toLowerCase() causes unexpected 
> behavior if the the locale is Turkish
> Not sure about String.equalsIgnoreCase(..).
> Here is the error :
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile 
> (default-testCompile) on project hadoop-common: Compilation failure
> [ERROR] 
> /Users/serkan/programlar/dev/hadooptest/hadoop-trunk/hadoop-common-project/hadoop-common/target/generated-test-sources/java/org/apache/hadoop/io/serializer/avro/AvroRecord.java:[10,244]
>  unmappable character for encoding UTF-8
> [ERROR] -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile 
> (default-testCompile) on project hadoop-common: Compilation failure
> /Users/serkan/programlar/dev/hadooptest/hadoop-trunk/hadoop-common-project/hadoop-common/target/generated-test-sources/java/org/apache/hadoop/io/serializer/avro/AvroRecord.java:[10,244]
>  unmappable character for encoding UTF-8
> I f i check the code i discovered the reason for error :
>  public static final org.apache.avro.Schema SCHEMA$ = new 
> org.apache.avro.Schema.Parser().parse("{\"type\":\"record\",\"name\":\"AvroRecord\",\"namespace\":\"org.apache.hadoop.io.serializer.avro\",\"fields\":[{\"name\":\"intField\",\"type\":\"Ýnt\"}]}");
> For the code generated from schema, locale dependent capitalization of letter 
> "i" turns in to "Ý" should be the same for "I" to "ı".
> Same bug exist in OPENEJB-1071, OAK-260, IBATIS-218.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Can somebody look at: https://issues.apache.org/jira/browse/AVRO-1723

2015-12-09 Thread Zoltan Farkas 
Sure, 

The patch I have attached to AVRO-1723 
 includes the fixes from 
AVRO-1667 (The included unit test would fail otherwise). 
I have created the patch based on the latest trunk at the time. (Doug asked for 
it in order to review the changes easier)

you can either use the attached patch or merge the branch: 
https://github.com/zolyfarkas/avro/tree/trunk-AVRO-1723  (rebased it now to 
latest trunk)

Let me know if there is anything I can do to help with this.


—Z


> On Dec 9, 2015, at 4:49 PM, Ryan Blue  wrote:
> 
> Zoltan, thanks for taking the time to contribute this patch. I've been 
> watching this one and IIRC it is pretty close.
> 
> One thing that makes it more difficult is that it depends on AVRO-1667 and 
> the linked branch includes another uncommitted fix. Sorting that out actually 
> prevented me from looking at it a couple weeks ago and it would be awesome if 
> you had the time to determine what is really blocking the patch and work with 
> me on the other issues first.
> 
> Lets focus on getting AVRO-1667 in and then we'll get this done. Does that 
> work for you?
> 
> rb
> 
> On 12/09/2015 12:10 PM, Zoltan Farkas wrote:
>> This is quite a nice improvement to the avro IDL, and the amount of code is 
>> small.
>> 
>> thank you
>> 
>> —Z
>> 
> 
> 
> -- 
> Ryan Blue
> Software Engineer
> Cloudera, Inc.

[jira] [Commented] (AVRO-1747) JavaScript implementation

2015-12-09 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049985#comment-15049985
 ] 

ASF GitHub Bot commented on AVRO-1747:
--

Github user mtth closed the pull request at:

https://github.com/apache/avro/pull/58


> JavaScript implementation
> -
>
> Key: AVRO-1747
> URL: https://issues.apache.org/jira/browse/AVRO-1747
> Project: Avro
>  Issue Type: Improvement
>  Components: javascript
>Reporter: Matthieu Monsch
>Assignee: Matthieu Monsch
> Attachments: AVRO-1747.patch, AVRO-1747.patch, AVRO-1747.patch
>
>
> Hello,
> I'm not sure if there is still interest in a JavaScript implementation of the 
> Avro spec, or if this is the right place for this message (apologies if not), 
> but in case it's useful here is one: https://github.com/mtth/avsc
> It's pretty fast, fully featured aside from protocols (AFAIK), and runs in 
> the browser.
> Disclaimer: I wrote this library. (I initially searched around for existing 
> implementations, and even saw a few tickets on this board about JavaScript 
> decoders, but couldn't find one to support the schemas I have to process.)
> Best,
> -Matthieu



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (AVRO-1747) JavaScript implementation

2015-12-09 Thread Matthieu Monsch (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AVRO-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthieu Monsch resolved AVRO-1747.
---
Resolution: Fixed

> JavaScript implementation
> -
>
> Key: AVRO-1747
> URL: https://issues.apache.org/jira/browse/AVRO-1747
> Project: Avro
>  Issue Type: Improvement
>  Components: javascript
>Reporter: Matthieu Monsch
>Assignee: Matthieu Monsch
> Attachments: AVRO-1747.patch, AVRO-1747.patch, AVRO-1747.patch
>
>
> Hello,
> I'm not sure if there is still interest in a JavaScript implementation of the 
> Avro spec, or if this is the right place for this message (apologies if not), 
> but in case it's useful here is one: https://github.com/mtth/avsc
> It's pretty fast, fully featured aside from protocols (AFAIK), and runs in 
> the browser.
> Disclaimer: I wrote this library. (I initially searched around for existing 
> implementations, and even saw a few tickets on this board about JavaScript 
> decoders, but couldn't find one to support the schemas I have to process.)
> Best,
> -Matthieu



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] avro pull request: AVRO-1747: JavaScript: Add implementation.

2015-12-09 Thread mtth 
Github user mtth closed the pull request at:

https://github.com/apache/avro/pull/58


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---