[jira] [Commented] (AVRO-1738) add java tool for outputting schema fingerprints

2017-02-15 Thread Ben Osheroff (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869169#comment-15869169
 ] 

Ben Osheroff commented on AVRO-1738:


Zendesk would love to be early adopters of schema-fingerprinting and the more 
general schema-evolution-via-data-header stuff.  Any chance to get some action 
here?  

thanks!

> add java tool for outputting schema fingerprints
> 
>
> Key: AVRO-1738
> URL: https://issues.apache.org/jira/browse/AVRO-1738
> Project: Avro
>  Issue Type: New Feature
>  Components: java
>Reporter: Sean Busbey
>Assignee: Sean Busbey
> Fix For: 1.9.0
>
> Attachments: AVRO-1738.1.patch
>
>
> over in AVRO-1694 I wanted to quickly check that the Java library came up 
> with the same md5/sha fingerprint for some shcemas that the proposed Ruby 
> implementation does.
> I noticed we don't have a tool that exposes the functionality yet, which 
> seems like a commonly useful thing to do.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Issue Comment Deleted] (AVRO-2001) Add support for doc in the Ruby library for Avro schemas

2017-02-15 Thread Wyatt Anderse (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AVRO-2001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wyatt Anderse updated AVRO-2001:

Comment: was deleted

(was: Here's a PR, 1 commit, with unit tests: 
https://github.com/apache/avro/pull/197. How can I improve it?)

> Add support for doc in the Ruby library for Avro schemas
> 
>
> Key: AVRO-2001
> URL: https://issues.apache.org/jira/browse/AVRO-2001
> Project: Avro
>  Issue Type: New Feature
>  Components: ruby
>Reporter: Wyatt Anderse
>Priority: Minor
>  Labels: features
>
> The Avro schema and protocol 
> specification(http://avro.apache.org/docs/current/spec.html) allows for an 
> optional attribute "doc" defined as, "a JSON string describing this field for 
> users." Users should be able to include a doc with each field for "records" 
> (http://avro.apache.org/docs/current/spec.html#N10085) and "enums" 
> (http://avro.apache.org/docs/current/spec.html#N10186).
> This doc attribute should be stripped when transforming into "Parsing 
> Canonical Form" (http://avro.apache.org/docs/current/spec.html#N10815).
> Currently, the Ruby Avro library 
> (https://github.com/apache/avro/tree/master/lang/ruby) does not support doc. 
> It is supported in other languages libraries, for example PHP 
> (https://github.com/apache/avro/blob/master/lang/php/lib/avro/schema.php#L46).
> Not supporting doc in Ruby causes confusion for the end user, and prevents 
> that end use from taking advantage of a useful tool for managing complex 
> schema. 
> doc should be supported for schema and protocols in the Apache Ruby Avro 
> library.
> Possible implementation: 
> https://github.com/lostphilosopher/avro/tree/add_doc_to_schema based on 
> https://github.com/apache/avro/pull/14/commits/164fb49f80dfaac421639e6249a6f92c802a50b6.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AVRO-2001) Add support for doc in the Ruby library for Avro schemas

2017-02-15 Thread Wyatt Anderse (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-2001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868645#comment-15868645
 ] 

Wyatt Anderse commented on AVRO-2001:
-

Here's a PR, 1 commit, with unit tests: 
https://github.com/apache/avro/pull/197. How can I improve it?

> Add support for doc in the Ruby library for Avro schemas
> 
>
> Key: AVRO-2001
> URL: https://issues.apache.org/jira/browse/AVRO-2001
> Project: Avro
>  Issue Type: New Feature
>  Components: ruby
>Reporter: Wyatt Anderse
>Priority: Minor
>  Labels: features
>
> The Avro schema and protocol 
> specification(http://avro.apache.org/docs/current/spec.html) allows for an 
> optional attribute "doc" defined as, "a JSON string describing this field for 
> users." Users should be able to include a doc with each field for "records" 
> (http://avro.apache.org/docs/current/spec.html#N10085) and "enums" 
> (http://avro.apache.org/docs/current/spec.html#N10186).
> This doc attribute should be stripped when transforming into "Parsing 
> Canonical Form" (http://avro.apache.org/docs/current/spec.html#N10815).
> Currently, the Ruby Avro library 
> (https://github.com/apache/avro/tree/master/lang/ruby) does not support doc. 
> It is supported in other languages libraries, for example PHP 
> (https://github.com/apache/avro/blob/master/lang/php/lib/avro/schema.php#L46).
> Not supporting doc in Ruby causes confusion for the end user, and prevents 
> that end use from taking advantage of a useful tool for managing complex 
> schema. 
> doc should be supported for schema and protocols in the Apache Ruby Avro 
> library.
> Possible implementation: 
> https://github.com/lostphilosopher/avro/tree/add_doc_to_schema based on 
> https://github.com/apache/avro/pull/14/commits/164fb49f80dfaac421639e6249a6f92c802a50b6.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AVRO-2001) Add support for doc in the Ruby library for Avro schemas

2017-02-15 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-2001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868643#comment-15868643
 ] 

ASF GitHub Bot commented on AVRO-2001:
--

GitHub user lostphilosopher opened a pull request:

https://github.com/apache/avro/pull/197

AVRO-2001 Adding support for doc attribute

The Avro schema and protocol 
specification(http://avro.apache.org/docs/current/spec.html) allows for an 
optional attribute "doc" defined as, "a JSON string describing this field for 
users." Users should be able to include a doc with each field for "records" 
(http://avro.apache.org/docs/current/spec.html#N10085) and "enums" 
(http://avro.apache.org/docs/current/spec.html#N10186).

This doc attribute should be stripped when transforming into "Parsing 
Canonical Form" (http://avro.apache.org/docs/current/spec.html#N10815).

Currently, the Ruby Avro library 
(https://github.com/apache/avro/tree/master/lang/ruby) does not support doc. It 
is supported in other languages libraries, for example PHP 
(https://github.com/apache/avro/blob/master/lang/php/lib/avro/schema.php#L46).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lostphilosopher/avro avro_2001

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/avro/pull/197.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #197


commit 654bedf194196b5f6d8ba5f28f516010c0be89b8
Author: lostphilosopher 
Date:   2017-02-15T19:05:43Z

AVRO-2001 Adding support for doc attribute




> Add support for doc in the Ruby library for Avro schemas
> 
>
> Key: AVRO-2001
> URL: https://issues.apache.org/jira/browse/AVRO-2001
> Project: Avro
>  Issue Type: New Feature
>  Components: ruby
>Reporter: Wyatt Anderse
>Priority: Minor
>  Labels: features
>
> The Avro schema and protocol 
> specification(http://avro.apache.org/docs/current/spec.html) allows for an 
> optional attribute "doc" defined as, "a JSON string describing this field for 
> users." Users should be able to include a doc with each field for "records" 
> (http://avro.apache.org/docs/current/spec.html#N10085) and "enums" 
> (http://avro.apache.org/docs/current/spec.html#N10186).
> This doc attribute should be stripped when transforming into "Parsing 
> Canonical Form" (http://avro.apache.org/docs/current/spec.html#N10815).
> Currently, the Ruby Avro library 
> (https://github.com/apache/avro/tree/master/lang/ruby) does not support doc. 
> It is supported in other languages libraries, for example PHP 
> (https://github.com/apache/avro/blob/master/lang/php/lib/avro/schema.php#L46).
> Not supporting doc in Ruby causes confusion for the end user, and prevents 
> that end use from taking advantage of a useful tool for managing complex 
> schema. 
> doc should be supported for schema and protocols in the Apache Ruby Avro 
> library.
> Possible implementation: 
> https://github.com/lostphilosopher/avro/tree/add_doc_to_schema based on 
> https://github.com/apache/avro/pull/14/commits/164fb49f80dfaac421639e6249a6f92c802a50b6.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] avro pull request #197: AVRO-2001 Adding support for doc attribute

2017-02-15 Thread lostphilosopher 
GitHub user lostphilosopher opened a pull request:

https://github.com/apache/avro/pull/197

AVRO-2001 Adding support for doc attribute

The Avro schema and protocol 
specification(http://avro.apache.org/docs/current/spec.html) allows for an 
optional attribute "doc" defined as, "a JSON string describing this field for 
users." Users should be able to include a doc with each field for "records" 
(http://avro.apache.org/docs/current/spec.html#N10085) and "enums" 
(http://avro.apache.org/docs/current/spec.html#N10186).

This doc attribute should be stripped when transforming into "Parsing 
Canonical Form" (http://avro.apache.org/docs/current/spec.html#N10815).

Currently, the Ruby Avro library 
(https://github.com/apache/avro/tree/master/lang/ruby) does not support doc. It 
is supported in other languages libraries, for example PHP 
(https://github.com/apache/avro/blob/master/lang/php/lib/avro/schema.php#L46).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lostphilosopher/avro avro_2001

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/avro/pull/197.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #197


commit 654bedf194196b5f6d8ba5f28f516010c0be89b8
Author: lostphilosopher 
Date:   2017-02-15T19:05:43Z

AVRO-2001 Adding support for doc attribute




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (AVRO-2001) Add support for doc in the Ruby library for Avro schemas

2017-02-15 Thread Wyatt Anderse (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-2001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868470#comment-15868470
 ] 

Wyatt Anderse edited comment on AVRO-2001 at 2/15/17 7:52 PM:
--

This issue is holding up a project I'm working on. While we can use a fork with 
support for doc I'd much prefer to stay with the core project. I'm happy to 
write the code to resolve this issue, and I'm actively working on a GitHub PR 
with proper unit tests and formatting. (First PR to this project so feel free 
to let me know if I'm going about anything incorrectly.)


was (Author: wandersen02):
This issue is holding up a project I'm working on. While we can use a fork with 
support for doc I'd much prefer to stay with the core project. I'm happy to 
write the code to resolve this issue, and I'm actively working on a GitHub PR 
with proper unit tests and formatting. First PR to this project to feel free to 
let me know if I'm going about anything incorrectly.

> Add support for doc in the Ruby library for Avro schemas
> 
>
> Key: AVRO-2001
> URL: https://issues.apache.org/jira/browse/AVRO-2001
> Project: Avro
>  Issue Type: New Feature
>  Components: ruby
>Reporter: Wyatt Anderse
>Priority: Minor
>  Labels: features
>
> The Avro schema and protocol 
> specification(http://avro.apache.org/docs/current/spec.html) allows for an 
> optional attribute "doc" defined as, "a JSON string describing this field for 
> users." Users should be able to include a doc with each field for "records" 
> (http://avro.apache.org/docs/current/spec.html#N10085) and "enums" 
> (http://avro.apache.org/docs/current/spec.html#N10186).
> This doc attribute should be stripped when transforming into "Parsing 
> Canonical Form" (http://avro.apache.org/docs/current/spec.html#N10815).
> Currently, the Ruby Avro library 
> (https://github.com/apache/avro/tree/master/lang/ruby) does not support doc. 
> It is supported in other languages libraries, for example PHP 
> (https://github.com/apache/avro/blob/master/lang/php/lib/avro/schema.php#L46).
> Not supporting doc in Ruby causes confusion for the end user, and prevents 
> that end use from taking advantage of a useful tool for managing complex 
> schema. 
> doc should be supported for schema and protocols in the Apache Ruby Avro 
> library.
> Possible implementation: 
> https://github.com/lostphilosopher/avro/tree/add_doc_to_schema based on 
> https://github.com/apache/avro/pull/14/commits/164fb49f80dfaac421639e6249a6f92c802a50b6.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AVRO-2001) Add support for doc in the Ruby library for Avro schemas

2017-02-15 Thread Wyatt Anderse (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-2001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868470#comment-15868470
 ] 

Wyatt Anderse commented on AVRO-2001:
-

This issue is holding up a project I'm working on. While we can use a fork with 
support for doc I'd much prefer to stay with the core project. I'm happy to 
write the code to resolve this issue, and I'm actively working on a GitHub PR 
with proper unit tests and formatting. First PR to this project to feel free to 
let me know if I'm going about anything incorrectly.

> Add support for doc in the Ruby library for Avro schemas
> 
>
> Key: AVRO-2001
> URL: https://issues.apache.org/jira/browse/AVRO-2001
> Project: Avro
>  Issue Type: New Feature
>  Components: ruby
>Reporter: Wyatt Anderse
>Priority: Minor
>  Labels: features
>
> The Avro schema and protocol 
> specification(http://avro.apache.org/docs/current/spec.html) allows for an 
> optional attribute "doc" defined as, "a JSON string describing this field for 
> users." Users should be able to include a doc with each field for "records" 
> (http://avro.apache.org/docs/current/spec.html#N10085) and "enums" 
> (http://avro.apache.org/docs/current/spec.html#N10186).
> This doc attribute should be stripped when transforming into "Parsing 
> Canonical Form" (http://avro.apache.org/docs/current/spec.html#N10815).
> Currently, the Ruby Avro library 
> (https://github.com/apache/avro/tree/master/lang/ruby) does not support doc. 
> It is supported in other languages libraries, for example PHP 
> (https://github.com/apache/avro/blob/master/lang/php/lib/avro/schema.php#L46).
> Not supporting doc in Ruby causes confusion for the end user, and prevents 
> that end use from taking advantage of a useful tool for managing complex 
> schema. 
> doc should be supported for schema and protocols in the Apache Ruby Avro 
> library.
> Possible implementation: 
> https://github.com/lostphilosopher/avro/tree/add_doc_to_schema based on 
> https://github.com/apache/avro/pull/14/commits/164fb49f80dfaac421639e6249a6f92c802a50b6.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (AVRO-2001) Add support for doc in the Ruby library for Avro schemas

2017-02-15 Thread Wyatt Anderse (JIRA) 
Wyatt Anderse created AVRO-2001:
---

 Summary: Add support for doc in the Ruby library for Avro schemas
 Key: AVRO-2001
 URL: https://issues.apache.org/jira/browse/AVRO-2001
 Project: Avro
  Issue Type: New Feature
  Components: ruby
Reporter: Wyatt Anderse
Priority: Minor


The Avro schema and protocol 
specification(http://avro.apache.org/docs/current/spec.html) allows for an 
optional attribute "doc" defined as, "a JSON string describing this field for 
users." Users should be able to include a doc with each field for "records" 
(http://avro.apache.org/docs/current/spec.html#N10085) and "enums" 
(http://avro.apache.org/docs/current/spec.html#N10186).

This doc attribute should be stripped when transforming into "Parsing Canonical 
Form" (http://avro.apache.org/docs/current/spec.html#N10815).

Currently, the Ruby Avro library 
(https://github.com/apache/avro/tree/master/lang/ruby) does not support doc. It 
is supported in other languages libraries, for example PHP 
(https://github.com/apache/avro/blob/master/lang/php/lib/avro/schema.php#L46).

Not supporting doc in Ruby causes confusion for the end user, and prevents that 
end use from taking advantage of a useful tool for managing complex schema. 

doc should be supported for schema and protocols in the Apache Ruby Avro 
library.

Possible implementation: 
https://github.com/lostphilosopher/avro/tree/add_doc_to_schema based on 
https://github.com/apache/avro/pull/14/commits/164fb49f80dfaac421639e6249a6f92c802a50b6.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: java.math.BigDecimal to Avro .avdl file help please

2017-02-15 Thread Nathan Fisher 
Hi Steve,

I'm seeing the same issue. I think you'll need to drop down to JSON. The
IDL appears to provide a subset of the Avro specification.

I've started mapping the gaps with the following record type:

enum YesNo { YES, NO }
fixed TWO(2);

record IdlHarness {
// primitives
boolean bool = true;
int i = 0;
int iN = null;
long l = 0;
long lN = null;
float f = 0.1;
float fN = 0.1;
double d = 0.1;
double dN = null;
bytes b = "banana";
bytes bN = null;
string s = "s";
string sN = null;

// complex
YesNo en = "YES";
YesNo enN = null;
array a = [1];
array aN = null;
map m = { "hello": 1 };
map mN = null;
union {null, string} un = "hello";
union {null, string} unN = null;
TWO fix = "12";
TWO fixN = null;

// logical
// decimal dec; // error
date dt = "2017-01-01";
date dtN = null;
// time-millis ms; // fails
// time-micros us; // fails
// timestamp-millis tsms; // fails
// timestamp-micros tsus; // fails
// duration du; // fails
}

I've put my results into the Google sheet below. It's still a work in
progress and I'll try to turn it into a JUnit test scenario.
https://docs.google.com/spreadsheets/d/1dQAyWbrD5PmHzMJ7sVovHBoZug6jlt_x_Htc7ixdbP8/edit?usp=sharing

Kind regards,
Nathan

On Wed, 1 Feb 2017 at 05:12 Steve Sun  wrote:

Hi Avro users mailing list,

I'm having trouble writing an Avro schema for java.math.BigDecimal type, I
tried the following:

1. Based on Avro official doc:
https://avro.apache.org/docs/1.8.1/spec.html#Decimal, I know I need to
define Logical Types myself to support BigDecimal, but that link gives
example only in avsc, I'm trying to figure it out in avdl.
2. Based on Avro dochttps://
avro.apache.org/docs/1.7.6/api/java/org/apache/avro/reflect/package-summary.html
and this example:
https://github.com/apache/avro/blob/master/share/test/schemas/stringables.avdl,
I wrote below avdl:



@namespace("test")
protocol My_Protocol_v1 {

  record BigDecimal {
@java-class("java.math.BigDecimal") string value;
  }



But it's not working:
This IDL schema compiles fine and can generate a Java class called
BigDecimal, but I cannot really use the generated BigDecimal as
java.math.BigDecimal, what's wrong? or How should I do it?

Thanks a lot
Steve

-- 
- from my thumbs to your eyes

[jira] [Comment Edited] (AVRO-1933) SchemaCompatibility class could be more user-friendly about incompatibilities

2017-02-15 Thread Elliot West (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867887#comment-15867887
 ] 

Elliot West edited comment on AVRO-1933 at 2/15/17 2:24 PM:


I agree with the motivations behind this effort. There is a lot of room for 
improvement with regards to the reporting of incompatibilities within schemas. 
This is especially important as the end user focus of Avro Schema development 
shifts from developers to analysts  and I know of a few projects that are 
heading in this direction. For example, one can envisage systems built upon 
Avro that have web-based UI tooling for schema submission etc.

In these cases it would be very handy to model situations that would enable us 
to make statements such as:

* "_Field 'uuid' in record 'namespace.record_name' in the existing schema 
cannot be read by the new schema because it has changed to an incompatible 
type; was 'string', now 'long'_"
* "_Field 'address' in record 'namespace.record_name' in the new schema does 
not exist in the earlier schema and does not declare a default_"
* "_Field union branch 1 at 'namespace.record_name.uField' in the earlier 
schema does not exist in the new schema_"

Certainly these conditions should generated as an accessible model and not 
simply returned as a message so that integrations with editors will be 
possible. With this is mind, would it be possible for 
{{SchemaCompatibilityDetails}} to also encapsulate a machine readable, fully 
qualified path to the incompatible node within the schema tree?

Nice to see the patch and hope to see such a feature present in a future 
version of Avro.



was (Author: teabot):
I agree with the motivations behind this effort. There is a lot of room for 
improvement with regards to the reporting of incompatibilities within schemas. 
This is especially important as the end user focus of Avro Schema development 
shifts from developers to analysts  and I know of a few projects that are 
heading in this direction. For example, one can envisage systems built upon 
Avro that have web-based UI tooling for schema submission etc.

In these cases it would be very handy to model situations that would later 
enable us to make statements such as:

* "_Field 'uuid' in record 'namespace.record_name' in the existing schema 
cannot be read by the new schema because it has changed to an incompatible 
type; was 'string', now 'long'_"
* "_Field 'address' in record 'namespace.record_name' in the new schema does 
not exist in the earlier schema and does not declare a default_"
* "_Field union branch 1 at 'namespace.record_name.uField' in the earlier 
schema does not exist in the new schema_"

Certainly these conditions should generated as an accessible model and not 
simply returned as a message so that integrations with editors will be 
possible. With this is mind, would it be possible for 
{{SchemaCompatibilityDetails}} to also encapsulate a machine readable, fully 
qualified path to the incompatible node within the schema tree?

Nice to see the patch and hope to see such a feature present in a future 
version of Avro.


> SchemaCompatibility class could be more user-friendly about incompatibilities
> -
>
> Key: AVRO-1933
> URL: https://issues.apache.org/jira/browse/AVRO-1933
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.1
> Environment: Any Java env
>Reporter: Anders Sundelin
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: AVRO-1933-compatible-with-AVRO-1931.patch, 
> AVRO-1933.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Today, the class SchemaCompatibility reports incompatibilities with quite 
> little detail. The whole reader and the whole writer schema is listed, and no 
> particular detail about what was incompatible.
> The attached patch fixes this, introducing a new enum 
> (SchemaIncompatibilityType), and more specific sub-schemas that were 
> incompatible.
> The old, overall picture, is still there - the new compatibility state is 
> encapsulated in the SchemaCompatibilityDetails class.
> Lots of test cases have been added, and there has been refactoring done in 
> the TestSchemaCompatibility and other test classes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (AVRO-1933) SchemaCompatibility class could be more user-friendly about incompatibilities

2017-02-15 Thread Elliot West (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867887#comment-15867887
 ] 

Elliot West edited comment on AVRO-1933 at 2/15/17 2:20 PM:


I agree with the motivations behind this effort. There is a lot of room for 
improvement with regards to the reporting of incompatibilities within schemas. 
This is especially important as the end user focus of Avro Schema development 
shifts from developers to analysts  and I know of a few projects that are 
heading in this direction. For example, one can envisage systems built upon 
Avro that have web-based UI tooling for schema submission etc.

In these cases it would be very handy to model situations that would later 
enable us to make statements such as:

* "_Field 'uuid' in record 'namespace.record_name' in the existing schema 
cannot be read by the new schema because it has changed to an incompatible 
type; was 'string', now 'long'_"
* "_Field 'address' in record 'namespace.record_name' in the new schema does 
not exist in the earlier schema and does not declare a default_"
* "_Field union branch 1 at 'namespace.record_name.uField' in the earlier 
schema does not exist in the new schema_"

Certainly these conditions should generated as an accessible model and not 
simply returned as a message so that integrations with editors will be 
possible. With this is mind, would it be possible for 
{{SchemaCompatibilityDetails}} to also encapsulate a machine readable, fully 
qualified path to the incompatible node within the schema tree?

Nice to see the patch and hope to see such a feature present in a future 
version of Avro.



was (Author: teabot):
I agree with the motivations behind this effort. There is a lot of room for 
improvement with regards to the reporting of incompatibilities within schemas. 
This is especially important as the end user focus of Avro Schema development 
shifts from developers to analysts  and I know of a few projects that are 
heading in this direction. For example, one can envisage systems built upon 
Avro that have web-based UI tooling for schema submission etc.

In these cases it would be very handy to model situations that would later 
enable us to make statements such as:

* "_Field 'uuid' in record 'namespace.record_name' in the existing schema 
cannot be read by the new schema because it has changed to an incompatible 
type; was 'string', now 'long'_"
* "_Field 'address' in record 'namespace.record_name' in the new schema does 
not exist in the earlier schema and does not declare a default_"
* "_Field union branch 1 at 'namespace.record_name.uField' in the earlier 
schema does not exist in the new schema_"

Certainly these conditions should generated as an accessible model and not 
simply returned as a message so that integrations with editors will be 
possible. With this is mind, would it be possible for 
{{SchemaCompatibilityDetails}} to also encapsulate a fully qualified path to 
the incompatible node within the schema tree?

Nice to see the patch and hope to see such a feature present in a future 
version of Avro.


> SchemaCompatibility class could be more user-friendly about incompatibilities
> -
>
> Key: AVRO-1933
> URL: https://issues.apache.org/jira/browse/AVRO-1933
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.1
> Environment: Any Java env
>Reporter: Anders Sundelin
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: AVRO-1933-compatible-with-AVRO-1931.patch, 
> AVRO-1933.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Today, the class SchemaCompatibility reports incompatibilities with quite 
> little detail. The whole reader and the whole writer schema is listed, and no 
> particular detail about what was incompatible.
> The attached patch fixes this, introducing a new enum 
> (SchemaIncompatibilityType), and more specific sub-schemas that were 
> incompatible.
> The old, overall picture, is still there - the new compatibility state is 
> encapsulated in the SchemaCompatibilityDetails class.
> Lots of test cases have been added, and there has been refactoring done in 
> the TestSchemaCompatibility and other test classes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AVRO-1997) Avro Field.defaultVal broken for Fixed fields.

2017-02-15 Thread Zoltan Farkas (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867909#comment-15867909
 ] 

Zoltan Farkas commented on AVRO-1997:
-

I see this JIRA has been assigned to me, however I am not sure what do I have 
to do next... See my previous comment with the pull request... Somebody should 
review the pull request and merge it in...

> Avro Field.defaultVal broken for Fixed fields.
> --
>
> Key: AVRO-1997
> URL: https://issues.apache.org/jira/browse/AVRO-1997
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.1, 1.8.2
>Reporter: Zoltan Farkas
>Assignee: Zoltan Farkas
>
> here is a unit test to reproduce the issue:
> {code}
> package org.apache.avro;
> import java.nio.ByteBuffer;
> import org.junit.Assert;
> import org.junit.Test;
> public class TestFixed {
>   @Test
>   public void testFixedDefaultValueDrop() {
> Schema md5 = SchemaBuilder.builder().fixed("MD5").size(16);
> Schema frec = SchemaBuilder.builder().record("test")
> .fields().name("hash").type(md5).withDefault(ByteBuffer.wrap(new 
> byte[16])).endRecord();
> Schema.Field field = frec.getField("hash");
> Assert.assertNotNull(field.defaultVal());
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (AVRO-1933) SchemaCompatibility class could be more user-friendly about incompatibilities

2017-02-15 Thread Elliot West (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867887#comment-15867887
 ] 

Elliot West edited comment on AVRO-1933 at 2/15/17 2:19 PM:


I agree with the motivations behind this effort. There is a lot of room for 
improvement with regards to the reporting of incompatibilities within schemas. 
This is especially important as the end user focus of Avro Schema development 
shifts from developers to analysts  and I know of a few projects that are 
heading in this direction. For example, one can envisage systems built upon 
Avro that have web-based UI tooling for schema submission etc.

In these cases it would be very handy to model situations that would later 
enable us to make statements such as:

* "_Field 'uuid' in record 'namespace.record_name' in the existing schema 
cannot be read by the new schema because it has changed to an incompatible 
type; was 'string', now 'long'_"
* "_Field 'address' in record 'namespace.record_name' in the new schema does 
not exist in the earlier schema and does not declare a default_"
* "_Field union branch 1 at 'namespace.record_name.uField' in the earlier 
schema does not exist in the new schema_"

Certainly these conditions should generated as an accessible model and not 
simply returned as a message so that integrations with editors will be 
possible. With this is mind, would it be possible for 
{{SchemaCompatibilityDetails}} to also encapsulate a fully qualified path to 
the incompatible node within the schema tree?

Nice to see the patch and hope to see such a feature present in a future 
version of Avro.



was (Author: teabot):
I agree with the motivations behind this effort. There is a lot of room for 
improvement with regards to the reporting of incompatibilities within schemas. 
This is especially important as the end user focus of Avro Schema development 
shifts from developers to analysts  and I know of a few projects that are 
heading in this direction. For example, one can envisage systems built upon 
Avro that have web-based UI tooling for schema submission etc.

In these cases it would be very handy to model situations that would later 
enable us to make statements such as:

* "_Field 'uuid' in record 'namespace.record_name' in the existing schema 
cannot be read by the new schema because it has changed to an incompatible 
type; was 'string', now 'long'_"
* "_Field 'address' in record 'namespace.record_name' in the new schema does 
not exist in the earlier schema and does not declare a default_"
* "_Field union branch 1 at 'namespace.record_name.uField' in the earlier 
schema does not exist in the new schema_"

Certainly these conditions should generated as an accessible model and not 
simply returned as a message so that integrations with editors will be possible.

Although this patch is moving in the right direction I think it could be 
further improved as follows:
# Instead of providing a separate implementation of the compatibility rules, 
extend the implementation provided in 
{{org.apache.avro.io.parsing.ResolvingGrammarGenerator}}, or refactor in such a 
way that they are declared only once.
# Perhaps {{SchemaCompatibilityResult}} would be a more precise class name for 
{{SchemaCompatibilityDetails}}?

Nice to see the patch and hope to see such a feature present in a future 
version of Avro.


> SchemaCompatibility class could be more user-friendly about incompatibilities
> -
>
> Key: AVRO-1933
> URL: https://issues.apache.org/jira/browse/AVRO-1933
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.1
> Environment: Any Java env
>Reporter: Anders Sundelin
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: AVRO-1933-compatible-with-AVRO-1931.patch, 
> AVRO-1933.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Today, the class SchemaCompatibility reports incompatibilities with quite 
> little detail. The whole reader and the whole writer schema is listed, and no 
> particular detail about what was incompatible.
> The attached patch fixes this, introducing a new enum 
> (SchemaIncompatibilityType), and more specific sub-schemas that were 
> incompatible.
> The old, overall picture, is still there - the new compatibility state is 
> encapsulated in the SchemaCompatibilityDetails class.
> Lots of test cases have been added, and there has been refactoring done in 
> the TestSchemaCompatibility and other test classes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (AVRO-1933) SchemaCompatibility class could be more user-friendly about incompatibilities

2017-02-15 Thread Elliot West (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867887#comment-15867887
 ] 

Elliot West edited comment on AVRO-1933 at 2/15/17 2:16 PM:


I agree with the motivations behind this effort. There is a lot of room for 
improvement with regards to the reporting of incompatibilities within schemas. 
This is especially important as the end user focus of Avro Schema development 
shifts from developers to analysts  and I know of a few projects that are 
heading in this direction. For example, one can envisage systems built upon 
Avro that have web-based UI tooling for schema submission etc.

In these cases it would be very handy to model situations that would later 
enable us to make statements such as:

* "_Field 'uuid' in record 'namespace.record_name' in the existing schema 
cannot be read by the new schema because it has changed to an incompatible 
type; was 'string', now 'long'_"
* "_Field 'address' in record 'namespace.record_name' in the new schema does 
not exist in the earlier schema and does not declare a default_"
* "_Field union branch 1 at 'namespace.record_name.uField' in the earlier 
schema does not exist in the new schema_"

Certainly these conditions should generated as an accessible model and not 
simply returned as a message so that integrations with editors will be possible.

Although this patch is moving in the right direction I think it could be 
further improved as follows:
# Instead of providing a separate implementation of the compatibility rules, 
extend the implementation provided in 
{{org.apache.avro.io.parsing.ResolvingGrammarGenerator}}, or refactor in such a 
way that they are declared only once.
# Perhaps {{SchemaCompatibilityResult}} would be a more precise class name for 
{{SchemaCompatibilityDetails}}?

Nice to see the patch and hope to see such a feature present in a future 
version of Avro.



was (Author: teabot):
I agree with the motivations behind this effort. There is a lot of room for 
improvement with regards to the reporting of incompatibilities within schemas. 
This is especially important as the end user focus of Avro Schema development 
shifts from developers to analysts  and I know of a few projects that are 
heading in this direction. For example, one can envisage systems built upon 
Avro that have web-based UI tooling for schema submission etc.

In these cases it would be very handy to model situations that would later 
enable us to make statements such as:

* "_Field 'uuid' in record 'namespace.record_name' in the existing schema 
cannot be read by the new schema because it has changed to an incompatible 
type; was 'string', now 'long'_"
* "_Field 'address' in record 'namespace.record_name' in the new schema does 
not exist in the earlier schema and does not declare a default_"
* "_Field union branch 1 at 'namespace.record_name.uField' in the earlier 
schema does not exist in the new schema_"

Certainly these conditions should generated as an accessible modeled and not 
simply returned as a string message, so that integrations with editors etc. 
will be possible.

Although this patch is moving in the right direction I think it could be 
further improved as follows:
# Instead of providing a separate implementation of the compatibility rules, 
extend the implementation provided in 
{{org.apache.avro.io.parsing.ResolvingGrammarGenerator}}, or refactor in such a 
way that they are declared only once.
# Perhaps {{SchemaCompatibilityResult}} would be a more precise class name for 
{{SchemaCompatibilityDetails}}?

Nice to see the patch and hope to see such a feature present in a future 
version of Avro.


> SchemaCompatibility class could be more user-friendly about incompatibilities
> -
>
> Key: AVRO-1933
> URL: https://issues.apache.org/jira/browse/AVRO-1933
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.1
> Environment: Any Java env
>Reporter: Anders Sundelin
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: AVRO-1933-compatible-with-AVRO-1931.patch, 
> AVRO-1933.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Today, the class SchemaCompatibility reports incompatibilities with quite 
> little detail. The whole reader and the whole writer schema is listed, and no 
> particular detail about what was incompatible.
> The attached patch fixes this, introducing a new enum 
> (SchemaIncompatibilityType), and more specific sub-schemas that were 
> incompatible.
> The old, overall picture, is still there - the new compatibility state is 
> encapsulated in the SchemaCompatibilityDetails class.
> Lots of test cases have been added, and there has been refactoring done in 
> the

[jira] [Comment Edited] (AVRO-1933) SchemaCompatibility class could be more user-friendly about incompatibilities

2017-02-15 Thread Elliot West (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867887#comment-15867887
 ] 

Elliot West edited comment on AVRO-1933 at 2/15/17 2:15 PM:


I agree with the motivations behind this effort. There is a lot of room for 
improvement with regards to the reporting of incompatibilities within schemas. 
This is especially important as the end user focus of Avro Schema development 
shifts from developers to analysts  and I know of a few projects that are 
heading in this direction. For example, one can envisage systems built upon 
Avro that have web-based UI tooling for schema submission etc.

In these cases it would be very handy to model situations that would later 
enable us to make statements such as:

* "_Field 'uuid' in record 'namespace.record_name' in the existing schema 
cannot be read by the new schema because it has changed to an incompatible 
type; was 'string', now 'long'_"
* "_Field 'address' in record 'namespace.record_name' in the new schema does 
not exist in the earlier schema and does not declare a default_"
* "_Field union branch 1 at 'namespace.record_name.uField' in the earlier 
schema does not exist in the new schema_"

Certainly these conditions should generated as an accessible modeled and not 
simply returned as a string message, so that integrations with editors etc. 
will be possible.

Although this patch is moving in the right direction I think it could be 
further improved as follows:
# Instead of providing a separate implementation of the compatibility rules, 
extend the implementation provided in 
{{org.apache.avro.io.parsing.ResolvingGrammarGenerator}}, or refactor in such a 
way that they are declared only once.
# Perhaps {{SchemaCompatibilityResult}} would be a more precise class name for 
{{SchemaCompatibilityDetails}}?

Nice to see the patch and hope to see such a feature present in a future 
version of Avro.



was (Author: teabot):
I agree with the motivations behind this effort. There is a lot of room for 
improvement with regards to the reporting of incompatibilities within schemas. 
This is especially important as the end user focus of Avro Schema development 
shifts from developers to analysts  and I know of a few projects that are 
heading in this direction. For example, one can envisage systems built upon 
Avro that have web-based UI tooling for schema submission etc.

In these cases it would be very handy to model situations that would later 
enable us to make statements such as:

* "_Field 'uuid' in record 'namespace.record_name' in the existing schema 
cannot be read by the new schema because it has changed to an incompatible 
type; was 'string', now 'long'_"
* "_Field 'address' in record 'namespace.record_name' in the new schema does 
not exist in the earlier schema and does not declare a default_"
* "_Field union branch 1 at 'namespace.record_name.uField' in the earlier 
schema does not exist in the new schema_"

Certainly these conditions should generated as an accessible modeled and not 
simply returned as a string message, so that integrations with editors etc. 
will be possible.

Although this patch is moving in the right direction I think it could be 
further improved as follows:
# Instead of providing a separate implementation of the compatibility rules, 
extend the implementation provided in 
{{org.apache.avro.io.parsing.ResolvingGrammarGenerator}}, or refactor in such a 
way that they are declared only once.
# {{SchemaCompatibilityDetails}} should include a property that describes the 
fully qualified path to the incompatible node in the schema tree. Initially 
this would help users navigate to the problematic location in the schema. Later 
it would allow future tooling to highlight the relevant sections of the schema 
source code.
# Perhaps {{SchemaCompatibilityResult}} would be a more precise class name for 
{{SchemaCompatibilityDetails}}.

Nice to see the patch and hope to see such a feature present in a future 
version of Avro.


> SchemaCompatibility class could be more user-friendly about incompatibilities
> -
>
> Key: AVRO-1933
> URL: https://issues.apache.org/jira/browse/AVRO-1933
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.1
> Environment: Any Java env
>Reporter: Anders Sundelin
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: AVRO-1933-compatible-with-AVRO-1931.patch, 
> AVRO-1933.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Today, the class SchemaCompatibility reports incompatibilities with quite 
> little detail. The whole reader and the whole writer schema is listed, and no 
> particular detail about what was incompatible.
> The attached patch fixes this,

[jira] [Comment Edited] (AVRO-1933) SchemaCompatibility class could be more user-friendly about incompatibilities

2017-02-15 Thread Elliot West (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867887#comment-15867887
 ] 

Elliot West edited comment on AVRO-1933 at 2/15/17 2:13 PM:


I agree with the motivations behind this effort. There is a lot of room for 
improvement with regards to the reporting of incompatibilities within schemas. 
This is especially important as the end user focus of Avro Schema development 
shifts from developers to analysts  and I know of a few projects that are 
heading in this direction. For example, one can envisage systems built upon 
Avro that have web-based UI tooling for schema submission etc.

In these cases it would be very handy to model situations that would later 
enable us to make statements such as:

* "_Field 'uuid' in record 'namespace.record_name' in the existing schema 
cannot be read by the new schema because it has changed to an incompatible 
type; was 'string', now 'long'_"
* "_Field 'address' in record 'namespace.record_name' in the new schema does 
not exist in the earlier schema and does not declare a default_"
* "_Field union branch 1 at 'namespace.record_name.uField' in the earlier 
schema does not exist in the new schema_"

Certainly these conditions should generated as an accessible modeled and not 
simply returned as a string message, so that integrations with editors etc. 
will be possible.

Although this patch is moving in the right direction I think it could be 
further improved as follows:
# Instead of providing a separate implementation of the compatibility rules, 
extend the implementation provided in 
{{org.apache.avro.io.parsing.ResolvingGrammarGenerator}}, or refactor in such a 
way that they are declared only once.
# {{SchemaCompatibilityDetails}} should include a property that describes the 
fully qualified path to the incompatible node in the schema tree. Initially 
this would help users navigate to the problematic location in the schema. Later 
it would allow future tooling to highlight the relevant sections of the schema 
source code.
# Perhaps {{SchemaCompatibilityResult}} would be a more precise class name for 
{{SchemaCompatibilityDetails}}.

Nice to see the patch and hope to see such a feature present in a future 
version of Avro.



was (Author: teabot):
I agree with the motivations behind this effort. There is a lot of room for 
improvement with regards to the reporting of incompatibilities within schemas. 
This is especially important as the end user focus of Avro Schema development 
shifts from developers to analysts  and I know of a few projects that are 
heading in this direction. For example, one can envisage systems built upon 
Avro that have web-based UI tooling for schema submission etc.

In these cases it would be very handy to model situations that would later 
enable us to make statements such as:

* "_Field 'uuid' in record 'namespace.record_name' in the existing schema 
cannot be read by the new schema because it has changed to an incompatible 
type; was 'string', now 'long'_"
* "_Field 'address' in record 'namespace.record_name' in the new schema does 
not exist in the earlier schema and does not declare a default_"
* "_Field union branch 1 at 'namespace.record_name.uField' in the earlier 
schema does not exist in the new schema_"

Certainly these conditions should generated as an accessible modeled and not 
simply returned as a string message, so that integrations with editors etc. 
will be possible.

Although this patch is moving in the right direction I think it could be 
further improved as follows:
# Instead of providing a separate implementation of the compatibility rules, 
extend the implementation provided in 
{{org.apache.avro.io.parsing.ResolvingGrammarGenerator}}, or refactor in such a 
way that they are declared only once.
# {{SchemaCompatibilityDetails}} should include a property that describes the 
fully qualified path to the incompatible node in the schema tree. Initially 
this would help users navigate to the problematic location in the schema. Later 
it would allow future tooling to highlight the relevant sections of the schema 
source code.
# Perhaps {{SchemaCompatibilityResult}} would be {{SchemaCompatibilityDetails}} 





> SchemaCompatibility class could be more user-friendly about incompatibilities
> -
>
> Key: AVRO-1933
> URL: https://issues.apache.org/jira/browse/AVRO-1933
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.1
> Environment: Any Java env
>Reporter: Anders Sundelin
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: AVRO-1933-compatible-with-AVRO-1931.patch, 
> AVRO-1933.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Today, the class

[jira] [Commented] (AVRO-1933) SchemaCompatibility class could be more user-friendly about incompatibilities

2017-02-15 Thread Elliot West (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867887#comment-15867887
 ] 

Elliot West commented on AVRO-1933:
---

I agree with the motivations behind this effort. There is a lot of room for 
improvement with regards to the reporting of incompatibilities within schemas. 
This is especially important as the end user focus of Avro Schema development 
shifts from developers to analysts  and I know of a few projects that are 
heading in this direction. For example, one can envisage systems built upon 
Avro that have web-based UI tooling for schema submission etc.

In these cases it would be very handy to model situations that would later 
enable us to make statements such as:

* "_Field 'uuid' in record 'namespace.record_name' in the existing schema 
cannot be read by the new schema because it has changed to an incompatible 
type; was 'string', now 'long'_"
* "_Field 'address' in record 'namespace.record_name' in the new schema does 
not exist in the earlier schema and does not declare a default_"
* "_Field union branch 1 at 'namespace.record_name.uField' in the earlier 
schema does not exist in the new schema_"

Certainly these conditions should generated as an accessible modeled and not 
simply returned as a string message, so that integrations with editors etc. 
will be possible.

Although this patch is moving in the right direction I think it could be 
further improved as follows:
# Instead of providing a separate implementation of the compatibility rules, 
extend the implementation provided in 
{{org.apache.avro.io.parsing.ResolvingGrammarGenerator}}, or refactor in such a 
way that they are declared only once.
# {{SchemaCompatibilityDetails}} should include a property that describes the 
fully qualified path to the incompatible node in the schema tree. Initially 
this would help users navigate to the problematic location in the schema. Later 
it would allow future tooling to highlight the relevant sections of the schema 
source code.
# Perhaps {{SchemaCompatibilityResult}} would be {{SchemaCompatibilityDetails}} 





> SchemaCompatibility class could be more user-friendly about incompatibilities
> -
>
> Key: AVRO-1933
> URL: https://issues.apache.org/jira/browse/AVRO-1933
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.1
> Environment: Any Java env
>Reporter: Anders Sundelin
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: AVRO-1933-compatible-with-AVRO-1931.patch, 
> AVRO-1933.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Today, the class SchemaCompatibility reports incompatibilities with quite 
> little detail. The whole reader and the whole writer schema is listed, and no 
> particular detail about what was incompatible.
> The attached patch fixes this, introducing a new enum 
> (SchemaIncompatibilityType), and more specific sub-schemas that were 
> incompatible.
> The old, overall picture, is still there - the new compatibility state is 
> encapsulated in the SchemaCompatibilityDetails class.
> Lots of test cases have been added, and there has been refactoring done in 
> the TestSchemaCompatibility and other test classes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)