[jira] [Commented] (AVRO-1547) AvroApp Schema Tool

[Lewis John McGibbney (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809423#comment-16809423
 ] 

Lewis John McGibbney commented on AVRO-1547:


[~brianolsen87] go for it :) I never got around to this!

> AvroApp Schema Tool
> ---
>
> Key: AVRO-1547
> URL: https://issues.apache.org/jira/browse/AVRO-1547
> Project: Apache Avro
>  Issue Type: New Feature
>  Components: java
>Reporter: Lewis John McGibbney
>Assignee: Lewis John McGibbney
>Priority: Minor
>
> Over in Gora, I have been thinking for a while that the process of writing 
> JSON data beans is rather time consuming when beans are LARGE.
> I wanted to open this ticket for a while and now only get around to. I 
> proposed to have the following
> A simple HTML webpage that defines a form of sorts, the form will enable 
> users to create JSON schemas and will be driven by enabling users to enter 
> Object values based on the current Avro specification document e.g. ti will 
> be restrictive in scope.
> On top of this I propose to then use simple JQuery to send a request to the 
> JSONBlob API [0], obtain a JSON representation of the data and then pretty 
> print write this information to a file within the browser. The users can then 
> save this file focally and do with it what they wish.
> I think that this page can easily be hosted alongside the current static Avro 
> website and that there is no need to write a web application for this yet.
> I'll try to work on it sooner rather than later as this would also lower the 
> barrier for users of Gora (as I am sure it would Users of other technologies 
> requiring definition of Objects via JSOn schemas).
> I've not assigned this against any component as there is none which I feel 
> appropriate.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AVRO-2367) FieldActionAdjust should include default value

[Anthony Miyaguchi (JIRA)]

Anthony Miyaguchi created AVRO-2367:
---

 Summary: FieldActionAdjust should include default value
 Key: AVRO-2367
 URL: https://issues.apache.org/jira/browse/AVRO-2367
 Project: Apache Avro
  Issue Type: Improvement
  Components: java
Affects Versions: 1.8.2
Reporter: Anthony Miyaguchi


The FieldAdjustAction is an implicit action symbol used within the 
JsonGrammarGenerator. It provides the production rules for reading and writing 
object fields.

 

I am currently writing a custom Json decoder that can handle 
missing/non-required fields based on the reader's schema. With a [small 
workaround|https://github.com/acmiyaguchi/gcp-ingestion/commit/fc6c676dc0784ca3470c3ef19b4896663052ee45],
 I can handle fields where the intended default is null. However, I would like 
to insert the non-null default values. This is not possible because the default 
value in the schemas are missing from the symbol set.

 
{code:java}
{"type": ["null", "boolean"], "default": null} // can be emulated
{"type": ["boolean", "null"], "default": true} // not enough information
{code}
The solution would be to include the default value (JsonNode) as a member of 
the FieldAdjustAction. This would be backwards compatible with the current 
code, while allowing custom decoders to insert default values for union values.

 

[https://github.com/apache/avro/blob/3a7786ed1a3bedf05e6bce7924aea2f75b576912/lang/java/avro/src/main/java/org/apache/avro/io/parsing/JsonGrammarGenerator.java#L85]

[https://github.com/apache/avro/blob/3a7786ed1a3bedf05e6bce7924aea2f75b576912/lang/java/avro/src/main/java/org/apache/avro/io/parsing/Symbol.java#L582-L597]

 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AVRO-2332) Support java_multiple_files protobuf option

[Nandor Kollar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nandor Kollar reassigned AVRO-2332:
---

Assignee: Ryo Okubo

> Support java_multiple_files protobuf option
> ---
>
> Key: AVRO-2332
> URL: https://issues.apache.org/jira/browse/AVRO-2332
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Ryo Okubo
>Assignee: Ryo Okubo
>Priority: Major
>
> Support java_multiple_files option,
> [https://developers.google.com/protocol-buffers/docs/proto3#options]
> The option helps to get classes via reflection.
>  
> protoc doesn't generate outer class if the option enables, but current 
> getNamespace() doesn't consider that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AVRO-2344) Handle google.protobuf.Timestamp as timestamp logical type in Avro

[Nandor Kollar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nandor Kollar reassigned AVRO-2344:
---

Assignee: Ryo Okubo

> Handle google.protobuf.Timestamp as timestamp logical type in Avro 
> ---
>
> Key: AVRO-2344
> URL: https://issues.apache.org/jira/browse/AVRO-2344
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Reporter: Ryo Okubo
>Assignee: Ryo Okubo
>Priority: Major
>
> Current avro-protobuf converts google.protobuf.Timestamp to simply record but 
> I'd like to get timestamp logical type. 
>  
> refs"
>  * proto 
> [https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/timestamp.proto]
>  * avro [https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2200) Option to fail when extra fields are in the payload

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808545#comment-16808545
 ] 

Hudson commented on AVRO-2200:
--

FAILURE: Integrated in Jenkins build AvroJava #636 (See 
[https://builds.apache.org/job/AvroJava/636/])
AVRO-2200: Option to fail when extra fields are in the payload (#321) (fokko: 
[https://github.com/apache/avro/commit/3a7786ed1a3bedf05e6bce7924aea2f75b576912])
* (edit) lang/ruby/lib/avro/schema_validator.rb
* (edit) lang/ruby/test/test_schema_validator.rb


> Option to fail when extra fields are in the payload
> ---
>
> Key: AVRO-2200
> URL: https://issues.apache.org/jira/browse/AVRO-2200
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: ruby
>Reporter: Daniel Orner
>Assignee: Tim Perkins
>Priority: Minor
> Fix For: 1.9.0
>
>
> Currently, when passing a datum into the schema validator, the datum will 
> pass even if the datum has more fields than the schema does. This can be 
> dangerous because the client code is assuming that the schema has these 
> values, but they are actually being silently swallowed when they are encoded.
> I'd like to open a PR where we can provide an option to make the validator 
> fail on any extra fields. See https://github.com/salsify/avro-patches/pull/15



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: AvroJava #636

[Apache Jenkins Server]

See 

Changes:

[Fokko] AVRO-2200: Option to fail when extra fields are in the payload (#321)

--
[...truncated 22.28 KB...]
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ avro 
---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:3.0.0-M3:test (default-test) @ avro ---
[INFO] 
[INFO] ---
[INFO]  T E S T S
[INFO] ---
[INFO] Running org.apache.avro.generic.TestGenericData
[INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.066 s 
- in org.apache.avro.generic.TestGenericData
[INFO] Running org.apache.avro.reflect.TestReflect
[INFO] Tests run: 68, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.273 s 
- in org.apache.avro.reflect.TestReflect
[INFO] Running org.apache.avro.io.parsing.TestResolvingGrammarGenerator
[INFO] Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.13 s 
- in org.apache.avro.io.parsing.TestResolvingGrammarGenerator
[INFO] Running org.apache.avro.util.internal.TestJacksonUtils
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017 s 
- in org.apache.avro.util.internal.TestJacksonUtils
[INFO] Running org.apache.avro.TestSchemaCompatibilityMissingEnumSymbols
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 s 
- in org.apache.avro.TestSchemaCompatibilityMissingEnumSymbols
[INFO] Running org.apache.avro.specific.TestSpecificToFromByteArray
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.187 s 
- in org.apache.avro.specific.TestSpecificToFromByteArray
[INFO] Running org.apache.avro.file.TestSeekableByteArrayInput
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in 
org.apache.avro.file.TestSeekableByteArrayInput
[INFO] Running org.apache.avro.generic.TestGenericRecordBuilder
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 s 
- in org.apache.avro.generic.TestGenericRecordBuilder
[INFO] Running org.apache.avro.specific.TestSpecificData
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.026 s 
- in org.apache.avro.specific.TestSpecificData
[INFO] Running org.apache.avro.TestSchemaValidation
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.098 s 
- in org.apache.avro.TestSchemaValidation
[INFO] Running org.apache.avro.file.TestZstandardCodec
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in 
org.apache.avro.file.TestZstandardCodec
[INFO] Running org.apache.avro.TestSchemaCompatibilityTypeMismatch
[INFO] Tests run: 29, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.042 s 
- in org.apache.avro.TestSchemaCompatibilityTypeMismatch
[INFO] Running org.apache.avro.TestDataFileCorruption
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.019 s 
- in org.apache.avro.TestDataFileCorruption
[INFO] Running org.apache.avro.TestDataFileDeflate
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s 
- in org.apache.avro.TestDataFileDeflate
[INFO] Running org.apache.avro.io.TestJsonDecoder
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 s 
- in org.apache.avro.io.TestJsonDecoder
[INFO] Running org.apache.avro.TestLogicalType
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.033 s 
- in org.apache.avro.TestLogicalType
[INFO] Running org.apache.avro.io.parsing.TestResolvingGrammarGenerator2
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 s 
- in org.apache.avro.io.parsing.TestResolvingGrammarGenerator2
[INFO] Running org.apache.avro.io.TestValidatingIO
[INFO] Tests run: 972, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.478 
s - in org.apache.avro.io.TestValidatingIO
[INFO] Running 
org.apache.avro.TestSchemaCompatibilityReaderFieldMissingDefaultValue
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in 
org.apache.avro.TestSchemaCompatibilityReaderFieldMissingDefaultValue
[INFO] Running org.apache.avro.TestDataFileConcat
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.017 s 
- in org.apache.avro.TestDataFileConcat
[INFO] Running org.apache.avro.io.TestResolvingIO
[INFO] Tests run: 816, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.946 
s - in org.apache.avro.io.TestResolvingIO
[INFO] Running org.apache.avro.reflect.TestNonStringMapKeys
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s 
- in org.apache.avro.reflect.TestNonStringMapKeys
[INFO] Running org.apache.avro.data.TestJodaTimeConversions
[INFO] Tests run: 12, Failures: 0, Errors: 0, 

[jira] [Commented] (AVRO-2200) Option to fail when extra fields are in the payload

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808488#comment-16808488
 ] 

ASF subversion and git services commented on AVRO-2200:
---

Commit 3a7786ed1a3bedf05e6bce7924aea2f75b576912 in avro's branch 
refs/heads/master from Daniel Orner
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=3a7786e ]

AVRO-2200: Option to fail when extra fields are in the payload (#321)

* AVRO-2200: Option to fail when extra fields are in the payload

* Check for extra fields recursively


> Option to fail when extra fields are in the payload
> ---
>
> Key: AVRO-2200
> URL: https://issues.apache.org/jira/browse/AVRO-2200
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: ruby
>Reporter: Daniel Orner
>Assignee: Tim Perkins
>Priority: Minor
> Fix For: 1.9.0
>
>
> Currently, when passing a datum into the schema validator, the datum will 
> pass even if the datum has more fields than the schema does. This can be 
> dangerous because the client code is assuming that the schema has these 
> values, but they are actually being silently swallowed when they are encoded.
> I'd like to open a PR where we can provide an option to make the validator 
> fail on any extra fields. See https://github.com/salsify/avro-patches/pull/15



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2200) Option to fail when extra fields are in the payload

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808489#comment-16808489
 ] 

ASF subversion and git services commented on AVRO-2200:
---

Commit 3a7786ed1a3bedf05e6bce7924aea2f75b576912 in avro's branch 
refs/heads/master from Daniel Orner
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=3a7786e ]

AVRO-2200: Option to fail when extra fields are in the payload (#321)

* AVRO-2200: Option to fail when extra fields are in the payload

* Check for extra fields recursively


> Option to fail when extra fields are in the payload
> ---
>
> Key: AVRO-2200
> URL: https://issues.apache.org/jira/browse/AVRO-2200
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: ruby
>Reporter: Daniel Orner
>Assignee: Tim Perkins
>Priority: Minor
> Fix For: 1.9.0
>
>
> Currently, when passing a datum into the schema validator, the datum will 
> pass even if the datum has more fields than the schema does. This can be 
> dangerous because the client code is assuming that the schema has these 
> values, but they are actually being silently swallowed when they are encoded.
> I'd like to open a PR where we can provide an option to make the validator 
> fail on any extra fields. See https://github.com/salsify/avro-patches/pull/15



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AVRO-2200) Option to fail when extra fields are in the payload

[Fokko Driesprong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong resolved AVRO-2200.

   Resolution: Fixed
Fix Version/s: 1.9.0

> Option to fail when extra fields are in the payload
> ---
>
> Key: AVRO-2200
> URL: https://issues.apache.org/jira/browse/AVRO-2200
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: ruby
>Reporter: Daniel Orner
>Assignee: Tim Perkins
>Priority: Minor
> Fix For: 1.9.0
>
>
> Currently, when passing a datum into the schema validator, the datum will 
> pass even if the datum has more fields than the schema does. This can be 
> dangerous because the client code is assuming that the schema has these 
> values, but they are actually being silently swallowed when they are encoded.
> I'd like to open a PR where we can provide an option to make the validator 
> fail on any extra fields. See https://github.com/salsify/avro-patches/pull/15



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AVRO-2200) Option to fail when extra fields are in the payload

[Fokko Driesprong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong reassigned AVRO-2200:
--

Assignee: Fokko Driesprong

> Option to fail when extra fields are in the payload
> ---
>
> Key: AVRO-2200
> URL: https://issues.apache.org/jira/browse/AVRO-2200
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: ruby
>Reporter: Daniel Orner
>Assignee: Fokko Driesprong
>Priority: Minor
>
> Currently, when passing a datum into the schema validator, the datum will 
> pass even if the datum has more fields than the schema does. This can be 
> dangerous because the client code is assuming that the schema has these 
> values, but they are actually being silently swallowed when they are encoded.
> I'd like to open a PR where we can provide an option to make the validator 
> fail on any extra fields. See https://github.com/salsify/avro-patches/pull/15



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AVRO-2200) Option to fail when extra fields are in the payload

[Fokko Driesprong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong reassigned AVRO-2200:
--

Assignee: Tim Perkins  (was: Fokko Driesprong)

> Option to fail when extra fields are in the payload
> ---
>
> Key: AVRO-2200
> URL: https://issues.apache.org/jira/browse/AVRO-2200
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: ruby
>Reporter: Daniel Orner
>Assignee: Tim Perkins
>Priority: Minor
>
> Currently, when passing a datum into the schema validator, the datum will 
> pass even if the datum has more fields than the schema does. This can be 
> dangerous because the client code is assuming that the schema has these 
> values, but they are actually being silently swallowed when they are encoded.
> I'd like to open a PR where we can provide an option to make the validator 
> fail on any extra fields. See https://github.com/salsify/avro-patches/pull/15



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)