[jira] [Commented] (AVRO-1991) Java concat tool -appendToFirst option doesn't work with local filesystem

2017-06-26 Thread Sean Busbey (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063540#comment-16063540
 ] 

Sean Busbey commented on AVRO-1991:
---

this is a great question. I can remember the jira that added it to the tool, 
but similarly can find no proof of this option now. ¯\_(ツ)_/¯

> Java concat tool -appendToFirst option doesn't work with local filesystem
> -
>
> Key: AVRO-1991
> URL: https://issues.apache.org/jira/browse/AVRO-1991
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0
>Reporter: Sean Busbey
>
> We should either make it work (e.g. by switching to RawLocalFilesytem instead 
> of LocalFileSystem) or update the cli help.
> {code}
>  java -jar lang/java/tools/target/avro-tools-1.9.0-SNAPSHOT.jar concat 
> -appendToFirst ~/Downloads/append.0.avro 
> '/Users/busbey/Downloads/example.*.avro'
> log4j:WARN No appenders could be found for logger 
> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> Exception in thread "main" java.io.IOException: Not supported
>   at 
> org.apache.hadoop.fs.ChecksumFileSystem.append(ChecksumFileSystem.java:352)
>   at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1161)
>   at org.apache.avro.tool.Util.createFromFS(Util.java:150)
>   at org.apache.avro.tool.Util.fileOrStdout(Util.java:76)
>   at org.apache.avro.tool.ConcatTool.run(ConcatTool.java:76)
>   at org.apache.avro.tool.Main.run(Main.java:87)
>   at org.apache.avro.tool.Main.main(Main.java:76)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AVRO-2035) enable validation of default values in schemas by default

2017-06-26 Thread Doug Cutting (JIRA) 

 [ 
https://issues.apache.org/jira/browse/AVRO-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Cutting updated AVRO-2035:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> enable validation of default values in schemas by default
> -
>
> Key: AVRO-2035
> URL: https://issues.apache.org/jira/browse/AVRO-2035
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.1
>Reporter: radai rosenblatt
>Assignee: Doug Cutting
> Fix For: 1.9.0
>
> Attachments: AVRO-2035.patch
>
>
> suppose i have the following schema evolution:
> {code}
> {
>   "name": "Bob",
>   "type": "record",
>   "fields": [
> {"name": "f1", "type": "int"}
>   ]
> }
> {code}
> and then:
> {code}
> {
>   "name": "Bob",
>   "type": "record",
>   "fields": [
> {"name": "f1", "type": "int"},
> {"name": "f2", "type": "boolean", "default": "true"}
>   ]
> }
> {code}
> the default value for "f2" is specified as the _STRING_ "true" (and not the 
> literal boolean true). 
> if this default value is ever accessed (when reading a gen1-serialized object 
> as a gen2) we get this:
> {code}
> org.apache.avro.AvroTypeException: Non-boolean default for boolean: "true"
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:408)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.getBinary(ResolvingGrammarGenerator.java:307)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.resolveRecords(ResolvingGrammarGenerator.java:285)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:118)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:50)
>   at org.apache.avro.io.ResolvingDecoder.resolve(ResolvingDecoder.java:85)
>   at org.apache.avro.io.ResolvingDecoder.(ResolvingDecoder.java:49)
>   at 
> org.apache.avro.io.DecoderFactory.resolvingDecoder(DecoderFactory.java:307)
>   at 
> org.apache.avro.generic.GenericDatumReader.getResolver(GenericDatumReader.java:127)
>   at 
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
> {code}
> yet Schema.parse() passes for this



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AVRO-2035) enable validation of default values in schemas by default

2017-06-26 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063300#comment-16063300
 ] 

ASF subversion and git services commented on AVRO-2035:
---

Commit e5ae42e5703b8b193e5c2a61458e28a931c5774e in avro's branch 
refs/heads/master from [~cutting]
[ https://git-wip-us.apache.org/repos/asf?p=avro.git;h=e5ae42e ]

AVRO-2035: Java: validate default values when parsing schemas.


> enable validation of default values in schemas by default
> -
>
> Key: AVRO-2035
> URL: https://issues.apache.org/jira/browse/AVRO-2035
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.1
>Reporter: radai rosenblatt
>Assignee: Doug Cutting
> Fix For: 1.9.0
>
> Attachments: AVRO-2035.patch
>
>
> suppose i have the following schema evolution:
> {code}
> {
>   "name": "Bob",
>   "type": "record",
>   "fields": [
> {"name": "f1", "type": "int"}
>   ]
> }
> {code}
> and then:
> {code}
> {
>   "name": "Bob",
>   "type": "record",
>   "fields": [
> {"name": "f1", "type": "int"},
> {"name": "f2", "type": "boolean", "default": "true"}
>   ]
> }
> {code}
> the default value for "f2" is specified as the _STRING_ "true" (and not the 
> literal boolean true). 
> if this default value is ever accessed (when reading a gen1-serialized object 
> as a gen2) we get this:
> {code}
> org.apache.avro.AvroTypeException: Non-boolean default for boolean: "true"
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:408)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.getBinary(ResolvingGrammarGenerator.java:307)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.resolveRecords(ResolvingGrammarGenerator.java:285)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:118)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:50)
>   at org.apache.avro.io.ResolvingDecoder.resolve(ResolvingDecoder.java:85)
>   at org.apache.avro.io.ResolvingDecoder.(ResolvingDecoder.java:49)
>   at 
> org.apache.avro.io.DecoderFactory.resolvingDecoder(DecoderFactory.java:307)
>   at 
> org.apache.avro.generic.GenericDatumReader.getResolver(GenericDatumReader.java:127)
>   at 
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
> {code}
> yet Schema.parse() passes for this



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AVRO-1991) Java concat tool -appendToFirst option doesn't work with local filesystem

2017-06-26 Thread Nandor Kollar (JIRA) 

[ 
https://issues.apache.org/jira/browse/AVRO-1991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063060#comment-16063060
 ] 

Nandor Kollar commented on AVRO-1991:
-

[~busbey] is -appendToFirst a valid option for concat? I couldn't find this 
string in the project, and the example you mentioned here fails with a 
different exception: Exception in thread "main" java.io.FileNotFoundException: 
-appendFirst

The documentation of concat doesn't mention appendToFirst option:
{code}
concat [input-file...] output-file

Concatenates one or more input files into a new output file
by appending the input blocks without decoding them. The input
files must have the same schema, metadata and codec. If they
do not the tool will return the following error codes:
  1 if the schemas don't match
  2 if the metadata doesn't match
  3 if the codecs don't match
If no input files are given stdin will be used. The tool
0 on success. A dash ('-') can be given as an input file
to use stdin, and as an output file to use stdout. If a directory
is given as an input-file all the files within this directory
are used.
{code}

> Java concat tool -appendToFirst option doesn't work with local filesystem
> -
>
> Key: AVRO-1991
> URL: https://issues.apache.org/jira/browse/AVRO-1991
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.9.0
>Reporter: Sean Busbey
>
> We should either make it work (e.g. by switching to RawLocalFilesytem instead 
> of LocalFileSystem) or update the cli help.
> {code}
>  java -jar lang/java/tools/target/avro-tools-1.9.0-SNAPSHOT.jar concat 
> -appendToFirst ~/Downloads/append.0.avro 
> '/Users/busbey/Downloads/example.*.avro'
> log4j:WARN No appenders could be found for logger 
> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> Exception in thread "main" java.io.IOException: Not supported
>   at 
> org.apache.hadoop.fs.ChecksumFileSystem.append(ChecksumFileSystem.java:352)
>   at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1161)
>   at org.apache.avro.tool.Util.createFromFS(Util.java:150)
>   at org.apache.avro.tool.Util.fileOrStdout(Util.java:76)
>   at org.apache.avro.tool.ConcatTool.run(ConcatTool.java:76)
>   at org.apache.avro.tool.Main.run(Main.java:87)
>   at org.apache.avro.tool.Main.main(Main.java:76)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)