[jira] [Commented] (AVRO-2078) Avro does not enforce schema resolution rules for Decimal type
[
https://issues.apache.org/jira/browse/AVRO-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175799#comment-16175799
]
Anthony Hsu commented on AVRO-2078:
---
[~nkollar], thanks for looking at this! Your pull request on GitHub looks good
to me.
> Avro does not enforce schema resolution rules for Decimal type
> --
>
> Key: AVRO-2078
> URL: https://issues.apache.org/jira/browse/AVRO-2078
> Project: Avro
> Issue Type: Bug
>Reporter: Anthony Hsu
>Assignee: Nandor Kollar
> Fix For: 1.8.2
>
> Attachments: dec.avro
>
>
> According to http://avro.apache.org/docs/1.8.2/spec.html#Decimal
> bq. For the purposes of schema resolution, two schemas that are {{decimal}}
> logical types _match_ if their scales and precisions match.
> This is not enforced.
> I wrote a file with (precision 5, scale 2) and tried to read it with a reader
> schema with (precision 3, scale 1). I expected an AvroTypeException to be
> thrown, but none was thrown.
> Test data file attached. The code to read it is:
> {noformat:title=ReadDecimal.java}
> import java.io.File;
> import org.apache.avro.Schema;
> import org.apache.avro.file.DataFileReader;
> import org.apache.avro.generic.GenericDatumReader;
> import org.apache.avro.generic.GenericRecord;
> import org.apache.avro.io.DatumReader;
> public class ReadDecimal {
> public static void main(String[] args) throws Exception {
> Schema schema = new Schema.Parser().parse("{\n" + " \"type\" :
> \"record\",\n" + " \"name\" : \"some_schema\",\n"
> + " \"namespace\" : \"com.howdy\",\n" + " \"fields\" : [ {\n" + "
> \"name\" : \"name\",\n"
> + "\"type\" : \"string\"\n" + " }, {\n" + "\"name\" :
> \"value\",\n" + "\"type\" : {\n"
> + " \"type\" : \"bytes\",\n" + " \"logicalType\" :
> \"decimal\",\n" + " \"precision\" : 3,\n"
> + " \"scale\" : 1\n" + "}\n" + " } ]\n" + "}");
> DatumReader datumReader = new GenericDatumReader<>(schema);
> // dec.avro has precision 5, scale 2
> DataFileReader dataFileReader = new DataFileReader<>(
> new File("/tmp/dec.avro"), datumReader);
> GenericRecord foo = null;
> while (dataFileReader.hasNext()) {
> foo = dataFileReader.next(foo); // AvroTypeException expected due to
> change in scale/precision but none occurs
> }
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-2078) Avro does not enforce schema resolution rules for Decimal type
[
https://issues.apache.org/jira/browse/AVRO-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-2078:
Assignee: Nandor Kollar
Status: Patch Available (was: Open)
> Avro does not enforce schema resolution rules for Decimal type
> --
>
> Key: AVRO-2078
> URL: https://issues.apache.org/jira/browse/AVRO-2078
> Project: Avro
> Issue Type: Bug
>Reporter: Anthony Hsu
>Assignee: Nandor Kollar
> Fix For: 1.8.2
>
> Attachments: dec.avro
>
>
> According to http://avro.apache.org/docs/1.8.2/spec.html#Decimal
> bq. For the purposes of schema resolution, two schemas that are {{decimal}}
> logical types _match_ if their scales and precisions match.
> This is not enforced.
> I wrote a file with (precision 5, scale 2) and tried to read it with a reader
> schema with (precision 3, scale 1). I expected an AvroTypeException to be
> thrown, but none was thrown.
> Test data file attached. The code to read it is:
> {noformat:title=ReadDecimal.java}
> import java.io.File;
> import org.apache.avro.Schema;
> import org.apache.avro.file.DataFileReader;
> import org.apache.avro.generic.GenericDatumReader;
> import org.apache.avro.generic.GenericRecord;
> import org.apache.avro.io.DatumReader;
> public class ReadDecimal {
> public static void main(String[] args) throws Exception {
> Schema schema = new Schema.Parser().parse("{\n" + " \"type\" :
> \"record\",\n" + " \"name\" : \"some_schema\",\n"
> + " \"namespace\" : \"com.howdy\",\n" + " \"fields\" : [ {\n" + "
> \"name\" : \"name\",\n"
> + "\"type\" : \"string\"\n" + " }, {\n" + "\"name\" :
> \"value\",\n" + "\"type\" : {\n"
> + " \"type\" : \"bytes\",\n" + " \"logicalType\" :
> \"decimal\",\n" + " \"precision\" : 3,\n"
> + " \"scale\" : 1\n" + "}\n" + " } ]\n" + "}");
> DatumReader datumReader = new GenericDatumReader<>(schema);
> // dec.avro has precision 5, scale 2
> DataFileReader dataFileReader = new DataFileReader<>(
> new File("/tmp/dec.avro"), datumReader);
> GenericRecord foo = null;
> while (dataFileReader.hasNext()) {
> foo = dataFileReader.next(foo); // AvroTypeException expected due to
> change in scale/precision but none occurs
> }
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-2078) Avro does not enforce schema resolution rules for Decimal type
[
https://issues.apache.org/jira/browse/AVRO-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174738#comment-16174738
]
Nandor Kollar commented on AVRO-2078:
-
Looks like a bug for me, the schema resolution logic is not compliant with the
spec. Created a PR with my proposed modifications.
> Avro does not enforce schema resolution rules for Decimal type
> --
>
> Key: AVRO-2078
> URL: https://issues.apache.org/jira/browse/AVRO-2078
> Project: Avro
> Issue Type: Bug
>Reporter: Anthony Hsu
> Fix For: 1.8.2
>
> Attachments: dec.avro
>
>
> According to http://avro.apache.org/docs/1.8.2/spec.html#Decimal
> bq. For the purposes of schema resolution, two schemas that are {{decimal}}
> logical types _match_ if their scales and precisions match.
> This is not enforced.
> I wrote a file with (precision 5, scale 2) and tried to read it with a reader
> schema with (precision 3, scale 1). I expected an AvroTypeException to be
> thrown, but none was thrown.
> Test data file attached. The code to read it is:
> {noformat:title=ReadDecimal.java}
> import java.io.File;
> import org.apache.avro.Schema;
> import org.apache.avro.file.DataFileReader;
> import org.apache.avro.generic.GenericDatumReader;
> import org.apache.avro.generic.GenericRecord;
> import org.apache.avro.io.DatumReader;
> public class ReadDecimal {
> public static void main(String[] args) throws Exception {
> Schema schema = new Schema.Parser().parse("{\n" + " \"type\" :
> \"record\",\n" + " \"name\" : \"some_schema\",\n"
> + " \"namespace\" : \"com.howdy\",\n" + " \"fields\" : [ {\n" + "
> \"name\" : \"name\",\n"
> + "\"type\" : \"string\"\n" + " }, {\n" + "\"name\" :
> \"value\",\n" + "\"type\" : {\n"
> + " \"type\" : \"bytes\",\n" + " \"logicalType\" :
> \"decimal\",\n" + " \"precision\" : 3,\n"
> + " \"scale\" : 1\n" + "}\n" + " } ]\n" + "}");
> DatumReader datumReader = new GenericDatumReader<>(schema);
> // dec.avro has precision 5, scale 2
> DataFileReader dataFileReader = new DataFileReader<>(
> new File("/tmp/dec.avro"), datumReader);
> GenericRecord foo = null;
> while (dataFileReader.hasNext()) {
> foo = dataFileReader.next(foo); // AvroTypeException expected due to
> change in scale/precision but none occurs
> }
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
