[jira] [Commented] (AVRO-2036) Avro fails to compile for C
[
https://issues.apache.org/jira/browse/AVRO-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16857130#comment-16857130
]
Anthony Hsu commented on AVRO-2036:
---
I noticed this too while trying to build the Avro C library from source using
Bazel for a Bazel project. Seems like [avro CMakeLists.txt
file|https://github.com/apache/avro/blob/master/lang/c/src/CMakeLists.txt]
purposely excludes {{schema_specific.c}}, where the compilation error happens.
After I exclude {{schema_specific.c}} from the build, the build succeeds.
> Avro fails to compile for C
> ---
>
> Key: AVRO-2036
> URL: https://issues.apache.org/jira/browse/AVRO-2036
> Project: Apache Avro
> Issue Type: Bug
> Components: c
>Affects Versions: 1.8.2
> Environment: Linux
>Reporter: Don Snedigar
>Priority: Major
>
> AVRO for C fails to compile.
> Could be duplicate of AVRO-778
> First instance of failure (of many) occures in avro_schema_to_header :
> static int avro_schema_to_header(avro_schema_t schema, specific_ctx * ctx)
> {
> size_t i;
> FILE *fp = ctx->header;
> indent(ctx, fp);
> ctx->depth++;
> if (is_avro_primitive(schema) && !ctx->name) {
>
> ctx has no member name.
> The definition of the struct above is :
> enum specific_state {
> START_STATE,
> };
> typedef enum specific_state specific_state;
> struct specific_ctx {
> FILE *header;
> FILE *source;
> int depth;
> specific_state state;
> };
> typedef struct specific_ctx specific_ctx;
> So every instance of ctx->name fails.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-1741) Python 3 code should check if avro.codec is None before decoding
[
https://issues.apache.org/jira/browse/AVRO-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726175#comment-16726175
]
Anthony Hsu commented on AVRO-1741:
---
This Jira can be resolved. The fix is committed
[here|https://github.com/apache/avro/commit/191badb80281498611ffcbc0e79b9b020a3d5096#diff-4c24d0cfd2675026f0b87b9bfcaac54e]
and present in avro-python3 1.8.2.
> Python 3 code should check if avro.codec is None before decoding
>
>
> Key: AVRO-1741
> URL: https://issues.apache.org/jira/browse/AVRO-1741
> Project: Apache Avro
> Issue Type: Bug
>Reporter: Matthew Hayes
>Assignee: Matthew Hayes
>Priority: Major
> Fix For: 1.7.7
>
> Attachments: AVRO-1741.diff
>
>
> I'm getting the error below when trying to read an Avro file in Python 3.
> FWIW, I generated this Avro file from Pig. I don't know if that makes a
> difference.
> {code}
> avro.datafile.DataFileReader(open("test.avro","rb"),avro.io.DatumReader())
> Traceback (most recent call last):
> File "", line 1, in
> File
> "/Users/mhayes/.pyenv/versions/3.4.2/lib/python3.4/site-packages/avro/datafile.py",
> line 352, in __init__
> self.codec = self.GetMeta('avro.codec').decode('utf-8')
> AttributeError: 'NoneType' object has no attribute 'decode'
> {code}
> It appears the problem is in the following code. It should check if GetMeta
> returns None before trying to decode it. The Python 2 code doesn't have this
> problem because it is already decoded.
> {code}
> # ensure codec is valid
> self.codec = self.GetMeta('avro.codec').decode('utf-8')
> if self.codec is None:
> self.codec = "null"
> if self.codec not in VALID_CODECS:
> raise DataFileException('Unknown codec: %s.' % self.codec)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2078) Avro does not enforce schema resolution rules for Decimal type
[
https://issues.apache.org/jira/browse/AVRO-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175799#comment-16175799
]
Anthony Hsu commented on AVRO-2078:
---
[~nkollar], thanks for looking at this! Your pull request on GitHub looks good
to me.
> Avro does not enforce schema resolution rules for Decimal type
> --
>
> Key: AVRO-2078
> URL: https://issues.apache.org/jira/browse/AVRO-2078
> Project: Avro
> Issue Type: Bug
>Reporter: Anthony Hsu
>Assignee: Nandor Kollar
> Fix For: 1.8.2
>
> Attachments: dec.avro
>
>
> According to http://avro.apache.org/docs/1.8.2/spec.html#Decimal
> bq. For the purposes of schema resolution, two schemas that are {{decimal}}
> logical types _match_ if their scales and precisions match.
> This is not enforced.
> I wrote a file with (precision 5, scale 2) and tried to read it with a reader
> schema with (precision 3, scale 1). I expected an AvroTypeException to be
> thrown, but none was thrown.
> Test data file attached. The code to read it is:
> {noformat:title=ReadDecimal.java}
> import java.io.File;
> import org.apache.avro.Schema;
> import org.apache.avro.file.DataFileReader;
> import org.apache.avro.generic.GenericDatumReader;
> import org.apache.avro.generic.GenericRecord;
> import org.apache.avro.io.DatumReader;
> public class ReadDecimal {
> public static void main(String[] args) throws Exception {
> Schema schema = new Schema.Parser().parse("{\n" + " \"type\" :
> \"record\",\n" + " \"name\" : \"some_schema\",\n"
> + " \"namespace\" : \"com.howdy\",\n" + " \"fields\" : [ {\n" + "
> \"name\" : \"name\",\n"
> + "\"type\" : \"string\"\n" + " }, {\n" + "\"name\" :
> \"value\",\n" + "\"type\" : {\n"
> + " \"type\" : \"bytes\",\n" + " \"logicalType\" :
> \"decimal\",\n" + " \"precision\" : 3,\n"
> + " \"scale\" : 1\n" + "}\n" + " } ]\n" + "}");
> DatumReader datumReader = new GenericDatumReader<>(schema);
> // dec.avro has precision 5, scale 2
> DataFileReader dataFileReader = new DataFileReader<>(
> new File("/tmp/dec.avro"), datumReader);
> GenericRecord foo = null;
> while (dataFileReader.hasNext()) {
> foo = dataFileReader.next(foo); // AvroTypeException expected due to
> change in scale/precision but none occurs
> }
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-2078) Avro does not enforce schema resolution rules for Decimal type
[
https://issues.apache.org/jira/browse/AVRO-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anthony Hsu updated AVRO-2078:
--
Attachment: dec.avro
> Avro does not enforce schema resolution rules for Decimal type
> --
>
> Key: AVRO-2078
> URL: https://issues.apache.org/jira/browse/AVRO-2078
> Project: Avro
> Issue Type: Bug
>Reporter: Anthony Hsu
> Fix For: 1.8.2
>
> Attachments: dec.avro
>
>
> According to http://avro.apache.org/docs/1.8.2/spec.html#Decimal
> bq. For the purposes of schema resolution, two schemas that are {{decimal}}
> logical types _match_ if their scales and precisions match.
> This is not enforced.
> I wrote a file with (precision 5, scale 2) and tried to read it with a reader
> schema with (precision 3, scale 1). I expected an AvroTypeException to be
> thrown, but none was thrown.
> Test data file attached. The code to read it is:
> {noformat:title=ReadDecimal.java}
> import java.io.File;
> import org.apache.avro.Schema;
> import org.apache.avro.file.DataFileReader;
> import org.apache.avro.generic.GenericDatumReader;
> import org.apache.avro.generic.GenericRecord;
> import org.apache.avro.io.DatumReader;
> public class ReadDecimal {
> public static void main(String[] args) throws Exception {
> Schema schema = new Schema.Parser().parse("{\n" + " \"type\" :
> \"record\",\n" + " \"name\" : \"some_schema\",\n"
> + " \"namespace\" : \"com.howdy\",\n" + " \"fields\" : [ {\n" + "
> \"name\" : \"name\",\n"
> + "\"type\" : \"string\"\n" + " }, {\n" + "\"name\" :
> \"value\",\n" + "\"type\" : {\n"
> + " \"type\" : \"bytes\",\n" + " \"logicalType\" :
> \"decimal\",\n" + " \"precision\" : 3,\n"
> + " \"scale\" : 1\n" + "}\n" + " } ]\n" + "}");
> DatumReader datumReader = new GenericDatumReader<>(schema);
> // dec.avro has precision 5, scale 2
> DataFileReader dataFileReader = new DataFileReader<>(
> new File("/tmp/dec.avro"), datumReader);
> GenericRecord foo = null;
> while (dataFileReader.hasNext()) {
> foo = dataFileReader.next(foo); // AvroTypeException expected due to
> change in scale/precision but none occurs
> }
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (AVRO-2078) Avro does not enforce schema resolution rules for Decimal type
Anthony Hsu created AVRO-2078:
-
Summary: Avro does not enforce schema resolution rules for Decimal
type
Key: AVRO-2078
URL: https://issues.apache.org/jira/browse/AVRO-2078
Project: Avro
Issue Type: Bug
Reporter: Anthony Hsu
Fix For: 1.8.2
According to http://avro.apache.org/docs/1.8.2/spec.html#Decimal
bq. For the purposes of schema resolution, two schemas that are {{decimal}}
logical types _match_ if their scales and precisions match.
This is not enforced.
I wrote a file with (precision 5, scale 2) and tried to read it with a reader
schema with (precision 3, scale 1). I expected an AvroTypeException to be
thrown, but none was thrown.
Test data file attached. The code to read it is:
{noformat:title=ReadDecimal.java}
import java.io.File;
import org.apache.avro.Schema;
import org.apache.avro.file.DataFileReader;
import org.apache.avro.generic.GenericDatumReader;
import org.apache.avro.generic.GenericRecord;
import org.apache.avro.io.DatumReader;
public class ReadDecimal {
public static void main(String[] args) throws Exception {
Schema schema = new Schema.Parser().parse("{\n" + " \"type\" :
\"record\",\n" + " \"name\" : \"some_schema\",\n"
+ " \"namespace\" : \"com.howdy\",\n" + " \"fields\" : [ {\n" + "
\"name\" : \"name\",\n"
+ "\"type\" : \"string\"\n" + " }, {\n" + "\"name\" :
\"value\",\n" + "\"type\" : {\n"
+ " \"type\" : \"bytes\",\n" + " \"logicalType\" :
\"decimal\",\n" + " \"precision\" : 3,\n"
+ " \"scale\" : 1\n" + "}\n" + " } ]\n" + "}");
DatumReader datumReader = new GenericDatumReader<>(schema);
// dec.avro has precision 5, scale 2
DataFileReader dataFileReader = new DataFileReader<>(
new File("/tmp/dec.avro"), datumReader);
GenericRecord foo = null;
while (dataFileReader.hasNext()) {
foo = dataFileReader.next(foo); // AvroTypeException expected due to
change in scale/precision but none occurs
}
}
}
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
