[jira] [Updated] (AVRO-1857) GenericDatumWriter.write using BufferedBinaryEncoder leaves ByteBuffer in indeterminate state
[
https://issues.apache.org/jira/browse/AVRO-1857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-1857:
Status: Patch Available (was: Open)
> GenericDatumWriter.write using BufferedBinaryEncoder leaves ByteBuffer in
> indeterminate state
> -
>
> Key: AVRO-1857
> URL: https://issues.apache.org/jira/browse/AVRO-1857
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.7.7
>Reporter: Matt Grimwade
>Assignee: Nandor Kollar
>
> Calling {{GenericDatumWriter.write(Object, Encoder)}} using a
> BufferedBinaryEncoder leaves any ByteBuffers within the object (representing
> BYTES or FIXED types) in an indeterminate state. Specifically, each buffer
> may be left either in its initial state, with (position, remaining) = (0, N)
> or in its "consumed" state of (N, 0).
> Although I cannot find it documented, I believe the correct behaviour is that
> the state of the object being written should be unmodified.
> This is an actual problem in our project where we save a copy of an object to
> disk before performing some action on it. This later action fails
> indeterminately because some of the ByteBuffers in the object are "consumed"
> and some are not.
> I think the fault lies in
> {{org.apache.avro.io.BufferedBinaryEncoder#writeFixed(java.nio.ByteBuffer)}},
> wherein the first branch advances the buffer's position but the second does
> not:
> {code}
> @Override
> public void writeFixed(ByteBuffer bytes) throws IOException {
> if (!bytes.hasArray() && bytes.remaining() > bulkLimit) {
> flushBuffer();
> sink.innerWrite(bytes); // bypass the buffer
> } else {
> super.writeFixed(bytes);
> }
> }
> {code}
> Here is a failing test case:
> {code}
> import static java.util.Arrays.asList;
> import static org.hamcrest.Matchers.equalTo;
> import static org.hamcrest.Matchers.is;
> import static org.junit.Assert.assertThat;
> import java.io.ByteArrayOutputStream;
> import java.io.IOException;
> import java.nio.ByteBuffer;
> import java.nio.MappedByteBuffer;
> import java.nio.channels.FileChannel;
> import java.nio.file.Files;
> import java.nio.file.Path;
> import java.nio.file.StandardOpenOption;
> import org.apache.avro.Schema;
> import org.apache.avro.generic.GenericDatumWriter;
> import org.apache.avro.io.Encoder;
> import org.apache.avro.io.EncoderFactory;
> import org.junit.Test;
> public class BugTest {
> private static final int ENCODER_BUFFER_SIZE = 32;
> private static final int EXAMPLE_DATA_SIZE = 17;
> @Test
> public void testArrayBackedByteBuffer() throws IOException {
> ByteBuffer buffer = ByteBuffer.wrap(someBytes(EXAMPLE_DATA_SIZE));
> doTest(buffer);
> }
> @Test
> public void testMappedByteBuffer() throws IOException {
> Path file = Files.createTempFile("test", "data");
> Files.write(file, someBytes(EXAMPLE_DATA_SIZE));
> MappedByteBuffer buffer = FileChannel.open(file,
> StandardOpenOption.READ).map(FileChannel.MapMode.READ_ONLY, 0,
> EXAMPLE_DATA_SIZE);
> doTest(buffer);
> }
> private static void doTest(ByteBuffer buffer) throws IOException {
> assertThat(asList(buffer.position(), buffer.remaining()),
> is(asList(0, EXAMPLE_DATA_SIZE)));
> ByteArrayOutputStream output = new
> ByteArrayOutputStream(EXAMPLE_DATA_SIZE * 2);
> EncoderFactory encoderFactory = new EncoderFactory();
> encoderFactory.configureBufferSize(ENCODER_BUFFER_SIZE);
> Encoder encoder = encoderFactory.binaryEncoder(output, null);
> new
> GenericDatumWriter(Schema.create(Schema.Type.BYTES)).write(buffer,
> encoder);
> encoder.flush();
> assertThat(output.toByteArray(),
> equalTo(avroEncoded(someBytes(EXAMPLE_DATA_SIZE;
> assertThat(asList(buffer.position(), buffer.remaining()),
> is(asList(0, EXAMPLE_DATA_SIZE))); // fails if buffer is not array-backed and
> buffer overflow occurs
> }
> private static byte[] someBytes(int size) {
> byte[] result = new byte[size];
> for (int i = 0; i < size; i++) {
> result[i] = (byte) i;
> }
> return result;
> }
> private static byte[] avroEncoded(byte[] bytes) {
> assert bytes.length < 64;
> byte[] result = new byte[1 + bytes.length];
> result[0] = (byte) (bytes.length * 2); // zig-zag encoding
> System.arraycopy(bytes, 0, result, 1, bytes.length);
> return result;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (AVRO-1857) GenericDatumWriter.write using BufferedBinaryEncoder leaves ByteBuffer in indeterminate state
[
https://issues.apache.org/jira/browse/AVRO-1857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar reassigned AVRO-1857:
---
Assignee: Nandor Kollar
> GenericDatumWriter.write using BufferedBinaryEncoder leaves ByteBuffer in
> indeterminate state
> -
>
> Key: AVRO-1857
> URL: https://issues.apache.org/jira/browse/AVRO-1857
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.7.7
>Reporter: Matt Grimwade
>Assignee: Nandor Kollar
>
> Calling {{GenericDatumWriter.write(Object, Encoder)}} using a
> BufferedBinaryEncoder leaves any ByteBuffers within the object (representing
> BYTES or FIXED types) in an indeterminate state. Specifically, each buffer
> may be left either in its initial state, with (position, remaining) = (0, N)
> or in its "consumed" state of (N, 0).
> Although I cannot find it documented, I believe the correct behaviour is that
> the state of the object being written should be unmodified.
> This is an actual problem in our project where we save a copy of an object to
> disk before performing some action on it. This later action fails
> indeterminately because some of the ByteBuffers in the object are "consumed"
> and some are not.
> I think the fault lies in
> {{org.apache.avro.io.BufferedBinaryEncoder#writeFixed(java.nio.ByteBuffer)}},
> wherein the first branch advances the buffer's position but the second does
> not:
> {code}
> @Override
> public void writeFixed(ByteBuffer bytes) throws IOException {
> if (!bytes.hasArray() && bytes.remaining() > bulkLimit) {
> flushBuffer();
> sink.innerWrite(bytes); // bypass the buffer
> } else {
> super.writeFixed(bytes);
> }
> }
> {code}
> Here is a failing test case:
> {code}
> import static java.util.Arrays.asList;
> import static org.hamcrest.Matchers.equalTo;
> import static org.hamcrest.Matchers.is;
> import static org.junit.Assert.assertThat;
> import java.io.ByteArrayOutputStream;
> import java.io.IOException;
> import java.nio.ByteBuffer;
> import java.nio.MappedByteBuffer;
> import java.nio.channels.FileChannel;
> import java.nio.file.Files;
> import java.nio.file.Path;
> import java.nio.file.StandardOpenOption;
> import org.apache.avro.Schema;
> import org.apache.avro.generic.GenericDatumWriter;
> import org.apache.avro.io.Encoder;
> import org.apache.avro.io.EncoderFactory;
> import org.junit.Test;
> public class BugTest {
> private static final int ENCODER_BUFFER_SIZE = 32;
> private static final int EXAMPLE_DATA_SIZE = 17;
> @Test
> public void testArrayBackedByteBuffer() throws IOException {
> ByteBuffer buffer = ByteBuffer.wrap(someBytes(EXAMPLE_DATA_SIZE));
> doTest(buffer);
> }
> @Test
> public void testMappedByteBuffer() throws IOException {
> Path file = Files.createTempFile("test", "data");
> Files.write(file, someBytes(EXAMPLE_DATA_SIZE));
> MappedByteBuffer buffer = FileChannel.open(file,
> StandardOpenOption.READ).map(FileChannel.MapMode.READ_ONLY, 0,
> EXAMPLE_DATA_SIZE);
> doTest(buffer);
> }
> private static void doTest(ByteBuffer buffer) throws IOException {
> assertThat(asList(buffer.position(), buffer.remaining()),
> is(asList(0, EXAMPLE_DATA_SIZE)));
> ByteArrayOutputStream output = new
> ByteArrayOutputStream(EXAMPLE_DATA_SIZE * 2);
> EncoderFactory encoderFactory = new EncoderFactory();
> encoderFactory.configureBufferSize(ENCODER_BUFFER_SIZE);
> Encoder encoder = encoderFactory.binaryEncoder(output, null);
> new
> GenericDatumWriter(Schema.create(Schema.Type.BYTES)).write(buffer,
> encoder);
> encoder.flush();
> assertThat(output.toByteArray(),
> equalTo(avroEncoded(someBytes(EXAMPLE_DATA_SIZE;
> assertThat(asList(buffer.position(), buffer.remaining()),
> is(asList(0, EXAMPLE_DATA_SIZE))); // fails if buffer is not array-backed and
> buffer overflow occurs
> }
> private static byte[] someBytes(int size) {
> byte[] result = new byte[size];
> for (int i = 0; i < size; i++) {
> result[i] = (byte) i;
> }
> return result;
> }
> private static byte[] avroEncoded(byte[] bytes) {
> assert bytes.length < 64;
> byte[] result = new byte[1 + bytes.length];
> result[0] = (byte) (bytes.length * 2); // zig-zag encoding
> System.arraycopy(bytes, 0, result, 1, bytes.length);
> return result;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (AVRO-1818) Avoid buffer copy in DeflateCodec.compress and decompress
[
https://issues.apache.org/jira/browse/AVRO-1818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar reassigned AVRO-1818:
---
Assignee: Nandor Kollar
> Avoid buffer copy in DeflateCodec.compress and decompress
> -
>
> Key: AVRO-1818
> URL: https://issues.apache.org/jira/browse/AVRO-1818
> Project: Avro
> Issue Type: Improvement
>Reporter: Rohini Palaniswamy
>Assignee: Nandor Kollar
>
> One of our jobs reading avro hit OOM due to the buffer copy in compress and
> decompress methods which is very inefficient.
> https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/file/DeflateCodec.java#L71-L86
> {code}
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3236)
> at
> java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:191)
> at org.apache.avro.file.DeflateCodec.decompress(DeflateCodec.java:84)
> {code}
> I would suggest using a class that extends ByteArrrayOutputStream like
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/DataOutputBuffer.java#L51-L53
> and do
> ByteBuffer result = ByteBuffer.wrap(buf.getData(), 0, buf.getLength());
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (AVRO-1954) Schema.Field.defaultVal() generates: Unknown datum type org.apache.avro.JsonProperties$Null
[ https://issues.apache.org/jira/browse/AVRO-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-1954: Status: Patch Available (was: Open) > Schema.Field.defaultVal() generates: Unknown datum type > org.apache.avro.JsonProperties$Null > --- > > Key: AVRO-1954 > URL: https://issues.apache.org/jira/browse/AVRO-1954 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.1, 1.9.0 >Reporter: rui miranda >Assignee: Nandor Kollar >Priority: Minor > Attachments: unitTestDefaultNull.patch > > > I was creating GenericRecords and populating some fields -- which i could not > find the content on some json files -- with the Schema.Field.defaultVal(). > It seems if the schema has explicitly set the default value to be null, the > records generated this way can't be written. In this case, if default value > is null in the schema, an instance of > org.apache.avro.JsonProperties.NULL_VALUE is returned by > Schema.Field.defaultVal(). > I created an unit test which replicates the bug. I was thinking modify the > class org.apache.avro.generic.GenericData to evaluate > org.apache.avro.JsonProperties.NULL_VALUE as null. Is this the way to go? or > org.apache.avro.JsonProperties.NULL_VALUE is intend for other purposes? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (AVRO-1954) Schema.Field.defaultVal() generates: Unknown datum type org.apache.avro.JsonProperties$Null
[ https://issues.apache.org/jira/browse/AVRO-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar reassigned AVRO-1954: --- Assignee: Nandor Kollar > Schema.Field.defaultVal() generates: Unknown datum type > org.apache.avro.JsonProperties$Null > --- > > Key: AVRO-1954 > URL: https://issues.apache.org/jira/browse/AVRO-1954 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.1, 1.9.0 >Reporter: rui miranda >Assignee: Nandor Kollar >Priority: Minor > Attachments: unitTestDefaultNull.patch > > > I was creating GenericRecords and populating some fields -- which i could not > find the content on some json files -- with the Schema.Field.defaultVal(). > It seems if the schema has explicitly set the default value to be null, the > records generated this way can't be written. In this case, if default value > is null in the schema, an instance of > org.apache.avro.JsonProperties.NULL_VALUE is returned by > Schema.Field.defaultVal(). > I created an unit test which replicates the bug. I was thinking modify the > class org.apache.avro.generic.GenericData to evaluate > org.apache.avro.JsonProperties.NULL_VALUE as null. Is this the way to go? or > org.apache.avro.JsonProperties.NULL_VALUE is intend for other purposes? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1881) Avro (Java) Memory Leak when reusing JsonDecoder instance
[
https://issues.apache.org/jira/browse/AVRO-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15801220#comment-15801220
]
Nandor Kollar commented on AVRO-1881:
-
Yes, I have, and it didn't get OutOfMemory error. I also noticed that the
reorderBuffers was filled with null elements, wondering if it make sense to add
nulls when the currentReorderBuffer is null:
{code}
} else if (top == Symbol.RECORD_START) {
if (in.getCurrentToken() == JsonToken.START_OBJECT) {
in.nextToken();
reorderBuffers.push(currentReorderBuffer);
currentReorderBuffer = null;
} else {
throw error("record-start");
}
{code}
Nevertheless, JsonDecoderMemoryLeak didn't fail with OOM in my pull request.
> Avro (Java) Memory Leak when reusing JsonDecoder instance
> -
>
> Key: AVRO-1881
> URL: https://issues.apache.org/jira/browse/AVRO-1881
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.1
> Environment: Ubuntu 15.04
> Oracle 1.8.0_91 and OpenJDK 1.8.0_45
>Reporter: Matt Allen
>Assignee: Nandor Kollar
>
> {{JsonDecoder}} maintains state for each record decoded, leading to a memory
> leak if the same instance is used for multiple inputs. Using
> {{JsonDecoder.configure}} to change the input does not correctly clean up the
> state stored in {{JsonDecoder.reorderBuffers}}, which leads to an unbounded
> number of {{ReorderBuffer}} instances being accumulated. If a new
> {{JsonDecoder}} is created for each input there is no memory leak, but it is
> significantly more expensive than reusing the same instance.
> This problem seems to only occur when the input schema contains a record,
> which is consistent with the {{reorderBuffers}} being the source of the leak.
> My first look at the {{JsonDecoder}} code leads me to believe that the
> {{reorderBuffers}} stack should be empty after a record is fully processed,
> so there may be other behavior at play here.
> The following is a minimal example which will exhaust a 50MB heap (-Xmx50m)
> after about 5.25 million iterations. The first section demonstrates that no
> memory leak is encountered when creating a fresh {{JsonDecoder}} instance for
> each input.
> {code:title=JsonDecoderMemoryLeak.java|borderStyle=solid}
> import org.apache.avro.Schema;
> import org.apache.avro.io.*;
> import org.apache.avro.generic.*;
> import java.io.IOException;
> public class JsonDecoderMemoryLeak {
> public static DecoderFactory decoderFactory = DecoderFactory.get();
> public static JsonDecoder createDecoder(String input, Schema schema)
> throws IOException {
> return decoderFactory.jsonDecoder(schema, input);
> }
> public static Object decodeAvro(String input, Schema schema, JsonDecoder
> decoder) throws IOException {
> if (decoder == null) {
> decoder = createDecoder(input, schema);
> } else {
> decoder.configure(input);
> }
> GenericDatumReader reader = new
> GenericDatumReader(schema);
> return reader.read(null, decoder);
> }
> public static Schema.Parser parser = new Schema.Parser();
> public static Schema schema = parser.parse("{\"name\": \"TestRecord\",
> \"type\": \"record\", \"fields\": [{\"name\": \"field1\", \"type\":
> \"long\"}]}");
> public static String record(long i) {
> StringBuilder builder = new StringBuilder("{\"field1\": ");
> builder.append(i);
> builder.append("}");
> return builder.toString();
> }
> public static void main(String[] args) throws IOException {
> // No memory issues when creating a new decoder for each record
> System.out.println("Running with fresh JsonDecoder instances for
> 600 iterations");
> for(long i = 0; i < 600; i++) {
> decodeAvro(record(i), schema, null);
> }
>
> // Runs out of memory after ~525 records
> System.out.println("Running with a single reused JsonDecoder
> instance");
> long count = 0;
> try {
> JsonDecoder decoder = createDecoder(record(0), schema);
> while(true) {
> decodeAvro(record(count), schema, decoder);
> count++;
> }
> } catch (OutOfMemoryError e) {
> System.out.println("Out of memory after " + count + " records");
> e.printStackTrace();
> }
> }
> }
> {code}
> {code:title=Output|borderStyle=solid}
> $ java -Xmx50m -jar json-decoder-memory-leak.jar
> Running with fresh JsonDecoder instances for 600 iterations
> Running with a single reused JsonDecoder instance
> Out of memory after 5242880 records
> java.lang.OutOfMemoryError: Java heap space
> at
[jira] [Assigned] (AVRO-1881) Avro (Java) Memory Leak when reusing JsonDecoder instance
[
https://issues.apache.org/jira/browse/AVRO-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar reassigned AVRO-1881:
---
Assignee: Nandor Kollar
> Avro (Java) Memory Leak when reusing JsonDecoder instance
> -
>
> Key: AVRO-1881
> URL: https://issues.apache.org/jira/browse/AVRO-1881
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.1
> Environment: Ubuntu 15.04
> Oracle 1.8.0_91 and OpenJDK 1.8.0_45
>Reporter: Matt Allen
>Assignee: Nandor Kollar
>
> {{JsonDecoder}} maintains state for each record decoded, leading to a memory
> leak if the same instance is used for multiple inputs. Using
> {{JsonDecoder.configure}} to change the input does not correctly clean up the
> state stored in {{JsonDecoder.reorderBuffers}}, which leads to an unbounded
> number of {{ReorderBuffer}} instances being accumulated. If a new
> {{JsonDecoder}} is created for each input there is no memory leak, but it is
> significantly more expensive than reusing the same instance.
> This problem seems to only occur when the input schema contains a record,
> which is consistent with the {{reorderBuffers}} being the source of the leak.
> My first look at the {{JsonDecoder}} code leads me to believe that the
> {{reorderBuffers}} stack should be empty after a record is fully processed,
> so there may be other behavior at play here.
> The following is a minimal example which will exhaust a 50MB heap (-Xmx50m)
> after about 5.25 million iterations. The first section demonstrates that no
> memory leak is encountered when creating a fresh {{JsonDecoder}} instance for
> each input.
> {code:title=JsonDecoderMemoryLeak.java|borderStyle=solid}
> import org.apache.avro.Schema;
> import org.apache.avro.io.*;
> import org.apache.avro.generic.*;
> import java.io.IOException;
> public class JsonDecoderMemoryLeak {
> public static DecoderFactory decoderFactory = DecoderFactory.get();
> public static JsonDecoder createDecoder(String input, Schema schema)
> throws IOException {
> return decoderFactory.jsonDecoder(schema, input);
> }
> public static Object decodeAvro(String input, Schema schema, JsonDecoder
> decoder) throws IOException {
> if (decoder == null) {
> decoder = createDecoder(input, schema);
> } else {
> decoder.configure(input);
> }
> GenericDatumReader reader = new
> GenericDatumReader(schema);
> return reader.read(null, decoder);
> }
> public static Schema.Parser parser = new Schema.Parser();
> public static Schema schema = parser.parse("{\"name\": \"TestRecord\",
> \"type\": \"record\", \"fields\": [{\"name\": \"field1\", \"type\":
> \"long\"}]}");
> public static String record(long i) {
> StringBuilder builder = new StringBuilder("{\"field1\": ");
> builder.append(i);
> builder.append("}");
> return builder.toString();
> }
> public static void main(String[] args) throws IOException {
> // No memory issues when creating a new decoder for each record
> System.out.println("Running with fresh JsonDecoder instances for
> 600 iterations");
> for(long i = 0; i < 600; i++) {
> decodeAvro(record(i), schema, null);
> }
>
> // Runs out of memory after ~525 records
> System.out.println("Running with a single reused JsonDecoder
> instance");
> long count = 0;
> try {
> JsonDecoder decoder = createDecoder(record(0), schema);
> while(true) {
> decodeAvro(record(count), schema, decoder);
> count++;
> }
> } catch (OutOfMemoryError e) {
> System.out.println("Out of memory after " + count + " records");
> e.printStackTrace();
> }
> }
> }
> {code}
> {code:title=Output|borderStyle=solid}
> $ java -Xmx50m -jar json-decoder-memory-leak.jar
> Running with fresh JsonDecoder instances for 600 iterations
> Running with a single reused JsonDecoder instance
> Out of memory after 5242880 records
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3210)
> at java.util.Arrays.copyOf(Arrays.java:3181)
> at java.util.Vector.grow(Vector.java:266)
> at java.util.Vector.ensureCapacityHelper(Vector.java:246)
> at java.util.Vector.addElement(Vector.java:620)
> at java.util.Stack.push(Stack.java:67)
> at org.apache.avro.io.JsonDecoder.doAction(JsonDecoder.java:487)
> at org.apache.avro.io.parsing.Parser.advance(Parser.java:88)
> at org.apache.avro.io.JsonDecoder.advance(JsonDecoder.java:139)
> at
[jira] [Commented] (AVRO-1931) SchemaCompatibility fails to recognize reader compatible with all branches of a union
[ https://issues.apache.org/jira/browse/AVRO-1931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127296#comment-16127296 ] Nandor Kollar commented on AVRO-1931: - The patch looks good to me, though it was a bit hard to understand the wording of the relevant part of the specification. [~epkanol] since the specification says that int and long are promotable to float, I think we should allow float unions to read int or long unions, and if this behavior is questionable, that should be a different Jira (and we should also modify the specification). [~sacharya] looks like this is important for Avro users, I saw an email related to this Jira popping up in the user's list, and Anders addressed your concern on the pull request. Is there anything else required to merge this PR to master? Is it an incompatible change? > SchemaCompatibility fails to recognize reader compatible with all branches of > a union > - > > Key: AVRO-1931 > URL: https://issues.apache.org/jira/browse/AVRO-1931 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.7.7, 1.8.1 > Environment: Java >Reporter: Anders Sundelin >Priority: Minor > Labels: patch > Attachments: AVRO-1931-2.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > It is stated in the Avro spec > "if writer's is a union, but reader's is not: > If the reader's schema matches the selected writer's schema, it is > recursively resolved against it. If they do not match, an error is signalled." > In case a the chosen reader is compatible with all branches of the union in > the writer, then the class SchemaCompatibility should reflect this. Currently > it does not. > The submitted patch corrects this (also added tests showing this behaviour in > Avro) > The new tests, in the class TestReadingWritingDataInEvolvedSchemas, could be > redundant, but they were very useful when exploring how Avro actually works > during de-/serialization > I will try to continue working a little bit on the SchemaCompatibility class, > adding more user-friendly error messages (suitable for deeper structures than > todays error message). Feel free to contact me if you have any ideas or > pointers to existing work. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (AVRO-2066) Wrong schema file used in one TestSpecificCompiler test case
[
https://issues.apache.org/jira/browse/AVRO-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar reassigned AVRO-2066:
---
Assignee: Nandor Kollar
> Wrong schema file used in one TestSpecificCompiler test case
>
>
> Key: AVRO-2066
> URL: https://issues.apache.org/jira/browse/AVRO-2066
> Project: Avro
> Issue Type: Bug
> Components: java
>Reporter: Nandor Kollar
>Assignee: Nandor Kollar
>Priority: Trivial
> Attachments: AVRO-2066.patch
>
>
> I think instead of simple_record.avsc
> {{TestSpecificCompiler#testLogicalTypesWithMultipleFields}} meant to use
> logical_types_with_multiple_fields.avsc
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-2066) Wrong schema file used in one TestSpecificCompiler test case
[
https://issues.apache.org/jira/browse/AVRO-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-2066:
Status: Patch Available (was: Open)
> Wrong schema file used in one TestSpecificCompiler test case
>
>
> Key: AVRO-2066
> URL: https://issues.apache.org/jira/browse/AVRO-2066
> Project: Avro
> Issue Type: Bug
> Components: java
>Reporter: Nandor Kollar
>Priority: Trivial
> Attachments: AVRO-2066.patch
>
>
> I think instead of simple_record.avsc
> {{TestSpecificCompiler#testLogicalTypesWithMultipleFields}} meant to use
> logical_types_with_multiple_fields.avsc
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-1658) Add avroDoc on reflect
[
https://issues.apache.org/jira/browse/AVRO-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123468#comment-16123468
]
Nandor Kollar commented on AVRO-1658:
-
I didn't see any test failure after merging Evan's PR, all unit test passed. I
just had to resolve a merge conflict in the test class, the patch looks good to
me.
> Add avroDoc on reflect
> --
>
> Key: AVRO-1658
> URL: https://issues.apache.org/jira/browse/AVRO-1658
> Project: Avro
> Issue Type: New Feature
> Components: java
>Affects Versions: 1.7.7
>Reporter: Zhaonan Sun
>Assignee: Zhaonan Sun
> Labels: reflection
> Attachments:
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch,
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch,
> 0001-AVRO-1658-Java-Add-reflection-annotation-AvroDoc.patch
>
>
> Looks like @AvroMeta can't add reserved fields, like @AvroMeta("doc", "some
> doc") will have exceptions.
> I would be greate if we have a @AvroDoc("some documentations") in
> org.apache.avro.reflect
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-2066) Wrong schema file used in one TestSpecificCompiler test case
[
https://issues.apache.org/jira/browse/AVRO-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125438#comment-16125438
]
Nandor Kollar commented on AVRO-2066:
-
[~blue_impala_48d6] it seems you you did this test, could you please have a
look at the attached patch? I think the file reference in the test is not the
one you wanted to use.
> Wrong schema file used in one TestSpecificCompiler test case
>
>
> Key: AVRO-2066
> URL: https://issues.apache.org/jira/browse/AVRO-2066
> Project: Avro
> Issue Type: Bug
> Components: java
>Reporter: Nandor Kollar
>Priority: Trivial
> Attachments: AVRO-2066.patch
>
>
> I think instead of simple_record.avsc
> {{TestSpecificCompiler#testLogicalTypesWithMultipleFields}} meant to use
> logical_types_with_multiple_fields.avsc
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (AVRO-2066) Wrong schema file used in one TestSpecificCompiler test case
Nandor Kollar created AVRO-2066:
---
Summary: Wrong schema file used in one TestSpecificCompiler test
case
Key: AVRO-2066
URL: https://issues.apache.org/jira/browse/AVRO-2066
Project: Avro
Issue Type: Bug
Components: java
Reporter: Nandor Kollar
Priority: Trivial
I think instead of simple_record.avsc
{{TestSpecificCompiler#testLogicalTypesWithMultipleFields}} meant to use
logical_types_with_multiple_fields.avsc
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-2066) Wrong schema file used in one TestSpecificCompiler test case
[
https://issues.apache.org/jira/browse/AVRO-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-2066:
Attachment: AVRO-2066.patch
> Wrong schema file used in one TestSpecificCompiler test case
>
>
> Key: AVRO-2066
> URL: https://issues.apache.org/jira/browse/AVRO-2066
> Project: Avro
> Issue Type: Bug
> Components: java
>Reporter: Nandor Kollar
>Priority: Trivial
> Attachments: AVRO-2066.patch
>
>
> I think instead of simple_record.avsc
> {{TestSpecificCompiler#testLogicalTypesWithMultipleFields}} meant to use
> logical_types_with_multiple_fields.avsc
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-2050) Clear Array To Allow GC
[
https://issues.apache.org/jira/browse/AVRO-2050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16089518#comment-16089518
]
Nandor Kollar commented on AVRO-2050:
-
I'm wondering why {{clear()}} method is in overridden. It looks like the base
class is AbstractList, which has clear method implemented correctly, so we
might instead implement the {{Array}} iterator's {{remove()}} method no?
> Clear Array To Allow GC
> ---
>
> Key: AVRO-2050
> URL: https://issues.apache.org/jira/browse/AVRO-2050
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.7.7, 1.8.2
>Reporter: BELUGA BEHR
>Priority: Minor
> Attachments: AVRO-2050.1.patch
>
>
> Java's {{ArrayList}} implementation clears all Objects from the internal
> buffer when the {{clear()}} method is called. This allows the Objects to be
> free for GC. We should do the same in Avro
> {{org.apache.avro.generic.GenericData}}
> [ArrayList
> Source|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8u40-b25/java/util/ArrayList.java#ArrayList.clear%28%29]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-1485) Specification says Record field type can be record name but implementation allows any named type.
[ https://issues.apache.org/jira/browse/AVRO-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092826#comment-16092826 ] Nandor Kollar commented on AVRO-1485: - [~gszadovszky] could you please review this item? > Specification says Record field type can be record name but implementation > allows any named type. > - > > Key: AVRO-1485 > URL: https://issues.apache.org/jira/browse/AVRO-1485 > Project: Avro > Issue Type: Bug > Components: java, spec >Affects Versions: 1.7.6 >Reporter: Sean Busbey >Assignee: Nandor Kollar > Attachments: AVRO-1485_1.patch > > > The [specification for Record > fields|http://avro.apache.org/docs/1.7.6/spec.html#schema_record] says that > the type is > bq. A JSON object defining a schema, or a JSON string naming a record > definition (required). > AFAICT, the Java implementation allows for any [named > type|http://avro.apache.org/docs/1.7.6/spec.html#Names]. > The specification should be updated to state any named type is allowed or the > Java implementation should restrict what can be used. The former seems less > likely to disturb current users. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2049) Remove Superfluous Configuration From AvroSerializer
[
https://issues.apache.org/jira/browse/AVRO-2049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101553#comment-16101553
]
Nandor Kollar commented on AVRO-2049:
-
Looks good to me, just one minor comment, in {{AvroSerialization}} I think we
can reuse the {{encoder}} by passing it to the factory method instead of null:
{{this.encoder = new EncoderFactory().binaryEncoder(out, encoder);}}
> Remove Superfluous Configuration From AvroSerializer
>
>
> Key: AVRO-2049
> URL: https://issues.apache.org/jira/browse/AVRO-2049
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.7.7, 1.8.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: AVRO-2049.1.patch, AVRO-2049.2.patch
>
>
> In the class {{org.apache.avro.hadoop.io.AvroSerializer}}, we see that the
> Avro block size is configured with a hard-coded value and there is a request
> to benchmark different buffer sizes.
> {code:title=org.apache.avro.hadoop.io.AvroSerializer}
> /**
>* The block size for the Avro encoder.
>*
>* This number was copied from the AvroSerialization of
> org.apache.avro.mapred in Avro 1.5.1.
>*
>* TODO(gwu): Do some benchmarking with different numbers here to see if it
> is important.
>*/
> private static final int AVRO_ENCODER_BLOCK_SIZE_BYTES = 512;
> /** An factory for creating Avro datum encoders. */
> private static EncoderFactory mEncoderFactory
> = new
> EncoderFactory().configureBlockSize(AVRO_ENCODER_BLOCK_SIZE_BYTES);
> {code}
> However, there is no need to benchmark, this setting is superfluous and is
> ignored with the current implementation.
> {code:title=org.apache.avro.hadoop.io.AvroSerializer}
> @Override
> public void open(OutputStream outputStream) throws IOException {
> mOutputStream = outputStream;
> mAvroEncoder = mEncoderFactory.binaryEncoder(outputStream, mAvroEncoder);
> }
> {code}
> {{org.apache.avro.io.EncoderFactory.binaryEncoder}} ignores this setting.
> This setting is only relevant for calls to
> {{org.apache.avro.io.EncoderFactory.blockingBinaryEncoder}}
> which considers the configured "Block Size" for doing binary encoding of
> blocked Array types as laid out in the
> [specs|https://avro.apache.org/docs/1.8.2/spec.html#binary_encode_complex].
> It can simply be removed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (AVRO-1991) Java concat tool -appendToFirst option doesn't work with local filesystem
[
https://issues.apache.org/jira/browse/AVRO-1991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar reassigned AVRO-1991:
---
Assignee: Nandor Kollar
> Java concat tool -appendToFirst option doesn't work with local filesystem
> -
>
> Key: AVRO-1991
> URL: https://issues.apache.org/jira/browse/AVRO-1991
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Sean Busbey
>Assignee: Nandor Kollar
>
> We should either make it work (e.g. by switching to RawLocalFilesytem instead
> of LocalFileSystem) or update the cli help.
> {code}
> java -jar lang/java/tools/target/avro-tools-1.9.0-SNAPSHOT.jar concat
> -appendToFirst ~/Downloads/append.0.avro
> '/Users/busbey/Downloads/example.*.avro'
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> Exception in thread "main" java.io.IOException: Not supported
> at
> org.apache.hadoop.fs.ChecksumFileSystem.append(ChecksumFileSystem.java:352)
> at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1161)
> at org.apache.avro.tool.Util.createFromFS(Util.java:150)
> at org.apache.avro.tool.Util.fileOrStdout(Util.java:76)
> at org.apache.avro.tool.ConcatTool.run(ConcatTool.java:76)
> at org.apache.avro.tool.Main.run(Main.java:87)
> at org.apache.avro.tool.Main.main(Main.java:76)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (AVRO-1815) Incompatible schema change not detected when wrapped in a UNION
[
https://issues.apache.org/jira/browse/AVRO-1815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072480#comment-16072480
]
Nandor Kollar edited comment on AVRO-1815 at 7/3/17 2:16 PM:
-
[~gdeschut] it looks like this issue is similar to AVRO-1883, which is fixed on
[master|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/io/parsing/Symbol.java#L378]
and already included into 1.8.2 release. Is it possible for you to upgrade to
this release, and check if this is still an outstanding issue? I quickly
assembled a unit test to check the schema you provided, and looks like
validation fails with the expected incompatibility error.
was (Author: nkollar):
[~gdeschut] it looks like this issue is similar to AVRO-1883, which is fixed on
master and already included into 1.8.2 release. Is it possible for you to
upgrade to this release, and check if this is still an outstanding issue? I
quickly assembled a unit test to check the schema you provided, and looks like
validation fails with the expected incompatibility error.
> Incompatible schema change not detected when wrapped in a UNION
> ---
>
> Key: AVRO-1815
> URL: https://issues.apache.org/jira/browse/AVRO-1815
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.7.7
>Reporter: Martin Boyle
> Attachments: AVRO-1815.patch
>
>
> An incompatible schema change is not detected when it is in a UNION and the
> change is to the value type of a map e.g.
> field
> { "name": "segmentEkv", "type": ["null", {"type": "map", "values": {"type":
> "map", "values": "string"}}], "default": null},
> changes to
> { "name": "segmentEkv", "type": ["null", {"type": "map", "values": {"type":
> "array", "items": "int"}}], "default": null},
> The SchemaValidatorBuilder() will pass this as being compatible. Whereas
> SchemaCompatibility.check_reader_writer_compatibility will return an
> incompatible result. The problem for me is that the Confluent Schema
> Registry uses SchemaValidatorBuilder.
> Problem appears to be that while the ResolvingGrammerGenerator correctly
> marks the field as being an incompatible change, the check for errors on the
> Symbol object does not descend into the UnionAdjustActionField
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-1815) Incompatible schema change not detected when wrapped in a UNION
[
https://issues.apache.org/jira/browse/AVRO-1815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072480#comment-16072480
]
Nandor Kollar commented on AVRO-1815:
-
[~gdeschut] it looks like this issue is similar to AVRO-1883, which is fixed on
master and already included into 1.8.2 release. Is it possible for you to
upgrade to this release, and check if this is still an outstanding issue? I
quickly assembled a unit test to check the schema you provided, and looks like
validation fails with the expected incompatibility error.
> Incompatible schema change not detected when wrapped in a UNION
> ---
>
> Key: AVRO-1815
> URL: https://issues.apache.org/jira/browse/AVRO-1815
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.7.7
>Reporter: Martin Boyle
> Attachments: AVRO-1815.patch
>
>
> An incompatible schema change is not detected when it is in a UNION and the
> change is to the value type of a map e.g.
> field
> { "name": "segmentEkv", "type": ["null", {"type": "map", "values": {"type":
> "map", "values": "string"}}], "default": null},
> changes to
> { "name": "segmentEkv", "type": ["null", {"type": "map", "values": {"type":
> "array", "items": "int"}}], "default": null},
> The SchemaValidatorBuilder() will pass this as being compatible. Whereas
> SchemaCompatibility.check_reader_writer_compatibility will return an
> incompatible result. The problem for me is that the Confluent Schema
> Registry uses SchemaValidatorBuilder.
> Problem appears to be that while the ResolvingGrammerGenerator correctly
> marks the field as being an incompatible change, the check for errors on the
> Symbol object does not descend into the UnionAdjustActionField
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-1991) Java concat tool -appendToFirst option doesn't work with local filesystem
[
https://issues.apache.org/jira/browse/AVRO-1991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068341#comment-16068341
]
Nandor Kollar commented on AVRO-1991:
-
[~busbey] I think I just found a reference to this feature! Here it is:
AVRO-1856. It seems it is still an open item waiting for review.
> Java concat tool -appendToFirst option doesn't work with local filesystem
> -
>
> Key: AVRO-1991
> URL: https://issues.apache.org/jira/browse/AVRO-1991
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Sean Busbey
>
> We should either make it work (e.g. by switching to RawLocalFilesytem instead
> of LocalFileSystem) or update the cli help.
> {code}
> java -jar lang/java/tools/target/avro-tools-1.9.0-SNAPSHOT.jar concat
> -appendToFirst ~/Downloads/append.0.avro
> '/Users/busbey/Downloads/example.*.avro'
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> Exception in thread "main" java.io.IOException: Not supported
> at
> org.apache.hadoop.fs.ChecksumFileSystem.append(ChecksumFileSystem.java:352)
> at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1161)
> at org.apache.avro.tool.Util.createFromFS(Util.java:150)
> at org.apache.avro.tool.Util.fileOrStdout(Util.java:76)
> at org.apache.avro.tool.ConcatTool.run(ConcatTool.java:76)
> at org.apache.avro.tool.Main.run(Main.java:87)
> at org.apache.avro.tool.Main.main(Main.java:76)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (AVRO-2021) uuid logical type is not documented
[ https://issues.apache.org/jira/browse/AVRO-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar reassigned AVRO-2021: --- Assignee: Nandor Kollar > uuid logical type is not documented > --- > > Key: AVRO-2021 > URL: https://issues.apache.org/jira/browse/AVRO-2021 > Project: Avro > Issue Type: Improvement >Reporter: Andrew Rosca >Assignee: Nandor Kollar >Priority: Minor > Attachments: AVRO-2021_1.patch > > > The documentation does not mention anything about the _uuid_ logical type, > which is in fact implemented in LogicalTypes.java. > Add documentation for _uuid_ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-2021) uuid logical type is not documented
[ https://issues.apache.org/jira/browse/AVRO-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2021: Attachment: AVRO-2021_1.patch > uuid logical type is not documented > --- > > Key: AVRO-2021 > URL: https://issues.apache.org/jira/browse/AVRO-2021 > Project: Avro > Issue Type: Improvement >Reporter: Andrew Rosca >Assignee: Nandor Kollar >Priority: Minor > Attachments: AVRO-2021_1.patch > > > The documentation does not mention anything about the _uuid_ logical type, > which is in fact implemented in LogicalTypes.java. > Add documentation for _uuid_ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2021) uuid logical type is not documented
[ https://issues.apache.org/jira/browse/AVRO-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066548#comment-16066548 ] Nandor Kollar commented on AVRO-2021: - [~gszadovszky] could you please have a look at the patch? I'm wondering if we should also document that uuid can annotate a nullable union too, Avro has a test case for this: TestGenericLogicalTypes#testWriteNullableUUID. > uuid logical type is not documented > --- > > Key: AVRO-2021 > URL: https://issues.apache.org/jira/browse/AVRO-2021 > Project: Avro > Issue Type: Improvement >Reporter: Andrew Rosca >Assignee: Nandor Kollar >Priority: Minor > Attachments: AVRO-2021_1.patch > > > The documentation does not mention anything about the _uuid_ logical type, > which is in fact implemented in LogicalTypes.java. > Add documentation for _uuid_ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-2021) uuid logical type is not documented
[ https://issues.apache.org/jira/browse/AVRO-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2021: Status: Patch Available (was: Open) > uuid logical type is not documented > --- > > Key: AVRO-2021 > URL: https://issues.apache.org/jira/browse/AVRO-2021 > Project: Avro > Issue Type: Improvement >Reporter: Andrew Rosca >Assignee: Nandor Kollar >Priority: Minor > Attachments: AVRO-2021_1.patch > > > The documentation does not mention anything about the _uuid_ logical type, > which is in fact implemented in LogicalTypes.java. > Add documentation for _uuid_ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2061) Improve Invalid File Format Error Message
[
https://issues.apache.org/jira/browse/AVRO-2061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116446#comment-16116446
]
Nandor Kollar commented on AVRO-2061:
-
Looks good to me.
> Improve Invalid File Format Error Message
> -
>
> Key: AVRO-2061
> URL: https://issues.apache.org/jira/browse/AVRO-2061
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.9.0, 1.8.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Attachments: AVRO-2061.1.patch
>
>
> {code:title=org.apache.avro.file.DataFileStream}
> try {
> vin.readFixed(magic); // read magic
> } catch (IOException e) {
> throw new IOException("Not a data file.", e);
> }
> if (!Arrays.equals(DataFileConstants.MAGIC, magic))
> throw new IOException("Not a data file.");
> {code}
> Please consider improving the error message here. I just saw a MapReduce job
> fail with an IOException with the message "Not a data file." There was
> definitely data files in the input directory, however, they were not Avro
> files. It would have been much more helpful if it told me that.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-1957) TimeConversions do not implement getRecommendedSchema()
[
https://issues.apache.org/jira/browse/AVRO-1957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116500#comment-16116500
]
Nandor Kollar commented on AVRO-1957:
-
Looks good to me, {{mvn clean verify}} passed.
> TimeConversions do not implement getRecommendedSchema()
> ---
>
> Key: AVRO-1957
> URL: https://issues.apache.org/jira/browse/AVRO-1957
> Project: Avro
> Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: Sean Timm
>Assignee: Sean Timm
> Fix For: 1.8.3
>
>
> org.apache.avro.data.TimeConversions.TimestampConversion and other date and
> time conversions do not implement getRecommendedSchema(). When trying to
> dynamically generate an Avro schema from a pojo that contains a DateTime
> object using ReflectData, I get an unsupported operation exception.
> I think the implementation should be as simple as
> {code}
> @Override
> public Schema getRecommendedSchema() {
> return
> LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-1821) Avro (Java) Memory Leak in ReflectData Caching
[
https://issues.apache.org/jira/browse/AVRO-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-1821:
Fix Version/s: 1.8.1
> Avro (Java) Memory Leak in ReflectData Caching
> --
>
> Key: AVRO-1821
> URL: https://issues.apache.org/jira/browse/AVRO-1821
> Project: Avro
> Issue Type: Bug
> Components: java
> Environment: OS X 10.11.3
> {code}java version "1.8.0_66"
> Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
> Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode){code}
>Reporter: Bryan Harclerode
>Assignee: Bryan Harclerode
> Fix For: 1.8.1
>
> Attachments:
> 0001-AVRO-1821-Fix-memory-leak-of-Schemas-in-ReflectData.patch
>
>
> I think I have encountered one of the memory leaks described by AVRO-1283 in
> the way Java Avro implements field accessor caching in {{ReflectData}}. When
> a reflected object is serialized, the key of {{ClassAccessorData.bySchema}}
> (as retained by {{ReflectData.ACCESSOR_CACHE}}) retains a strong reference to
> the schema that was used to serialize the object, but there exists no code
> path for clearing these references after a schema will no longer be used.
> While in most cases, a class will probably only have one schema associated
> with it (created and cached by {{ReflectData.getSchema(Type)}}), I
> experienced {{OutOfMemoryError}} when serializing generic classes with
> dynamically-generated schemas. The following is a minimal example which will
> exhaust a 50MiB heap ({{-Xmx50m}}) after about 190K iterations:
> {code:title=AvroMemoryLeakMinimal.java|borderStyle=solid}
> import java.io.ByteArrayOutputStream;
> import java.io.IOException;
> import java.util.Collections;
> import org.apache.avro.Schema;
> import org.apache.avro.io.BinaryEncoder;
> import org.apache.avro.io.EncoderFactory;
> import org.apache.avro.reflect.ReflectDatumWriter;
> public class AvroMemoryLeakMinimal {
> public static void main(String[] args) throws IOException {
> long count = 0;
> EncoderFactory encFactory = EncoderFactory.get();
> try {
> while (true) {
> // Create schema
> Schema schema = Schema.createRecord("schema", null, null,
> false);
> schema.setFields(Collections.emptyList());
> // serialize
> ByteArrayOutputStream baos = new ByteArrayOutputStream(1024);
> BinaryEncoder encoder = encFactory.binaryEncoder(baos, null);
> (new ReflectDatumWriter(schema)).write(new Object(),
> encoder);
> byte[] result = baos.toByteArray();
> count++;
> }
> } catch (OutOfMemoryError e) {
> System.out.print("Memory exhausted after ");
> System.out.print(count);
> System.out.println(" schemas");
> throw e;
> }
> }
> }
> {code}
> I was able to fix the bug in the latest 1.9.0-SNAPSHOT from git with the
> following patch to {{ClassAccessorData.bySchema}} to use weak keys so that it
> properly released the {{Schema}} objects if no other threads are still
> referencing them:
> {code:title=ReflectData.java.patch|borderStyle=solid}
> --- a/lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectData.java
> +++ b/lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectData.java
> @@ -57,6 +57,7 @@ import org.apache.avro.io.DatumWriter;
> import org.apache.avro.specific.FixedSize;
> import org.apache.avro.specific.SpecificData;
> import org.apache.avro.SchemaNormalization;
> +import org.apache.avro.util.WeakIdentityHashMap;
> import org.codehaus.jackson.JsonNode;
> import org.codehaus.jackson.node.NullNode;
>
> @@ -234,8 +235,8 @@ public class ReflectData extends SpecificData {
> private final Class clazz;
> private final Map byName =
> new HashMap();
> -private final IdentityHashMap bySchema =
> -new IdentityHashMap();
> +private final WeakIdentityHashMap bySchema =
> +new WeakIdentityHashMap();
>
> private ClassAccessorData(Class c) {
>clazz = c;
> {code}
> Additionally, I'm not sure why an {{IdentityHashMap}} was used instead of a
> standard {{HashMap}}, since two equivalent schemas have the same set of
> {{FieldAccessor}}. Everything appears to work and all tests pass if I use a
> {{WeakHashMap}} instead of an {{WeakIdentityHashMap}}, but I don't know if
> there was some other reason object identity was important for this map. If a
> non-identity map can be used, this will help reduce memory/CPU usage further
> by not regenerating all the field
[jira] [Commented] (AVRO-1821) Avro (Java) Memory Leak in ReflectData Caching
[
https://issues.apache.org/jira/browse/AVRO-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055373#comment-16055373
]
Nandor Kollar commented on AVRO-1821:
-
[~alunarbeach] it looks like
[this|https://github.com/apache/avro/commit/ec8a091819a25bccf03adc868449f57f9c076d19]
is already committed and released in 1.8.1.
> Avro (Java) Memory Leak in ReflectData Caching
> --
>
> Key: AVRO-1821
> URL: https://issues.apache.org/jira/browse/AVRO-1821
> Project: Avro
> Issue Type: Bug
> Components: java
> Environment: OS X 10.11.3
> {code}java version "1.8.0_66"
> Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
> Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode){code}
>Reporter: Bryan Harclerode
>Assignee: Bryan Harclerode
> Fix For: 1.8.1
>
> Attachments:
> 0001-AVRO-1821-Fix-memory-leak-of-Schemas-in-ReflectData.patch
>
>
> I think I have encountered one of the memory leaks described by AVRO-1283 in
> the way Java Avro implements field accessor caching in {{ReflectData}}. When
> a reflected object is serialized, the key of {{ClassAccessorData.bySchema}}
> (as retained by {{ReflectData.ACCESSOR_CACHE}}) retains a strong reference to
> the schema that was used to serialize the object, but there exists no code
> path for clearing these references after a schema will no longer be used.
> While in most cases, a class will probably only have one schema associated
> with it (created and cached by {{ReflectData.getSchema(Type)}}), I
> experienced {{OutOfMemoryError}} when serializing generic classes with
> dynamically-generated schemas. The following is a minimal example which will
> exhaust a 50MiB heap ({{-Xmx50m}}) after about 190K iterations:
> {code:title=AvroMemoryLeakMinimal.java|borderStyle=solid}
> import java.io.ByteArrayOutputStream;
> import java.io.IOException;
> import java.util.Collections;
> import org.apache.avro.Schema;
> import org.apache.avro.io.BinaryEncoder;
> import org.apache.avro.io.EncoderFactory;
> import org.apache.avro.reflect.ReflectDatumWriter;
> public class AvroMemoryLeakMinimal {
> public static void main(String[] args) throws IOException {
> long count = 0;
> EncoderFactory encFactory = EncoderFactory.get();
> try {
> while (true) {
> // Create schema
> Schema schema = Schema.createRecord("schema", null, null,
> false);
> schema.setFields(Collections.emptyList());
> // serialize
> ByteArrayOutputStream baos = new ByteArrayOutputStream(1024);
> BinaryEncoder encoder = encFactory.binaryEncoder(baos, null);
> (new ReflectDatumWriter(schema)).write(new Object(),
> encoder);
> byte[] result = baos.toByteArray();
> count++;
> }
> } catch (OutOfMemoryError e) {
> System.out.print("Memory exhausted after ");
> System.out.print(count);
> System.out.println(" schemas");
> throw e;
> }
> }
> }
> {code}
> I was able to fix the bug in the latest 1.9.0-SNAPSHOT from git with the
> following patch to {{ClassAccessorData.bySchema}} to use weak keys so that it
> properly released the {{Schema}} objects if no other threads are still
> referencing them:
> {code:title=ReflectData.java.patch|borderStyle=solid}
> --- a/lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectData.java
> +++ b/lang/java/avro/src/main/java/org/apache/avro/reflect/ReflectData.java
> @@ -57,6 +57,7 @@ import org.apache.avro.io.DatumWriter;
> import org.apache.avro.specific.FixedSize;
> import org.apache.avro.specific.SpecificData;
> import org.apache.avro.SchemaNormalization;
> +import org.apache.avro.util.WeakIdentityHashMap;
> import org.codehaus.jackson.JsonNode;
> import org.codehaus.jackson.node.NullNode;
>
> @@ -234,8 +235,8 @@ public class ReflectData extends SpecificData {
> private final Class clazz;
> private final Map byName =
> new HashMap();
> -private final IdentityHashMap bySchema =
> -new IdentityHashMap();
> +private final WeakIdentityHashMap bySchema =
> +new WeakIdentityHashMap();
>
> private ClassAccessorData(Class c) {
>clazz = c;
> {code}
> Additionally, I'm not sure why an {{IdentityHashMap}} was used instead of a
> standard {{HashMap}}, since two equivalent schemas have the same set of
> {{FieldAccessor}}. Everything appears to work and all tests pass if I use a
> {{WeakHashMap}} instead of an {{WeakIdentityHashMap}}, but I don't know if
> there was some other
[jira] [Updated] (AVRO-1064) Encoder.writeString throws NullPointerException when the charSequence is null
[
https://issues.apache.org/jira/browse/AVRO-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-1064:
Resolution: Not A Bug
Status: Resolved (was: Patch Available)
> Encoder.writeString throws NullPointerException when the charSequence is null
> -
>
> Key: AVRO-1064
> URL: https://issues.apache.org/jira/browse/AVRO-1064
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.6.3
>Reporter: Kevin Zhao
> Attachments: Encoder.java.patch
>
>
> org.apache.avro.io.Encoder.writeString(CharSequence charSequence) throws
> NullPointerException when dealing with a javabean which has a null value in
> one of its fields, like the following one:
> Person person = new Person();
> person.setName("Kevin");
> person.setId(null);
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-1965) Reparsing an existing schema mutates the schema
[
https://issues.apache.org/jira/browse/AVRO-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16059305#comment-16059305
]
Nandor Kollar commented on AVRO-1965:
-
Looks good to me too.
> Reparsing an existing schema mutates the schema
> ---
>
> Key: AVRO-1965
> URL: https://issues.apache.org/jira/browse/AVRO-1965
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.1
>Reporter: David Maughan
>Assignee: Doug Cutting
> Fix For: 1.8.3
>
> Attachments: AVRO-1965.patch
>
>
> h2. Overview
> In certain scenarios re-parsing a schema constructed via {{SchemaBuilder}} or
> {{Schema.createRecord}} produces a schema that is no longer equal to the
> original. In the below example, after parsing the existing schema, the
> namespace of the top level record is cascaded down to the bottom level
> record. Note that the way the top level record is being created with the
> namespace as part of the name {{default.a}} is the same way Hive's
> {{AvroSerDe}} constructs a schema from a Hive table's metadata.
> h2. Failing test case
> {code}
> Schema d = SchemaBuilder.builder().intType();
> Schema c =
> SchemaBuilder.record("c").fields().name("d").type(d).noDefault().endRecord();
> Schema b =
> SchemaBuilder.record("b").fields().name("c").type(c).noDefault().endRecord();
> Schema a1 =
> SchemaBuilder.record("default.a").fields().name("b").type(b).noDefault().endRecord();
> Schema a2 = new Schema.Parser().parse(a1.toString());
> // a1 =
> {"type":"record","name":"a","namespace":"default","fields":[{"name":"b","type":{"type":"record","name":"b","namespace":"","fields":[{"name":"c","type":{"type":"record","name":"c","fields":[{"name":"d","type":"int"}]}}]}}]}
> // a2 =
> {"type":"record","name":"a","namespace":"default","fields":[{"name":"b","type":{"type":"record","name":"b","namespace":"","fields":[{"name":"c","type":{"type":"record","name":"c","namespace":"default","fields":[{"name":"d","type":"int"}]}}]}}]}
> assertThat(a2, is(a1));
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-1991) Java concat tool -appendToFirst option doesn't work with local filesystem
[
https://issues.apache.org/jira/browse/AVRO-1991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063060#comment-16063060
]
Nandor Kollar commented on AVRO-1991:
-
[~busbey] is -appendToFirst a valid option for concat? I couldn't find this
string in the project, and the example you mentioned here fails with a
different exception: Exception in thread "main" java.io.FileNotFoundException:
-appendFirst
The documentation of concat doesn't mention appendToFirst option:
{code}
concat [input-file...] output-file
Concatenates one or more input files into a new output file
by appending the input blocks without decoding them. The input
files must have the same schema, metadata and codec. If they
do not the tool will return the following error codes:
1 if the schemas don't match
2 if the metadata doesn't match
3 if the codecs don't match
If no input files are given stdin will be used. The tool
0 on success. A dash ('-') can be given as an input file
to use stdin, and as an output file to use stdout. If a directory
is given as an input-file all the files within this directory
are used.
{code}
> Java concat tool -appendToFirst option doesn't work with local filesystem
> -
>
> Key: AVRO-1991
> URL: https://issues.apache.org/jira/browse/AVRO-1991
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Sean Busbey
>
> We should either make it work (e.g. by switching to RawLocalFilesytem instead
> of LocalFileSystem) or update the cli help.
> {code}
> java -jar lang/java/tools/target/avro-tools-1.9.0-SNAPSHOT.jar concat
> -appendToFirst ~/Downloads/append.0.avro
> '/Users/busbey/Downloads/example.*.avro'
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> Exception in thread "main" java.io.IOException: Not supported
> at
> org.apache.hadoop.fs.ChecksumFileSystem.append(ChecksumFileSystem.java:352)
> at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1161)
> at org.apache.avro.tool.Util.createFromFS(Util.java:150)
> at org.apache.avro.tool.Util.fileOrStdout(Util.java:76)
> at org.apache.avro.tool.ConcatTool.run(ConcatTool.java:76)
> at org.apache.avro.tool.Main.run(Main.java:87)
> at org.apache.avro.tool.Main.main(Main.java:76)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-1990) CreateRandomFileTool should validate arguments
[
https://issues.apache.org/jira/browse/AVRO-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026366#comment-16026366
]
Nandor Kollar commented on AVRO-1990:
-
[~busbey] could you please review my pull request?
> CreateRandomFileTool should validate arguments
> --
>
> Key: AVRO-1990
> URL: https://issues.apache.org/jira/browse/AVRO-1990
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Sean Busbey
>Assignee: Nandor Kollar
> Labels: beginner
>
> Running CreateRandomFileTool without the {{--count}} argument results in a
> NPE. it should instead give a useful error message.
> {code}
> java -jar lang/java/tools/target/avro-tools-1.9.0-SNAPSHOT.jar random
> --schema '{ "type": "record", "name": "foobar", "fields": [ {"name":
> "field0", "type": "string"}, {"name":"field2", "type":"int"}]}'
> ~/Downloads/example.0.avro
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.avro.tool.CreateRandomFileTool.run(CreateRandomFileTool.java:89)
> at org.apache.avro.tool.Main.run(Main.java:87)
> at org.apache.avro.tool.Main.main(Main.java:76)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (AVRO-1990) CreateRandomFileTool should validate arguments
[
https://issues.apache.org/jira/browse/AVRO-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-1990:
Status: Patch Available (was: Open)
> CreateRandomFileTool should validate arguments
> --
>
> Key: AVRO-1990
> URL: https://issues.apache.org/jira/browse/AVRO-1990
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Sean Busbey
>Assignee: Nandor Kollar
> Labels: beginner
>
> Running CreateRandomFileTool without the {{--count}} argument results in a
> NPE. it should instead give a useful error message.
> {code}
> java -jar lang/java/tools/target/avro-tools-1.9.0-SNAPSHOT.jar random
> --schema '{ "type": "record", "name": "foobar", "fields": [ {"name":
> "field0", "type": "string"}, {"name":"field2", "type":"int"}]}'
> ~/Downloads/example.0.avro
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.avro.tool.CreateRandomFileTool.run(CreateRandomFileTool.java:89)
> at org.apache.avro.tool.Main.run(Main.java:87)
> at org.apache.avro.tool.Main.main(Main.java:76)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (AVRO-1990) CreateRandomFileTool should validate arguments
[
https://issues.apache.org/jira/browse/AVRO-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar reassigned AVRO-1990:
---
Assignee: Nandor Kollar
> CreateRandomFileTool should validate arguments
> --
>
> Key: AVRO-1990
> URL: https://issues.apache.org/jira/browse/AVRO-1990
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Sean Busbey
>Assignee: Nandor Kollar
> Labels: beginner
>
> Running CreateRandomFileTool without the {{--count}} argument results in a
> NPE. it should instead give a useful error message.
> {code}
> java -jar lang/java/tools/target/avro-tools-1.9.0-SNAPSHOT.jar random
> --schema '{ "type": "record", "name": "foobar", "fields": [ {"name":
> "field0", "type": "string"}, {"name":"field2", "type":"int"}]}'
> ~/Downloads/example.0.avro
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.avro.tool.CreateRandomFileTool.run(CreateRandomFileTool.java:89)
> at org.apache.avro.tool.Main.run(Main.java:87)
> at org.apache.avro.tool.Main.main(Main.java:76)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AVRO-1575) Platform specific end of line hardcoded in unit test causes test failure on Windows.
[ https://issues.apache.org/jira/browse/AVRO-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029218#comment-16029218 ] Nandor Kollar commented on AVRO-1575: - It seems this is fixed with the duplicate issue. [~zolyfarkas] is this still an outstanding issue? If not, may I close this? > Platform specific end of line hardcoded in unit test causes test failure on > Windows. > > > Key: AVRO-1575 > URL: https://issues.apache.org/jira/browse/AVRO-1575 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.7.7 > Environment: Windows >Reporter: Zoltan Farkas >Priority: Trivial > Attachments: Avro-TestSchemaCompatibility.patch > > Original Estimate: 5m > Remaining Estimate: 5m > > diff --git > a/lang/java/avro/src/test/java/org/apache/avro/TestSchemaCompatibility.java > b/lang/java/avro/src/test/java/org/apache/avro/TestSchemaCompatibility.java > index 10d87df..6114981 100644 > --- > a/lang/java/avro/src/test/java/org/apache/avro/TestSchemaCompatibility.java > +++ > b/lang/java/avro/src/test/java/org/apache/avro/TestSchemaCompatibility.java > @@ -246,8 +246,8 @@ > reader, > WRITER_SCHEMA, > String.format( > -"Data encoded using writer schema:\n%s\n" > -+ "will or may fail to decode using reader schema:\n%s\n", > +"Data encoded using writer schema:%n%s%n" > ++ "will or may fail to decode using reader schema:%n%s%n", > WRITER_SCHEMA.toString(true), > reader.toString(true))); > > @@ -271,8 +271,8 @@ > invalidReader, > STRING_ARRAY_SCHEMA, > String.format( > -"Data encoded using writer schema:\n%s\n" > -+ "will or may fail to decode using reader schema:\n%s\n", > +"Data encoded using writer schema:%n%s%n" > ++ "will or may fail to decode using reader schema:%n%s%n", > STRING_ARRAY_SCHEMA.toString(true), > invalidReader.toString(true))); > > @@ -299,8 +299,8 @@ > INT_SCHEMA, > STRING_SCHEMA, > String.format( > -"Data encoded using writer schema:\n%s\n" > -+ "will or may fail to decode using reader schema:\n%s\n", > +"Data encoded using writer schema:%n%s%n" > ++ "will or may fail to decode using reader schema:%n%s%n", > STRING_SCHEMA.toString(true), > INT_SCHEMA.toString(true))); > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AVRO-1575) Platform specific end of line hardcoded in unit test causes test failure on Windows.
[ https://issues.apache.org/jira/browse/AVRO-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029388#comment-16029388 ] Nandor Kollar commented on AVRO-1575: - Thanks, resolved as duplicate. > Platform specific end of line hardcoded in unit test causes test failure on > Windows. > > > Key: AVRO-1575 > URL: https://issues.apache.org/jira/browse/AVRO-1575 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.7.7 > Environment: Windows >Reporter: Zoltan Farkas >Priority: Trivial > Attachments: Avro-TestSchemaCompatibility.patch > > Original Estimate: 5m > Remaining Estimate: 5m > > diff --git > a/lang/java/avro/src/test/java/org/apache/avro/TestSchemaCompatibility.java > b/lang/java/avro/src/test/java/org/apache/avro/TestSchemaCompatibility.java > index 10d87df..6114981 100644 > --- > a/lang/java/avro/src/test/java/org/apache/avro/TestSchemaCompatibility.java > +++ > b/lang/java/avro/src/test/java/org/apache/avro/TestSchemaCompatibility.java > @@ -246,8 +246,8 @@ > reader, > WRITER_SCHEMA, > String.format( > -"Data encoded using writer schema:\n%s\n" > -+ "will or may fail to decode using reader schema:\n%s\n", > +"Data encoded using writer schema:%n%s%n" > ++ "will or may fail to decode using reader schema:%n%s%n", > WRITER_SCHEMA.toString(true), > reader.toString(true))); > > @@ -271,8 +271,8 @@ > invalidReader, > STRING_ARRAY_SCHEMA, > String.format( > -"Data encoded using writer schema:\n%s\n" > -+ "will or may fail to decode using reader schema:\n%s\n", > +"Data encoded using writer schema:%n%s%n" > ++ "will or may fail to decode using reader schema:%n%s%n", > STRING_ARRAY_SCHEMA.toString(true), > invalidReader.toString(true))); > > @@ -299,8 +299,8 @@ > INT_SCHEMA, > STRING_SCHEMA, > String.format( > -"Data encoded using writer schema:\n%s\n" > -+ "will or may fail to decode using reader schema:\n%s\n", > +"Data encoded using writer schema:%n%s%n" > ++ "will or may fail to decode using reader schema:%n%s%n", > STRING_SCHEMA.toString(true), > INT_SCHEMA.toString(true))); > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (AVRO-1575) Platform specific end of line hardcoded in unit test causes test failure on Windows.
[ https://issues.apache.org/jira/browse/AVRO-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar resolved AVRO-1575. - Resolution: Duplicate > Platform specific end of line hardcoded in unit test causes test failure on > Windows. > > > Key: AVRO-1575 > URL: https://issues.apache.org/jira/browse/AVRO-1575 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.7.7 > Environment: Windows >Reporter: Zoltan Farkas >Priority: Trivial > Attachments: Avro-TestSchemaCompatibility.patch > > Original Estimate: 5m > Remaining Estimate: 5m > > diff --git > a/lang/java/avro/src/test/java/org/apache/avro/TestSchemaCompatibility.java > b/lang/java/avro/src/test/java/org/apache/avro/TestSchemaCompatibility.java > index 10d87df..6114981 100644 > --- > a/lang/java/avro/src/test/java/org/apache/avro/TestSchemaCompatibility.java > +++ > b/lang/java/avro/src/test/java/org/apache/avro/TestSchemaCompatibility.java > @@ -246,8 +246,8 @@ > reader, > WRITER_SCHEMA, > String.format( > -"Data encoded using writer schema:\n%s\n" > -+ "will or may fail to decode using reader schema:\n%s\n", > +"Data encoded using writer schema:%n%s%n" > ++ "will or may fail to decode using reader schema:%n%s%n", > WRITER_SCHEMA.toString(true), > reader.toString(true))); > > @@ -271,8 +271,8 @@ > invalidReader, > STRING_ARRAY_SCHEMA, > String.format( > -"Data encoded using writer schema:\n%s\n" > -+ "will or may fail to decode using reader schema:\n%s\n", > +"Data encoded using writer schema:%n%s%n" > ++ "will or may fail to decode using reader schema:%n%s%n", > STRING_ARRAY_SCHEMA.toString(true), > invalidReader.toString(true))); > > @@ -299,8 +299,8 @@ > INT_SCHEMA, > STRING_SCHEMA, > String.format( > -"Data encoded using writer schema:\n%s\n" > -+ "will or may fail to decode using reader schema:\n%s\n", > +"Data encoded using writer schema:%n%s%n" > ++ "will or may fail to decode using reader schema:%n%s%n", > STRING_SCHEMA.toString(true), > INT_SCHEMA.toString(true))); > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AVRO-1401) @Nullable does not work with byte[]
[
https://issues.apache.org/jira/browse/AVRO-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043568#comment-16043568
]
Nandor Kollar commented on AVRO-1401:
-
[~gszadovszky] could you please review my pull request?
> @Nullable does not work with byte[]
> ---
>
> Key: AVRO-1401
> URL: https://issues.apache.org/jira/browse/AVRO-1401
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.7.5
>Reporter: dennis lucero
> Labels: java, reflection, union
>
> @Nullable does not seem to be compatible with byte[] (Avro type bytes)
> {code:java}
> public static void main(String[] args) throws IOException
> {
> Schema schema = ReflectData.get().getSchema(MyRecord.class);
> DatumWriter protocol = ReflectData.get().createDatumWriter(schema);
> DataFileWriter writer = new
> DataFileWriter(protocol).create(schema, System.out);
> writer.append(new MyRecord());
> writer.close();
> }
> public static class MyRecord {
> @Nullable
> byte[] bytes = "foo".getBytes();
> }
> {code}
> {code}
> org.apache.avro.UnresolvedUnionException: Not in union
> ["null",{"type":"bytes","java-class":"[B"}]: [B@6d3f1f92
> at
> org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:600)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:151)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:71)
> at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114)
> at
> org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:175)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
> at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:257)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (AVRO-1401) @Nullable does not work with byte[]
[
https://issues.apache.org/jira/browse/AVRO-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-1401:
Status: Patch Available (was: Open)
> @Nullable does not work with byte[]
> ---
>
> Key: AVRO-1401
> URL: https://issues.apache.org/jira/browse/AVRO-1401
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.7.5
>Reporter: dennis lucero
>Assignee: Nandor Kollar
> Labels: java, reflection, union
>
> @Nullable does not seem to be compatible with byte[] (Avro type bytes)
> {code:java}
> public static void main(String[] args) throws IOException
> {
> Schema schema = ReflectData.get().getSchema(MyRecord.class);
> DatumWriter protocol = ReflectData.get().createDatumWriter(schema);
> DataFileWriter writer = new
> DataFileWriter(protocol).create(schema, System.out);
> writer.append(new MyRecord());
> writer.close();
> }
> public static class MyRecord {
> @Nullable
> byte[] bytes = "foo".getBytes();
> }
> {code}
> {code}
> org.apache.avro.UnresolvedUnionException: Not in union
> ["null",{"type":"bytes","java-class":"[B"}]: [B@6d3f1f92
> at
> org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:600)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:151)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:71)
> at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114)
> at
> org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:175)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
> at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:257)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (AVRO-1401) @Nullable does not work with byte[]
[
https://issues.apache.org/jira/browse/AVRO-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar reassigned AVRO-1401:
---
Assignee: Nandor Kollar
> @Nullable does not work with byte[]
> ---
>
> Key: AVRO-1401
> URL: https://issues.apache.org/jira/browse/AVRO-1401
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.7.5
>Reporter: dennis lucero
>Assignee: Nandor Kollar
> Labels: java, reflection, union
>
> @Nullable does not seem to be compatible with byte[] (Avro type bytes)
> {code:java}
> public static void main(String[] args) throws IOException
> {
> Schema schema = ReflectData.get().getSchema(MyRecord.class);
> DatumWriter protocol = ReflectData.get().createDatumWriter(schema);
> DataFileWriter writer = new
> DataFileWriter(protocol).create(schema, System.out);
> writer.append(new MyRecord());
> writer.close();
> }
> public static class MyRecord {
> @Nullable
> byte[] bytes = "foo".getBytes();
> }
> {code}
> {code}
> org.apache.avro.UnresolvedUnionException: Not in union
> ["null",{"type":"bytes","java-class":"[B"}]: [B@6d3f1f92
> at
> org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:600)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:151)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:71)
> at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114)
> at
> org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:175)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
> at
> org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:257)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AVRO-2019) Improve documentation for logical type annotations in IDL
[
https://issues.apache.org/jira/browse/AVRO-2019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053733#comment-16053733
]
Nandor Kollar commented on AVRO-2019:
-
Indeed, looks like this is missing from the documentation. Is this covered with
tests too? I was looking for a relevant test case, but couldn't find any.
[~arosca] do you know any relevant test case for this feature?
> Improve documentation for logical type annotations in IDL
> -
>
> Key: AVRO-2019
> URL: https://issues.apache.org/jira/browse/AVRO-2019
> Project: Avro
> Issue Type: Improvement
> Components: doc, logical types
>Reporter: Andrew Rosca
>Priority: Minor
> Attachments: AVRO-2019.patch
>
>
> The IDL documentation lacks information for how annotations can be specified
> for logical types, like in the following example:
> {code}
> protocol test {
> record test {
> @logicalType("timestamp-millis")
> long time;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-1990) CreateRandomFileTool should validate arguments
[
https://issues.apache.org/jira/browse/AVRO-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-1990:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> CreateRandomFileTool should validate arguments
> --
>
> Key: AVRO-1990
> URL: https://issues.apache.org/jira/browse/AVRO-1990
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Sean Busbey
>Assignee: Nandor Kollar
> Labels: beginner
>
> Running CreateRandomFileTool without the {{--count}} argument results in a
> NPE. it should instead give a useful error message.
> {code}
> java -jar lang/java/tools/target/avro-tools-1.9.0-SNAPSHOT.jar random
> --schema '{ "type": "record", "name": "foobar", "fields": [ {"name":
> "field0", "type": "string"}, {"name":"field2", "type":"int"}]}'
> ~/Downloads/example.0.avro
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.avro.tool.CreateRandomFileTool.run(CreateRandomFileTool.java:89)
> at org.apache.avro.tool.Main.run(Main.java:87)
> at org.apache.avro.tool.Main.main(Main.java:76)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-1990) CreateRandomFileTool should validate arguments
[
https://issues.apache.org/jira/browse/AVRO-1990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-1990:
Fix Version/s: 1.9.0
> CreateRandomFileTool should validate arguments
> --
>
> Key: AVRO-1990
> URL: https://issues.apache.org/jira/browse/AVRO-1990
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.0
>Reporter: Sean Busbey
>Assignee: Nandor Kollar
> Labels: beginner
> Fix For: 1.9.0
>
>
> Running CreateRandomFileTool without the {{--count}} argument results in a
> NPE. it should instead give a useful error message.
> {code}
> java -jar lang/java/tools/target/avro-tools-1.9.0-SNAPSHOT.jar random
> --schema '{ "type": "record", "name": "foobar", "fields": [ {"name":
> "field0", "type": "string"}, {"name":"field2", "type":"int"}]}'
> ~/Downloads/example.0.avro
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.avro.tool.CreateRandomFileTool.run(CreateRandomFileTool.java:89)
> at org.apache.avro.tool.Main.run(Main.java:87)
> at org.apache.avro.tool.Main.main(Main.java:76)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-1574) Suboptimal arraylist creation
[ https://issues.apache.org/jira/browse/AVRO-1574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048038#comment-16048038 ] Nandor Kollar commented on AVRO-1574: - LGTM > Suboptimal arraylist creation > - > > Key: AVRO-1574 > URL: https://issues.apache.org/jira/browse/AVRO-1574 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.7 >Reporter: Zoltan Farkas >Priority: Trivial > Attachments: AVRO-1574.patch > > Original Estimate: 5m > Remaining Estimate: 5m > > in Schema.java: > 1231 LockableArrayList symbols = new > LockableArrayList(); > > 1232 for (JsonNode n : symbolsNode) > > 1233 symbols.add(n.getTextValue()); > should be changed to: > 1231 LockableArrayList symbols = new > LockableArrayList(symbolsNode.size()); > > 1232 for (JsonNode n : symbolsNode) > > 1233 symbols.add(n.getTextValue()); > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-1485) Specification says Record field type can be record name but implementation allows any named type.
[ https://issues.apache.org/jira/browse/AVRO-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-1485: Status: Patch Available (was: Open) > Specification says Record field type can be record name but implementation > allows any named type. > - > > Key: AVRO-1485 > URL: https://issues.apache.org/jira/browse/AVRO-1485 > Project: Avro > Issue Type: Bug > Components: java, spec >Affects Versions: 1.7.6 >Reporter: Sean Busbey >Assignee: Nandor Kollar > Attachments: AVRO-1485_1.patch > > > The [specification for Record > fields|http://avro.apache.org/docs/1.7.6/spec.html#schema_record] says that > the type is > bq. A JSON object defining a schema, or a JSON string naming a record > definition (required). > AFAICT, the Java implementation allows for any [named > type|http://avro.apache.org/docs/1.7.6/spec.html#Names]. > The specification should be updated to state any named type is allowed or the > Java implementation should restrict what can be used. The former seems less > likely to disturb current users. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (AVRO-1485) Specification says Record field type can be record name but implementation allows any named type.
[ https://issues.apache.org/jira/browse/AVRO-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar reassigned AVRO-1485: --- Assignee: Nandor Kollar > Specification says Record field type can be record name but implementation > allows any named type. > - > > Key: AVRO-1485 > URL: https://issues.apache.org/jira/browse/AVRO-1485 > Project: Avro > Issue Type: Bug > Components: java, spec >Affects Versions: 1.7.6 >Reporter: Sean Busbey >Assignee: Nandor Kollar > Attachments: AVRO-1485_1.patch > > > The [specification for Record > fields|http://avro.apache.org/docs/1.7.6/spec.html#schema_record] says that > the type is > bq. A JSON object defining a schema, or a JSON string naming a record > definition (required). > AFAICT, the Java implementation allows for any [named > type|http://avro.apache.org/docs/1.7.6/spec.html#Names]. > The specification should be updated to state any named type is allowed or the > Java implementation should restrict what can be used. The former seems less > likely to disturb current users. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-1485) Specification says Record field type can be record name but implementation allows any named type.
[ https://issues.apache.org/jira/browse/AVRO-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-1485: Attachment: AVRO-1485_1.patch > Specification says Record field type can be record name but implementation > allows any named type. > - > > Key: AVRO-1485 > URL: https://issues.apache.org/jira/browse/AVRO-1485 > Project: Avro > Issue Type: Bug > Components: java, spec >Affects Versions: 1.7.6 >Reporter: Sean Busbey > Attachments: AVRO-1485_1.patch > > > The [specification for Record > fields|http://avro.apache.org/docs/1.7.6/spec.html#schema_record] says that > the type is > bq. A JSON object defining a schema, or a JSON string naming a record > definition (required). > AFAICT, the Java implementation allows for any [named > type|http://avro.apache.org/docs/1.7.6/spec.html#Names]. > The specification should be updated to state any named type is allowed or the > Java implementation should restrict what can be used. The former seems less > likely to disturb current users. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-1485) Specification says Record field type can be record name but implementation allows any named type.
[ https://issues.apache.org/jira/browse/AVRO-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066080#comment-16066080 ] Nandor Kollar commented on AVRO-1485: - [~busbey] modified the doc according to Doug's comment, could you please review the patch? > Specification says Record field type can be record name but implementation > allows any named type. > - > > Key: AVRO-1485 > URL: https://issues.apache.org/jira/browse/AVRO-1485 > Project: Avro > Issue Type: Bug > Components: java, spec >Affects Versions: 1.7.6 >Reporter: Sean Busbey >Assignee: Nandor Kollar > Attachments: AVRO-1485_1.patch > > > The [specification for Record > fields|http://avro.apache.org/docs/1.7.6/spec.html#schema_record] says that > the type is > bq. A JSON object defining a schema, or a JSON string naming a record > definition (required). > AFAICT, the Java implementation allows for any [named > type|http://avro.apache.org/docs/1.7.6/spec.html#Names]. > The specification should be updated to state any named type is allowed or the > Java implementation should restrict what can be used. The former seems less > likely to disturb current users. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AVRO-2077) Avro tools should have an option to check reader-writer compatibility
Nandor Kollar created AVRO-2077: --- Summary: Avro tools should have an option to check reader-writer compatibility Key: AVRO-2077 URL: https://issues.apache.org/jira/browse/AVRO-2077 Project: Avro Issue Type: Bug Components: java Reporter: Nandor Kollar Priority: Minor Fix For: 1.9.0 It seems that avro-tools doesn't have any option to check for a given Avro file and a given reader schema (JSON file or URL) that the reader's schema is compatible with the writer's schema. This tool should report every compatibility problem (if there's any). According my knowledge, there's no such option for avro-tools as of now. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-1815) Incompatible schema change not detected when wrapped in a UNION
[
https://issues.apache.org/jira/browse/AVRO-1815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169932#comment-16169932
]
Nandor Kollar commented on AVRO-1815:
-
[~busbey], [~gszadovszky] it seems this problem is the same as AVRO-1883 but is
only fixed in 1.8.x. What do you think, would it make sense to backport that
change to 1.7.x so users won't have to migrate to other major release, and
close this one as duplicate?
> Incompatible schema change not detected when wrapped in a UNION
> ---
>
> Key: AVRO-1815
> URL: https://issues.apache.org/jira/browse/AVRO-1815
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.7.7
>Reporter: Martin Boyle
> Attachments: AVRO-1815.patch
>
>
> An incompatible schema change is not detected when it is in a UNION and the
> change is to the value type of a map e.g.
> field
> { "name": "segmentEkv", "type": ["null", {"type": "map", "values": {"type":
> "map", "values": "string"}}], "default": null},
> changes to
> { "name": "segmentEkv", "type": ["null", {"type": "map", "values": {"type":
> "array", "items": "int"}}], "default": null},
> The SchemaValidatorBuilder() will pass this as being compatible. Whereas
> SchemaCompatibility.check_reader_writer_compatibility will return an
> incompatible result. The problem for me is that the Confluent Schema
> Registry uses SchemaValidatorBuilder.
> Problem appears to be that while the ResolvingGrammerGenerator correctly
> marks the field as being an incompatible change, the check for errors on the
> Symbol object does not descend into the UnionAdjustActionField
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-2074) SchemaValidator can't cope with recursive schemas?
[
https://issues.apache.org/jira/browse/AVRO-2074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173372#comment-16173372
]
Nandor Kollar commented on AVRO-2074:
-
Hm, interesting, it seems those lines which stuck in infinite recursion are
submitted with AVRO-328, which was committed quite a long ago. Do you have any
idea what could be the root cause, what has changed between the two releases
which broke this logic?
> SchemaValidator can't cope with recursive schemas?
> --
>
> Key: AVRO-2074
> URL: https://issues.apache.org/jira/browse/AVRO-2074
> Project: Avro
> Issue Type: Bug
> Components: java
>Reporter: Nandor Kollar
>
> When trying to validate a record which is recursively defined:
> {code}
> Schema record = Schema.createRecord("List", null, null, false);
> record.setFields(list(new Schema.Field("head",
> Schema.create(Schema.Type.INT), null, null),
> new Schema.Field("tail", record, null, null)));
> {code}
> then schema validation
> {code}
> new SchemaValidatorBuilder().canReadStrategy().validateAll().validate(record,
> asList(record));
> {code}
> fails with StackOverflowError:
> {code}
> java.lang.StackOverflowError
> at
> org.apache.avro.io.parsing.Symbol$Sequence.flattenedSize(Symbol.java:337)
> at org.apache.avro.io.parsing.Symbol.flattenedSize(Symbol.java:230)
> at
> org.apache.avro.io.parsing.Symbol$Sequence.flattenedSize(Symbol.java:337)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-2078) Avro does not enforce schema resolution rules for Decimal type
[
https://issues.apache.org/jira/browse/AVRO-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-2078:
Assignee: Nandor Kollar
Status: Patch Available (was: Open)
> Avro does not enforce schema resolution rules for Decimal type
> --
>
> Key: AVRO-2078
> URL: https://issues.apache.org/jira/browse/AVRO-2078
> Project: Avro
> Issue Type: Bug
>Reporter: Anthony Hsu
>Assignee: Nandor Kollar
> Fix For: 1.8.2
>
> Attachments: dec.avro
>
>
> According to http://avro.apache.org/docs/1.8.2/spec.html#Decimal
> bq. For the purposes of schema resolution, two schemas that are {{decimal}}
> logical types _match_ if their scales and precisions match.
> This is not enforced.
> I wrote a file with (precision 5, scale 2) and tried to read it with a reader
> schema with (precision 3, scale 1). I expected an AvroTypeException to be
> thrown, but none was thrown.
> Test data file attached. The code to read it is:
> {noformat:title=ReadDecimal.java}
> import java.io.File;
> import org.apache.avro.Schema;
> import org.apache.avro.file.DataFileReader;
> import org.apache.avro.generic.GenericDatumReader;
> import org.apache.avro.generic.GenericRecord;
> import org.apache.avro.io.DatumReader;
> public class ReadDecimal {
> public static void main(String[] args) throws Exception {
> Schema schema = new Schema.Parser().parse("{\n" + " \"type\" :
> \"record\",\n" + " \"name\" : \"some_schema\",\n"
> + " \"namespace\" : \"com.howdy\",\n" + " \"fields\" : [ {\n" + "
> \"name\" : \"name\",\n"
> + "\"type\" : \"string\"\n" + " }, {\n" + "\"name\" :
> \"value\",\n" + "\"type\" : {\n"
> + " \"type\" : \"bytes\",\n" + " \"logicalType\" :
> \"decimal\",\n" + " \"precision\" : 3,\n"
> + " \"scale\" : 1\n" + "}\n" + " } ]\n" + "}");
> DatumReader datumReader = new GenericDatumReader<>(schema);
> // dec.avro has precision 5, scale 2
> DataFileReader dataFileReader = new DataFileReader<>(
> new File("/tmp/dec.avro"), datumReader);
> GenericRecord foo = null;
> while (dataFileReader.hasNext()) {
> foo = dataFileReader.next(foo); // AvroTypeException expected due to
> change in scale/precision but none occurs
> }
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-2078) Avro does not enforce schema resolution rules for Decimal type
[
https://issues.apache.org/jira/browse/AVRO-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174738#comment-16174738
]
Nandor Kollar commented on AVRO-2078:
-
Looks like a bug for me, the schema resolution logic is not compliant with the
spec. Created a PR with my proposed modifications.
> Avro does not enforce schema resolution rules for Decimal type
> --
>
> Key: AVRO-2078
> URL: https://issues.apache.org/jira/browse/AVRO-2078
> Project: Avro
> Issue Type: Bug
>Reporter: Anthony Hsu
> Fix For: 1.8.2
>
> Attachments: dec.avro
>
>
> According to http://avro.apache.org/docs/1.8.2/spec.html#Decimal
> bq. For the purposes of schema resolution, two schemas that are {{decimal}}
> logical types _match_ if their scales and precisions match.
> This is not enforced.
> I wrote a file with (precision 5, scale 2) and tried to read it with a reader
> schema with (precision 3, scale 1). I expected an AvroTypeException to be
> thrown, but none was thrown.
> Test data file attached. The code to read it is:
> {noformat:title=ReadDecimal.java}
> import java.io.File;
> import org.apache.avro.Schema;
> import org.apache.avro.file.DataFileReader;
> import org.apache.avro.generic.GenericDatumReader;
> import org.apache.avro.generic.GenericRecord;
> import org.apache.avro.io.DatumReader;
> public class ReadDecimal {
> public static void main(String[] args) throws Exception {
> Schema schema = new Schema.Parser().parse("{\n" + " \"type\" :
> \"record\",\n" + " \"name\" : \"some_schema\",\n"
> + " \"namespace\" : \"com.howdy\",\n" + " \"fields\" : [ {\n" + "
> \"name\" : \"name\",\n"
> + "\"type\" : \"string\"\n" + " }, {\n" + "\"name\" :
> \"value\",\n" + "\"type\" : {\n"
> + " \"type\" : \"bytes\",\n" + " \"logicalType\" :
> \"decimal\",\n" + " \"precision\" : 3,\n"
> + " \"scale\" : 1\n" + "}\n" + " } ]\n" + "}");
> DatumReader datumReader = new GenericDatumReader<>(schema);
> // dec.avro has precision 5, scale 2
> DataFileReader dataFileReader = new DataFileReader<>(
> new File("/tmp/dec.avro"), datumReader);
> GenericRecord foo = null;
> while (dataFileReader.hasNext()) {
> foo = dataFileReader.next(foo); // AvroTypeException expected due to
> change in scale/precision but none occurs
> }
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-2003) Report specific location of schema incompatibilities
[
https://issues.apache.org/jira/browse/AVRO-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173153#comment-16173153
]
Nandor Kollar commented on AVRO-2003:
-
Thanks [~teabot], I'll help the committers with a review soon!
> Report specific location of schema incompatibilities
>
>
> Key: AVRO-2003
> URL: https://issues.apache.org/jira/browse/AVRO-2003
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.8.1
> Environment: Any java env
>Reporter: Elliot West
>Assignee: Elliot West
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: AVRO-2003.patch
>
>
> h2. Overview
> Building on the work to improve schema incompatibility reporting in
> AVRO-1933, it would be useful if the {{SchemaCompatibility}} classes could
> also report the location in the schema where any incompatibility was
> encountered.
> It is recommended that the location reported is both easily readable by
> humans and machines. In the first case this would assist schema developers to
> pin-point issues in there schema documents, and in the latter case it
> provides a useful mechanism to schema tooling, such as IDEs and editors, to
> easily select the pertinent nodes in the Schema document tree.
> h2. Implementation specifics
> To meet this requirements it is suggested that the location is encoded using
> the [JSON Pointer specification|https://tools.ietf.org/html/rfc6901]. This is
> both easily parsed by users, but is also supported by a number of libraries
> for a range of common programming languages and platforms.
> h2. Examples
> Given the following example schema, consider some incompatibility scenarios.
> For each case an expected JSON Pointer description of the incompatibility
> location is described:
> {code}
> {
> "type": "record",
> "name": "myRecord",
> "fields" : [
> {"name": "pField", "type": "long"},
> {"name": "uField", "type":
> ["null", "int", "string"]
> },
> {"name": "eField", "type":
> { "type": "enum", "name": "Suit", "symbols" : ["SPADES", "HEARTS",
> "DIAMONDS", "CLUBS"] }
> },
> {"name": "aField", "type":
> {"type": "array", "items": "string"}
> },
> {"name": "mField", "type":
> {"type": "map", "values": "long"}
> },
> {"name": "fField", "type":
> {"type": "fixed", "size": 16, "name": "md5"}
> }
> ]
> }
> {code}
> Possible incompatibility scenarions and the location that would be reported
> back to the user/tool:
> * Root type incompatibility; report location: {{/}}
> * Record name mismatch; report location: {{/name}}
> * {{pField}} type incompatibility; report location: {{/fields/0/type}}
> * {{uField}} field type incompatibility; report location: {{/fields/1/type}}
> * {{uField}} missing union branch {{string}}; report location:
> {{/fields/1/type/2}}
> * {{eField}} field type incompatibility; report location: {{/fields/2/type}}
> * {{eField}} missing enum symbol; report location: {{/fields/2/type/symbols}}
> * {{eField}} enum name mismatch; report location: {{/fields/2/type/name}}
> * {{aField}} field type incompatibility; report location: {{/fields/3/type}}
> * {{aField}} array element type incompatibility; report location:
> {{/fields/3/type/items}}
> * {{mField}} field type incompatibility; report location: {{/fields/4/type}}
> * {{mField}} map value type incompatibility; report location:
> {{/fields/4/type/values}}
> * {{fField}} field type incompatibility; report location: {{/fields/5/type}}
> * {{fField}} fixed name mismatch; report location: {{/fields/5/type/name}}
> * {{fField}} fixed size type incompatibility; report location:
> {{/fields/5/type/size}}
> * {{fField}} missing default value; report location: {{/fields/5}}
> h2. Notes
> * This ticket depends on AVRO-1933 and associated patches.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (AVRO-2086) Can't create specific record with a single Decimal field with default value
Nandor Kollar created AVRO-2086:
---
Summary: Can't create specific record with a single Decimal field
with default value
Key: AVRO-2086
URL: https://issues.apache.org/jira/browse/AVRO-2086
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.8.2
Reporter: Nandor Kollar
Given a simple record schema with a single byte field annotated with Decimal
logical type and has default value set:
{code}
{
"type":"record",
"name":"TestRecordWithDecimalFieldWithDefault",
"namespace":"org.apache.avro.specific",
"fields":[
{
"name":"f1",
"type":{
"type":"bytes",
"logicalType":"decimal",
"precision":9,
"scale":2
},
"default":"0.00"
}
]
}
{code}
compiled with
{{avro-tools compile -bigDecimal schema
TestRecordWithDecimalFieldWithDefault.avsc .}}
The following test case (which should use the decimal's default value):
{code}
@Test
public void testRecordWithDecimalLogicalTypeWithDefault() throws IOException {
SpecificData.get().addLogicalTypeConversion(new
Conversions.DecimalConversion());
TestRecordWithDecimalFieldWithDefault record =
TestRecordWithDecimalFieldWithDefault.newBuilder().build();
File data = write(TestRecordWithDecimalFieldWithDefault.getClassSchema(),
record);
List actual = read(
TestRecordWithDecimalFieldWithDefault.getClassSchema(), data);
assertEquals(new BigDecimal(0.00).setScale(2), actual.get(0).getF1());
}
{code}
fails with
{code}
org.apache.avro.AvroRuntimeException: org.apache.avro.AvroRuntimeException:
Cannot convert 808.60:BigDecimal: expected generic type
at
org.apache.avro.specific.TestRecordWithDecimalFieldWithDefault$Builder.build(TestRecordWithDecimalFieldWithDefault.java:239)
at
org.apache.avro.specific.TestSpecificLogicalTypes.testRecordWithDecimalLogicalTypeWithDefault(TestSpecificLogicalTypes.java:123)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:119)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: org.apache.avro.AvroRuntimeException: Cannot convert
808.60:BigDecimal: expected generic type
at
org.apache.avro.Conversions.convertToLogicalType(Conversions.java:172)
at
org.apache.avro.data.RecordBuilderBase.defaultValue(RecordBuilderBase.java:159)
at
org.apache.avro.specific.TestRecordWithDecimalFieldWithDefault$Builder.build(TestRecordWithDecimalFieldWithDefault.java:236)
... 30 more
Caused by: java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
[jira] [Updated] (AVRO-2086) Can't create specific record with a single Decimal field with default value
[
https://issues.apache.org/jira/browse/AVRO-2086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-2086:
Status: Patch Available (was: Open)
> Can't create specific record with a single Decimal field with default value
> ---
>
> Key: AVRO-2086
> URL: https://issues.apache.org/jira/browse/AVRO-2086
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.2
>Reporter: Nandor Kollar
>Assignee: Nandor Kollar
>
> Given a simple record schema with a single byte field annotated with Decimal
> logical type and has default value set:
> {code}
> {
>"type":"record",
>"name":"TestRecordWithDecimalFieldWithDefault",
>"namespace":"org.apache.avro.specific",
>"fields":[
> {
> "name":"f1",
> "type":{
> "type":"bytes",
> "logicalType":"decimal",
> "precision":9,
> "scale":2
> },
> "default":"0.00"
> }
>]
> }
> {code}
> compiled with
> {{avro-tools compile -bigDecimal schema
> TestRecordWithDecimalFieldWithDefault.avsc .}}
> The following test case (which should use the decimal's default value):
> {code}
> @Test
> public void testRecordWithDecimalLogicalTypeWithDefault() throws
> IOException {
> SpecificData.get().addLogicalTypeConversion(new
> Conversions.DecimalConversion());
> TestRecordWithDecimalFieldWithDefault record =
> TestRecordWithDecimalFieldWithDefault.newBuilder().build();
> File data = write(TestRecordWithDecimalFieldWithDefault.getClassSchema(),
> record);
> List actual = read(
> TestRecordWithDecimalFieldWithDefault.getClassSchema(), data);
> assertEquals(new BigDecimal(0.00).setScale(2), actual.get(0).getF1());
> }
> {code}
> fails with
> {code}
> org.apache.avro.AvroRuntimeException: org.apache.avro.AvroRuntimeException:
> Cannot convert 808.60:BigDecimal: expected generic type
> at
> org.apache.avro.specific.TestRecordWithDecimalFieldWithDefault$Builder.build(TestRecordWithDecimalFieldWithDefault.java:239)
> at
> org.apache.avro.specific.TestSpecificLogicalTypes.testRecordWithDecimalLogicalTypeWithDefault(TestSpecificLogicalTypes.java:123)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:119)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
> at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
> Caused by: org.apache.avro.AvroRuntimeException: Cannot convert
> 808.60:BigDecimal: expected generic type
> at
>
[jira] [Assigned] (AVRO-2086) Can't create specific record with a single Decimal field with default value
[
https://issues.apache.org/jira/browse/AVRO-2086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar reassigned AVRO-2086:
---
Assignee: Nandor Kollar
> Can't create specific record with a single Decimal field with default value
> ---
>
> Key: AVRO-2086
> URL: https://issues.apache.org/jira/browse/AVRO-2086
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.2
>Reporter: Nandor Kollar
>Assignee: Nandor Kollar
>
> Given a simple record schema with a single byte field annotated with Decimal
> logical type and has default value set:
> {code}
> {
>"type":"record",
>"name":"TestRecordWithDecimalFieldWithDefault",
>"namespace":"org.apache.avro.specific",
>"fields":[
> {
> "name":"f1",
> "type":{
> "type":"bytes",
> "logicalType":"decimal",
> "precision":9,
> "scale":2
> },
> "default":"0.00"
> }
>]
> }
> {code}
> compiled with
> {{avro-tools compile -bigDecimal schema
> TestRecordWithDecimalFieldWithDefault.avsc .}}
> The following test case (which should use the decimal's default value):
> {code}
> @Test
> public void testRecordWithDecimalLogicalTypeWithDefault() throws
> IOException {
> SpecificData.get().addLogicalTypeConversion(new
> Conversions.DecimalConversion());
> TestRecordWithDecimalFieldWithDefault record =
> TestRecordWithDecimalFieldWithDefault.newBuilder().build();
> File data = write(TestRecordWithDecimalFieldWithDefault.getClassSchema(),
> record);
> List actual = read(
> TestRecordWithDecimalFieldWithDefault.getClassSchema(), data);
> assertEquals(new BigDecimal(0.00).setScale(2), actual.get(0).getF1());
> }
> {code}
> fails with
> {code}
> org.apache.avro.AvroRuntimeException: org.apache.avro.AvroRuntimeException:
> Cannot convert 808.60:BigDecimal: expected generic type
> at
> org.apache.avro.specific.TestRecordWithDecimalFieldWithDefault$Builder.build(TestRecordWithDecimalFieldWithDefault.java:239)
> at
> org.apache.avro.specific.TestSpecificLogicalTypes.testRecordWithDecimalLogicalTypeWithDefault(TestSpecificLogicalTypes.java:123)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:119)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
> at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
> Caused by: org.apache.avro.AvroRuntimeException: Cannot convert
> 808.60:BigDecimal: expected generic type
> at
>
[jira] [Created] (AVRO-2097) Improve documentation: emphasize that default values are interpreted in physical type and not in logical
Nandor Kollar created AVRO-2097: --- Summary: Improve documentation: emphasize that default values are interpreted in physical type and not in logical Key: AVRO-2097 URL: https://issues.apache.org/jira/browse/AVRO-2097 Project: Avro Issue Type: Improvement Components: doc Affects Versions: 1.8.2 Reporter: Nandor Kollar Priority: Minor Documentation is not clear how default values specified in the schema are interpreted. It is very easy to misunderstand, and think that setting a decimal filed's default value to "0.00" is interpreted as 0.00 decimal, but in fact it isn't! The default value is interpreted in the underlying *physical type*, and not in the logical type. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2003) Report specific location of schema incompatibilities
[
https://issues.apache.org/jira/browse/AVRO-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207135#comment-16207135
]
Nandor Kollar commented on AVRO-2003:
-
[~busbey] I reviewed Elliot's pull request, and looks good to me! Could you
please have a look at it too, as the changed are non trivial?
> Report specific location of schema incompatibilities
>
>
> Key: AVRO-2003
> URL: https://issues.apache.org/jira/browse/AVRO-2003
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.8.1
> Environment: Any java env
>Reporter: Elliot West
>Assignee: Elliot West
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: AVRO-2003.patch
>
>
> h2. Overview
> Building on the work to improve schema incompatibility reporting in
> AVRO-1933, it would be useful if the {{SchemaCompatibility}} classes could
> also report the location in the schema where any incompatibility was
> encountered.
> It is recommended that the location reported is both easily readable by
> humans and machines. In the first case this would assist schema developers to
> pin-point issues in there schema documents, and in the latter case it
> provides a useful mechanism to schema tooling, such as IDEs and editors, to
> easily select the pertinent nodes in the Schema document tree.
> h2. Implementation specifics
> To meet this requirements it is suggested that the location is encoded using
> the [JSON Pointer specification|https://tools.ietf.org/html/rfc6901]. This is
> both easily parsed by users, but is also supported by a number of libraries
> for a range of common programming languages and platforms.
> h2. Examples
> Given the following example schema, consider some incompatibility scenarios.
> For each case an expected JSON Pointer description of the incompatibility
> location is described:
> {code}
> {
> "type": "record",
> "name": "myRecord",
> "fields" : [
> {"name": "pField", "type": "long"},
> {"name": "uField", "type":
> ["null", "int", "string"]
> },
> {"name": "eField", "type":
> { "type": "enum", "name": "Suit", "symbols" : ["SPADES", "HEARTS",
> "DIAMONDS", "CLUBS"] }
> },
> {"name": "aField", "type":
> {"type": "array", "items": "string"}
> },
> {"name": "mField", "type":
> {"type": "map", "values": "long"}
> },
> {"name": "fField", "type":
> {"type": "fixed", "size": 16, "name": "md5"}
> }
> ]
> }
> {code}
> Possible incompatibility scenarions and the location that would be reported
> back to the user/tool:
> * Root type incompatibility; report location: {{/}}
> * Record name mismatch; report location: {{/name}}
> * {{pField}} type incompatibility; report location: {{/fields/0/type}}
> * {{uField}} field type incompatibility; report location: {{/fields/1/type}}
> * {{uField}} missing union branch {{string}}; report location:
> {{/fields/1/type/2}}
> * {{eField}} field type incompatibility; report location: {{/fields/2/type}}
> * {{eField}} missing enum symbol; report location: {{/fields/2/type/symbols}}
> * {{eField}} enum name mismatch; report location: {{/fields/2/type/name}}
> * {{aField}} field type incompatibility; report location: {{/fields/3/type}}
> * {{aField}} array element type incompatibility; report location:
> {{/fields/3/type/items}}
> * {{mField}} field type incompatibility; report location: {{/fields/4/type}}
> * {{mField}} map value type incompatibility; report location:
> {{/fields/4/type/values}}
> * {{fField}} field type incompatibility; report location: {{/fields/5/type}}
> * {{fField}} fixed name mismatch; report location: {{/fields/5/type/name}}
> * {{fField}} fixed size type incompatibility; report location:
> {{/fields/5/type/size}}
> * {{fField}} missing default value; report location: {{/fields/5}}
> h2. Notes
> * This ticket depends on AVRO-1933 and associated patches.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-2072) ResolvingGrammarGenerator doesn't implement schema resolution correctly for unions
[ https://issues.apache.org/jira/browse/AVRO-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2072: Component/s: java > ResolvingGrammarGenerator doesn't implement schema resolution correctly for > unions > -- > > Key: AVRO-2072 > URL: https://issues.apache.org/jira/browse/AVRO-2072 > Project: Avro > Issue Type: Bug > Components: java >Reporter: Nandor Kollar >Assignee: Nandor Kollar > Attachments: 0001-AVRO-1931-Additional-test-cases.patch, > AVRO-2072_2.patch, AVRO-2072.patch > > > According to > [specification|https://avro.apache.org/docs/current/spec.html#Schema+Resolution], > int and long is promotable to float, but when using SchemaValidator, a union > with a single int or long branch is not readable by an union with a float > branch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-2072) ResolvingGrammarGenerator doesn't implement schema resolution correctly for unions
[ https://issues.apache.org/jira/browse/AVRO-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2072: Affects Version/s: 1.7.7 1.8.1 > ResolvingGrammarGenerator doesn't implement schema resolution correctly for > unions > -- > > Key: AVRO-2072 > URL: https://issues.apache.org/jira/browse/AVRO-2072 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.7.7, 1.8.1 >Reporter: Nandor Kollar >Assignee: Nandor Kollar > Attachments: 0001-AVRO-1931-Additional-test-cases.patch, > AVRO-2072_2.patch, AVRO-2072.patch > > > According to > [specification|https://avro.apache.org/docs/current/spec.html#Schema+Resolution], > int and long is promotable to float, but when using SchemaValidator, a union > with a single int or long branch is not readable by an union with a float > branch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-2072) ResolvingGrammarGenerator doesn't implement schema resolution correctly for unions
[ https://issues.apache.org/jira/browse/AVRO-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2072: Attachment: AVRO-2072.patch > ResolvingGrammarGenerator doesn't implement schema resolution correctly for > unions > -- > > Key: AVRO-2072 > URL: https://issues.apache.org/jira/browse/AVRO-2072 > Project: Avro > Issue Type: Bug >Reporter: Nandor Kollar >Assignee: Nandor Kollar > Attachments: AVRO-2072.patch > > > According to > [specification|https://avro.apache.org/docs/current/spec.html#Schema+Resolution], > int and long is promotable to float, but when using SchemaValidator, a union > with a single int or long branch is not readable by an union with a float > branch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-2072) ResolvingGrammarGenerator doesn't implement schema resolution correctly for unions
[ https://issues.apache.org/jira/browse/AVRO-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2072: Attachment: AVRO-2072_3.patch > ResolvingGrammarGenerator doesn't implement schema resolution correctly for > unions > -- > > Key: AVRO-2072 > URL: https://issues.apache.org/jira/browse/AVRO-2072 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.7.7, 1.8.1 >Reporter: Nandor Kollar >Assignee: Nandor Kollar > Attachments: 0001-AVRO-1931-Additional-test-cases.patch, > AVRO-2072_2.patch, AVRO-2072_3.patch, AVRO-2072.patch > > > According to > [specification|https://avro.apache.org/docs/current/spec.html#Schema+Resolution], > int and long is promotable to float, but when using SchemaValidator, a union > with a single int or long branch is not readable by an union with a float > branch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2072) ResolvingGrammarGenerator doesn't implement schema resolution correctly for unions
[ https://issues.apache.org/jira/browse/AVRO-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16160994#comment-16160994 ] Nandor Kollar commented on AVRO-2072: - [~epkanol] thanks for the review, in AVRO-2072_3.patch I modified the indentation according to your comment, and renamed the method to firstMatchingBranch to reflect the logic better. > ResolvingGrammarGenerator doesn't implement schema resolution correctly for > unions > -- > > Key: AVRO-2072 > URL: https://issues.apache.org/jira/browse/AVRO-2072 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.7.7, 1.8.1 >Reporter: Nandor Kollar >Assignee: Nandor Kollar > Attachments: 0001-AVRO-1931-Additional-test-cases.patch, > AVRO-2072_2.patch, AVRO-2072_3.patch, AVRO-2072.patch > > > According to > [specification|https://avro.apache.org/docs/current/spec.html#Schema+Resolution], > int and long is promotable to float, but when using SchemaValidator, a union > with a single int or long branch is not readable by an union with a float > branch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (AVRO-2075) Allow SchemaCompatibility to report possibly lossy conversions
[ https://issues.apache.org/jira/browse/AVRO-2075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar reassigned AVRO-2075: --- Assignee: Anders Sundelin > Allow SchemaCompatibility to report possibly lossy conversions > -- > > Key: AVRO-2075 > URL: https://issues.apache.org/jira/browse/AVRO-2075 > Project: Avro > Issue Type: Improvement >Affects Versions: 1.7.7, 1.8.2 > Environment: Java >Reporter: Anders Sundelin >Assignee: Anders Sundelin >Priority: Minor > Attachments: > 0001-AVRO-2075-Add-option-to-report-possible-data-loss-in.patch > > > It is stated in the Avro spec that int and long values are promotable to > floats and doubles. > However, numeric promotions to floats are lossy (losing precision), as is > long promotion to double. > It is suggested that the SchemaCompatibility class is updated to be able to > flag conversions that have the possibility to be lossy as errors. The > attached patch does just that, by adding a new boolean flag (allowDataLoss), > preserving backwards compatibility by defaulting this flag to true. > Testcases illustrating the problem has been added to the unit test class > TestReadingWritingDataInEvolvedSchemas -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-2072) ResolvingGrammarGenerator doesn't implement schema resolution correctly for unions
[ https://issues.apache.org/jira/browse/AVRO-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2072: Attachment: AVRO-2072_2.patch > ResolvingGrammarGenerator doesn't implement schema resolution correctly for > unions > -- > > Key: AVRO-2072 > URL: https://issues.apache.org/jira/browse/AVRO-2072 > Project: Avro > Issue Type: Bug >Reporter: Nandor Kollar >Assignee: Nandor Kollar > Attachments: 0001-AVRO-1931-Additional-test-cases.patch, > AVRO-2072_2.patch, AVRO-2072.patch > > > According to > [specification|https://avro.apache.org/docs/current/spec.html#Schema+Resolution], > int and long is promotable to float, but when using SchemaValidator, a union > with a single int or long branch is not readable by an union with a float > branch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2072) ResolvingGrammarGenerator doesn't implement schema resolution correctly for unions
[
https://issues.apache.org/jira/browse/AVRO-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156991#comment-16156991
]
Nandor Kollar commented on AVRO-2072:
-
Thanks [~epkanol], I uploaded AVRO-2072_2.patch with your additional test
cases, modified the failing ones in {{TestReadingWritingDataInEvolvedSchemas}}
which shouldn't fail after the fix in {{ResolvingGrammarGenerator}}, and
deleted the ignored case from {{TestSchemaCompatibility}}, since we should
support those promotions: changing the specification is not the best idea
according to [~cutting] and [~busbey], so we should allow this promotion, thus
this is a bug in the implementation (see discussion on Avro user's mailing
list). Anders, could you please help with a review?
> ResolvingGrammarGenerator doesn't implement schema resolution correctly for
> unions
> --
>
> Key: AVRO-2072
> URL: https://issues.apache.org/jira/browse/AVRO-2072
> Project: Avro
> Issue Type: Bug
>Reporter: Nandor Kollar
>Assignee: Nandor Kollar
> Attachments: 0001-AVRO-1931-Additional-test-cases.patch,
> AVRO-2072_2.patch, AVRO-2072.patch
>
>
> According to
> [specification|https://avro.apache.org/docs/current/spec.html#Schema+Resolution],
> int and long is promotable to float, but when using SchemaValidator, a union
> with a single int or long branch is not readable by an union with a float
> branch.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-2003) Report specific location of schema incompatibilities
[
https://issues.apache.org/jira/browse/AVRO-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156938#comment-16156938
]
Nandor Kollar commented on AVRO-2003:
-
[~teabot] now AVRO-1933 is committed to master, due to this your PR has several
merge conflicts. Could you please update it with merging your changes to latest
Apache master?
> Report specific location of schema incompatibilities
>
>
> Key: AVRO-2003
> URL: https://issues.apache.org/jira/browse/AVRO-2003
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.8.1
> Environment: Any java env
>Reporter: Elliot West
>Assignee: Elliot West
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: AVRO-2003.patch
>
>
> h2. Overview
> Building on the work to improve schema incompatibility reporting in
> AVRO-1933, it would be useful if the {{SchemaCompatibility}} classes could
> also report the location in the schema where any incompatibility was
> encountered.
> It is recommended that the location reported is both easily readable by
> humans and machines. In the first case this would assist schema developers to
> pin-point issues in there schema documents, and in the latter case it
> provides a useful mechanism to schema tooling, such as IDEs and editors, to
> easily select the pertinent nodes in the Schema document tree.
> h2. Implementation specifics
> To meet this requirements it is suggested that the location is encoded using
> the [JSON Pointer specification|https://tools.ietf.org/html/rfc6901]. This is
> both easily parsed by users, but is also supported by a number of libraries
> for a range of common programming languages and platforms.
> h2. Examples
> Given the following example schema, consider some incompatibility scenarios.
> For each case an expected JSON Pointer description of the incompatibility
> location is described:
> {code}
> {
> "type": "record",
> "name": "myRecord",
> "fields" : [
> {"name": "pField", "type": "long"},
> {"name": "uField", "type":
> ["null", "int", "string"]
> },
> {"name": "eField", "type":
> { "type": "enum", "name": "Suit", "symbols" : ["SPADES", "HEARTS",
> "DIAMONDS", "CLUBS"] }
> },
> {"name": "aField", "type":
> {"type": "array", "items": "string"}
> },
> {"name": "mField", "type":
> {"type": "map", "values": "long"}
> },
> {"name": "fField", "type":
> {"type": "fixed", "size": 16, "name": "md5"}
> }
> ]
> }
> {code}
> Possible incompatibility scenarions and the location that would be reported
> back to the user/tool:
> * Root type incompatibility; report location: {{/}}
> * Record name mismatch; report location: {{/name}}
> * {{pField}} type incompatibility; report location: {{/fields/0/type}}
> * {{uField}} field type incompatibility; report location: {{/fields/1/type}}
> * {{uField}} missing union branch {{string}}; report location:
> {{/fields/1/type/2}}
> * {{eField}} field type incompatibility; report location: {{/fields/2/type}}
> * {{eField}} missing enum symbol; report location: {{/fields/2/type/symbols}}
> * {{eField}} enum name mismatch; report location: {{/fields/2/type/name}}
> * {{aField}} field type incompatibility; report location: {{/fields/3/type}}
> * {{aField}} array element type incompatibility; report location:
> {{/fields/3/type/items}}
> * {{mField}} field type incompatibility; report location: {{/fields/4/type}}
> * {{mField}} map value type incompatibility; report location:
> {{/fields/4/type/values}}
> * {{fField}} field type incompatibility; report location: {{/fields/5/type}}
> * {{fField}} fixed name mismatch; report location: {{/fields/5/type/name}}
> * {{fField}} fixed size type incompatibility; report location:
> {{/fields/5/type/size}}
> * {{fField}} missing default value; report location: {{/fields/5}}
> h2. Notes
> * This ticket depends on AVRO-1933 and associated patches.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-2072) ResolvingGrammarGenerator doesn't implement schema resolution correctly for unions
[ https://issues.apache.org/jira/browse/AVRO-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16162605#comment-16162605 ] Nandor Kollar commented on AVRO-2072: - Thanks [~gszadovszky] and [~epkanol] for reviewing and committing this patch! > ResolvingGrammarGenerator doesn't implement schema resolution correctly for > unions > -- > > Key: AVRO-2072 > URL: https://issues.apache.org/jira/browse/AVRO-2072 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.7.7, 1.8.1 >Reporter: Nandor Kollar >Assignee: Nandor Kollar > Attachments: 0001-AVRO-1931-Additional-test-cases.patch, > AVRO-2072_2.patch, AVRO-2072_3.patch, AVRO-2072.patch > > > According to > [specification|https://avro.apache.org/docs/current/spec.html#Schema+Resolution], > int and long is promotable to float, but when using SchemaValidator, a union > with a single int or long branch is not readable by an union with a float > branch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-2072) ResolvingGrammarGenerator doesn't implement schema resolution correctly for unions
[ https://issues.apache.org/jira/browse/AVRO-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2072: Resolution: Fixed Hadoop Flags: Incompatible change,Reviewed (was: Incompatible change) Fix Version/s: 1.9.0 Status: Resolved (was: Patch Available) > ResolvingGrammarGenerator doesn't implement schema resolution correctly for > unions > -- > > Key: AVRO-2072 > URL: https://issues.apache.org/jira/browse/AVRO-2072 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.7.7, 1.8.1 >Reporter: Nandor Kollar >Assignee: Nandor Kollar > Fix For: 1.9.0 > > Attachments: 0001-AVRO-1931-Additional-test-cases.patch, > AVRO-2072_2.patch, AVRO-2072_3.patch, AVRO-2072.patch > > > According to > [specification|https://avro.apache.org/docs/current/spec.html#Schema+Resolution], > int and long is promotable to float, but when using SchemaValidator, a union > with a single int or long branch is not readable by an union with a float > branch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2003) Report specific location of schema incompatibilities
[
https://issues.apache.org/jira/browse/AVRO-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188327#comment-16188327
]
Nandor Kollar commented on AVRO-2003:
-
[~teabot] I added some comments to the PR.
> Report specific location of schema incompatibilities
>
>
> Key: AVRO-2003
> URL: https://issues.apache.org/jira/browse/AVRO-2003
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.8.1
> Environment: Any java env
>Reporter: Elliot West
>Assignee: Elliot West
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: AVRO-2003.patch
>
>
> h2. Overview
> Building on the work to improve schema incompatibility reporting in
> AVRO-1933, it would be useful if the {{SchemaCompatibility}} classes could
> also report the location in the schema where any incompatibility was
> encountered.
> It is recommended that the location reported is both easily readable by
> humans and machines. In the first case this would assist schema developers to
> pin-point issues in there schema documents, and in the latter case it
> provides a useful mechanism to schema tooling, such as IDEs and editors, to
> easily select the pertinent nodes in the Schema document tree.
> h2. Implementation specifics
> To meet this requirements it is suggested that the location is encoded using
> the [JSON Pointer specification|https://tools.ietf.org/html/rfc6901]. This is
> both easily parsed by users, but is also supported by a number of libraries
> for a range of common programming languages and platforms.
> h2. Examples
> Given the following example schema, consider some incompatibility scenarios.
> For each case an expected JSON Pointer description of the incompatibility
> location is described:
> {code}
> {
> "type": "record",
> "name": "myRecord",
> "fields" : [
> {"name": "pField", "type": "long"},
> {"name": "uField", "type":
> ["null", "int", "string"]
> },
> {"name": "eField", "type":
> { "type": "enum", "name": "Suit", "symbols" : ["SPADES", "HEARTS",
> "DIAMONDS", "CLUBS"] }
> },
> {"name": "aField", "type":
> {"type": "array", "items": "string"}
> },
> {"name": "mField", "type":
> {"type": "map", "values": "long"}
> },
> {"name": "fField", "type":
> {"type": "fixed", "size": 16, "name": "md5"}
> }
> ]
> }
> {code}
> Possible incompatibility scenarions and the location that would be reported
> back to the user/tool:
> * Root type incompatibility; report location: {{/}}
> * Record name mismatch; report location: {{/name}}
> * {{pField}} type incompatibility; report location: {{/fields/0/type}}
> * {{uField}} field type incompatibility; report location: {{/fields/1/type}}
> * {{uField}} missing union branch {{string}}; report location:
> {{/fields/1/type/2}}
> * {{eField}} field type incompatibility; report location: {{/fields/2/type}}
> * {{eField}} missing enum symbol; report location: {{/fields/2/type/symbols}}
> * {{eField}} enum name mismatch; report location: {{/fields/2/type/name}}
> * {{aField}} field type incompatibility; report location: {{/fields/3/type}}
> * {{aField}} array element type incompatibility; report location:
> {{/fields/3/type/items}}
> * {{mField}} field type incompatibility; report location: {{/fields/4/type}}
> * {{mField}} map value type incompatibility; report location:
> {{/fields/4/type/values}}
> * {{fField}} field type incompatibility; report location: {{/fields/5/type}}
> * {{fField}} fixed name mismatch; report location: {{/fields/5/type/name}}
> * {{fField}} fixed size type incompatibility; report location:
> {{/fields/5/type/size}}
> * {{fField}} missing default value; report location: {{/fields/5}}
> h2. Notes
> * This ticket depends on AVRO-1933 and associated patches.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-2078) Avro does not enforce schema resolution rules for Decimal type
[
https://issues.apache.org/jira/browse/AVRO-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-2078:
Fix Version/s: (was: 1.8.2)
> Avro does not enforce schema resolution rules for Decimal type
> --
>
> Key: AVRO-2078
> URL: https://issues.apache.org/jira/browse/AVRO-2078
> Project: Avro
> Issue Type: Bug
>Affects Versions: 1.8.2
>Reporter: Anthony Hsu
>Assignee: Nandor Kollar
> Attachments: dec.avro
>
>
> According to http://avro.apache.org/docs/1.8.2/spec.html#Decimal
> bq. For the purposes of schema resolution, two schemas that are {{decimal}}
> logical types _match_ if their scales and precisions match.
> This is not enforced.
> I wrote a file with (precision 5, scale 2) and tried to read it with a reader
> schema with (precision 3, scale 1). I expected an AvroTypeException to be
> thrown, but none was thrown.
> Test data file attached. The code to read it is:
> {noformat:title=ReadDecimal.java}
> import java.io.File;
> import org.apache.avro.Schema;
> import org.apache.avro.file.DataFileReader;
> import org.apache.avro.generic.GenericDatumReader;
> import org.apache.avro.generic.GenericRecord;
> import org.apache.avro.io.DatumReader;
> public class ReadDecimal {
> public static void main(String[] args) throws Exception {
> Schema schema = new Schema.Parser().parse("{\n" + " \"type\" :
> \"record\",\n" + " \"name\" : \"some_schema\",\n"
> + " \"namespace\" : \"com.howdy\",\n" + " \"fields\" : [ {\n" + "
> \"name\" : \"name\",\n"
> + "\"type\" : \"string\"\n" + " }, {\n" + "\"name\" :
> \"value\",\n" + "\"type\" : {\n"
> + " \"type\" : \"bytes\",\n" + " \"logicalType\" :
> \"decimal\",\n" + " \"precision\" : 3,\n"
> + " \"scale\" : 1\n" + "}\n" + " } ]\n" + "}");
> DatumReader datumReader = new GenericDatumReader<>(schema);
> // dec.avro has precision 5, scale 2
> DataFileReader dataFileReader = new DataFileReader<>(
> new File("/tmp/dec.avro"), datumReader);
> GenericRecord foo = null;
> while (dataFileReader.hasNext()) {
> foo = dataFileReader.next(foo); // AvroTypeException expected due to
> change in scale/precision but none occurs
> }
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-2078) Avro does not enforce schema resolution rules for Decimal type
[
https://issues.apache.org/jira/browse/AVRO-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-2078:
Affects Version/s: 1.8.2
> Avro does not enforce schema resolution rules for Decimal type
> --
>
> Key: AVRO-2078
> URL: https://issues.apache.org/jira/browse/AVRO-2078
> Project: Avro
> Issue Type: Bug
>Affects Versions: 1.8.2
>Reporter: Anthony Hsu
>Assignee: Nandor Kollar
> Fix For: 1.8.2
>
> Attachments: dec.avro
>
>
> According to http://avro.apache.org/docs/1.8.2/spec.html#Decimal
> bq. For the purposes of schema resolution, two schemas that are {{decimal}}
> logical types _match_ if their scales and precisions match.
> This is not enforced.
> I wrote a file with (precision 5, scale 2) and tried to read it with a reader
> schema with (precision 3, scale 1). I expected an AvroTypeException to be
> thrown, but none was thrown.
> Test data file attached. The code to read it is:
> {noformat:title=ReadDecimal.java}
> import java.io.File;
> import org.apache.avro.Schema;
> import org.apache.avro.file.DataFileReader;
> import org.apache.avro.generic.GenericDatumReader;
> import org.apache.avro.generic.GenericRecord;
> import org.apache.avro.io.DatumReader;
> public class ReadDecimal {
> public static void main(String[] args) throws Exception {
> Schema schema = new Schema.Parser().parse("{\n" + " \"type\" :
> \"record\",\n" + " \"name\" : \"some_schema\",\n"
> + " \"namespace\" : \"com.howdy\",\n" + " \"fields\" : [ {\n" + "
> \"name\" : \"name\",\n"
> + "\"type\" : \"string\"\n" + " }, {\n" + "\"name\" :
> \"value\",\n" + "\"type\" : {\n"
> + " \"type\" : \"bytes\",\n" + " \"logicalType\" :
> \"decimal\",\n" + " \"precision\" : 3,\n"
> + " \"scale\" : 1\n" + "}\n" + " } ]\n" + "}");
> DatumReader datumReader = new GenericDatumReader<>(schema);
> // dec.avro has precision 5, scale 2
> DataFileReader dataFileReader = new DataFileReader<>(
> new File("/tmp/dec.avro"), datumReader);
> GenericRecord foo = null;
> while (dataFileReader.hasNext()) {
> foo = dataFileReader.next(foo); // AvroTypeException expected due to
> change in scale/precision but none occurs
> }
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (AVRO-2083) Add Yetus InterfaceAudience annotations to Avro
Nandor Kollar created AVRO-2083: --- Summary: Add Yetus InterfaceAudience annotations to Avro Key: AVRO-2083 URL: https://issues.apache.org/jira/browse/AVRO-2083 Project: Avro Issue Type: Improvement Components: java Affects Versions: 1.8.2 Reporter: Nandor Kollar Priority: Minor Avro should use Yetus [InterfaceAudience annotations|https://yetus.apache.org/documentation/in-progress/audience-annotations-apidocs/org/apache/yetus/audience/package-summary.html] to mark classes which are intended for public use (these should rarely change), and classes for private use. This could be beneficial for both clients and API compatibility checker tools. This package also has annotations for stability, we should also consider using those. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2077) Avro tools should have an option to check reader-writer compatibility
[
https://issues.apache.org/jira/browse/AVRO-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189672#comment-16189672
]
Nandor Kollar commented on AVRO-2077:
-
[~howellbridger] AVRO-1933 and AVRO-2003 provides details about
incompatibilities between reader and writer schemas. It seems that there's no
tool in avro-tools to display these for a given file and a (reader) schema,
with this Jira I wanted to address this gap: {{schemaCompatTool input_Avro_file
reader_schema_file}}. This can be made bi-directional, but I don't see any
point for checking if a given (writer) schema present in an existing Avro file
could be used to as reader schema.
For your case, I'd create an other tool with different parameters:
{{mutualSchemaCompatTool schema_1 schema_2}} (either URLs or files with Avro
JSON schema), and it would list the bi-directional compatibility (which could
be implemented as part of this Jira too). How about this approach?
> Avro tools should have an option to check reader-writer compatibility
> -
>
> Key: AVRO-2077
> URL: https://issues.apache.org/jira/browse/AVRO-2077
> Project: Avro
> Issue Type: New Feature
> Components: java, tools
>Reporter: Nandor Kollar
>Priority: Minor
> Fix For: 1.9.0
>
>
> It seems that avro-tools doesn't have any option to check for a given Avro
> file and a given reader schema (JSON file or URL) that the reader's schema is
> compatible with the writer's schema. This tool should report every
> compatibility problem (if there's any).
> According my knowledge, there's no such option for avro-tools as of now.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-2077) Avro tools should have an option to check reader-writer compatibility
[ https://issues.apache.org/jira/browse/AVRO-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189811#comment-16189811 ] Nandor Kollar commented on AVRO-2077: - [~howellbridger] I like the idea you proposed! Let me think about the details, but I think with these additional options would work nicely! > Avro tools should have an option to check reader-writer compatibility > - > > Key: AVRO-2077 > URL: https://issues.apache.org/jira/browse/AVRO-2077 > Project: Avro > Issue Type: New Feature > Components: java, tools >Reporter: Nandor Kollar >Priority: Minor > Fix For: 1.9.0 > > > It seems that avro-tools doesn't have any option to check for a given Avro > file and a given reader schema (JSON file or URL) that the reader's schema is > compatible with the writer's schema. This tool should report every > compatibility problem (if there's any). > According my knowledge, there's no such option for avro-tools as of now. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-1933) SchemaCompatibility class could be more user-friendly about incompatibilities
[ https://issues.apache.org/jira/browse/AVRO-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134891#comment-16134891 ] Nandor Kollar commented on AVRO-1933: - [~epkanol] since AVRO-1931 is committed to trunk now, it looks like your pull request has several merge conflicts. Could you please merge Avro trunk into your fork, and resolve the merge conflicts? > SchemaCompatibility class could be more user-friendly about incompatibilities > - > > Key: AVRO-1933 > URL: https://issues.apache.org/jira/browse/AVRO-1933 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.7, 1.8.1 > Environment: Any Java env >Reporter: Anders Sundelin >Assignee: Anders Sundelin >Priority: Minor > Fix For: 1.9.0 > > Attachments: AVRO-1933-compatible-with-AVRO-1931.patch, > AVRO-1933.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > Today, the class SchemaCompatibility reports incompatibilities with quite > little detail. The whole reader and the whole writer schema is listed, and no > particular detail about what was incompatible. > The attached patch fixes this, introducing a new enum > (SchemaIncompatibilityType), and more specific sub-schemas that were > incompatible. > The old, overall picture, is still there - the new compatibility state is > encapsulated in the SchemaCompatibilityDetails class. > Lots of test cases have been added, and there has been refactoring done in > the TestSchemaCompatibility and other test classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-2069) Use primitive fields in generated getters & setters in Java code
[ https://issues.apache.org/jira/browse/AVRO-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-2069: Status: Patch Available (was: Open) > Use primitive fields in generated getters & setters in Java code > > > Key: AVRO-2069 > URL: https://issues.apache.org/jira/browse/AVRO-2069 > Project: Avro > Issue Type: Improvement >Affects Versions: 1.8.2 >Reporter: Daniil Gitelson >Assignee: Daniil Gitelson > > Currently, for primitive types (such as int, long, etc) generated getters and > setters return and accept java.lang.* boxed (while fields actually holds > primitive values). This is inefeccient and produces code boilerplate. > Changed this behaviour in pull request: > https://github.com/apache/avro/pull/243 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2069) Use primitive fields in generated getters & setters in Java code
[ https://issues.apache.org/jira/browse/AVRO-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148697#comment-16148697 ] Nandor Kollar commented on AVRO-2069: - Thanks Daniil, now all tests passed. Looks good to me! > Use primitive fields in generated getters & setters in Java code > > > Key: AVRO-2069 > URL: https://issues.apache.org/jira/browse/AVRO-2069 > Project: Avro > Issue Type: Improvement >Affects Versions: 1.8.2 >Reporter: Daniil Gitelson >Assignee: Daniil Gitelson > > Currently, for primitive types (such as int, long, etc) generated getters and > setters return and accept java.lang.* boxed (while fields actually holds > primitive values). This is inefeccient and produces code boilerplate. > Changed this behaviour in pull request: > https://github.com/apache/avro/pull/243 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2069) Use primitive fields in generated getters & setters in Java code
[
https://issues.apache.org/jira/browse/AVRO-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150565#comment-16150565
]
Nandor Kollar commented on AVRO-2069:
-
[~dguitelson] looks like we still have test failures,
TestSpecificRecordBuilder#testInterop fails on my desktop with
{{java.lang.AssertionError: Use assertEquals(expected, actual, delta) to
compare floating-point numbers}}. There are two asserts for float and double in
this test, could you please replace them to one with an additional delta
parameter?
> Use primitive fields in generated getters & setters in Java code
>
>
> Key: AVRO-2069
> URL: https://issues.apache.org/jira/browse/AVRO-2069
> Project: Avro
> Issue Type: Improvement
>Affects Versions: 1.8.2
>Reporter: Daniil Gitelson
>Assignee: Daniil Gitelson
>
> Currently, for primitive types (such as int, long, etc) generated getters and
> setters return and accept java.lang.* boxed (while fields actually holds
> primitive values). This is inefeccient and produces code boilerplate.
> Changed this behaviour in pull request:
> https://github.com/apache/avro/pull/243
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-2069) Use primitive fields in generated getters & setters in Java code
[ https://issues.apache.org/jira/browse/AVRO-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150660#comment-16150660 ] Nandor Kollar commented on AVRO-2069: - Perfect, looks good, tests are green again, thanks! > Use primitive fields in generated getters & setters in Java code > > > Key: AVRO-2069 > URL: https://issues.apache.org/jira/browse/AVRO-2069 > Project: Avro > Issue Type: Improvement >Affects Versions: 1.8.2 >Reporter: Daniil Gitelson >Assignee: Daniil Gitelson > > Currently, for primitive types (such as int, long, etc) generated getters and > setters return and accept java.lang.* boxed (while fields actually holds > primitive values). This is inefeccient and produces code boilerplate. > Changed this behaviour in pull request: > https://github.com/apache/avro/pull/243 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2069) Use primitive fields in generated getters & setters in Java code
[ https://issues.apache.org/jira/browse/AVRO-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150636#comment-16150636 ] Nandor Kollar commented on AVRO-2069: - Ahhh, yes you're right, my bad, sorry for that! :) > Use primitive fields in generated getters & setters in Java code > > > Key: AVRO-2069 > URL: https://issues.apache.org/jira/browse/AVRO-2069 > Project: Avro > Issue Type: Improvement >Affects Versions: 1.8.2 >Reporter: Daniil Gitelson >Assignee: Daniil Gitelson > > Currently, for primitive types (such as int, long, etc) generated getters and > setters return and accept java.lang.* boxed (while fields actually holds > primitive values). This is inefeccient and produces code boilerplate. > Changed this behaviour in pull request: > https://github.com/apache/avro/pull/243 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-1933) SchemaCompatibility class could be more user-friendly about incompatibilities
[ https://issues.apache.org/jira/browse/AVRO-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152301#comment-16152301 ] Nandor Kollar commented on AVRO-1933: - Thanks [~epkanol], I'll help the committers with review, do my best to get this committed! > SchemaCompatibility class could be more user-friendly about incompatibilities > - > > Key: AVRO-1933 > URL: https://issues.apache.org/jira/browse/AVRO-1933 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.7, 1.8.1 > Environment: Any Java env >Reporter: Anders Sundelin >Assignee: Anders Sundelin >Priority: Minor > Fix For: 1.9.0 > > Attachments: AVRO-1933-compatible-with-AVRO-1931.patch, > AVRO-1933.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > Today, the class SchemaCompatibility reports incompatibilities with quite > little detail. The whole reader and the whole writer schema is listed, and no > particular detail about what was incompatible. > The attached patch fixes this, introducing a new enum > (SchemaIncompatibilityType), and more specific sub-schemas that were > incompatible. > The old, overall picture, is still there - the new compatibility state is > encapsulated in the SchemaCompatibilityDetails class. > Lots of test cases have been added, and there has been refactoring done in > the TestSchemaCompatibility and other test classes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AVRO-2074) SchemaValidator can't cope with recursive schemas?
Nandor Kollar created AVRO-2074:
---
Summary: SchemaValidator can't cope with recursive schemas?
Key: AVRO-2074
URL: https://issues.apache.org/jira/browse/AVRO-2074
Project: Avro
Issue Type: Bug
Components: java
Reporter: Nandor Kollar
When trying to validate a record which is recursively defined:
{code}
Schema record = Schema.createRecord("List", null, null, false);
record.setFields(list(new Schema.Field("head",
Schema.create(Schema.Type.INT), null, null),
new Schema.Field("tail", record, null, null)));
{code}
then schema validation
{code}
new SchemaValidatorBuilder().canReadStrategy().validateAll().validate(record,
asList(record));
{code}
fails with StackOverflowError:
{code}
java.lang.StackOverflowError
at
org.apache.avro.io.parsing.Symbol$Sequence.flattenedSize(Symbol.java:337)
at org.apache.avro.io.parsing.Symbol.flattenedSize(Symbol.java:230)
at
org.apache.avro.io.parsing.Symbol$Sequence.flattenedSize(Symbol.java:337)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-2072) ResolvingGrammarGenerator doesn't implement schema resolution correctly for unions
[ https://issues.apache.org/jira/browse/AVRO-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147116#comment-16147116 ] Nandor Kollar commented on AVRO-2072: - Thanks Zoltan, I'll have a look at the failing tests! > ResolvingGrammarGenerator doesn't implement schema resolution correctly for > unions > -- > > Key: AVRO-2072 > URL: https://issues.apache.org/jira/browse/AVRO-2072 > Project: Avro > Issue Type: Bug >Reporter: Nandor Kollar >Assignee: Nandor Kollar > Attachments: AVRO-2072.patch > > > According to > [specification|https://avro.apache.org/docs/current/spec.html#Schema+Resolution], > int and long is promotable to float, but when using SchemaValidator, a union > with a single int or long branch is not readable by an union with a float > branch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (AVRO-2069) Use primitive fields in generated getters & setters in Java code
[ https://issues.apache.org/jira/browse/AVRO-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147129#comment-16147129 ] Nandor Kollar edited comment on AVRO-2069 at 8/30/17 12:09 PM: --- [~dguitelson] looks like three tests fail in TestSpecificCompilerTool: testCompileSchemaTwoFiles, testCompileSchemasUsingString and testCompileSchemaFileAndDirectory. You should change the expected files (lang/java/tools/src/test/compiler/output/Player.java and lang/java/tools/src/test/compiler/output/Position.java), the methods should return primitive types there according to your modification. was (Author: nkollar): [~dguitelson] looks like three test fails in TestSpecificCompilerTool: testCompileSchemaTwoFiles, testCompileSchemasUsingString and testCompileSchemaFileAndDirectory. You should change the expected files (lang/java/tools/src/test/compiler/output/Player.java and lang/java/tools/src/test/compiler/output/Position.java), the methods should return primitive types there according to your modification. > Use primitive fields in generated getters & setters in Java code > > > Key: AVRO-2069 > URL: https://issues.apache.org/jira/browse/AVRO-2069 > Project: Avro > Issue Type: Improvement >Affects Versions: 1.8.2 >Reporter: Daniil Gitelson > > Currently, for primitive types (such as int, long, etc) generated getters and > setters return and accept java.lang.* boxed (while fields actually holds > primitive values). This is inefeccient and produces code boilerplate. > Changed this behaviour in pull request: > https://github.com/apache/avro/pull/243 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2069) Use primitive fields in generated getters & setters in Java code
[ https://issues.apache.org/jira/browse/AVRO-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147129#comment-16147129 ] Nandor Kollar commented on AVRO-2069: - [~dguitelson] looks like three test fails in TestSpecificCompilerTool: testCompileSchemaTwoFiles, testCompileSchemasUsingString and testCompileSchemaFileAndDirectory. You should change the expected files (lang/java/tools/src/test/compiler/output/Player.java and lang/java/tools/src/test/compiler/output/Position.java), the methods should return primitive types there according to your modification. > Use primitive fields in generated getters & setters in Java code > > > Key: AVRO-2069 > URL: https://issues.apache.org/jira/browse/AVRO-2069 > Project: Avro > Issue Type: Improvement >Affects Versions: 1.8.2 >Reporter: Daniil Gitelson > > Currently, for primitive types (such as int, long, etc) generated getters and > setters return and accept java.lang.* boxed (while fields actually holds > primitive values). This is inefeccient and produces code boilerplate. > Changed this behaviour in pull request: > https://github.com/apache/avro/pull/243 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (AVRO-2069) Use primitive fields in generated getters & setters in Java code
[ https://issues.apache.org/jira/browse/AVRO-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar reassigned AVRO-2069: --- Assignee: Daniil Gitelson > Use primitive fields in generated getters & setters in Java code > > > Key: AVRO-2069 > URL: https://issues.apache.org/jira/browse/AVRO-2069 > Project: Avro > Issue Type: Improvement >Affects Versions: 1.8.2 >Reporter: Daniil Gitelson >Assignee: Daniil Gitelson > > Currently, for primitive types (such as int, long, etc) generated getters and > setters return and accept java.lang.* boxed (while fields actually holds > primitive values). This is inefeccient and produces code boilerplate. > Changed this behaviour in pull request: > https://github.com/apache/avro/pull/243 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2072) ResolvingGrammarGenerator doesn't implement schema resolution correctly for unions
[ https://issues.apache.org/jira/browse/AVRO-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147147#comment-16147147 ] Nandor Kollar commented on AVRO-2072: - [~zolyfarkas] I don't see any test failure, guess you executed tests on your forked version instead of Apache master. Looks like [intUnionIsNotConvertedToFloatUnion |https://github.com/zolyfarkas/avro/blob/trunk/lang/java/avro/src/test/java/org/apache/avro/TestReadingWritingDataInEvolvedSchemas.java#L158] and [longUnionIsNotConvertedToFloatUnion|https://github.com/zolyfarkas/avro/blob/trunk/lang/java/avro/src/test/java/org/apache/avro/TestReadingWritingDataInEvolvedSchemas.java#L181] are negative test: testing exactly what this Jira intends to fix right? Are these tests your own test cases, or this was merged with a patch available Jira, which is not yet merged to upstream trunk? As of the last test case, I'm not sure what is the problem. > ResolvingGrammarGenerator doesn't implement schema resolution correctly for > unions > -- > > Key: AVRO-2072 > URL: https://issues.apache.org/jira/browse/AVRO-2072 > Project: Avro > Issue Type: Bug >Reporter: Nandor Kollar >Assignee: Nandor Kollar > Attachments: AVRO-2072.patch > > > According to > [specification|https://avro.apache.org/docs/current/spec.html#Schema+Resolution], > int and long is promotable to float, but when using SchemaValidator, a union > with a single int or long branch is not readable by an union with a float > branch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2072) ResolvingGrammarGenerator doesn't implement schema resolution correctly for unions
[ https://issues.apache.org/jira/browse/AVRO-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147294#comment-16147294 ] Nandor Kollar commented on AVRO-2072: - [~zolyfarkas] I think this is fine, the test should expect float instead of Double in this case, since the spec says: *if reader's is a union, but writer's is not* The first schema in the reader's union that matches the writer's schema is recursively resolved against it. If none match, an error is signalled. and int field will (with the attach patch) match with float branch now, and not with double. I think this is exactly what the spec says, no? > ResolvingGrammarGenerator doesn't implement schema resolution correctly for > unions > -- > > Key: AVRO-2072 > URL: https://issues.apache.org/jira/browse/AVRO-2072 > Project: Avro > Issue Type: Bug >Reporter: Nandor Kollar >Assignee: Nandor Kollar > Attachments: AVRO-2072.patch > > > According to > [specification|https://avro.apache.org/docs/current/spec.html#Schema+Resolution], > int and long is promotable to float, but when using SchemaValidator, a union > with a single int or long branch is not readable by an union with a float > branch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-2072) ResolvingGrammarGenerator doesn't implement schema resolution correctly for unions
[ https://issues.apache.org/jira/browse/AVRO-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147348#comment-16147348 ] Nandor Kollar commented on AVRO-2072: - Ah okay, I think the pull request for AVRO-1931 didn't include these tests, just the attached patch, that's why it was closed but the tests are not there. [~epkanol] could you please help identifying the difference between your attached patch and your PR? Looks like some test cases are missing. > ResolvingGrammarGenerator doesn't implement schema resolution correctly for > unions > -- > > Key: AVRO-2072 > URL: https://issues.apache.org/jira/browse/AVRO-2072 > Project: Avro > Issue Type: Bug >Reporter: Nandor Kollar >Assignee: Nandor Kollar > Attachments: AVRO-2072.patch > > > According to > [specification|https://avro.apache.org/docs/current/spec.html#Schema+Resolution], > int and long is promotable to float, but when using SchemaValidator, a union > with a single int or long branch is not readable by an union with a float > branch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-1827) Handling correctly optional fields when converting Protobuf to Avro
[
https://issues.apache.org/jira/browse/AVRO-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212536#comment-16212536
]
Nandor Kollar commented on AVRO-1827:
-
Took a look at the latest patch, looks good to me. Though I'm not sure this can
go in 1.8.x or 1.7.x release, since this is not a backward compatible
modification. [~busbey] what do you think?
> Handling correctly optional fields when converting Protobuf to Avro
> ---
>
> Key: AVRO-1827
> URL: https://issues.apache.org/jira/browse/AVRO-1827
> Project: Avro
> Issue Type: Improvement
>Affects Versions: 1.7.7, 1.8.0
>Reporter: Jakub Kahovec
>Assignee: Karel Fuka
> Attachments: AVRO-1827.patch, AVRO-1827.patch, AVRO-1827.patch,
> AVRO-1827.patch
>
>
> Hello,
> as of the current implementation of converting protobuf files into avro
> format, protobuf optional fields are being given default values in the avro
> schema if not specified explicitly.
> So for instance when the protobuf field is defined as
> {quote}
> optional int64 fieldInt64 = 1;
> {quote}
> in the avro schema it appears as
> {quote}
> "name" : "fieldInt64",
> "type" : "long",
> "default" : 0
> {quote}
> The problem with this implementation is that we are losing information about
> whether the field was present or not in the original protobuf, as when we ask
> for this field's value in avro we will be given the default value.
> What I'm proposing instead is that if the field in the protobuf is defined as
> optional and has no default value then the generated avro schema type will us
> a union comprising the matching type and null type with default value null.
> It is going to look like this:
> {quote}
> "name" : "fieldIn64",
> "type" : [ "null", "long" ],
> "default" : null
> {quote}
> I'm aware that is a breaking change but I think that is the proper way how to
> handle optional fields.
> I've also created a patch which fixes the conversion
> Jakub
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-2003) Report specific location of schema incompatibilities
[
https://issues.apache.org/jira/browse/AVRO-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar updated AVRO-2003:
Resolution: Fixed
Hadoop Flags: Incompatible change
Release Note: SchemaCompatibility reports location of incompatibility as a
JSON Pointer. This is a backward incompatible change, because the public
interface of SchemaCompatibility class changed. (was: SchemaCompatibility
reports location of incompatibility as a JSON Pointer.)
Status: Resolved (was: Patch Available)
> Report specific location of schema incompatibilities
>
>
> Key: AVRO-2003
> URL: https://issues.apache.org/jira/browse/AVRO-2003
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.8.1
> Environment: Any java env
>Reporter: Elliot West
>Assignee: Elliot West
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: AVRO-2003.patch
>
>
> h2. Overview
> Building on the work to improve schema incompatibility reporting in
> AVRO-1933, it would be useful if the {{SchemaCompatibility}} classes could
> also report the location in the schema where any incompatibility was
> encountered.
> It is recommended that the location reported is both easily readable by
> humans and machines. In the first case this would assist schema developers to
> pin-point issues in there schema documents, and in the latter case it
> provides a useful mechanism to schema tooling, such as IDEs and editors, to
> easily select the pertinent nodes in the Schema document tree.
> h2. Implementation specifics
> To meet this requirements it is suggested that the location is encoded using
> the [JSON Pointer specification|https://tools.ietf.org/html/rfc6901]. This is
> both easily parsed by users, but is also supported by a number of libraries
> for a range of common programming languages and platforms.
> h2. Examples
> Given the following example schema, consider some incompatibility scenarios.
> For each case an expected JSON Pointer description of the incompatibility
> location is described:
> {code}
> {
> "type": "record",
> "name": "myRecord",
> "fields" : [
> {"name": "pField", "type": "long"},
> {"name": "uField", "type":
> ["null", "int", "string"]
> },
> {"name": "eField", "type":
> { "type": "enum", "name": "Suit", "symbols" : ["SPADES", "HEARTS",
> "DIAMONDS", "CLUBS"] }
> },
> {"name": "aField", "type":
> {"type": "array", "items": "string"}
> },
> {"name": "mField", "type":
> {"type": "map", "values": "long"}
> },
> {"name": "fField", "type":
> {"type": "fixed", "size": 16, "name": "md5"}
> }
> ]
> }
> {code}
> Possible incompatibility scenarions and the location that would be reported
> back to the user/tool:
> * Root type incompatibility; report location: {{/}}
> * Record name mismatch; report location: {{/name}}
> * {{pField}} type incompatibility; report location: {{/fields/0/type}}
> * {{uField}} field type incompatibility; report location: {{/fields/1/type}}
> * {{uField}} missing union branch {{string}}; report location:
> {{/fields/1/type/2}}
> * {{eField}} field type incompatibility; report location: {{/fields/2/type}}
> * {{eField}} missing enum symbol; report location: {{/fields/2/type/symbols}}
> * {{eField}} enum name mismatch; report location: {{/fields/2/type/name}}
> * {{aField}} field type incompatibility; report location: {{/fields/3/type}}
> * {{aField}} array element type incompatibility; report location:
> {{/fields/3/type/items}}
> * {{mField}} field type incompatibility; report location: {{/fields/4/type}}
> * {{mField}} map value type incompatibility; report location:
> {{/fields/4/type/values}}
> * {{fField}} field type incompatibility; report location: {{/fields/5/type}}
> * {{fField}} fixed name mismatch; report location: {{/fields/5/type/name}}
> * {{fField}} fixed size type incompatibility; report location:
> {{/fields/5/type/size}}
> * {{fField}} missing default value; report location: {{/fields/5}}
> h2. Notes
> * This ticket depends on AVRO-1933 and associated patches.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AVRO-2003) Report specific location of schema incompatibilities
[
https://issues.apache.org/jira/browse/AVRO-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278235#comment-16278235
]
Nandor Kollar commented on AVRO-2003:
-
Committed to master, thanks Elliot for working on this!
> Report specific location of schema incompatibilities
>
>
> Key: AVRO-2003
> URL: https://issues.apache.org/jira/browse/AVRO-2003
> Project: Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.8.1
> Environment: Any java env
>Reporter: Elliot West
>Assignee: Elliot West
>Priority: Minor
> Fix For: 1.9.0
>
> Attachments: AVRO-2003.patch
>
>
> h2. Overview
> Building on the work to improve schema incompatibility reporting in
> AVRO-1933, it would be useful if the {{SchemaCompatibility}} classes could
> also report the location in the schema where any incompatibility was
> encountered.
> It is recommended that the location reported is both easily readable by
> humans and machines. In the first case this would assist schema developers to
> pin-point issues in there schema documents, and in the latter case it
> provides a useful mechanism to schema tooling, such as IDEs and editors, to
> easily select the pertinent nodes in the Schema document tree.
> h2. Implementation specifics
> To meet this requirements it is suggested that the location is encoded using
> the [JSON Pointer specification|https://tools.ietf.org/html/rfc6901]. This is
> both easily parsed by users, but is also supported by a number of libraries
> for a range of common programming languages and platforms.
> h2. Examples
> Given the following example schema, consider some incompatibility scenarios.
> For each case an expected JSON Pointer description of the incompatibility
> location is described:
> {code}
> {
> "type": "record",
> "name": "myRecord",
> "fields" : [
> {"name": "pField", "type": "long"},
> {"name": "uField", "type":
> ["null", "int", "string"]
> },
> {"name": "eField", "type":
> { "type": "enum", "name": "Suit", "symbols" : ["SPADES", "HEARTS",
> "DIAMONDS", "CLUBS"] }
> },
> {"name": "aField", "type":
> {"type": "array", "items": "string"}
> },
> {"name": "mField", "type":
> {"type": "map", "values": "long"}
> },
> {"name": "fField", "type":
> {"type": "fixed", "size": 16, "name": "md5"}
> }
> ]
> }
> {code}
> Possible incompatibility scenarions and the location that would be reported
> back to the user/tool:
> * Root type incompatibility; report location: {{/}}
> * Record name mismatch; report location: {{/name}}
> * {{pField}} type incompatibility; report location: {{/fields/0/type}}
> * {{uField}} field type incompatibility; report location: {{/fields/1/type}}
> * {{uField}} missing union branch {{string}}; report location:
> {{/fields/1/type/2}}
> * {{eField}} field type incompatibility; report location: {{/fields/2/type}}
> * {{eField}} missing enum symbol; report location: {{/fields/2/type/symbols}}
> * {{eField}} enum name mismatch; report location: {{/fields/2/type/name}}
> * {{aField}} field type incompatibility; report location: {{/fields/3/type}}
> * {{aField}} array element type incompatibility; report location:
> {{/fields/3/type/items}}
> * {{mField}} field type incompatibility; report location: {{/fields/4/type}}
> * {{mField}} map value type incompatibility; report location:
> {{/fields/4/type/values}}
> * {{fField}} field type incompatibility; report location: {{/fields/5/type}}
> * {{fField}} fixed name mismatch; report location: {{/fields/5/type/name}}
> * {{fField}} fixed size type incompatibility; report location:
> {{/fields/5/type/size}}
> * {{fField}} missing default value; report location: {{/fields/5}}
> h2. Notes
> * This ticket depends on AVRO-1933 and associated patches.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (AVRO-1213) Update to latest release of Jetty
[ https://issues.apache.org/jira/browse/AVRO-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-1213: Resolution: Fixed Status: Resolved (was: Patch Available) > Update to latest release of Jetty > - > > Key: AVRO-1213 > URL: https://issues.apache.org/jira/browse/AVRO-1213 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.2 >Reporter: Sharmarke Aden >Assignee: Daniel Kulp >Priority: Blocker > Fix For: 1.9.0 > > > The compile scoped dependency on jetty servlet-api in the IPC pom file can be > problematic if using Avro in a webapp environment. Would it be possible to > make this dependency either optional or provided? Or maybe Avro modularize > into sub-modules in such a way that desired features can be assembled > piecemeal? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AVRO-2106) Revisit dependencies on Jetty, servlet-api, and Netty
Nandor Kollar created AVRO-2106: --- Summary: Revisit dependencies on Jetty, servlet-api, and Netty Key: AVRO-2106 URL: https://issues.apache.org/jira/browse/AVRO-2106 Project: Avro Issue Type: Improvement Components: java Affects Versions: 1.8.0, 1.7.2 Reporter: Nandor Kollar The compile scoped dependency on jetty servlet-api in the IPC pom file can be problematic if using Avro in a webapp environment. Would it be possible to make this dependency either optional or provided? Or maybe Avro modularize into sub-modules in such a way that desired features can be assembled piecemeal? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-1213) Update to latest release of Jetty
[ https://issues.apache.org/jira/browse/AVRO-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16256740#comment-16256740 ] Nandor Kollar commented on AVRO-1213: - Actually the PR just updates the Jetty version, doesn't address the other (breaking up IPC module into Jetty and Netty sub-modules). Created AVRO-2106 for this. > Update to latest release of Jetty > - > > Key: AVRO-1213 > URL: https://issues.apache.org/jira/browse/AVRO-1213 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.2 >Reporter: Sharmarke Aden >Assignee: Daniel Kulp >Priority: Blocker > Fix For: 1.9.0 > > > The compile scoped dependency on jetty servlet-api in the IPC pom file can be > problematic if using Avro in a webapp environment. Would it be possible to > make this dependency either optional or provided? Or maybe Avro modularize > into sub-modules in such a way that desired features can be assembled > piecemeal? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-1213) Update to latest release of Jetty
[ https://issues.apache.org/jira/browse/AVRO-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16256724#comment-16256724 ] Nandor Kollar commented on AVRO-1213: - Committed to trunk, thanks Daniel for fixing this! > Update to latest release of Jetty > - > > Key: AVRO-1213 > URL: https://issues.apache.org/jira/browse/AVRO-1213 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.2 >Reporter: Sharmarke Aden >Assignee: Daniel Kulp >Priority: Blocker > Fix For: 1.9.0 > > > The compile scoped dependency on jetty servlet-api in the IPC pom file can be > problematic if using Avro in a webapp environment. Would it be possible to > make this dependency either optional or provided? Or maybe Avro modularize > into sub-modules in such a way that desired features can be assembled > piecemeal? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AVRO-1213) Update to latest release of Jetty
[ https://issues.apache.org/jira/browse/AVRO-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nandor Kollar updated AVRO-1213: Hadoop Flags: Incompatible change Release Note: Jetty version was updated from 2.5-20081211 to 9.4.6.v20170531. This is a breaking change due to package name change in Jetty, which leads to interface change in Avro. The type of constructor parameters in HttpServer.java changed. > Update to latest release of Jetty > - > > Key: AVRO-1213 > URL: https://issues.apache.org/jira/browse/AVRO-1213 > Project: Avro > Issue Type: Improvement > Components: java >Affects Versions: 1.7.2 >Reporter: Sharmarke Aden >Assignee: Daniel Kulp >Priority: Blocker > Fix For: 1.9.0 > > > The compile scoped dependency on jetty servlet-api in the IPC pom file can be > problematic if using Avro in a webapp environment. Would it be possible to > make this dependency either optional or provided? Or maybe Avro modularize > into sub-modules in such a way that desired features can be assembled > piecemeal? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
