[jira] [Commented] (AVRO-2780) ProtobufData and ThriftData Can Get Into Endless Loop
[
https://issues.apache.org/jira/browse/AVRO-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128663#comment-17128663
]
Allen Wang commented on AVRO-2780:
--
How can I try it out? Is there a snapshot release in maven?
> ProtobufData and ThriftData Can Get Into Endless Loop
> -
>
> Key: AVRO-2780
> URL: https://issues.apache.org/jira/browse/AVRO-2780
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.2
>Reporter: Tianyu Lang
>Assignee: David Mollitor
>Priority: Critical
> Fix For: 1.10.0
>
>
> We have found a bug related, but probably not limited to ProtobufDatumReader.
> In Protobuf, if the ".proto" file has the same name as the message, in the
> generated ".java" file, "OuterClass" will be appended to the outer class's
> name. This is documented here:
> [https://developers.google.com/protocol-buffers/docs/reference/java-generated#invocation]
> Now we have a test protobuf class defined like so:
>
> {code:java}
> syntax = "proto2";
> package test.avrobug;
> option java_package = "com.test.protos.avrobug";
> option java_generic_services = true;
> enum State {
> BAD = 0;
> VERY_BAD = 1;
> SUPER_BAD = 2;
> }
> message Dummy {
> optional string token = 1;
> optional string description = 2;
> optional int64 count = 3;
> optional State state = 4;
> }
> {code}
>
>
> If we first create a Protobuf object, write it to a file as Avro through
> ProtobufDatumReader, then read it into a Protobuf with ProtobufDatumReader, a
> stack overflow exception will happen. Code to reproduce is as follows:
>
> {code:java}
> @Test
> public void ProtoToAvroOuterClassBug() throws Exception {
> DummyOuterClass.Dummy dummy = DummyOuterClass.Dummy.newBuilder()
> .setCount(50)
> .setDescription("hehe")
> .setToken("abc123")
> .setState(DummyOuterClass.State.BAD)
> .build();
>
> ProtobufDatumWriter pbWriter =
> new ProtobufDatumWriter<>(DummyOuterClass.Dummy.class);
> DataFileWriter dataFileWriter = new
> DataFileWriter<>(pbWriter);
> Schema schema = ProtobufData.get().getSchema(DummyOuterClass.Dummy.class);
> dataFileWriter.create(schema,
> new File("/Users/me/Documents/DummyAvroNoDefaultValues.avro"));
> dataFileWriter.append(dummy);
> dataFileWriter.close();
> ProtobufDatumReader pbReader =
> new ProtobufDatumReader<>(DummyOuterClass.Dummy.class);
> DataFileReader dataFileReader =
> new DataFileReader<>(new
> File("/Users/me/Documents/DummyAvroNoDefaultValues.avro"),
> pbReader);
> while(dataFileReader.hasNext()) {
> DummyOuterClass.Dummy record = dataFileReader.next();
> String recordStr = record.toString();
> String originStr = dummy.toString();
> System.out.println(recordStr);
> System.out.println(originStr);
> }
> }
> {code}
>
>
> When this is run, a stack overflow exception will happen with the following
> stack trace:
>
> {code:java}
> java.lang.StackOverflowErrorjava.lang.StackOverflowError
> at
> java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
> at
> org.apache.avro.specific.SpecificData.getClass(SpecificData.java:250)
> at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:141)
> at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:143)
> at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:143)
> at
> ..{code}
> newRecord() is recursed infinitely until stack explodes
> I did a bit code tracing and I found out when Avro tries to load the
> corresponding Java class from the schema, it tries to load the class
> "com.test.protos.avrobug.Dummy.Dummy". I suspect the double "Dummy" at the
> end is related to the stack overflow, but I don't have enough knowledge on
> Proto or Avro to know the exact reason.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AVRO-2862) C# Primitive Schema losing metadata
[
https://issues.apache.org/jira/browse/AVRO-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cody updated AVRO-2862:
---
Description:
Using this ticket as a base reference:
https://issues.apache.org/jira/browse/AVRO-1545
The same problem is happening in the C# version as well.
Given:
{code:java}
{
"type" : "record",
"name" : "BrokenRecord",
"namespace" : "whatever",
"fields" : [
{
"name" : "a_string",
"type" : {
"type" : "string",
"foobar" : "some example here"
}
}
]
}
{code}
The property "foobar" will be lost when parsing the schema.
The fix that my team has implemented is modifying this line (file used from
release 1.9.2)
[https://github.com/apache/avro/blob/bf20128ca6138a830b2ea13e0490f3df6b035639/lang/csharp/src/apache/main/Schema/PrimitiveSchema.cs#L85]
And replacing it with the following:
{code:java}
if(this.Props != null && this.Props.Count > 0)
{
w.WriteStartObject();
w.WritePropertyName("type");
w.WriteValue(Name);
foreach(var prop in Props)
{
w.WritePropertyName(prop.Key);
w.WriteRawValue(prop.Value);
}
w.WriteEndObject();
}
else w.WriteValue(Name);{code}
Another somewhat related ticket:
https://issues.apache.org/jira/browse/AVRO-1346
was:
Using this ticket as a base reference:
https://issues.apache.org/jira/browse/AVRO-1545
The same problem is happening in the C# version as well.
Given:
{code:java}
{
"type" : "record",
"name" : "BrokenRecord",
"namespace" : "whatever",
"fields" : [
{
"name" : "a_string",
"type" : {
"type" : "string",
"foobar" : "some example here"
}
}
]
}
{code}
The property "foobar" will be lost when parsing the schema.
The fix that my team has implemented is modifying this line (file used from
release 1.9.2)
[https://github.com/apache/avro/blob/bf20128ca6138a830b2ea13e0490f3df6b035639/lang/csharp/src/apache/main/Schema/PrimitiveSchema.cs#L85]
And replacing it with the following:
{code:java}
if(this.Props != null && this.Props.Count > 0)
{
w.WriteStartObject();
w.WritePropertyName("type");
w.WriteValue(Name);
foreach(var prop in Props)
{
w.WritePropertyName(prop.Key);
w.WriteRawValue(prop.Value);
}
w.WriteEndObject();
}
else w.WriteValue(Name);{code}
> C# Primitive Schema losing metadata
> ---
>
> Key: AVRO-2862
> URL: https://issues.apache.org/jira/browse/AVRO-2862
> Project: Apache Avro
> Issue Type: Bug
> Components: csharp
>Affects Versions: 1.9.2
>Reporter: Cody
>Priority: Major
>
> Using this ticket as a base reference:
> https://issues.apache.org/jira/browse/AVRO-1545
> The same problem is happening in the C# version as well.
> Given:
> {code:java}
> {
> "type" : "record",
> "name" : "BrokenRecord",
> "namespace" : "whatever",
> "fields" : [
> {
> "name" : "a_string",
> "type" : {
> "type" : "string",
> "foobar" : "some example here"
> }
> }
> ]
> }
> {code}
> The property "foobar" will be lost when parsing the schema.
> The fix that my team has implemented is modifying this line (file used from
> release 1.9.2)
>
> [https://github.com/apache/avro/blob/bf20128ca6138a830b2ea13e0490f3df6b035639/lang/csharp/src/apache/main/Schema/PrimitiveSchema.cs#L85]
> And replacing it with the following:
> {code:java}
> if(this.Props != null && this.Props.Count > 0)
> {
> w.WriteStartObject();
> w.WritePropertyName("type");
> w.WriteValue(Name);
> foreach(var prop in Props)
> {
> w.WritePropertyName(prop.Key);
> w.WriteRawValue(prop.Value);
> }
> w.WriteEndObject();
> }
> else w.WriteValue(Name);{code}
> Another somewhat related ticket:
> https://issues.apache.org/jira/browse/AVRO-1346
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AVRO-2862) C# Primitive Schema losing metadata
[
https://issues.apache.org/jira/browse/AVRO-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cody updated AVRO-2862:
---
Description:
Using this ticket as a base reference:
https://issues.apache.org/jira/browse/AVRO-1545
The same problem is happening in the C# version as well.
Given:
{code:java}
{
"type" : "record",
"name" : "BrokenRecord",
"namespace" : "whatever",
"fields" : [
{
"name" : "a_string",
"type" : {
"type" : "string",
"foobar" : "some example here"
}
}
]
}
{code}
The property "foobar" will be lost when parsing the schema.
The fix that my team has implemented is modifying this line (file used from
release 1.9.2)
[https://github.com/apache/avro/blob/bf20128ca6138a830b2ea13e0490f3df6b035639/lang/csharp/src/apache/main/Schema/PrimitiveSchema.cs#L85]
And replacing it with the following:
{code:java}
if(this.Props != null && this.Props.Count > 0)
{
w.WriteStartObject();
w.WritePropertyName("type");
w.WriteValue(Name);
foreach(var prop in Props)
{
w.WritePropertyName(prop.Key);
w.WriteRawValue(prop.Value);
}
w.WriteEndObject();
}
else w.WriteValue(Name);{code}
was:
Using this ticket as a base reference:
https://issues.apache.org/jira/browse/AVRO-1545
The same problem is happening in the C# version as well.
Given:
{code:java}
{
"type" : "record",
"name" : "BrokenRecord",
"namespace" : "whatever",
"fields" : [
{
"name" : "a_string",
"type" : {
"type" : "string",
"foobar" : "some example here"
}
}
]
}
{code}
The property "foobar" will be lost when parsing the schema.
The fix that my team has implemented is modifying this line (file used from
release 1.9.2)
[https://github.com/apache/avro/blob/bf20128ca6138a830b2ea13e0490f3df6b035639/lang/csharp/src/apache/main/Schema/PrimitiveSchema.cs#L85]
And replacing it with the following:
{code:java}
if(this.Props != null && this.Props.Count > 0)
{
w.WriteStartObject();
w.WritePropertyName("type");
w.WriteValue(Name);
foreach(var prop in Props)
{
w.WritePropertyName(prop.Key);
w.WriteRawValue(prop.Value);
}
w.WriteEndObject();
}
else w.WriteValue(Name);{code}
> C# Primitive Schema losing metadata
> ---
>
> Key: AVRO-2862
> URL: https://issues.apache.org/jira/browse/AVRO-2862
> Project: Apache Avro
> Issue Type: Bug
> Components: csharp
>Affects Versions: 1.9.2
>Reporter: Cody
>Priority: Major
>
> Using this ticket as a base reference:
> https://issues.apache.org/jira/browse/AVRO-1545
> The same problem is happening in the C# version as well.
> Given:
> {code:java}
> {
> "type" : "record",
> "name" : "BrokenRecord",
> "namespace" : "whatever",
> "fields" : [
> {
> "name" : "a_string",
> "type" : {
> "type" : "string",
> "foobar" : "some example here"
> }
> }
> ]
> }
> {code}
> The property "foobar" will be lost when parsing the schema.
> The fix that my team has implemented is modifying this line (file used from
> release 1.9.2)
>
> [https://github.com/apache/avro/blob/bf20128ca6138a830b2ea13e0490f3df6b035639/lang/csharp/src/apache/main/Schema/PrimitiveSchema.cs#L85]
> And replacing it with the following:
> {code:java}
> if(this.Props != null && this.Props.Count > 0)
> {
> w.WriteStartObject();
> w.WritePropertyName("type");
> w.WriteValue(Name);
> foreach(var prop in Props)
> {
> w.WritePropertyName(prop.Key);
> w.WriteRawValue(prop.Value);
> }
> w.WriteEndObject();
> }
> else w.WriteValue(Name);{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (AVRO-2862) C# Primitive Schema losing metadata
Cody created AVRO-2862:
--
Summary: C# Primitive Schema losing metadata
Key: AVRO-2862
URL: https://issues.apache.org/jira/browse/AVRO-2862
Project: Apache Avro
Issue Type: Bug
Components: csharp
Affects Versions: 1.9.2
Reporter: Cody
Using this ticket as a base reference:
https://issues.apache.org/jira/browse/AVRO-1545
The same problem is happening in the C# version as well.
Given:
{code:java}
{
"type" : "record",
"name" : "BrokenRecord",
"namespace" : "whatever",
"fields" : [
{
"name" : "a_string",
"type" : {
"type" : "string",
"foobar" : "some example here"
}
}
]
}
{code}
The property "foobar" will be lost when parsing the schema.
The fix that my team has implemented is modifying this line (file used from
release 1.9.2)
[https://github.com/apache/avro/blob/bf20128ca6138a830b2ea13e0490f3df6b035639/lang/csharp/src/apache/main/Schema/PrimitiveSchema.cs#L85]
And replacing it with the following:
{code:java}
if(this.Props != null && this.Props.Count > 0)
{
w.WriteStartObject();
w.WritePropertyName("type");
w.WriteValue(Name);
foreach(var prop in Props)
{
w.WritePropertyName(prop.Key);
w.WriteRawValue(prop.Value);
}
w.WriteEndObject();
}
else w.WriteValue(Name);{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AVRO-2862) C# Primitive Schema losing metadata
[
https://issues.apache.org/jira/browse/AVRO-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cody updated AVRO-2862:
---
Description:
Using this ticket as a base reference:
https://issues.apache.org/jira/browse/AVRO-1545
The same problem is happening in the C# version as well.
Given:
{code:java}
{
"type" : "record",
"name" : "BrokenRecord",
"namespace" : "whatever",
"fields" : [
{
"name" : "a_string",
"type" : {
"type" : "string",
"foobar" : "some example here"
}
}
]
}
{code}
The property "foobar" will be lost when parsing the schema.
The fix that my team has implemented is modifying this line (file used from
release 1.9.2)
[https://github.com/apache/avro/blob/bf20128ca6138a830b2ea13e0490f3df6b035639/lang/csharp/src/apache/main/Schema/PrimitiveSchema.cs#L85]
And replacing it with the following:
{code:java}
if(this.Props != null && this.Props.Count > 0)
{
w.WriteStartObject();
w.WritePropertyName("type");
w.WriteValue(Name);
foreach(var prop in Props)
{
w.WritePropertyName(prop.Key);
w.WriteRawValue(prop.Value);
}
w.WriteEndObject();
}
else w.WriteValue(Name);{code}
was:
Using this ticket as a base reference:
https://issues.apache.org/jira/browse/AVRO-1545
The same problem is happening in the C# version as well.
Given:
{code:java}
{
"type" : "record",
"name" : "BrokenRecord",
"namespace" : "whatever",
"fields" : [
{
"name" : "a_string",
"type" : {
"type" : "string",
"foobar" : "some example here"
}
}
]
}
{code}
The property "foobar" will be lost when parsing the schema.
The fix that my team has implemented is modifying this line (file used from
release 1.9.2)
[https://github.com/apache/avro/blob/bf20128ca6138a830b2ea13e0490f3df6b035639/lang/csharp/src/apache/main/Schema/PrimitiveSchema.cs#L85]
And replacing it with the following:
{code:java}
if(this.Props != null && this.Props.Count > 0)
{
w.WriteStartObject();
w.WritePropertyName("type");
w.WriteValue(Name);
foreach(var prop in Props)
{
w.WritePropertyName(prop.Key);
w.WriteRawValue(prop.Value);
}
w.WriteEndObject();
}
else w.WriteValue(Name);{code}
> C# Primitive Schema losing metadata
> ---
>
> Key: AVRO-2862
> URL: https://issues.apache.org/jira/browse/AVRO-2862
> Project: Apache Avro
> Issue Type: Bug
> Components: csharp
>Affects Versions: 1.9.2
>Reporter: Cody
>Priority: Major
>
> Using this ticket as a base reference:
> https://issues.apache.org/jira/browse/AVRO-1545
> The same problem is happening in the C# version as well.
> Given:
> {code:java}
> {
> "type" : "record",
> "name" : "BrokenRecord",
> "namespace" : "whatever",
> "fields" : [
> {
> "name" : "a_string",
> "type" : {
> "type" : "string",
> "foobar" : "some example here"
> }
> }
> ]
> }
> {code}
> The property "foobar" will be lost when parsing the schema.
> The fix that my team has implemented is modifying this line (file used from
> release 1.9.2)
>
> [https://github.com/apache/avro/blob/bf20128ca6138a830b2ea13e0490f3df6b035639/lang/csharp/src/apache/main/Schema/PrimitiveSchema.cs#L85]
> And replacing it with the following:
>
> {code:java}
> if(this.Props != null && this.Props.Count > 0)
> {
> w.WriteStartObject();
> w.WritePropertyName("type");
> w.WriteValue(Name);
> foreach(var prop in Props)
> {
> w.WritePropertyName(prop.Key);
> w.WriteRawValue(prop.Value);
> }
> w.WriteEndObject();
> }
> else w.WriteValue(Name);{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2780) ProtobufData and ThriftData Can Get Into Endless Loop
[
https://issues.apache.org/jira/browse/AVRO-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128518#comment-17128518
]
David Mollitor commented on AVRO-2780:
--
[~allenxwang] ^^^
> ProtobufData and ThriftData Can Get Into Endless Loop
> -
>
> Key: AVRO-2780
> URL: https://issues.apache.org/jira/browse/AVRO-2780
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.2
>Reporter: Tianyu Lang
>Assignee: David Mollitor
>Priority: Critical
> Fix For: 1.10.0
>
>
> We have found a bug related, but probably not limited to ProtobufDatumReader.
> In Protobuf, if the ".proto" file has the same name as the message, in the
> generated ".java" file, "OuterClass" will be appended to the outer class's
> name. This is documented here:
> [https://developers.google.com/protocol-buffers/docs/reference/java-generated#invocation]
> Now we have a test protobuf class defined like so:
>
> {code:java}
> syntax = "proto2";
> package test.avrobug;
> option java_package = "com.test.protos.avrobug";
> option java_generic_services = true;
> enum State {
> BAD = 0;
> VERY_BAD = 1;
> SUPER_BAD = 2;
> }
> message Dummy {
> optional string token = 1;
> optional string description = 2;
> optional int64 count = 3;
> optional State state = 4;
> }
> {code}
>
>
> If we first create a Protobuf object, write it to a file as Avro through
> ProtobufDatumReader, then read it into a Protobuf with ProtobufDatumReader, a
> stack overflow exception will happen. Code to reproduce is as follows:
>
> {code:java}
> @Test
> public void ProtoToAvroOuterClassBug() throws Exception {
> DummyOuterClass.Dummy dummy = DummyOuterClass.Dummy.newBuilder()
> .setCount(50)
> .setDescription("hehe")
> .setToken("abc123")
> .setState(DummyOuterClass.State.BAD)
> .build();
>
> ProtobufDatumWriter pbWriter =
> new ProtobufDatumWriter<>(DummyOuterClass.Dummy.class);
> DataFileWriter dataFileWriter = new
> DataFileWriter<>(pbWriter);
> Schema schema = ProtobufData.get().getSchema(DummyOuterClass.Dummy.class);
> dataFileWriter.create(schema,
> new File("/Users/me/Documents/DummyAvroNoDefaultValues.avro"));
> dataFileWriter.append(dummy);
> dataFileWriter.close();
> ProtobufDatumReader pbReader =
> new ProtobufDatumReader<>(DummyOuterClass.Dummy.class);
> DataFileReader dataFileReader =
> new DataFileReader<>(new
> File("/Users/me/Documents/DummyAvroNoDefaultValues.avro"),
> pbReader);
> while(dataFileReader.hasNext()) {
> DummyOuterClass.Dummy record = dataFileReader.next();
> String recordStr = record.toString();
> String originStr = dummy.toString();
> System.out.println(recordStr);
> System.out.println(originStr);
> }
> }
> {code}
>
>
> When this is run, a stack overflow exception will happen with the following
> stack trace:
>
> {code:java}
> java.lang.StackOverflowErrorjava.lang.StackOverflowError
> at
> java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
> at
> org.apache.avro.specific.SpecificData.getClass(SpecificData.java:250)
> at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:141)
> at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:143)
> at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:143)
> at
> ..{code}
> newRecord() is recursed infinitely until stack explodes
> I did a bit code tracing and I found out when Avro tries to load the
> corresponding Java class from the schema, it tries to load the class
> "com.test.protos.avrobug.Dummy.Dummy". I suspect the double "Dummy" at the
> end is related to the stack overflow, but I don't have enough knowledge on
> Proto or Avro to know the exact reason.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2780) ProtobufData and ThriftData Can Get Into Endless Loop
[
https://issues.apache.org/jira/browse/AVRO-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128517#comment-17128517
]
David Mollitor commented on AVRO-2780:
--
I would think so! Thanks for bringing to my attention.
I've seen something similar before. Avro attempts to load classes unexpectedly
depending on the use case. Here, the class loading was failing and then trying
to fall back. The fall back was where things were not working. Try out your
use case again and see if this resolves it. Otherwise there is another issue
at play here and the library should not be falling back.
> ProtobufData and ThriftData Can Get Into Endless Loop
> -
>
> Key: AVRO-2780
> URL: https://issues.apache.org/jira/browse/AVRO-2780
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.2
>Reporter: Tianyu Lang
>Assignee: David Mollitor
>Priority: Critical
> Fix For: 1.10.0
>
>
> We have found a bug related, but probably not limited to ProtobufDatumReader.
> In Protobuf, if the ".proto" file has the same name as the message, in the
> generated ".java" file, "OuterClass" will be appended to the outer class's
> name. This is documented here:
> [https://developers.google.com/protocol-buffers/docs/reference/java-generated#invocation]
> Now we have a test protobuf class defined like so:
>
> {code:java}
> syntax = "proto2";
> package test.avrobug;
> option java_package = "com.test.protos.avrobug";
> option java_generic_services = true;
> enum State {
> BAD = 0;
> VERY_BAD = 1;
> SUPER_BAD = 2;
> }
> message Dummy {
> optional string token = 1;
> optional string description = 2;
> optional int64 count = 3;
> optional State state = 4;
> }
> {code}
>
>
> If we first create a Protobuf object, write it to a file as Avro through
> ProtobufDatumReader, then read it into a Protobuf with ProtobufDatumReader, a
> stack overflow exception will happen. Code to reproduce is as follows:
>
> {code:java}
> @Test
> public void ProtoToAvroOuterClassBug() throws Exception {
> DummyOuterClass.Dummy dummy = DummyOuterClass.Dummy.newBuilder()
> .setCount(50)
> .setDescription("hehe")
> .setToken("abc123")
> .setState(DummyOuterClass.State.BAD)
> .build();
>
> ProtobufDatumWriter pbWriter =
> new ProtobufDatumWriter<>(DummyOuterClass.Dummy.class);
> DataFileWriter dataFileWriter = new
> DataFileWriter<>(pbWriter);
> Schema schema = ProtobufData.get().getSchema(DummyOuterClass.Dummy.class);
> dataFileWriter.create(schema,
> new File("/Users/me/Documents/DummyAvroNoDefaultValues.avro"));
> dataFileWriter.append(dummy);
> dataFileWriter.close();
> ProtobufDatumReader pbReader =
> new ProtobufDatumReader<>(DummyOuterClass.Dummy.class);
> DataFileReader dataFileReader =
> new DataFileReader<>(new
> File("/Users/me/Documents/DummyAvroNoDefaultValues.avro"),
> pbReader);
> while(dataFileReader.hasNext()) {
> DummyOuterClass.Dummy record = dataFileReader.next();
> String recordStr = record.toString();
> String originStr = dummy.toString();
> System.out.println(recordStr);
> System.out.println(originStr);
> }
> }
> {code}
>
>
> When this is run, a stack overflow exception will happen with the following
> stack trace:
>
> {code:java}
> java.lang.StackOverflowErrorjava.lang.StackOverflowError
> at
> java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
> at
> org.apache.avro.specific.SpecificData.getClass(SpecificData.java:250)
> at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:141)
> at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:143)
> at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:143)
> at
> ..{code}
> newRecord() is recursed infinitely until stack explodes
> I did a bit code tracing and I found out when Avro tries to load the
> corresponding Java class from the schema, it tries to load the class
> "com.test.protos.avrobug.Dummy.Dummy". I suspect the double "Dummy" at the
> end is related to the stack overflow, but I don't have enough knowledge on
> Proto or Avro to know the exact reason.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2780) ProtobufData and ThriftData Can Get Into Endless Loop
[
https://issues.apache.org/jira/browse/AVRO-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128494#comment-17128494
]
Allen Wang commented on AVRO-2780:
--
Does this fix also apply to AVRO-2799? That one has nothing to do with
outerclass though.
> ProtobufData and ThriftData Can Get Into Endless Loop
> -
>
> Key: AVRO-2780
> URL: https://issues.apache.org/jira/browse/AVRO-2780
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.2
>Reporter: Tianyu Lang
>Assignee: David Mollitor
>Priority: Critical
> Fix For: 1.10.0
>
>
> We have found a bug related, but probably not limited to ProtobufDatumReader.
> In Protobuf, if the ".proto" file has the same name as the message, in the
> generated ".java" file, "OuterClass" will be appended to the outer class's
> name. This is documented here:
> [https://developers.google.com/protocol-buffers/docs/reference/java-generated#invocation]
> Now we have a test protobuf class defined like so:
>
> {code:java}
> syntax = "proto2";
> package test.avrobug;
> option java_package = "com.test.protos.avrobug";
> option java_generic_services = true;
> enum State {
> BAD = 0;
> VERY_BAD = 1;
> SUPER_BAD = 2;
> }
> message Dummy {
> optional string token = 1;
> optional string description = 2;
> optional int64 count = 3;
> optional State state = 4;
> }
> {code}
>
>
> If we first create a Protobuf object, write it to a file as Avro through
> ProtobufDatumReader, then read it into a Protobuf with ProtobufDatumReader, a
> stack overflow exception will happen. Code to reproduce is as follows:
>
> {code:java}
> @Test
> public void ProtoToAvroOuterClassBug() throws Exception {
> DummyOuterClass.Dummy dummy = DummyOuterClass.Dummy.newBuilder()
> .setCount(50)
> .setDescription("hehe")
> .setToken("abc123")
> .setState(DummyOuterClass.State.BAD)
> .build();
>
> ProtobufDatumWriter pbWriter =
> new ProtobufDatumWriter<>(DummyOuterClass.Dummy.class);
> DataFileWriter dataFileWriter = new
> DataFileWriter<>(pbWriter);
> Schema schema = ProtobufData.get().getSchema(DummyOuterClass.Dummy.class);
> dataFileWriter.create(schema,
> new File("/Users/me/Documents/DummyAvroNoDefaultValues.avro"));
> dataFileWriter.append(dummy);
> dataFileWriter.close();
> ProtobufDatumReader pbReader =
> new ProtobufDatumReader<>(DummyOuterClass.Dummy.class);
> DataFileReader dataFileReader =
> new DataFileReader<>(new
> File("/Users/me/Documents/DummyAvroNoDefaultValues.avro"),
> pbReader);
> while(dataFileReader.hasNext()) {
> DummyOuterClass.Dummy record = dataFileReader.next();
> String recordStr = record.toString();
> String originStr = dummy.toString();
> System.out.println(recordStr);
> System.out.println(originStr);
> }
> }
> {code}
>
>
> When this is run, a stack overflow exception will happen with the following
> stack trace:
>
> {code:java}
> java.lang.StackOverflowErrorjava.lang.StackOverflowError
> at
> java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
> at
> org.apache.avro.specific.SpecificData.getClass(SpecificData.java:250)
> at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:141)
> at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:143)
> at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:143)
> at
> ..{code}
> newRecord() is recursed infinitely until stack explodes
> I did a bit code tracing and I found out when Avro tries to load the
> corresponding Java class from the schema, it tries to load the class
> "com.test.protos.avrobug.Dummy.Dummy". I suspect the double "Dummy" at the
> end is related to the stack overflow, but I don't have enough knowledge on
> Proto or Avro to know the exact reason.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AVRO-1664) PHP library can't serialise records with optional (union-null) values
[
https://issues.apache.org/jira/browse/AVRO-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Skraba updated AVRO-1664:
--
Fix Version/s: 1.10.0
> PHP library can't serialise records with optional (union-null) values
> -
>
> Key: AVRO-1664
> URL: https://issues.apache.org/jira/browse/AVRO-1664
> Project: Apache Avro
> Issue Type: Bug
> Components: php
>Affects Versions: 1.7.7
> Environment: php 5.5.15 OS X 10.9
>Reporter: Paul Banks
>Assignee: Siad Ardroumli
>Priority: Major
> Fix For: 1.10.0
>
>
> PHP avro serialising doesn't appear to support "optional" fields in records.
> Consider the PHP script below:
> {code}
> require_once('lib/avro.php');
> $schema_json = <<<_JSON
> {"name":"member",
> "type":"record",
> "fields":[{"name":"one", "type":"int"},
>{"name":"two", "type":["null", "string"]}
>]}
> _JSON;
> $schema = AvroSchema::parse($schema_json);
> // Our datum is missing the 'optional' field (i.e. it's null)
> $datum = array("one" => 1);
> $io = new AvroStringIO();
> $writer = new AvroIODatumWriter($schema);
> $encoder = new AvroIOBinaryEncoder($io);
> $writer->write($datum, $encoder);
> $bin = $io->string();
> echo bin2hex($bin) . "\n";
> {code}
> My understanding from documentation is that this should work and output the
> encoded binary in hex.
> Instead it throws:
> {code}
> PHP Fatal error: Uncaught exception 'AvroIOTypeException' with message 'The
> datum array (
> 'one' => 1,
> ) is not an example of schema
> {"type":"record","name":"member","fields":[{"name":"one","type":"int"},{"name":"two","type":["null","string"]}]}'
> {code}
> It's possible that this is not a valid usage of Avro and I'm mistaken in my
> expectations, so I tried the python library as a comparison. Sure enough the
> following script works as expected:
> {code}
> from avro import schema
> from avro import io
> from StringIO import StringIO
> s = schema.parse("""
> {"name":"member",
> "type":"record",
> "fields":[{"name":"one", "type":"int"},
> {"name":"two", "type":["null", "string"]}
> ]}""")
> writer = StringIO()
> encoder = io.BinaryEncoder(writer)
> datum_writer = io.DatumWriter(s)
> datum_writer.write({"one": 1}, encoder)
> print writer.getvalue().encode("hex")
> {code}
> which outputs:
> {code}
> $ python avro_test.py
> 0200
> {code}
> As expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-1664) PHP library can't serialise records with optional (union-null) values
[
https://issues.apache.org/jira/browse/AVRO-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Skraba resolved AVRO-1664.
---
Resolution: Fixed
> PHP library can't serialise records with optional (union-null) values
> -
>
> Key: AVRO-1664
> URL: https://issues.apache.org/jira/browse/AVRO-1664
> Project: Apache Avro
> Issue Type: Bug
> Components: php
>Affects Versions: 1.7.7
> Environment: php 5.5.15 OS X 10.9
>Reporter: Paul Banks
>Assignee: Siad Ardroumli
>Priority: Major
>
> PHP avro serialising doesn't appear to support "optional" fields in records.
> Consider the PHP script below:
> {code}
> require_once('lib/avro.php');
> $schema_json = <<<_JSON
> {"name":"member",
> "type":"record",
> "fields":[{"name":"one", "type":"int"},
>{"name":"two", "type":["null", "string"]}
>]}
> _JSON;
> $schema = AvroSchema::parse($schema_json);
> // Our datum is missing the 'optional' field (i.e. it's null)
> $datum = array("one" => 1);
> $io = new AvroStringIO();
> $writer = new AvroIODatumWriter($schema);
> $encoder = new AvroIOBinaryEncoder($io);
> $writer->write($datum, $encoder);
> $bin = $io->string();
> echo bin2hex($bin) . "\n";
> {code}
> My understanding from documentation is that this should work and output the
> encoded binary in hex.
> Instead it throws:
> {code}
> PHP Fatal error: Uncaught exception 'AvroIOTypeException' with message 'The
> datum array (
> 'one' => 1,
> ) is not an example of schema
> {"type":"record","name":"member","fields":[{"name":"one","type":"int"},{"name":"two","type":["null","string"]}]}'
> {code}
> It's possible that this is not a valid usage of Avro and I'm mistaken in my
> expectations, so I tried the python library as a comparison. Sure enough the
> following script works as expected:
> {code}
> from avro import schema
> from avro import io
> from StringIO import StringIO
> s = schema.parse("""
> {"name":"member",
> "type":"record",
> "fields":[{"name":"one", "type":"int"},
> {"name":"two", "type":["null", "string"]}
> ]}""")
> writer = StringIO()
> encoder = io.BinaryEncoder(writer)
> datum_writer = io.DatumWriter(s)
> datum_writer.write({"one": 1}, encoder)
> print writer.getvalue().encode("hex")
> {code}
> which outputs:
> {code}
> $ python avro_test.py
> 0200
> {code}
> As expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-1669) Schema from JSON drops extra attributes
[ https://issues.apache.org/jira/browse/AVRO-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128412#comment-17128412 ] Ryan Skraba commented on AVRO-1669: --- Agreed, it looks like this was a typo for AVRO-1664. > Schema from JSON drops extra attributes > --- > > Key: AVRO-1669 > URL: https://issues.apache.org/jira/browse/AVRO-1669 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.7.7 >Reporter: Oleksandr Shulgin >Priority: Major > Attachments: avro-schema-from-json-logical-type-example.c > > > Schema produced by avro_schema_from_json can be missing substantial details > found in the JSON, e.g. logicalType. > The spec says that Decimal logical type can be derived from either fixed or > bytes type: http://avro.apache.org/docs/1.7.7/spec.html#Decimal > However, after parsing a correct JSON, the resulting schema is missing any > details of the logical type. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AVRO-1669) Schema from JSON drops extra attributes
[ https://issues.apache.org/jira/browse/AVRO-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Skraba updated AVRO-1669: -- Fix Version/s: (was: 1.10.0) > Schema from JSON drops extra attributes > --- > > Key: AVRO-1669 > URL: https://issues.apache.org/jira/browse/AVRO-1669 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.7.7 >Reporter: Oleksandr Shulgin >Priority: Major > Attachments: avro-schema-from-json-logical-type-example.c > > > Schema produced by avro_schema_from_json can be missing substantial details > found in the JSON, e.g. logicalType. > The spec says that Decimal logical type can be derived from either fixed or > bytes type: http://avro.apache.org/docs/1.7.7/spec.html#Decimal > However, after parsing a correct JSON, the resulting schema is missing any > details of the logical type. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AVRO-1669) Schema from JSON drops extra attributes
[ https://issues.apache.org/jira/browse/AVRO-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Skraba reassigned AVRO-1669: - Assignee: (was: Siad Ardroumli) > Schema from JSON drops extra attributes > --- > > Key: AVRO-1669 > URL: https://issues.apache.org/jira/browse/AVRO-1669 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.7.7 >Reporter: Oleksandr Shulgin >Priority: Major > Fix For: 1.10.0 > > Attachments: avro-schema-from-json-logical-type-example.c > > > Schema produced by avro_schema_from_json can be missing substantial details > found in the JSON, e.g. logicalType. > The spec says that Decimal logical type can be derived from either fixed or > bytes type: http://avro.apache.org/docs/1.7.7/spec.html#Decimal > However, after parsing a correct JSON, the resulting schema is missing any > details of the logical type. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AVRO-1664) PHP library can't serialise records with optional (union-null) values
[
https://issues.apache.org/jira/browse/AVRO-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Skraba reassigned AVRO-1664:
-
Assignee: Siad Ardroumli
> PHP library can't serialise records with optional (union-null) values
> -
>
> Key: AVRO-1664
> URL: https://issues.apache.org/jira/browse/AVRO-1664
> Project: Apache Avro
> Issue Type: Bug
> Components: php
>Affects Versions: 1.7.7
> Environment: php 5.5.15 OS X 10.9
>Reporter: Paul Banks
>Assignee: Siad Ardroumli
>Priority: Major
>
> PHP avro serialising doesn't appear to support "optional" fields in records.
> Consider the PHP script below:
> {code}
> require_once('lib/avro.php');
> $schema_json = <<<_JSON
> {"name":"member",
> "type":"record",
> "fields":[{"name":"one", "type":"int"},
>{"name":"two", "type":["null", "string"]}
>]}
> _JSON;
> $schema = AvroSchema::parse($schema_json);
> // Our datum is missing the 'optional' field (i.e. it's null)
> $datum = array("one" => 1);
> $io = new AvroStringIO();
> $writer = new AvroIODatumWriter($schema);
> $encoder = new AvroIOBinaryEncoder($io);
> $writer->write($datum, $encoder);
> $bin = $io->string();
> echo bin2hex($bin) . "\n";
> {code}
> My understanding from documentation is that this should work and output the
> encoded binary in hex.
> Instead it throws:
> {code}
> PHP Fatal error: Uncaught exception 'AvroIOTypeException' with message 'The
> datum array (
> 'one' => 1,
> ) is not an example of schema
> {"type":"record","name":"member","fields":[{"name":"one","type":"int"},{"name":"two","type":["null","string"]}]}'
> {code}
> It's possible that this is not a valid usage of Avro and I'm mistaken in my
> expectations, so I tried the python library as a comparison. Sure enough the
> following script works as expected:
> {code}
> from avro import schema
> from avro import io
> from StringIO import StringIO
> s = schema.parse("""
> {"name":"member",
> "type":"record",
> "fields":[{"name":"one", "type":"int"},
> {"name":"two", "type":["null", "string"]}
> ]}""")
> writer = StringIO()
> encoder = io.BinaryEncoder(writer)
> datum_writer = io.DatumWriter(s)
> datum_writer.write({"one": 1}, encoder)
> print writer.getvalue().encode("hex")
> {code}
> which outputs:
> {code}
> $ python avro_test.py
> 0200
> {code}
> As expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2407) Automate the version bumping of the test resources
[ https://issues.apache.org/jira/browse/AVRO-2407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128315#comment-17128315 ] Hudson commented on AVRO-2407: -- SUCCESS: Integrated in Jenkins build AvroJava #908 (See [https://builds.apache.org/job/AvroJava/908/]) AVRO-2407: Manually fix test pom.xml releases (#907) (github: [https://github.com/apache/avro/commit/f84ca85ab59cfef44b999262cde5fcd87f9aea19]) * (edit) lang/java/maven-plugin/src/test/resources/unit/protocol/pom.xml * (edit) lang/java/maven-plugin/src/test/resources/unit/idl/pom.xml * (edit) lang/java/maven-plugin/src/test/resources/unit/schema/pom.xml > Automate the version bumping of the test resources > -- > > Key: AVRO-2407 > URL: https://issues.apache.org/jira/browse/AVRO-2407 > Project: Apache Avro > Issue Type: Improvement > Components: build >Affects Versions: 1.9.0 >Reporter: Fokko Driesprong >Priority: Major > Fix For: 1.11.0, 1.10.1 > > > When performing a version bump using Maven, the pom's in the test directories > aren't updated. For example > lang/java/maven-plugin/src/test/resources/unit/idl/pom-joda.xml. > This is a source of errors and confusion, therefore we want to automatically > update these versions as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (AVRO-1669) Schema from JSON drops extra attributes
[ https://issues.apache.org/jira/browse/AVRO-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleksandr Shulgin reopened AVRO-1669: - The change committed in #913 doesn't seem to be related to this issue (PHP null fields?). > Schema from JSON drops extra attributes > --- > > Key: AVRO-1669 > URL: https://issues.apache.org/jira/browse/AVRO-1669 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.7.7 >Reporter: Oleksandr Shulgin >Assignee: Siad Ardroumli >Priority: Major > Fix For: 1.10.0 > > Attachments: avro-schema-from-json-logical-type-example.c > > > Schema produced by avro_schema_from_json can be missing substantial details > found in the JSON, e.g. logicalType. > The spec says that Decimal logical type can be derived from either fixed or > bytes type: http://avro.apache.org/docs/1.7.7/spec.html#Decimal > However, after parsing a correct JSON, the resulting schema is missing any > details of the logical type. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-2780) ProtobufData and ThriftData Can Get Into Endless Loop
[
https://issues.apache.org/jira/browse/AVRO-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128293#comment-17128293
]
Hudson commented on AVRO-2780:
--
SUCCESS: Integrated in Jenkins build AvroJava #907 (See
[https://builds.apache.org/job/AvroJava/907/])
AVRO-2780: ProtobufData and ThriftData Can Get Into Endless Loop (#908)
(github:
[https://github.com/apache/avro/commit/249db48c63a9ddf5bd736a8c020fa5ebb267c50d])
* (edit) lang/java/thrift/src/main/java/org/apache/avro/thrift/ThriftData.java
* (edit)
lang/java/protobuf/src/main/java/org/apache/avro/protobuf/ProtobufData.java
> ProtobufData and ThriftData Can Get Into Endless Loop
> -
>
> Key: AVRO-2780
> URL: https://issues.apache.org/jira/browse/AVRO-2780
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.2
>Reporter: Tianyu Lang
>Assignee: David Mollitor
>Priority: Critical
> Fix For: 1.10.0
>
>
> We have found a bug related, but probably not limited to ProtobufDatumReader.
> In Protobuf, if the ".proto" file has the same name as the message, in the
> generated ".java" file, "OuterClass" will be appended to the outer class's
> name. This is documented here:
> [https://developers.google.com/protocol-buffers/docs/reference/java-generated#invocation]
> Now we have a test protobuf class defined like so:
>
> {code:java}
> syntax = "proto2";
> package test.avrobug;
> option java_package = "com.test.protos.avrobug";
> option java_generic_services = true;
> enum State {
> BAD = 0;
> VERY_BAD = 1;
> SUPER_BAD = 2;
> }
> message Dummy {
> optional string token = 1;
> optional string description = 2;
> optional int64 count = 3;
> optional State state = 4;
> }
> {code}
>
>
> If we first create a Protobuf object, write it to a file as Avro through
> ProtobufDatumReader, then read it into a Protobuf with ProtobufDatumReader, a
> stack overflow exception will happen. Code to reproduce is as follows:
>
> {code:java}
> @Test
> public void ProtoToAvroOuterClassBug() throws Exception {
> DummyOuterClass.Dummy dummy = DummyOuterClass.Dummy.newBuilder()
> .setCount(50)
> .setDescription("hehe")
> .setToken("abc123")
> .setState(DummyOuterClass.State.BAD)
> .build();
>
> ProtobufDatumWriter pbWriter =
> new ProtobufDatumWriter<>(DummyOuterClass.Dummy.class);
> DataFileWriter dataFileWriter = new
> DataFileWriter<>(pbWriter);
> Schema schema = ProtobufData.get().getSchema(DummyOuterClass.Dummy.class);
> dataFileWriter.create(schema,
> new File("/Users/me/Documents/DummyAvroNoDefaultValues.avro"));
> dataFileWriter.append(dummy);
> dataFileWriter.close();
> ProtobufDatumReader pbReader =
> new ProtobufDatumReader<>(DummyOuterClass.Dummy.class);
> DataFileReader dataFileReader =
> new DataFileReader<>(new
> File("/Users/me/Documents/DummyAvroNoDefaultValues.avro"),
> pbReader);
> while(dataFileReader.hasNext()) {
> DummyOuterClass.Dummy record = dataFileReader.next();
> String recordStr = record.toString();
> String originStr = dummy.toString();
> System.out.println(recordStr);
> System.out.println(originStr);
> }
> }
> {code}
>
>
> When this is run, a stack overflow exception will happen with the following
> stack trace:
>
> {code:java}
> java.lang.StackOverflowErrorjava.lang.StackOverflowError
> at
> java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
> at
> org.apache.avro.specific.SpecificData.getClass(SpecificData.java:250)
> at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:141)
> at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:143)
> at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:143)
> at
> ..{code}
> newRecord() is recursed infinitely until stack explodes
> I did a bit code tracing and I found out when Avro tries to load the
> corresponding Java class from the schema, it tries to load the class
> "com.test.protos.avrobug.Dummy.Dummy". I suspect the double "Dummy" at the
> end is related to the stack overflow, but I don't have enough knowledge on
> Proto or Avro to know the exact reason.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2139) Add Test for Decimal @java-class Annotated Generated Classes
[
https://issues.apache.org/jira/browse/AVRO-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128294#comment-17128294
]
Hudson commented on AVRO-2139:
--
SUCCESS: Integrated in Jenkins build AvroJava #907 (See
[https://builds.apache.org/job/AvroJava/907/])
AVRO-2139: Add Test for Decimal @java-class Annotated Generated Classes
(github:
[https://github.com/apache/avro/commit/51d0fe8451bc15e3274ba2f0c5ae0260fcfcb78b])
* (add) lang/java/ipc/src/test/java/org/apache/avro/generic/TestBuilderCopy.java
> Add Test for Decimal @java-class Annotated Generated Classes
>
>
> Key: AVRO-2139
> URL: https://issues.apache.org/jira/browse/AVRO-2139
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.8.2
>Reporter: Auke van Leeuwen
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 1.10.0
>
>
> Description:
> When I'm using a schema with an {{@java-class}}, I can't seem to be using the
> 'copy' builder.
> I think a piece of code says more than a thousands words:
> {{.avdl}} snippet:
> {noformat}
> ...
> record Money {
> Currency currency;
> @java-class("java.math.BigDecimal")
> string amount;
> }
> ...
> {noformat}
> Test:
> {noformat}
> @Test
> void test_copy_builder() {
> Money original = Money.newBuilder()
> .setAmount(BigDecimal.TEN)
> .setCurrency(Currency.EUR)
> .build();
> Money duplicate = Money.newBuilder(original).build();
> assertThat(duplicate, is(equalTo(original)));
> }
> {noformat}
> Result:
> {noformat}
> java.lang.ClassCastException: org.apache.avro.util.Utf8 cannot be cast to
> java.math.BigDecimal
> at
> avro.moneyou.midlayer.common.financial.Money$Builder.(Money.java:193)
> at
> avro.moneyou.midlayer.common.financial.Money$Builder.(Money.java:155)
> at
> avro.moneyou.midlayer.common.financial.Money.newBuilder(Money.java:149)
> at
> avro.moneyou.midlayer.common.financial.MoneyTest.test_copy_builder(MoneyTest.java:20)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:389)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:115)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:167)
> at
> org.junit.jupiter.engine.execution.ThrowableCollector.execute(ThrowableCollector.java:40)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:163)
> ...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2407) Automate the version bumping of the test resources
[ https://issues.apache.org/jira/browse/AVRO-2407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128287#comment-17128287 ] ASF subversion and git services commented on AVRO-2407: --- Commit f84ca85ab59cfef44b999262cde5fcd87f9aea19 in avro's branch refs/heads/master from RyanSkraba [ https://gitbox.apache.org/repos/asf?p=avro.git;h=f84ca85 ] AVRO-2407: Manually fix test pom.xml releases (#907) > Automate the version bumping of the test resources > -- > > Key: AVRO-2407 > URL: https://issues.apache.org/jira/browse/AVRO-2407 > Project: Apache Avro > Issue Type: Improvement > Components: build >Affects Versions: 1.9.0 >Reporter: Fokko Driesprong >Priority: Major > Fix For: 1.11.0, 1.10.1 > > > When performing a version bump using Maven, the pom's in the test directories > aren't updated. For example > lang/java/maven-plugin/src/test/resources/unit/idl/pom-joda.xml. > This is a source of errors and confusion, therefore we want to automatically > update these versions as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AVRO-1669) Schema from JSON drops extra attributes
[ https://issues.apache.org/jira/browse/AVRO-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong reassigned AVRO-1669: -- Assignee: Siad Ardroumli > Schema from JSON drops extra attributes > --- > > Key: AVRO-1669 > URL: https://issues.apache.org/jira/browse/AVRO-1669 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.7.7 >Reporter: Oleksandr Shulgin >Assignee: Siad Ardroumli >Priority: Major > Fix For: 1.10.0 > > Attachments: avro-schema-from-json-logical-type-example.c > > > Schema produced by avro_schema_from_json can be missing substantial details > found in the JSON, e.g. logicalType. > The spec says that Decimal logical type can be derived from either fixed or > bytes type: http://avro.apache.org/docs/1.7.7/spec.html#Decimal > However, after parsing a correct JSON, the resulting schema is missing any > details of the logical type. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-1669) Schema from JSON drops extra attributes
[ https://issues.apache.org/jira/browse/AVRO-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128281#comment-17128281 ] ASF subversion and git services commented on AVRO-1669: --- Commit b8ccc2f6451343c1da00b288ed4161d055e5daef in avro's branch refs/heads/master from Siad Ardroumli [ https://gitbox.apache.org/repos/asf?p=avro.git;h=b8ccc2f ] AVRO-1669: PHP Fixed null handling on fields. (#913) > Schema from JSON drops extra attributes > --- > > Key: AVRO-1669 > URL: https://issues.apache.org/jira/browse/AVRO-1669 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.7.7 >Reporter: Oleksandr Shulgin >Priority: Major > Attachments: avro-schema-from-json-logical-type-example.c > > > Schema produced by avro_schema_from_json can be missing substantial details > found in the JSON, e.g. logicalType. > The spec says that Decimal logical type can be derived from either fixed or > bytes type: http://avro.apache.org/docs/1.7.7/spec.html#Decimal > However, after parsing a correct JSON, the resulting schema is missing any > details of the logical type. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AVRO-1669) Schema from JSON drops extra attributes
[ https://issues.apache.org/jira/browse/AVRO-1669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong resolved AVRO-1669. Fix Version/s: 1.10.0 Resolution: Fixed > Schema from JSON drops extra attributes > --- > > Key: AVRO-1669 > URL: https://issues.apache.org/jira/browse/AVRO-1669 > Project: Apache Avro > Issue Type: Bug > Components: c >Affects Versions: 1.7.7 >Reporter: Oleksandr Shulgin >Priority: Major > Fix For: 1.10.0 > > Attachments: avro-schema-from-json-logical-type-example.c > > > Schema produced by avro_schema_from_json can be missing substantial details > found in the JSON, e.g. logicalType. > The spec says that Decimal logical type can be derived from either fixed or > bytes type: http://avro.apache.org/docs/1.7.7/spec.html#Decimal > However, after parsing a correct JSON, the resulting schema is missing any > details of the logical type. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-2139) Add Test for Decimal @java-class Annotated Generated Classes
[
https://issues.apache.org/jira/browse/AVRO-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128280#comment-17128280
]
ASF subversion and git services commented on AVRO-2139:
---
Commit 51d0fe8451bc15e3274ba2f0c5ae0260fcfcb78b in avro's branch
refs/heads/master from belugabehr
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=51d0fe8 ]
AVRO-2139: Add Test for Decimal @java-class Annotated Generated Classes (#909)
> Add Test for Decimal @java-class Annotated Generated Classes
>
>
> Key: AVRO-2139
> URL: https://issues.apache.org/jira/browse/AVRO-2139
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.8.2
>Reporter: Auke van Leeuwen
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 1.10.0
>
>
> Description:
> When I'm using a schema with an {{@java-class}}, I can't seem to be using the
> 'copy' builder.
> I think a piece of code says more than a thousands words:
> {{.avdl}} snippet:
> {noformat}
> ...
> record Money {
> Currency currency;
> @java-class("java.math.BigDecimal")
> string amount;
> }
> ...
> {noformat}
> Test:
> {noformat}
> @Test
> void test_copy_builder() {
> Money original = Money.newBuilder()
> .setAmount(BigDecimal.TEN)
> .setCurrency(Currency.EUR)
> .build();
> Money duplicate = Money.newBuilder(original).build();
> assertThat(duplicate, is(equalTo(original)));
> }
> {noformat}
> Result:
> {noformat}
> java.lang.ClassCastException: org.apache.avro.util.Utf8 cannot be cast to
> java.math.BigDecimal
> at
> avro.moneyou.midlayer.common.financial.Money$Builder.(Money.java:193)
> at
> avro.moneyou.midlayer.common.financial.Money$Builder.(Money.java:155)
> at
> avro.moneyou.midlayer.common.financial.Money.newBuilder(Money.java:149)
> at
> avro.moneyou.midlayer.common.financial.MoneyTest.test_copy_builder(MoneyTest.java:20)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:389)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:115)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:167)
> at
> org.junit.jupiter.engine.execution.ThrowableCollector.execute(ThrowableCollector.java:40)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:163)
> ...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-2139) Add Test for Decimal @java-class Annotated Generated Classes
[
https://issues.apache.org/jira/browse/AVRO-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fokko Driesprong resolved AVRO-2139.
Fix Version/s: 1.10.0
Resolution: Fixed
> Add Test for Decimal @java-class Annotated Generated Classes
>
>
> Key: AVRO-2139
> URL: https://issues.apache.org/jira/browse/AVRO-2139
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
>Affects Versions: 1.8.2
>Reporter: Auke van Leeuwen
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 1.10.0
>
>
> Description:
> When I'm using a schema with an {{@java-class}}, I can't seem to be using the
> 'copy' builder.
> I think a piece of code says more than a thousands words:
> {{.avdl}} snippet:
> {noformat}
> ...
> record Money {
> Currency currency;
> @java-class("java.math.BigDecimal")
> string amount;
> }
> ...
> {noformat}
> Test:
> {noformat}
> @Test
> void test_copy_builder() {
> Money original = Money.newBuilder()
> .setAmount(BigDecimal.TEN)
> .setCurrency(Currency.EUR)
> .build();
> Money duplicate = Money.newBuilder(original).build();
> assertThat(duplicate, is(equalTo(original)));
> }
> {noformat}
> Result:
> {noformat}
> java.lang.ClassCastException: org.apache.avro.util.Utf8 cannot be cast to
> java.math.BigDecimal
> at
> avro.moneyou.midlayer.common.financial.Money$Builder.(Money.java:193)
> at
> avro.moneyou.midlayer.common.financial.Money$Builder.(Money.java:155)
> at
> avro.moneyou.midlayer.common.financial.Money.newBuilder(Money.java:149)
> at
> avro.moneyou.midlayer.common.financial.MoneyTest.test_copy_builder(MoneyTest.java:20)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:389)
> at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:115)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:167)
> at
> org.junit.jupiter.engine.execution.ThrowableCollector.execute(ThrowableCollector.java:40)
> at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:163)
> ...
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-2780) ProtobufData and ThriftData Can Get Into Endless Loop
[
https://issues.apache.org/jira/browse/AVRO-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128279#comment-17128279
]
ASF subversion and git services commented on AVRO-2780:
---
Commit 249db48c63a9ddf5bd736a8c020fa5ebb267c50d in avro's branch
refs/heads/master from belugabehr
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=249db48 ]
AVRO-2780: ProtobufData and ThriftData Can Get Into Endless Loop (#908)
> ProtobufData and ThriftData Can Get Into Endless Loop
> -
>
> Key: AVRO-2780
> URL: https://issues.apache.org/jira/browse/AVRO-2780
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.2
>Reporter: Tianyu Lang
>Assignee: David Mollitor
>Priority: Critical
>
> We have found a bug related, but probably not limited to ProtobufDatumReader.
> In Protobuf, if the ".proto" file has the same name as the message, in the
> generated ".java" file, "OuterClass" will be appended to the outer class's
> name. This is documented here:
> [https://developers.google.com/protocol-buffers/docs/reference/java-generated#invocation]
> Now we have a test protobuf class defined like so:
>
> {code:java}
> syntax = "proto2";
> package test.avrobug;
> option java_package = "com.test.protos.avrobug";
> option java_generic_services = true;
> enum State {
> BAD = 0;
> VERY_BAD = 1;
> SUPER_BAD = 2;
> }
> message Dummy {
> optional string token = 1;
> optional string description = 2;
> optional int64 count = 3;
> optional State state = 4;
> }
> {code}
>
>
> If we first create a Protobuf object, write it to a file as Avro through
> ProtobufDatumReader, then read it into a Protobuf with ProtobufDatumReader, a
> stack overflow exception will happen. Code to reproduce is as follows:
>
> {code:java}
> @Test
> public void ProtoToAvroOuterClassBug() throws Exception {
> DummyOuterClass.Dummy dummy = DummyOuterClass.Dummy.newBuilder()
> .setCount(50)
> .setDescription("hehe")
> .setToken("abc123")
> .setState(DummyOuterClass.State.BAD)
> .build();
>
> ProtobufDatumWriter pbWriter =
> new ProtobufDatumWriter<>(DummyOuterClass.Dummy.class);
> DataFileWriter dataFileWriter = new
> DataFileWriter<>(pbWriter);
> Schema schema = ProtobufData.get().getSchema(DummyOuterClass.Dummy.class);
> dataFileWriter.create(schema,
> new File("/Users/me/Documents/DummyAvroNoDefaultValues.avro"));
> dataFileWriter.append(dummy);
> dataFileWriter.close();
> ProtobufDatumReader pbReader =
> new ProtobufDatumReader<>(DummyOuterClass.Dummy.class);
> DataFileReader dataFileReader =
> new DataFileReader<>(new
> File("/Users/me/Documents/DummyAvroNoDefaultValues.avro"),
> pbReader);
> while(dataFileReader.hasNext()) {
> DummyOuterClass.Dummy record = dataFileReader.next();
> String recordStr = record.toString();
> String originStr = dummy.toString();
> System.out.println(recordStr);
> System.out.println(originStr);
> }
> }
> {code}
>
>
> When this is run, a stack overflow exception will happen with the following
> stack trace:
>
> {code:java}
> java.lang.StackOverflowErrorjava.lang.StackOverflowError
> at
> java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
> at
> org.apache.avro.specific.SpecificData.getClass(SpecificData.java:250)
> at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:141)
> at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:143)
> at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:143)
> at
> ..{code}
> newRecord() is recursed infinitely until stack explodes
> I did a bit code tracing and I found out when Avro tries to load the
> corresponding Java class from the schema, it tries to load the class
> "com.test.protos.avrobug.Dummy.Dummy". I suspect the double "Dummy" at the
> end is related to the stack overflow, but I don't have enough knowledge on
> Proto or Avro to know the exact reason.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AVRO-2780) ProtobufData and ThriftData Can Get Into Endless Loop
[
https://issues.apache.org/jira/browse/AVRO-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fokko Driesprong resolved AVRO-2780.
Fix Version/s: 1.10.0
Resolution: Fixed
> ProtobufData and ThriftData Can Get Into Endless Loop
> -
>
> Key: AVRO-2780
> URL: https://issues.apache.org/jira/browse/AVRO-2780
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.9.2
>Reporter: Tianyu Lang
>Assignee: David Mollitor
>Priority: Critical
> Fix For: 1.10.0
>
>
> We have found a bug related, but probably not limited to ProtobufDatumReader.
> In Protobuf, if the ".proto" file has the same name as the message, in the
> generated ".java" file, "OuterClass" will be appended to the outer class's
> name. This is documented here:
> [https://developers.google.com/protocol-buffers/docs/reference/java-generated#invocation]
> Now we have a test protobuf class defined like so:
>
> {code:java}
> syntax = "proto2";
> package test.avrobug;
> option java_package = "com.test.protos.avrobug";
> option java_generic_services = true;
> enum State {
> BAD = 0;
> VERY_BAD = 1;
> SUPER_BAD = 2;
> }
> message Dummy {
> optional string token = 1;
> optional string description = 2;
> optional int64 count = 3;
> optional State state = 4;
> }
> {code}
>
>
> If we first create a Protobuf object, write it to a file as Avro through
> ProtobufDatumReader, then read it into a Protobuf with ProtobufDatumReader, a
> stack overflow exception will happen. Code to reproduce is as follows:
>
> {code:java}
> @Test
> public void ProtoToAvroOuterClassBug() throws Exception {
> DummyOuterClass.Dummy dummy = DummyOuterClass.Dummy.newBuilder()
> .setCount(50)
> .setDescription("hehe")
> .setToken("abc123")
> .setState(DummyOuterClass.State.BAD)
> .build();
>
> ProtobufDatumWriter pbWriter =
> new ProtobufDatumWriter<>(DummyOuterClass.Dummy.class);
> DataFileWriter dataFileWriter = new
> DataFileWriter<>(pbWriter);
> Schema schema = ProtobufData.get().getSchema(DummyOuterClass.Dummy.class);
> dataFileWriter.create(schema,
> new File("/Users/me/Documents/DummyAvroNoDefaultValues.avro"));
> dataFileWriter.append(dummy);
> dataFileWriter.close();
> ProtobufDatumReader pbReader =
> new ProtobufDatumReader<>(DummyOuterClass.Dummy.class);
> DataFileReader dataFileReader =
> new DataFileReader<>(new
> File("/Users/me/Documents/DummyAvroNoDefaultValues.avro"),
> pbReader);
> while(dataFileReader.hasNext()) {
> DummyOuterClass.Dummy record = dataFileReader.next();
> String recordStr = record.toString();
> String originStr = dummy.toString();
> System.out.println(recordStr);
> System.out.println(originStr);
> }
> }
> {code}
>
>
> When this is run, a stack overflow exception will happen with the following
> stack trace:
>
> {code:java}
> java.lang.StackOverflowErrorjava.lang.StackOverflowError
> at
> java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
> at
> org.apache.avro.specific.SpecificData.getClass(SpecificData.java:250)
> at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:141)
> at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:143)
> at
> org.apache.avro.protobuf.ProtobufData.newRecord(ProtobufData.java:143)
> at
> ..{code}
> newRecord() is recursed infinitely until stack explodes
> I did a bit code tracing and I found out when Avro tries to load the
> corresponding Java class from the schema, it tries to load the class
> "com.test.protos.avrobug.Dummy.Dummy". I suspect the double "Dummy" at the
> end is related to the stack overflow, but I don't have enough knowledge on
> Proto or Avro to know the exact reason.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AVRO-2757) ReflectData still generates namespaces with dollar signs
[
https://issues.apache.org/jira/browse/AVRO-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Skraba updated AVRO-2757:
--
Status: Patch Available (was: Open)
> ReflectData still generates namespaces with dollar signs
>
>
> Key: AVRO-2757
> URL: https://issues.apache.org/jira/browse/AVRO-2757
> Project: Apache Avro
> Issue Type: Bug
> Components: java
>Reporter: Ryan Skraba
>Assignee: Ryan Skraba
>Priority: Major
>
> ReflectData should always generate valid Avro schemas for Java classes.
> Currently, the name and namespace are taken from the Java class name, which
> can contain dollar signs. AVRO-2143 fixed this for the most common cases for
> nested classes.
> It's still possible to generate namespaces with a dollar sign for classes
> that are nested more than one level. This can be observed in this unit test:
> [https://github.com/apache/avro/blob/70260919426f89825ca148f5ee815f3b2cf4764d/lang/java/avro/src/test/java/org/apache/avro/reflect/TestReflect.java#L842]
> which generates
> {code:java}
> {
> "type" : "record",
> "name" : "AnotherSampleRecord",
> "namespace" : "org.apache.avro.reflect.TestReflect$SampleRecord",
> "fields" : [ {
> "name" : "a",
> "type" : [ "null", "int" ],
> "default" : null
> }, {
> "name" : "s",
> "type" : [ "null", {
> "type" : "record",
> "name" : "SampleRecord",
> "namespace" : "org.apache.avro.reflect.TestReflect",
> "fields" : [ {
> "name" : "x",
> "type" : "int"
> }, {
> "name" : "y",
> "type" : "int"
> } ]
> } ],
> "default" : null
> } ]
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AVRO-1130) MapReduce Jobs can output write SortedKeyValueFiles directly
[ https://issues.apache.org/jira/browse/AVRO-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128208#comment-17128208 ] Ryan Skraba commented on AVRO-1130: --- If you've found this code useful and are willing to share it, I'm sure it would be useful to others! Please feel free to take this JIRA and submit a PR. > MapReduce Jobs can output write SortedKeyValueFiles directly > > > Key: AVRO-1130 > URL: https://issues.apache.org/jira/browse/AVRO-1130 > Project: Apache Avro > Issue Type: New Feature > Components: java >Affects Versions: 1.7.1 >Reporter: Jeremy Lewi >Priority: Minor > > It would be nice if MapReduce jobs could write directly to > SortedKeyValueFile's. > harsh@'s response on this thread http://goo.gl/OT1rN for some more > information on what needs to be done. -- This message was sent by Atlassian Jira (v8.3.4#803005)
