Hey RB,
I was trying out the circular refs with latest 1.8.1 version of Avro and it
doesn't seem to be working out of the box for me.
Perhaps I am missing something and would appreciate your help.
Thanks,
SG
Here is my test code:
public class CircularRefsTest
{
@Test
public void testSerialization() throws Exception
{
// Create a circular linked-list
ListNode first = new ListNode("first");
ListNode second = new ListNode("second");
second.next = first;
first.next = second;
ReflectData rdata = ReflectData.AllowNull.get();
// Print Schema
Schema schema = rdata.getSchema(ListNode.class);
System.out.println(schema);
// Serialize
DatumWriter<ListNode> datumWriter = new
ReflectDatumWriter<ListNode>(ListNode.class);
ByteArrayOutputStream byteArrayOutputStream = new
ByteArrayOutputStream();
Encoder encoder =
EncoderFactory.get().binaryEncoder(byteArrayOutputStream, null);
datumWriter.write(first, encoder);
encoder.flush();
byte[] bytes = byteArrayOutputStream.toByteArray();
assertTrue( bytes != null );
}
}
class ListNode
{
String data;
ListNode next;
public ListNode() {
}
public ListNode(String data) {
this.data = data;
}
}
On Thu, May 28, 2015 at 11:00 AM, Ryan Blue <[email protected]> wrote:
> SG,
>
> The data ends up looking like this:
>
> {"id":1,"p":"parent data!","child":{"c":"child data!","parent":{"long":1}}}
>
> I produced that with avro-tools 1.7.6 and tojson.
>
> Here's the schema: {
> "type" : "record",
> "name" : "Parent",
> "fields" : [ {
> "name" : "id",
> "type" : "long"
> }, {
> "name" : "p",
> "type" : "string"
> }, {
> "name" : "child",
> "type" : {
> "type" : "record",
> "name" : "Child",
> "fields" : [ {
> "name" : "c",
> "type" : "string"
> }, {
> "name" : "parent",
> "type" : [ "null", "long", "Parent" ]
> } ],
> "logicalType" : "reference",
> "ref-field-name" : "parent"
> }
> } ],
> "logicalType" : "referenceable",
> "id-field-name" : "id"
> }
>
> rb
>
>
> On 05/28/2015 09:50 AM, S G wrote:
>
>> RB,
>>
>> Could you please attach the schema and the JSON serialized output from
>> your
>> test-code as well?
>> My build environment is currently broken as I am grappling with some Java
>> 8
>> update issues.
>>
>> Thanks
>> SG
>>
>>
>> On Wed, May 27, 2015 at 3:28 PM, Ryan Blue <[email protected]> wrote:
>>
>> SG,
>>>
>>> Now that logical types are in, I had some time to look at this issue.
>>> Thanks for your patience on this.
>>>
>>> When I started looking at the use case, this began to look very much like
>>> a logical type issue. (I know, I've been saying that a lot.) When you
>>> write, you replace any referenced object with its id. When you read, you
>>> replace those ids with the correct object. I went ahead and implemented
>>> this using 2 logical types: Referenceable for an object with an id, and
>>> Reference for a record with a field that references another object by id.
>>>
>>> There were some trade-offs to this approach. For example, I had to use a
>>> logical type for the object containing the reference rather than for the
>>> reference itself because I used a callback to set the object once it is
>>> found. That happens because children with references to a parent are
>>> deserialized completely first. The parent is the last object to be
>>> assembled and passed to the logical type conversion.
>>>
>>> A bigger issue was that logical types are currently conservative and
>>> don't
>>> overlap with reflect types or anything that sets java-class. That means
>>> that this currently only works with generic types. But, I'd rather make
>>> logical types work for reflect than add more custom code to support this.
>>> Does that sound reasonable?
>>>
>>> I'm attaching a diff with the working test code so you can take a look at
>>> the approach. Let me know what you are thinking.
>>>
>>> rb
>>>
>>> On 05/20/2015 12:05 PM, S G wrote:
>>>
>>> I am requesting some help with AVRO-695.
>>>> Here are some pieces from the last conversation.
>>>>
>>>>
>>>> Doug Cutting
>>>> <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=cutting>
>>>> added
>>>> a comment - 02/Oct/14 21:19
>>>>
>>>> Here's a modified version of the patch. It moves all significant changes
>>>> to
>>>> reflect. Since reflect is the only implementation that can generate an
>>>> appropriate schema, changes should be confined to there.
>>>>
>>>> The tests need to be updated, as they still assume generic.
>>>> <https://issues.apache.org/jira/browse/AVRO-695#>
>>>> <
>>>>
>>>> https://issues.apache.org/jira/browse/AVRO-695?focusedCommentId=14286370&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14286370
>>>>
>>>>>
>>>>> Sachin Goyal
>>>> <
>>>> https://issues.apache.org/jira/secure/ViewProfile.jspa?name=sachingoyal
>>>> >
>>>> added
>>>> a comment - 21/Jan/15 21:56
>>>>
>>>> Here is a patch with the updated test-cases.
>>>> I also confirm that all my changes are there in the patch.
>>>>
>>>>
>>>> The patch was submitted in June 2014 and was very hot till October 2014.
>>>> Since then, there has been no action on this even though I have sent
>>>> many
>>>> reminders in this group.
>>>>
>>>> I understand that everyone is very busy with their own stuff but I would
>>>> really appreciate if someone could help a fellow engineer in getting his
>>>> patch accepted.
>>>> It would encourage more participation as well as help people wanting to
>>>> use
>>>> Avro for circular references.
>>>>
>>>> Regards
>>>> SG
>>>>
>>>>
>>>
>>> --
>>> Ryan Blue
>>> Software Engineer
>>> Cloudera, Inc.
>>>
>>>
>>
>
> --
> Ryan Blue
> Software Engineer
> Cloudera, Inc.
>
