[
https://issues.apache.org/jira/browse/AVRO-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981471#comment-13981471
]
Ramana Suvarapu commented on AVRO-1474:
---------------------------------------
Please see the email below for more details and attached is the patch to fix
this issue.
From: Suvarapu, Venkata - GMRT-GET
Sent: Friday, April 25, 2014 9:52 AM
To: 'Thiruvalluvan MG'
Cc: 'Srinidhi S'
Subject: RE: Resolving decoder bug's fix
Hi Thiru,
Just to make it clear that ResolvingGrammarGenerator::getWriterProduction
function doesn’t work with Schemas with Symbollic nodes. Because Production to
actual node of Symbollic node is not going to be available in the
node->production map created within getWriterProduction() method. If Schema
contains nested schema for its child nodes then the function works.
Thanks,
Ramana
From: Suvarapu, Venkata - GMRT-GET
Sent: Thursday, April 24, 2014 7:42 PM
To: 'Thiruvalluvan MG'
Cc: 'Srinidhi S'; Ye, Xiaolu - GMRT-EST
Subject: RE: Resolving decoder bug's fix
Hi Thiru,
I debugged the code and found the root cause for the issue. I suspect that
ProductionPtr ResolvingGrammarGenerator::getWriterProduction method has a bug.
Let’s say writer record schema has following additional and reader schema
doesn’t have this field.
{
"name": "optionalNestedrecord",
"type": ["null", "Nested"],
"default": null
}
When resolving the Writer and Readers record fields, Since this field is not
there in reader schema, ResolveRecords does skip the Production of this node
and ProductionPtr for this field will be retrieved by calling
getWriterProduction().
ProductionPtr ResolvingGrammarGenerator::getWriterProduction(
const NodePtr& n, const map<NodePtr, ProductionPtr>& m2)
{
const NodePtr& nn = (n->type() == AVRO_SYMBOLIC) ?
static_cast<const NodeSymbolic& >(*n).getNode() : n;
map<NodePtr, ProductionPtr>::const_iterator it2 = m2.find(nn);
return (it2 != m2.end()) ? it2->second :
ValidatingGrammarGenerator::generate(nn);
}
In the above code, when nn is not found in m2, it calls
ValidatingGrammarGenerator::generate(nn) to generate the ProductionPtr and this
function uses new node->production map instead of m2 which is already processed
during the first pass.
I think this is not correct. We should call ProductionPtr result =
ValidatingGrammarGenerator::doGenerate(nn, m2) to generate the ProductionPtr.
The reason for this is that in my above example schema, optionalNestedrecord is
union schema and since it doesn’t exists in m2, it generates the new production
using new node->production map. Since “Nested” node is not going to be
available in new map, it will generates null production. Later point of time,
this is causing
boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_weak_ptr>
> at memory location 0x00b5d3b0 exception during the performing skip operation
when decoding the record using reader schema. But If m2 is used , “Nested” node
is guaranteed to be available in m2 and that value is used instead of null or
empty production.
Modified function
ProductionPtr ResolvingGrammarGenerator::getWriterProduction(
const NodePtr& n, map<NodePtr, ProductionPtr>& m2)
{
const NodePtr& nn = (n->type() == AVRO_SYMBOLIC) ?
static_cast<const NodeSymbolic& >(*n).getNode() : n;
map<NodePtr, ProductionPtr>::const_iterator it2 = m2.find(nn);
if (it2 != m2.end())
return it2->second;
else
{
ProductionPtr result = ValidatingGrammarGenerator::doGenerate(nn,
m2);
fixup(result, m2);
return result;
}
}
I tested the fix locally and it seems to be working. But I had to modify
qualifiers of arguments of few functions in the code to make it compile.
Attached is the patch for this fix and please review it and see it makes sense.
Thanks,
Ramana
From: Suvarapu, Venkata - GMRT-GET
Sent: Wednesday, April 23, 2014 11:57 PM
To: Thiruvalluvan MG
Cc: 'Srinidhi S'; Ye, Xiaolu - GMRT-EST
Subject: RE: Resolving decoder bug
Hi Thiru,
Attached are writer and reader schema. In writer schema I added new field
“optionalNestedrecord”
to RootRecord which is optional and default value is null. This field doesn’t
exist in Reader schema.
{
"name": "optionalNestedrecord",
"type": ["null", "Nested"],
"default": null
}
When I populate this field in writers RootRecord and when I tried to
deserialize using reader schema, It’s throwing following exception.
boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_weak_ptr>
> at memory location 0x00b5d3b0..
This exception is thrown in the following section of void skip(Decoder& d) of
Symbol.hh
case Symbol::sSymbolic:
{
ProductionPtr pp(
t.extra<boost::weak_ptr<Production> >());
parsingStack.pop();
append(pp);
}
Steps to reproduce:
1. Use the attached files to regenerate bigrecord types for both reader
and writer
2. In writer RootRecord, populate optionalNestedrecord and encode it
3. Using ResolvingDecoder and using reader schema, try to deserialize
encoded writer’s record and you will see this exception.
I attached theses schema to AVRO-1474 jira.
Please take a look into this.
Thanks,
Ramana
On Wed, Apr 23, 2014 at 7:41 PM, Srinidhi S <[email protected]> wrote:
Hi Thiru
I found a bug with latest patch. Here is the scenario.
writer record schema has optional field which is defined as union [null,
another record] and here "another record" is defined as Symbolic to schema
instead of actual definition of full record schema.
If reader schema doesn't have this field and writer populates this field and
try to decode writer's record using reader schema it's throwing boost::bad_ptr
exception. The exception is thrown from symbol class where it's trying to
extract extra info from Production for Symbollic.
I have a test case to reproduce this issue and I will send it in my next email.
Thanks
Venkat
> C++ resolvind decoder doesn't work when reader schema has more fields than
> writer schema
> ----------------------------------------------------------------------------------------
>
> Key: AVRO-1474
> URL: https://issues.apache.org/jira/browse/AVRO-1474
> Project: Avro
> Issue Type: Bug
> Components: c++
> Affects Versions: 1.7.6
> Reporter: Ramana Suvarapu
> Assignee: Thiruvalluvan M. G.
> Attachments: AVRO-1474.patch, bigrecord, bigrecord_r, reader, writer
>
>
> When reader schema has more number of fields than writer schema, C++
> implementation of resolving decoder is throwing exception "throwing exception
> "Don't know how to handle excess fields for reader.” with out checking
> whether fields are optional or fields have default values.
> Attached are reader and writer schemas. Record in reader schema has 2
> additional fields than writer schema. One field is required field but it has
> default value and another one is optional field (union of null and string).
> Since one has default value and another is optional both reader and writer
> schemas are supposed to be compatible.
>
> {"name": "defaultField", "type": "string", "default": "DEFAULT",
> "declared":"true"},
> {"name": "optionalField", "type": ["string", "null"],"declared":"true"},
>
> main()
> {
> avro::ValidSchema readerSchema = load("reader.json");
> avro::ValidSchema writerSchema = load("writer.json");
> avro::DecoderPtr d = avro::resolvingDecoder(writerSchema,
> readerSchema,avro::binaryDecoder());
> }
>
> But when I tried to create resolving decoder, I am getting "Don't know how to
> handle excess fields for reader.” But Java implementation works.
> Also field ordering is not working.
> The same issue is reported in AVRO-1360.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
