[
https://issues.apache.org/jira/browse/NIFI-6654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ron Chittaro updated NIFI-6654:
-------------------------------
Description:
We (and our customers, thus why I marked this a blocker as we cannot proceed
with 1.9.2 and in some cases product depolyment at all) have run into a stack
overflow exception with the 'UpdateRecord' processor when upgrading from Nifi
1.7 to 1.9.2. I did some digging and it happens when:
* A schema comparison operation is performed, AND
* the schema in question is self referential
I was a bit confused initially as I found that a unit test case for this
already existed and was passing when I checked out and built 1.9.2. The test
case is:
{quote}testHashCodeAndEqualsWithSelfReferencingSchema() from
TestSimpleRecordSchema.java located in:
nifi\nifi-commons\nifi-record\src\test\java\org\apache\nifi\serialization
{quote}
However, if you dig a bit into this test case it is passing because the two
schemas being compared contain the same memory references for the fields within
them, thus comparisons do not exercise completely the underlying equals()
operators of the objects contained in the schema. See bold below:
{quote}final SimpleRecordSchema schema = new
SimpleRecordSchema(SchemaIdentifier.EMPTY);
final List<RecordField> personFields = new ArrayList<>();
personFields.add(new RecordField("name",
RecordFieldType.STRING.getDataType()));
personFields.add(new RecordField("sibling",
RecordFieldType.RECORD.getRecordDataType(schema)));
*schema.setFields(personFields);*
schema.hashCode();
assertTrue(schema.equals(schema));
final SimpleRecordSchema secondSchema = new
SimpleRecordSchema(SchemaIdentifier.EMPTY);
*secondSchema.setFields(personFields);*
assertTrue(schema.equals(secondSchema));
assertTrue(secondSchema.equals(schema));
{quote}
To reproduce the stack overflow I wrote the following test cases which I
believe behaves more like it would in the 'UpdateProcessor' (or anywhere else a
schema comparison is happening) where the schema object and all objects within
are completely different memory references.
@Test
public void testSchemaCompareSelfRef() {
// Set up first schema
final SimpleRecordSchema personSchema = new
SimpleRecordSchema(SchemaIdentifier.EMPTY);
final List<RecordField> personFields = new ArrayList<>();
personFields.add(new RecordField("name",
RecordFieldType.STRING.getDataType()));
personFields.add(new RecordField("address",
RecordFieldType.STRING.getDataType()));
personFields.add(new RecordField("SIN", RecordFieldType.STRING.getDataType()));
personFields.add(new RecordField("PersonalData",
RecordFieldType.RECORD.getRecordDataType(personSchema)));
personSchema.setFields(personFields);
// Set up second schema. Must be completely separate set of objects otherwise
overloaded comparison
// operators (particularly that from the SimpleRecordSchema class) will not be
called.
SimpleRecordSchema secondSchema = new
SimpleRecordSchema(SchemaIdentifier.EMPTY);
personFields.clear();
personFields.add(new RecordField("name",
RecordFieldType.STRING.getDataType()));
personFields.add(new RecordField("address",
RecordFieldType.STRING.getDataType()));
personFields.add(new RecordField("SIN", RecordFieldType.STRING.getDataType()));
personFields.add(new RecordField("PersonalData",
RecordFieldType.RECORD.getRecordDataType(secondSchema)));
secondSchema.setFields(personFields);
assertTrue(personSchema.equals(secondSchema));
}
was:
We (and our customers, thus why I marked this a blocker as we cannot proceed
with 1.9.2 and in some cases product depolyment at all) have run into a stack
overflow exception with the 'UpdateRecord' processor when upgrading from Nifi
1.7 to 1.9.2. I did some digging and it happens when:
* A schema comparison operation is performed, AND
* the schema in question is self referential
I was a bit confused initially as I found that a unit test case for this
already existed and was passing when I checked out and built 1.9.2. The test
case is:
{quote}testHashCodeAndEqualsWithSelfReferencingSchema() from
TestSimpleRecordSchema.java located in:
nifi\nifi-commons\nifi-record\src\test\java\org\apache\nifi\serialization
{quote}
However, if you dig a bit into this test case it is passing because the two
schemas being compared contain the same memory references for the fields within
them, thus comparisons do not exercise completely the underlying equals()
operators of the objects contained in the schema. See bold below:
{quote}final SimpleRecordSchema schema = new
SimpleRecordSchema(SchemaIdentifier.EMPTY);
final List<RecordField> personFields = new ArrayList<>();
personFields.add(new RecordField("name",
RecordFieldType.STRING.getDataType()));
personFields.add(new RecordField("sibling",
RecordFieldType.RECORD.getRecordDataType(schema)));
*schema.setFields(personFields);*
schema.hashCode();
assertTrue(schema.equals(schema));
final SimpleRecordSchema secondSchema = new
SimpleRecordSchema(SchemaIdentifier.EMPTY);
*secondSchema.setFields(personFields);*
assertTrue(schema.equals(secondSchema));
assertTrue(secondSchema.equals(schema));
{quote}
To reproduce the stack overflow I wrote the following test cases which I
believe behaves more like it would in the 'UpdateProcessor' (or anywhere else a
schema comparison is happening) where the schema object and all objects within
are completely different memory references.
{quote}@Test
public void testSchemaCompareSelfRef() {
// Set up first schema
final SimpleRecordSchema personSchema = new
SimpleRecordSchema(SchemaIdentifier.EMPTY);
final List<RecordField> personFields = new ArrayList<>();
personFields.add(new RecordField("name",
RecordFieldType.STRING.getDataType()));
personFields.add(new RecordField("address",
RecordFieldType.STRING.getDataType()));
personFields.add(new RecordField("SIN", RecordFieldType.STRING.getDataType()));
personFields.add(new RecordField("PersonalData",
RecordFieldType.RECORD.getRecordDataType(personSchema)));
personSchema.setFields(personFields);
// Set up second schema. Must be completely separate set of objects otherwise
overloaded comparison
// operators (particularly that from the SimpleRecordSchema class) will not be
called.
SimpleRecordSchema secondSchema = new
SimpleRecordSchema(SchemaIdentifier.EMPTY);
personFields.clear();
personFields.add(new RecordField("name",
RecordFieldType.STRING.getDataType()));
personFields.add(new RecordField("address",
RecordFieldType.STRING.getDataType()));
personFields.add(new RecordField("SIN", RecordFieldType.STRING.getDataType()));
personFields.add(new RecordField("PersonalData",
RecordFieldType.RECORD.getRecordDataType(secondSchema)));
secondSchema.setFields(personFields);
assertTrue(personSchema.equals(secondSchema));
}
{quote}
> Stack overflow with self referencing avro schema
> ------------------------------------------------
>
> Key: NIFI-6654
> URL: https://issues.apache.org/jira/browse/NIFI-6654
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.9.2
> Reporter: Ron Chittaro
> Priority: Major
>
> We (and our customers, thus why I marked this a blocker as we cannot proceed
> with 1.9.2 and in some cases product depolyment at all) have run into a stack
> overflow exception with the 'UpdateRecord' processor when upgrading from Nifi
> 1.7 to 1.9.2. I did some digging and it happens when:
> * A schema comparison operation is performed, AND
> * the schema in question is self referential
> I was a bit confused initially as I found that a unit test case for this
> already existed and was passing when I checked out and built 1.9.2. The test
> case is:
> {quote}testHashCodeAndEqualsWithSelfReferencingSchema() from
> TestSimpleRecordSchema.java located in:
> nifi\nifi-commons\nifi-record\src\test\java\org\apache\nifi\serialization
> {quote}
> However, if you dig a bit into this test case it is passing because the two
> schemas being compared contain the same memory references for the fields
> within them, thus comparisons do not exercise completely the underlying
> equals() operators of the objects contained in the schema. See bold below:
> {quote}final SimpleRecordSchema schema = new
> SimpleRecordSchema(SchemaIdentifier.EMPTY);
> final List<RecordField> personFields = new ArrayList<>();
> personFields.add(new RecordField("name",
> RecordFieldType.STRING.getDataType()));
> personFields.add(new RecordField("sibling",
> RecordFieldType.RECORD.getRecordDataType(schema)));
> *schema.setFields(personFields);*
> schema.hashCode();
> assertTrue(schema.equals(schema));
> final SimpleRecordSchema secondSchema = new
> SimpleRecordSchema(SchemaIdentifier.EMPTY);
> *secondSchema.setFields(personFields);*
> assertTrue(schema.equals(secondSchema));
> assertTrue(secondSchema.equals(schema));
> {quote}
>
> To reproduce the stack overflow I wrote the following test cases which I
> believe behaves more like it would in the 'UpdateProcessor' (or anywhere else
> a schema comparison is happening) where the schema object and all objects
> within are completely different memory references.
>
> @Test
> public void testSchemaCompareSelfRef() {
> // Set up first schema
> final SimpleRecordSchema personSchema = new
> SimpleRecordSchema(SchemaIdentifier.EMPTY);
> final List<RecordField> personFields = new ArrayList<>();
> personFields.add(new RecordField("name",
> RecordFieldType.STRING.getDataType()));
> personFields.add(new RecordField("address",
> RecordFieldType.STRING.getDataType()));
> personFields.add(new RecordField("SIN",
> RecordFieldType.STRING.getDataType()));
> personFields.add(new RecordField("PersonalData",
> RecordFieldType.RECORD.getRecordDataType(personSchema)));
> personSchema.setFields(personFields);
> // Set up second schema. Must be completely separate set of objects otherwise
> overloaded comparison
> // operators (particularly that from the SimpleRecordSchema class) will not
> be called.
> SimpleRecordSchema secondSchema = new
> SimpleRecordSchema(SchemaIdentifier.EMPTY);
> personFields.clear();
> personFields.add(new RecordField("name",
> RecordFieldType.STRING.getDataType()));
> personFields.add(new RecordField("address",
> RecordFieldType.STRING.getDataType()));
> personFields.add(new RecordField("SIN",
> RecordFieldType.STRING.getDataType()));
> personFields.add(new RecordField("PersonalData",
> RecordFieldType.RECORD.getRecordDataType(secondSchema)));
> secondSchema.setFields(personFields);
> assertTrue(personSchema.equals(secondSchema));
> }
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
