Re: Record path API chooses CHOICE[STRING, RECORD] when the field is missing

[Mark Payne]

Mike,

What Record Reader is being used here? The problem appears to be due to the 
Record Reader itself assigning that as the field type.

I created a dummy unit test to verify the RecordPath stuff is correct:


@Test
public void testFromEmail() {
    final List<RecordField> fields = new ArrayList<>();
    fields.add(new RecordField("firstName", 
RecordFieldType.STRING.getDataType()));
    fields.add(new RecordField("lastName", 
RecordFieldType.STRING.getDataType()));
    fields.add(new RecordField("creationDateTime", 
RecordFieldType.TIMESTAMP.getDataType(), true));
    final RecordSchema schema = new SimpleRecordSchema(fields);

    final Map<String, Object> values = new HashMap<>();
    values.put("firstName", "John");
    values.put("lastName", "Doe");
    values.put("creationDateTime", new Timestamp(System.currentTimeMillis()));
    final Record record = new MapRecord(schema, values);

    final Optional<FieldValue> optionalFieldValue = 
RecordPath.compile("/creationDateTime").evaluate(record).getSelectedFields().findFirst();
    final FieldValue fieldValue = optionalFieldValue.get();
    System.out.println(fieldValue.getField());
}

Which prints out the correct field type:

RecordField[name=creationDateTime, dataType=TIMESTAMP:yyyy-MM-dd HH:mm:ss, 
nullable=true]

So I presume the Record Reader may not be properly applying the schema to the 
Record that it returns.

Thanks
-Mark

On Jul 28, 2019, at 10:19 AM, Mike Thomsen 
<mikerthom...@gmail.com<mailto:mikerthom...@gmail.com>> wrote:

I have a simple avro schema in a test case that looks like this:

{
   "type": "record",
   "name": "PersonRecord",
   "fields": [
       { "name": "firstName", "type": "string" },
       { "name": "lastName", "type": "string" },
       { "name": "creationDateTime", "type": [ "null", "type": "long",
"logicalType": "timestamp-millis" }]
   ]
}

Then I try something like this...

RecordPath path = recordPathCache.getCompiled("/creationDateTime");
RecordPathResult rp = path.evaluate(targetRecord);
Optional<FieldValue> nodeField = rp.getSelectedFields().findFirst();

if (!nodeField.isPresent()) {
   throw new ProcessException("...");
}

FieldValue fieldValue = nodeField.get();
//fieldValue.getField() is a Choice of String, Record

Is there a way to get the correct field type here? I assume that
Choice[String, Record] default here was done to facilitate schema inference.

Thanks,

Mike