Koji Kawamura created NIFI-4844:
-----------------------------------
Summary: AvroRecordSetWriter should be able to convert a double
having less scale than intended target Avro schema instead of throwing an
AvroTypeException
Key: NIFI-4844
URL: https://issues.apache.org/jira/browse/NIFI-4844
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Affects Versions: 1.3.0
Reporter: Koji Kawamura
Assignee: Koji Kawamura
Current AvroTypeUtil conversion logic can throw AvroTypeException when it maps
double values into Avro decimal logical type fields if the double value has
less scale than the one defined at the target Avro decimal field schema.
For example, with following schema:
{code}
{
"type": "record",
"name": "logicalDecimalTest",
"fields": [
{"name": "id", "type": "int"},
{"name": "name", "type": "string"},
{
"name": "price",
"type": {
"type": "bytes",
"logicalType": "decimal",
"precision": 18,
"scale": 8
}}]}
{code}
And following CSV records:
{code}
id|name|price
1|one|1.23
2|two|2.34
{code}
Would produce this Exception:
{code}
2018-02-06 09:57:27,461 ERROR [Timer-Driven Process Thread-7]
o.a.n.processors.standard.ConvertRecord
ConvertRecord[id=6897bc30-0161-1000-a8e7-9ce0ce8eb9ae] Failed to process
StandardFlowFileRecord[uuid=a97366a0-79bb-42ff-9023-c5d62ecfdbc5,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1517878123416-2, container=default,
section=2], offset=5, length=48],offset=0,name=220105646548465,size=48]; will
route to failure: org.apache.avro.AvroTypeException: Cannot encode decimal with
scale 17 as scale 8
org.apache.avro.AvroTypeException: Cannot encode decimal with scale 17 as scale
8
at
org.apache.avro.Conversions$DecimalConversion.toBytes(Conversions.java:86)
at
org.apache.nifi.avro.AvroTypeUtil.convertToAvroObject(AvroTypeUtil.java:546)
at
org.apache.nifi.avro.AvroTypeUtil.createAvroRecord(AvroTypeUtil.java:457)
at
org.apache.nifi.avro.WriteAvroResultWithExternalSchema.writeRecord(WriteAvroResultWithExternalSchema.java:76)
at
org.apache.nifi.serialization.AbstractRecordSetWriter.write(AbstractRecordSetWriter.java:59)
at
org.apache.nifi.processors.standard.AbstractRecordProcessor$1.process(AbstractRecordProcessor.java:122)
at
org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2827)
at
org.apache.nifi.processors.standard.AbstractRecordProcessor.onTrigger(AbstractRecordProcessor.java:109)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
The same issue is reported in the Avro project,
[AVRO-1864|https://issues.apache.org/jira/browse/AVRO-1864]. The recommended
approach is to adjust the scale at NiFi side. Actually, for BigDecimal input
values, NiFi already does this, but not with double values. AvroTypeUtil should
do the same scale adjustment for double values, too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
