[
https://issues.apache.org/jira/browse/NIFI-5612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627434#comment-16627434
]
Bryan Bende commented on NIFI-5612:
-----------------------------------
[~colindean] thanks for digging in here... From quickly looking at your
findings, it looks like we conditionally decide to use int or long in the union
based on the precision:
{code:java}
case INTEGER:
if (meta.isSigned(i) || (meta.getPrecision(i) > 0 && meta.getPrecision(i) <
MAX_DIGITS_IN_INT)) {
builder.name(columnName).type().unionOf().nullBuilder().endNull().and().intType().endUnion().noDefault();
} else {
builder.name(columnName).type().unionOf().nullBuilder().endNull().and().longType().endUnion().noDefault();
}
break;{code}
So when creating the schema it must be going into the first if statement based
on precision being > 0 and < 11, so the schema now has [null, int].
Then later the JDBC driver returns a Long value, which seems like could be an
issue with the driver, but since we are deciding between int and long when
creating the schema, maybe we should be doing the same when handling the value?
It looks like a Long would fall into this block of code...
{code:java}
else if (value instanceof Number || value instanceof Boolean) {
if (javaSqlType == BIGINT) {
int precision = meta.getPrecision(i);
if (precision < 0 || precision > MAX_DIGITS_IN_BIGINT) {
rec.put(i - 1, value.toString());
} else {
rec.put(i - 1, value);
}
} else {
rec.put(i - 1, value);
}
}{code}
[https://github.com/apache/nifi/blob/e959630c22c9a52ec717141f6cf9f018830a38bf/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java#L442]
The value would be a Long which extends Number, and since its not BIGINT it
would fall into the second else block where it just writes value to the record.
Maybe in that second else block we should be doing something similar to what
was done when creating the schema, and do something like...
{code:java}
if ((value instanceof Long) && precision < MAX_DIGITS_IN_INT) {
int intValue = ((Long)value).intValue();
rec.put(i-1, intValue);
} else {
rec.put(i-1, value);
}{code}
> org.apache.avro.UnresolvedUnionException: Not in union ["null","int"]: 0
> ------------------------------------------------------------------------
>
> Key: NIFI-5612
> URL: https://issues.apache.org/jira/browse/NIFI-5612
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.5.0, 1.6.0, 1.7.1
> Environment: Microsoft Windows, MySQL Enterprise 5.0.80
> Reporter: Colin Dean
> Priority: Major
> Labels: ExecuteSQL, avro, nifi
>
> I'm seeing this when I execute {{SELECT * FROM <tablename>}} on a few tables
> but not on dozens of others in the same database.
> {code:java}
> 2018-09-13 01:11:31,434 WARN [Timer-Driven Process Thread-8]
> o.a.n.controller.tasks.ConnectableTask Administratively Yielding
> ExecuteSQL[id=cf5c0996-eddf-3e05-25a3-c407c5edf990] due to uncaught
> Exception: org.apache.avro.file.DataFileWriter$AppendWriteException:
> org.apache.avro.UnresolvedUnionException: Not in union ["null","int"]: 0
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> org.apache.avro.UnresolvedUnionException: Not in union ["null","int"]: 0
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:308)
> at
> org.apache.nifi.processors.standard.util.JdbcCommon.convertToAvroStream(JdbcCommon.java:462)
> at
> org.apache.nifi.processors.standard.ExecuteSQL.lambda$onTrigger$1(ExecuteSQL.java:252)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2625)
> at
> org.apache.nifi.processors.standard.ExecuteSQL.onTrigger(ExecuteSQL.java:242)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.avro.UnresolvedUnionException: Not in union
> ["null","int"]: 0
> at
> org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:709)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:153)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:302)
> ... 15 common frames omitted
> {code}
> I don't know if I can share the database schema – still working with my team
> on that – but looking at it, I think it has something to do with the
> signedness of int(1) or tinyint(1) because those two are the only numerical
> types common to all of the table.
>
> *Edit 2018-09-24, so that my update doesn't get buried:*
> I am able to reproduce the exception using
> * Vagrant 2.1.1
> * Virtualbox 5.2.18 r124319
> * Ubuntu 18.04
> * MySQL 5.0.81 (as close as I can get to the 5.0.80 Enterprise Edition in
> use on the system where I observed this failure first)
> * MySQL Connector/J 5.1.46
> * NiFi 1.7.1
> With this table definition and data:
> {code:sql}
> create table fails (
> fails int(1) unsigned NOT NULL default '0'
> ) ENGINE=InnoDB AUTO_INCREMENT=16527 DEFAULT CHARSET=latin1;
> insert into fails values ();
> {code}
> and an ExecuteSQL processor set up to access that table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
