Christian Gumpert created NIFI-8119:
---------------------------------------
Summary: ExecuteSQL does not properly free database ressources
Key: NIFI-8119
URL: https://issues.apache.org/jira/browse/NIFI-8119
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Affects Versions: 1.11.2
Reporter: Christian Gumpert
We are using Nifi to ingest data from a Teradata database into our S3-based
data lake using a typical pattern of GenerateTableFetch and ExecuteSQL
processors. Our Teradata database tables contain columns of type CLOB (which
contains some JSON data).
We have installed the Teradata JDBC driver from the Teradata Tools and
Utilities package version 16.10.26.00 as described in this [Cloudera community
article|https://community.cloudera.com/t5/Community-Articles/Using-Teradata-JDBC-connector-in-NiFi/ta-p/246783].
After having configured a DBConnectionPool service with the Teradata connection
parameters we are able to execute our flow. The GenerateTableFetch processors
generates flowfiles containing SQL Queries which are then executed by the
ExecuteSQL processor.
After having processed the first 15 flowfiles the ExecuteSQL processor yields
the following error:
{noformat}
2020-12-17T12:53:17+01:00 L921000109090A nifi-app.log: 2020-12-17 12:53:11,578
ERROR [Timer-Driven Process Thread-2] o.a.nifi.processors.standard.ExecuteSQL
ExecuteSQL[id=afa23b0f-2e57-1fb6-d047-13646de03ebf] Unable to execute SQL
select query call devezv_replworkedec.get_edec_meldung(561, 562, 2, 0); for
StandardFlowFileRecord[uuid=ff7219a7-14e9-404e-a57a-28121653fed8,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1607610786368-4986, container=repo0,
section=890], offset=701888,
length=1077672],offset=32266,name=ff7219a7-14e9-404e-a57a-28121653fed8,size=58]
due to java.sql.SQLException: [Teradata Database] [TeraJDBC 16.10.00.07] [Error
3130] [SQLState HY000] GET_EDEC_MELDUNG:Response limit exceeded.; routing to
failure: java.sql.SQLException: [Teradata Database] [TeraJDBC 16.10.00.07]
[Error 3130] [SQLState HY000] GET_EDEC_MELDUNG:Response limit exceeded.
java.sql.SQLException: [Teradata Database] [TeraJDBC 16.10.00.07] [Error 3130]
[SQLState HY000] GET_EDEC_MELDUNG:Response limit exceeded.
at
com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDatabaseSQLException(ErrorFactory.java:309)
at
com.teradata.jdbc.jdbc_4.statemachine.ReceiveInitSubState.action(ReceiveInitSubState.java:103)
at
com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.subStateMachine(StatementReceiveState.java:311)
at
com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.action(StatementReceiveState.java:200)
at
com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:137)
at
com.teradata.jdbc.jdbc_4.statemachine.PreparedStatementController.run(PreparedStatementController.java:46)
at
com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:389)
at
com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:331)
at
com.teradata.jdbc.jdbc_4.TDPreparedStatement.doPrepExecute(TDPreparedStatement.java:177)
at
com.teradata.jdbc.jdbc_4.TDPreparedStatement.execute(TDPreparedStatement.java:2778)
at
org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
at
org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
at
org.apache.nifi.processors.standard.AbstractExecuteSQL.onTrigger(AbstractExecuteSQL.java:266)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){noformat}
After googling for a while it appears that the important part is
*java.sql.SQLException: [Teradata Database] [TeraJDBC 16.10.00.07] [Error 3130]
[SQLState HY000] GET_EDEC_MELDUNG:Response limit exceeded.*
The [Teradata documentation for "Working with
LOBs"|https://teradata-docs.s3.amazonaws.com/doc/connectivity/jdbc/reference/current/jdbcug_chapter_5.html#CHDGCHBB]
contains information on how to properly free resources to avoid this limit.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
