There are several complications using Snappy but from what I can tell the SelectHiveQL Processor isn't trying to use it and instead the Hive JDBC Driver is trying to use it. Are you running a specific vendors distribution of NiFi? The reason I'm asking is when I go look at the source for the Hive JDBC Driver Class org.apache.hive.jdbc.Utils.verifySuccess where the error is being raised none of the line numbers are matching up with Apache. It may be that the error is actually coming from Hive not NiFi.
Thanks Shawn On 5/14/19, 6:37 PM, "David Yu" <[email protected]> wrote: Hi all, I need help finding the root cause on this issue. I occasionally get the following exception from SelectHiveQL and PutHiveQL processor. Sometimes the queries executed without any issue but next minute this exception could happen. Once it happened, by restarting only the NiFi does not resolve the issue. I have to reboot the Linux VM of the node and remove the flow.xml.gz file then restart nifi node. I am using NIFI version 1.7.1 in cluster mode with total 8 nodes Here is the stacktrace 2019-05-14 20:37:30,937 ERROR [Timer-Driven Process Thread-6] o.a.nifi.processors.hive.SelectHiveQL SelectHiveQL[id=365c2597-9483-302c-90d0-c7d2d7ce471d] Unable to execute HiveQL select query select max(version) as max_partition, min(version) as min_partition from Table_A for StandardFlowFileRecord[uuid=9253b1ab-b4a2-4547-b695-4dbf5f964ca2,claim=,offset=0,name=done,size=0] due to org.apache.nifi.processor.exception.ProcessException: Error during database query or conversion of records.; routing to failure: org.apache.nifi.processor.exception.ProcessException: Error during database query or conversion of records. org.apache.nifi.processor.exception.ProcessException: Error during database query or conversion of records. at org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$2(SelectHiveQL.java:407) at org.apache.nifi.processors.hive.SelectHiveQL$$Lambda$627/1348460623.process(Unknown Source) at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2625) at org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:395) at org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$0(SelectHiveQL.java:285) at org.apache.nifi.processors.hive.SelectHiveQL$$Lambda$622/285174376.execute(Unknown Source) at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114) at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:106) at org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:285) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.sql.SQLException: java.io.IOException: java.lang.RuntimeException: native snappy library not available at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:121) at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:109) at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:330) at org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:207) at org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:207) at org.apache.nifi.util.hive.HiveJdbcCommon.convertToCsvStream(HiveJdbcCommon.java:355) at org.apache.nifi.util.hive.HiveJdbcCommon.convertToCsvStream(HiveJdbcCommon.java:322) at org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$2(SelectHiveQL.java:401) ... 18 common frames omitted 2019-05-14 20:37:30,938 ERROR [Timer-Driven Process Thread-6] o.a.nifi.processors.hive.SelectHiveQL SelectHiveQL[id=365c2597-9483-302c-90d0-c7d2d7ce471d] org.apache.nifi.processors.hive.SelectHiveQL$$Lambda$622/285174376@54eb300d failed to process due to org.apache.nifi.processor.exception.FlowFileHandlingException: StandardFlowFileRecord[uuid=e2688819-edc9-406a-b5da-050ac072cd43,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1557687270864-64540, container=default, section=28], offset=81777, length=280],offset=0,name=done,size=280] transfer relationship not specified; rolling back session: org.apache.nifi.processor.exception.FlowFileHandlingException: StandardFlowFileRecord[uuid=e2688819-edc9-406a-b5da-050ac072cd43,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1557687270864-64540, container=default, section=28], offset=81777, length=280],offset=0,name=done,size=280] transfer relationship not specified org.apache.nifi.processor.exception.FlowFileHandlingException: StandardFlowFileRecord[uuid=e2688819-edc9-406a-b5da-050ac072cd43,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1557687270864-64540, container=default, section=28], offset=81777, length=280],offset=0,name=done,size=280] transfer relationship not specified at org.apache.nifi.controller.repository.StandardProcessSession.checkpoint(StandardProcessSession.java:264) at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:335) at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:115) at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:106) at org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:285) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Thanks David
