Hi Matt, Regarding sent information I have also additional questions:
- Are there any other tests I can perform to verify the source of the problem? - Do you have any idea what means/ what is the cause of the error ERROR [NiFi logging handler] org.apache.nifi.StdErr OK in NiFi bootstrap log and how to get rid of it? - Do you know what is the source of warning: WARN [Timer-Driven Process Thread-11] hive.metastore Unexpected increment of user count beyond one: 2 HCatClient: thread: 132 users=2 expired=false closed=false even in the case there is one PutHiveStreaming working as a single task on primary node only and max open connections property is set to 1? It seems like PutHiveStreaming is not closing previous connections properly... - Is there any method to set username/password that PutHiveStreaming is using to connect to Hive? Regards, Mike > -----Original Message----- > From: Michal Tomaszewski > Sent: Monday, February 19, 2018 4:10 PM > To: users@nifi.apache.org > Subject: RE: PutHiveStreaming NullPointerException error > > Matt, > > Regarding all mentioned points: > > > The post you mention has a comment in there suggesting perhaps the OS > > user running NiFi/Java might somehow be restricted from obtaining its > > own IP address, could that be an issue? Alternatively, if it cannot > > get a SessionState for some reason, I would presume there would be > > error(s) in the hive metastore log? > > a/ I think it is not possible. nifi is run under user "nifi", hadoop group, > without > any limits. NiFi was installed according to community documentation for > Ubuntu server. > To be 100% sure we tested starting nifi on "hive:hadoop" account - no > change We tested change hive server properties to run all queries on "hive" > user (hive.server2.enable.doAs) - no change > > b/ in logs: > > hivemetastore.log shows only warnings (and it seems to be connected to nifi > streaming): > 2018-02-19 00:45:06,492 WARN [pool-5-thread-224]: conf.HiveConf > (HiveConf.java:initialize(3093)) - HiveConf of name hive.log.dir does not > exist > 2018-02-19 00:45:06,492 WARN [pool-5-thread-224]: conf.HiveConf > (HiveConf.java:initialize(3093)) - HiveConf of name hive.log.file does not > exist > 2018-02-19 00:45:06,563 WARN [pool-5-thread-224]: conf.HiveConf > (HiveConf.java:initialize(3093)) - HiveConf of name hive.log.dir does not > exist > 2018-02-19 00:45:06,563 WARN [pool-5-thread-224]: conf.HiveConf > (HiveConf.java:initialize(3093)) - HiveConf of name hive.log.file does not > exist > 2018-02-19 00:45:08,399 WARN [pool-5-thread-163]: conf.HiveConf > (HiveConf.java:initialize(3093)) - HiveConf of name hive.log.dir does not > exist > 2018-02-19 00:45:08,399 WARN [pool-5-thread-163]: conf.HiveConf > (HiveConf.java:initialize(3093)) - HiveConf of name hive.log.file does not > exist > 2018-02-19 00:45:08,468 WARN [pool-5-thread-163]: conf.HiveConf > (HiveConf.java:initialize(3093)) - HiveConf of name hive.log.dir does not > exist > 2018-02-19 00:45:08,468 WARN [pool-5-thread-163]: conf.HiveConf > (HiveConf.java:initialize(3093)) - HiveConf of name hive.log.file does not > exist > 2018-02-19 00:45:08,919 WARN [pool-5-thread-324]: conf.HiveConf > (HiveConf.java:initialize(3093)) - HiveConf of name hive.log.dir does not > exist > 2018-02-19 00:45:08,919 WARN [pool-5-thread-324]: conf.HiveConf > (HiveConf.java:initialize(3093)) - HiveConf of name hive.log.file does not > exist > 2018-02-19 00:45:08,989 WARN [pool-5-thread-324]: conf.HiveConf > (HiveConf.java:initialize(3093)) - HiveConf of name hive.log.dir does not > exist > 2018-02-19 00:45:08,990 WARN [pool-5-thread-324]: conf.HiveConf > (HiveConf.java:initialize(3093)) - HiveConf of name hive.log.file does not > exist > > > hiveserver2.log has only such warnings: > > 2018-02-19 15:30:58,871 WARN [HiveServer2-Handler-Pool: Thread-68]: > conf.HiveConf (HiveConf.java:initialize(3093)) - HiveConf of name > hive.internal.ss.authz.settings.applied.marker does not exist > 2018-02-19 15:30:58,883 WARN [HiveServer2-Handler-Pool: Thread-68]: > metrics2.CodahaleMetrics (CodahaleMetrics.java:addGauge(299)) - A Gauge > with name [init_total_count_dbs] already exists. The old gauge will be > overwritten, but this is not recommended > 2018-02-19 15:30:58,884 WARN [HiveServer2-Handler-Pool: Thread-68]: > metrics2.CodahaleMetrics (CodahaleMetrics.java:addGauge(299)) - A Gauge > with name [init_total_count_tables] already exists. The old gauge will be > overwritten, but this is not recommended > 2018-02-19 15:30:58,884 WARN [HiveServer2-Handler-Pool: Thread-68]: > metrics2.CodahaleMetrics (CodahaleMetrics.java:addGauge(299)) - A Gauge > with name [init_total_count_partitions] already exists. The old gauge will be > overwritten, but this is not recommended > 2018-02-19 15:33:58,653 WARN [HiveServer2-Handler-Pool: Thread-68]: > conf.HiveConf (HiveConf.java:initialize(3093)) - HiveConf of name > hive.internal.ss.authz.settings.applied.marker does not exist > 2018-02-19 15:33:58,664 WARN [HiveServer2-Handler-Pool: Thread-68]: > metrics2.CodahaleMetrics (CodahaleMetrics.java:addGauge(299)) - A Gauge > with name [init_total_count_dbs] already exists. The old gauge will be > overwritten, but this is not recommended > 2018-02-19 15:33:58,664 WARN [HiveServer2-Handler-Pool: Thread-68]: > metrics2.CodahaleMetrics (CodahaleMetrics.java:addGauge(299)) - A Gauge > with name [init_total_count_tables] already exists. The old gauge will be > overwritten, but this is not recommended > 2018-02-19 15:33:58,664 WARN [HiveServer2-Handler-Pool: Thread-68]: > metrics2.CodahaleMetrics (CodahaleMetrics.java:addGauge(299)) - A Gauge > with name [init_total_count_partitions] already exists. The old gauge will be > overwritten, but this is not recommended > > NiFi bootstrap log shows a lot of such lines (couldn't find the cause of those > warnings): > 2018-02-19 15:37:31,103 ERROR [NiFi logging handler] org.apache.nifi.StdErr > OK > 2018-02-19 15:37:31,160 ERROR [NiFi logging handler] org.apache.nifi.StdErr > OK > 2018-02-19 15:38:32,167 ERROR [NiFi logging handler] org.apache.nifi.StdErr > OK > 2018-02-19 15:38:32,228 ERROR [NiFi logging handler] org.apache.nifi.StdErr > OK > 2018-02-19 15:39:33,248 ERROR [NiFi logging handler] org.apache.nifi.StdErr > OK > 2018-02-19 15:39:33,312 ERROR [NiFi logging handler] org.apache.nifi.StdErr > OK > 2018-02-19 15:40:35,201 ERROR [NiFi logging handler] org.apache.nifi.StdErr > OK > 2018-02-19 15:40:35,260 ERROR [NiFi logging handler] org.apache.nifi.StdErr > OK > 2018-02-19 15:41:36,825 ERROR [NiFi logging handler] org.apache.nifi.StdErr > OK > 2018-02-19 15:41:36,899 ERROR [NiFi logging handler] org.apache.nifi.StdErr > OK > > > > > > > I have seen this error with the Atlas hook, and evidently it also > > happens with the metastore hook. Are you overriding the metastore > > AuthorizationManager? If so the errors might be red herrings [1]. > > [1] https://issues.apache.org/jira/browse/HIVE-11190 > > [1] > We are not overriding anything - default installation with nearly no changes. > The only fact is that our Hive/Hadoop cluster is working in HA and is > unsecured - what (I think) is not very common... > > > Are you > > using multiple concurrent tasks for the PutHiveStreaming processor? > > It's possible you might be running into [2]. > > [2] https://issues.apache.org/jira/browse/HIVE-12409 > > [2] > In fact we are using HiveStreaming on each node (3 nodes nifi cluster) - so > they are some way concurrent. > I also thought it can be the cause, so I've tested it more thoughly: > There are 7 PutHiveStreaming components in the flow working on 3 node > cluster. All are streaming to different databases. All are set to 1 task per > node. > To verify concurrency problem I've made a test: > Reconfigured flow to run ONE PutHiiveStreaming component on NiFi Primary > Node only (the rest of PutHiveStreamings components were stopped during > the test), setting 1 task and 1 max open connections (in Hive streaming > properties). > In running such configuration the error appeared in logs as usual - so I > believe > it is not an issue caused by [2]. > > > Regards, > Mike > > > > On Mon, Feb 19, 2018 at 8:35 AM, Michal Tomaszewski > > <michal.tomaszew...@cca.pl> wrote: > > > Hi Matt, > > > Thank you for reply. > > > NiFi is compiled exactly like you mentioned. Otherwise we could not > > > use > > PutHiveQL processors. > > > > > > We compiled nifi 1.6 snapshot using exactly: > > > mvn -T C2.0 clean install -Phortonworks > > > -Dhive.version=1.2.1000.2.6.4.0-91 > > > -Dhive.hadoop.version=2.7.3.2.6.4.0-91 > > > -Dhadoop.version=2.7.3.2.6.4.0-91 -DskipTests –e > > > > > > > > > Regards, > > > Mike > > > > > >> -----Original Message----- > > >> From: Matt Burgess [mailto:mattyb...@apache.org] > > >> Sent: Monday, February 19, 2018 2:30 PM > > >> To: users@nifi.apache.org > > >> Subject: Re: PutHiveStreaming NullPointerException error > > >> > > >> Mike, > > >> > > >> Joe is correct, in order for Apache NiFi to interact with HDP Hive, > > >> the Hive client dependencies need to be swapped out, as HDP Hive > > >> 1.x components are not 100% compatible with Apache Hive 1.x > components. > > >> This can be done (in general) while building NiFi with Maven, by > > >> using a vendor profile and overriding the "hive.version" and > > "hive.hadoop.version" > > >> properties. There are currently 3 vendor profiles (hortonworks, > > >> cloudera, mapr), and there are examples of how to override the > > >> properties in the top- level pom.xml (in the vendor profile section). > > >> An example to build with HDP Hive components is: > > >> > > >> mvn clean install -Phortonworks -Dhive.version=1.2.1000.2.6.0.3-8 > > >> -Dhive.hadoop.version=2.7.3.2.6.0.3-8 > > >> > > >> You should be able to use this in the > > >> nifi-nar-bundles/nifi-hive-bundle directory to only build the Hive > > >> NAR(s), rather than a full rebuild. Then you can replace the Apache > > >> NiFi versions of the Hive NARs in your distribution with these > > >> vendor- > > specific ones. > > >> > > >> Regards, > > >> Matt > > >> > > >> On Mon, Feb 19, 2018 at 8:21 AM, Joe Witt <joe.w...@gmail.com> > wrote: > > >> > Mike - ah i see . Thanks for clarifying. > > >> > > > >> > I think the issue is that to interact properly with the Hive > > >> > version in HDP we have to swap out some dependencies at build time. > > >> > > > >> > Mattyb is definitely more knowledgeable so hopefully he can > > >> > comment > > >> soon. > > >> > > > >> > On Mon, Feb 19, 2018 at 8:06 AM, Michal Tomaszewski > > >> > <michal.tomaszew...@cca.pl> wrote: > > >> >> Hi Joe, > > >> >> Thanks for prompt answer. > > >> >> As I wrote - I'm currently using NiFi 1.6 snapshot compiled from > > >> >> latest > > >> community sources https://github.com/apache/nifi. > > >> >> I don't use Hortonworks' NiFi version. I verified Hortonworks' > > >> >> NiFi version > > >> only in order to be sure the problem is not connected with my > > >> compilation, as NiFi sends data to hive cluster on hortonworks. > > >> >> > > >> >> The problem is confirmed on community version of NiFi 1.6 > > >> >> (current > > >> snapshot), 1.5, 1.5 pre-RC and 1.4. > > >> >> > > >> >> Regards, > > >> >> Mike > > >> >> > > >> >> > > >> >> > > >> >> > > >> >>> -----Original Message----- > > >> >>> From: Joe Witt [mailto:joe.w...@gmail.com] > > >> >>> Sent: Monday, February 19, 2018 1:53 PM > > >> >>> To: users@nifi.apache.org > > >> >>> Subject: Re: PutHiveStreaming NullPointerException error > > >> >>> > > >> >>> Mike, > > >> >>> > > >> >>> Dev is fine but I think part of the difficultly for this group > > >> >>> here is you're referring to a vendor distribution the bundles > > >> >>> apache > > nifi. > > >> >>> The libraries involved are different than what we build/provide > > >> >>> directly. If you can recreate the same problem using apache > > >> >>> nifi as we provide it as a community it would be easier for > > >> >>> someone to help here and otherwise you might want to reach out > > >> >>> to the vendor for help > > >> with that configuration. > > >> >>> > > >> >>> Thanks > > >> >>> > > >> >>> On Mon, Feb 19, 2018 at 6:13 AM, Michal Tomaszewski > > >> >>> <michal.tomaszew...@cca.pl> wrote: > > >> >>> > Hi Team, > > >> >>> > > > >> >>> > Should I send this question to NiFi dev list instead of NiFi users? > > >> >>> > > > >> >>> > Regards, > > >> >>> > Mike > > >> >>> > > > >> >>> >>> 2018-02-15 17:42:29,901 ERROR [Timer-Driven Process > > >> >>> >>> Thread-11] > > >> >>> hive.log Got exception: java.lang.NullPointerException null > > >> >>> java.lang.NullPointerException: null > > >> >>> >>> at > > >> >>> org.apache.hadoop.hive.ql.security.authorization.plugin.Authori > > >> >>> za > > >> >>> tio > > >> >>> nMetaS > > >> >>> toreFilterHook.getFilteredObjects(AuthorizationMetaStoreFilterH > > >> >>> oo > > >> >>> k.j > > >> >>> ava:7 > > >> >>> 7) > > >> >>> >>> at > > >> >>> org.apache.hadoop.hive.ql.security.authorization.plugin.Authori > > >> >>> za > > >> >>> tio > > >> >>> nMetaS > > >> >>> toreFilterHook.filterDatabases(AuthorizationMetaStoreFilterHook > > >> >>> .j > > >> >>> ava > > >> >>> :54) > > >> >>> >>> at > > >> >>> > > >> > > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(Hi > > >> >>> ve > > >> >>> MetaStoreClient.java:1116) > > >> >>> >>> at > > >> >>> > > >> > > > org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaSto > > >> >>> r > > >> >>> eClient.isOpen(HiveClientCache.java:469) > > >> >>> >>> at > > >> >>> >>> sun.reflect.GeneratedMethodAccessor111.invoke(Unknown > > >> >>> Source) > > >> >>> >>> at > > >> >>> > > >> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce > > >> >>> s > > >> >>> sorImpl.java:43) > > >> >>> >>> at java.lang.reflect.Method.invoke(Method.java:498) > > >> >>> >>> at > > >> >>> > > >> > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(Ret > > >> r > > >> >>> yin > > >> >>> gMetaStoreClient.java:174) > > >> >>> >>> at com.sun.proxy.$Proxy341.isOpen(Unknown Source) > > >> >>> >>> at > > >> >>> > > >> > org.apache.hive.hcatalog.common.HiveClientCache.get(HiveClientCache. > > >> >>> jav > > >> >>> a:269) > > >> >>> >>> at > > >> >>> > > >> > org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(HCa > > >> t > > >> >>> Uti > > >> >>> l.java:558) > > >> >>> >>> at > > >> >>> org.apache.hive.hcatalog.streaming.AbstractRecordWriter.<init>( > > >> >>> Ab > > >> >>> str > > >> >>> actRe > > >> >>> cordWriter.java:94) > > >> >>> >>> at > > >> >>> org.apache.hive.hcatalog.streaming.StrictJsonWriter.<init>(Stri > > >> >>> ct > > >> >>> Jso > > >> >>> nWriter.j > > >> >>> ava:82) > > >> >>> >>> at > > >> >>> org.apache.hive.hcatalog.streaming.StrictJsonWriter.<init>(Stri > > >> >>> ct > > >> >>> Jso > > >> >>> nWriter.j > > >> >>> ava:60) > > >> >>> >>> at > > >> >>> org.apache.nifi.util.hive.HiveWriter.getRecordWriter(HiveWriter > > >> >>> .j > > >> >>> ava > > >> >>> :85) > > >> >>> >>> at > > org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:72) > > >> >>> >>> at > > >> >>> org.apache.nifi.util.hive.HiveUtils.makeHiveWriter(HiveUtils.ja > > >> >>> va > > >> >>> :46 > > >> >>> ) > > >> >>> >>> at > > >> >>> > > >> org.apache.nifi.processors.hive.PutHiveStreaming.makeHiveWriter(Put > > >> H > > >> >>> iveS > > >> >>> treaming.java:1036) > > >> >>> >>> at > > >> >>> org.apache.nifi.processors.hive.PutHiveStreaming.getOrCreateWri > > >> >>> te > > >> >>> r(P > > >> >>> utHiv > > >> >>> eStreaming.java:947) > > >> >>> >>> at > > >> >>> org.apache.nifi.processors.hive.PutHiveStreaming.lambda$null$8( > > >> >>> Pu > > >> >>> tHi > > >> >>> veStr > > >> >>> eaming.java:743) > > >> >>> >>> at > > >> >>> org.apache.nifi.processor.util.pattern.ExceptionHandler.execute > > >> >>> (E > > >> >>> xce > > >> >>> ptionH > > >> >>> andler.java:127) > > >> >>> >>> at > > >> >>> org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigg > > >> >>> er > > >> >>> $12 > > >> >>> (Put > > >> >>> HiveStreaming.java:740) > > >> >>> >>> at > > >> >>> org.apache.nifi.controller.repository.StandardProcessSession.re > > >> >>> ad > > >> >>> (St > > >> >>> andard > > >> >>> ProcessSession.java:2175) > > >> >>> >>> at > > >> >>> org.apache.nifi.controller.repository.StandardProcessSession.re > > >> >>> ad > > >> >>> (St > > >> >>> andard > > >> >>> ProcessSession.java:2145) > > >> >>> >>> at > > >> >>> org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutH > > >> >>> iv > > >> >>> eSt > > >> >>> reami > > >> >>> ng.java:694) > > >> >>> >>> at > > >> >>> org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigg > > >> >>> er > > >> >>> $4( > > >> >>> PutH > > >> >>> iveStreaming.java:572) > > >> >>> >>> at > > >> >>> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigg > > >> >>> er > > >> >>> (Pa > > >> >>> rtialFunct > > >> >>> ions.java:114) > > >> >>> >>> at > > >> >>> org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrig > > >> >>> ge > > >> >>> r(R > > >> >>> ollbackO > > >> >>> nFailure.java:184) > > >> >>> >>> at > > >> >>> org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutH > > >> >>> iv > > >> >>> eSt > > >> >>> reami > > >> >>> ng.java:572) > > >> >>> >>> at > > >> >>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(Stan > > >> >>> da > > >> >>> rdP > > >> >>> roces > > >> >>> sorNode.java:1122) > > >> >>> >>> at > > >> >>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.ca > > >> >>> ll > > >> >>> (Co > > >> >>> ntinually > > >> >>> RunProcessorTask.java:147) > > >> >>> >>> at > > >> >>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.ca > > >> >>> ll > > >> >>> (Co > > >> >>> ntinually > > >> >>> RunProcessorTask.java:47) > > >> >>> >>> at > > >> >>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgen > > >> >>> t$ > > >> >>> 1.r > > >> >>> un(Ti > > >> >>> merDrivenSchedulingAgent.java:128) > > >> >>> >>> at > > >> >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.j > > >> >>> av > > >> >>> a:5 > > >> >>> 11) > > >> >>> >>> at > > >> >>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308 > > >> >>> ) > > >> >>> >>> at > > >> >>> > > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas > > >> k > > >> >>> .ac > > >> >>> cess$301(ScheduledThreadPoolExecutor.java:180) > > >> >>> >>> at > > >> >>> > > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas > > >> k > > >> >>> .ru > > >> >>> n(ScheduledThreadPoolExecutor.java:294) > > >> >>> >>> at > > >> >>> > > >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto > > >> r > > >> >>> .jav > > >> >>> a:1149) > > >> >>> >>> at > > >> >>> > > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut > > >> o > > >> >>> r.ja > > >> >>> va:624) > > >> >>> >>> at java.lang.Thread.run(Thread.java:748) ________________________________________ Uwaga: Treść niniejszej wiadomości może być poufna i objęta zakazem jej ujawniania. Jeśli czytelnik tej wiadomości nie jest jej zamierzonym adresatem, pracownikiem lub pośrednikiem upoważnionym do jej przekazania adresatowi, informujemy że wszelkie rozprowadzanie, rozpowszechnianie lub powielanie niniejszej wiadomości jest zabronione. Jeśli otrzymałeś tę wiadomość omyłkowo, proszę bezzwłocznie odesłać ją nadawcy, a samą wiadomość usunąć z komputera. Dziękujemy. ________________________________ Note: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited.If you have received this communication in error, please notify the sender immediately by replying to the message and deleting it from your computer. Thank you. ________________________________
