Ryan Blue - any chance you are available to weigh in on this thread? Thanks Joe
On Tue, Dec 22, 2015 at 8:13 AM, Joe Percivall <[email protected]> wrote: > Are you saying the Hive dataset is not yet supported in Kite? > > Joe > - - - - - - > Joseph Percivall > linkedin.com/in/Percivall > e: [email protected] > > > > On Tuesday, December 22, 2015 7:41 AM, panfei <[email protected]> wrote: > > > hive dataset is not supported yet. > > > 2015年12月22日星期二，Chandu Koripella <[email protected]> 写道： >> Hi Joe, >> >> >> >> I am still facing this issue. any help is appreciated. >> >> I am not sure why dataset:hive:namespace/dataset_name didn’t get validated >> in NIFI. Same does work when I use kite-dataset CLI. >> >> >> >> Thanks, >> >> Chandra >> >> >> >> >> >> From: Joe Percivall [mailto:[email protected]] >> Sent: Monday, December 21, 2015 6:38 AM >> To: [email protected] >> Subject: Re: Nifi Kite-dataset URI issue >> >> >> >> Hello Chandra, >> >> >> >> Are you still running into issues? >> >> >> >> If so, can someone with more experience with Kite and Hadoop chime in? >> >> >> >> Joe >> >> - - - - - - >> >> Joseph Percivall >> >> linkedin.com/in/Percivall >> >> e: [email protected] >> >> >> >> >> >> On Wednesday, December 16, 2015 2:08 PM, Chandu Koripella >> <[email protected]> wrote: >> >> >> >> HI, >> >> >> >> I am running into another issue. I really appreciate your help. My apology >> if I am asking very basic question. I am first time user for both Nifi, Kite >> dataset and Hadoop >> >> >> >> My Nifi is service is running as HDFS user. >> >> >> >> · Create a dataset as HDFS and I could able to import just fine as >> HDFS >> >> >> >> </mail/u/1/s/?view=att&th=151c5ef250d3def9&attid=0.1&disp=emb&zw&atsh=1> >> >> >> >> · Permissions: >> >> >> >> </mail/u/1/s/?view=att&th=151c5ef250d3def9&attid=0.2&disp=emb&zw&atsh=1> >> >> >> >> >> >> · URI Validates just fine in NIfi >> >> </mail/u/1/s/?view=att&th=151c5ef250d3def9&attid=0.3&disp=emb&zw&atsh=1> >> >> >> >> </mail/u/1/s/?view=att&th=151c5ef250d3def9&attid=0.4&disp=emb&zw&atsh=1> >> >> >> >> · Hadoop Configuration files: >> /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/hdfs-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/core-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive/conf.dist/hive-site.xml >> >> >> >> Any help is appreciated. >> >> >> >> Thanks, >> >> Chandra >> >> >> >> From: Chandu Koripella [mailto:[email protected]] >> Sent: Monday, December 14, 2015 11:49 AM >> To: [email protected] >> Subject: RE: Nifi Kite-dataset URI issue >> >> >> >> You are right. It works just fine by changing the keyword to hdfs. >> Kit-dataset accepting the both hdfs and hive >> >> >> >> >> >> </mail/u/1/s/?view=att&th=151c5ef250d3def9&attid=0.5&disp=emb&zw&atsh=1> >> >> >> >> Thanks, >> >> Chandu >> >> >> >> >> >> From: Joe Witt [mailto:[email protected]] >> Sent: Monday, December 14, 2015 11:37 AM >> To: [email protected] >> Subject: Re: Nifi Kite-dataset URI issue >> >> >> >> the property to focus on is 'Target Dataset URI' >> >> >> >> I'm no expert on Kite but a quick check of their docs makes me think the >> current URI being used really is not valid. >> >> >> >> Try dataset:hdfs:/default/ctest1 >> >> >> >> >> >> On Mon, Dec 14, 2015 at 2:33 PM, Chandu Koripella <[email protected]> >> wrote: >> >> HI Alan, >> >> >> >> Thanks for responding. Hive dataset has 777 permissions. I am not sure why >> it still unable to validate it. I can load the data just fine if execute >> kite-dataset command on the server. can you please take a look? >> >> >> >> </mail/u/1/s/?view=att&th=151c5ef250d3def9&attid=0.6&disp=emb&zw&atsh=1> >> >> >> >> >> /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/hdfs-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/core-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive/conf.dist/hive-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive/conf.dist/hive-exec-log4j.properties,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive/conf.dist/hive-exec-log4j.properties,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/etc/hive/conf.dist/hive-log4j.properties >> >> >> >> Here is last few lines from nifi logs. >> >> >> >> 2015-12-14 11:07:26,671 INFO [pool-18-thread-1] >> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile >> Repository with 0 records in 27 milliseconds >> >> 2015-12-14 11:09:16,643 INFO [Flow Service Tasks Thread-2] >> o.a.nifi.controller.StandardFlowService Saved flow controller >> org.apache.nifi.controller.FlowController@66f2383d // Another save pending = >> false >> >> 2015-12-14 11:09:26,672 INFO [pool-18-thread-1] >> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile >> Repository >> >> 2015-12-14 11:09:26,700 INFO [pool-18-thread-1] >> org.wali.MinimalLockingWriteAheadLog >> org.wali.MinimalLockingWriteAheadLog@75a9cc24 checkpointed with 0 Records >> and 0 Swap Files in 28 milliseconds (Stop-the-world time = 12 milliseconds, >> Clear Edit Logs time = 13 millis), max Transaction ID -1 >> >> 2015-12-14 11:09:26,700 INFO [pool-18-thread-1] >> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile >> Repository with 0 records in 28 milliseconds >> >> 2015-12-14 11:09:42,649 INFO [Flow Service Tasks Thread-2] >> o.a.nifi.controller.StandardFlowService Saved flow controller >> org.apache.nifi.controller.FlowController@66f2383d // Another save pending = >> false >> >> 2015-12-14 11:10:33,658 INFO [Flow Service Tasks Thread-2] >> o.a.nifi.controller.StandardFlowService Saved flow controller >> org.apache.nifi.controller.FlowController@66f2383d // Another save pending = >> false >> >> 2015-12-14 11:11:26,700 INFO [pool-18-thread-1] >> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile >> Repository >> >> 2015-12-14 11:11:26,740 INFO [pool-18-thread-1] >> org.wali.MinimalLockingWriteAheadLog >> org.wali.MinimalLockingWriteAheadLog@75a9cc24 checkpointed with 0 Records >> and 0 Swap Files in 39 milliseconds (Stop-the-world time = 12 milliseconds, >> Clear Edit Logs time = 24 millis), max Transaction ID -1 >> >> 2015-12-14 11:11:26,740 INFO [pool-18-thread-1] >> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile >> Repository with 0 records in 39 milliseconds >> >> 2015-12-14 11:13:26,740 INFO [pool-18-thread-1] >> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile >> Repository >> >> 2015-12-14 11:13:26,769 INFO [pool-18-thread-1] >> org.wali.MinimalLockingWriteAheadLog >> org.wali.MinimalLockingWriteAheadLog@75a9cc24 checkpointed with 0 Records >> and 0 Swap Files in 28 milliseconds (Stop-the-world time = 12 milliseconds, >> Clear Edit Logs time = 13 millis), max Transaction ID -1 >> >> 2015-12-14 11:13:26,769 INFO [pool-18-thread-1] >> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile >> Repository with 0 records in 28 milliseconds >> >> 2015-12-14 11:15:21,314 INFO [Flow Service Tasks Thread-1] >> o.a.nifi.controller.StandardFlowService Saved flow controller >> org.apache.nifi.controller.FlowController@66f2383d // Another save pending = >> false >> >> 2015-12-14 11:15:26,769 INFO [pool-18-thread-1] >> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile >> Repository >> >> 2015-12-14 11:15:26,796 INFO [pool-18-thread-1] >> org.wali.MinimalLockingWriteAheadLog >> org.wali.MinimalLockingWriteAheadLog@75a9cc24 checkpointed with 0 Records >> and 0 Swap Files in 27 milliseconds (Stop-the-world time = 11 milliseconds, >> Clear Edit Logs time = 12 millis), max Transaction ID -1 >> >> 2015-12-14 11:15:26,796 INFO [pool-18-thread-1] >> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile >> Repository with 0 records in 27 milliseconds >> >> 2015-12-14 11:15:47,820 INFO [Flow Service Tasks Thread-1] >> o.a.nifi.controller.StandardFlowService Saved flow controller >> org.apache.nifi.controller.FlowController@66f2383d // Another save pending = >> false >> >> 2015-12-14 11:16:28,328 INFO [Flow Service Tasks Thread-1] >> o.a.nifi.controller.StandardFlowService Saved flow controller >> org.apache.nifi.controller.FlowController@66f2383d // Another save pending = >> false >> >> 2015-12-14 11:17:26,797 INFO [pool-18-thread-1] >> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile >> Repository >> >> 2015-12-14 11:17:26,824 INFO [pool-18-thread-1] >> org.wali.MinimalLockingWriteAheadLog >> org.wali.MinimalLockingWriteAheadLog@75a9cc24 checkpointed with 0 Records >> and 0 Swap Files in 27 milliseconds (Stop-the-world time = 10 milliseconds, >> Clear Edit Logs time = 13 millis), max Transaction ID -1 >> >> 2015-12-14 11:17:26,825 INFO [pool-18-thread-1] >> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile >> Repository with 0 records in 27 milliseconds >> >> 2015-12-14 11:19:26,825 INFO [pool-18-thread-1] >> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile >> Repository >> >> 2015-12-14 11:19:26,900 INFO [pool-18-thread-1] >> org.wali.MinimalLockingWriteAheadLog >> org.wali.MinimalLockingWriteAheadLog@75a9cc24 checkpointed with 0 Records >> and 0 Swap Files in 74 milliseconds (Stop-the-world time = 59 milliseconds, >> Clear Edit Logs time = 13 millis), max Transaction ID -1 >> >> 2015-12-14 11:19:26,900 INFO [pool-18-thread-1] >> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile >> Repository with 0 records in 75 milliseconds >> >> 2015-12-14 11:21:26,900 INFO [pool-18-thread-1] >> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile >> Repository >> >> 2015-12-14 11:21:26,928 INFO [pool-18-thread-1] >> org.wali.MinimalLockingWriteAheadLog >> org.wali.MinimalLockingWriteAheadLog@75a9cc24 checkpointed with 0 Records >> and 0 Swap Files in 27 milliseconds (Stop-the-world time = 12 milliseconds, >> Clear Edit Logs time = 13 millis), max Transaction ID -1 >> >> 2015-12-14 11:21:26,928 INFO [pool-18-thread-1] >> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile >> Repository with 0 record >> >> >> >> >> >> >> >> >> >> From: Alan Jackoway [mailto:[email protected]] >> Sent: Monday, December 14, 2015 11:05 AM >> >> To: [email protected] >> Subject: Re: Nifi Kite-dataset URI issue >> >> >> >> Try adding the path to hive-site.xml. >> >> >> >> Did it give you any error message beyond just "cannot validate url"? Is >> there anything interesting in the nifi-app.log? >> >> >> >> On Mon, Dec 14, 2015 at 12:54 PM, Chandu Koripella >> <[email protected]> wrote: >> >> It would be great help if someone can take a look at this. >> >> >> >> Thanks, >> >> Chandu >> >> >> >> From: Chandu Koripella >> Sent: Friday, December 11, 2015 11:44 AM >> To: '[email protected]' >> Subject: RE: Nifi Kite-dataset URI issue >> >> >> >> Hi, >> >> >> >> This is final step in implementing first NIfi job. I really appreciate any >> quick tips to resolve this this. >> >> >> >> Thanks, >> >> Chandu >> >> >> >> >> >> From: Chandu Koripella >> Sent: Friday, December 11, 2015 7:03 AM >> To: [email protected] >> Subject: RE: Nifi Kite-dataset URI issue >> >> >> >> Hi Allen, >> >> >> >> Thanks for helping me with this issue. Here is the full value Hadoop >> config property. >> >> >> >> >> /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/hdfs-site.xml,/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/core-site.xml >> >> >> >> </mail/u/1/s/?view=att&th=151c5ef250d3def9&attid=0.7&disp=emb&zw&atsh=1> >> >> >> >> This works just fine with putHDFS processor >> >> >> >> </mail/u/1/s/?view=att&th=151c5ef250d3def9&attid=0.8&disp=emb&zw&atsh=1> >> >> >> >> From: Alan Jackoway [mailto:[email protected]] >> Sent: Friday, December 11, 2015 5:42 AM >> To: [email protected] >> Subject: Re: Nifi Kite-dataset URI issue >> >> >> >> Can you give the full value of that hadoop configuration files property? >> >> On Dec 10, 2015 11:23 PM, "Chandu Koripella" <[email protected]> >> wrote: >> >> Hi, >> >> >> >> I am trying to load data into a hive external table. Nifi is unable to >> validate the URI. It is working just fine when I use kite-dataset. >> >> Can you please advice if am missing any parameters here? >> >> >> >> </mail/u/1/s/?view=att&th=151c5ef250d3def9&attid=0.9&disp=emb&zw&atsh=1> >> >> >> >> </mail/u/1/s/?view=att&th=151c5ef250d3def9&attid=0.10&disp=emb&zw&atsh=1> >> >> >> >> Thanks, >> >> Chandu >> >> >> >> >> >> > > -- > 不学习，不知道 > > >
