On Wed, Jul 27, 2016 at 5:30 PM Rajat Khandelwal <[email protected]> wrote:
> Still unsure whether the mail got sent. Unable to see it at > https://mail-archives.apache.org/mod_mbox/lens-user/201607.mbox/date. > Resending with attachments removed. > > > On Wed, Jul 27, 2016 at 5:01 PM Rajat Khandelwal <[email protected]> > wrote: > >> Apparently, Gmail doesn't allow tar.gz attachments. Attaching and sending >> again from my apache account. >> >> >> On Wed, Jul 27, 2016 at 4:56 PM Rajat Khandelwal <[email protected]> >> wrote: >> >>> Hi >>> >>> First off, I'd like to mention that the dependency on forked hive 0.13 >>> has been changed to apache hive in the development version of Lens. Lens >= >>> 2.6 will work with Apache Hive 2.1+. Release 2.6 is planned to go in a >>> couple of weeks. The steps of setup will be mostly same, though. With that, >>> let me focus on your questions. >>> >>> It looks like you have tried a bunch of things without success. It's >>> difficult for me to understand which step you might have missed or not done >>> correctly. So I'd like to start the setup from the beginning. >>> >>> >>> 1. Existing cluster deployment. >>> 2. Pick a machine for installing hive metastore, hive server and >>> lens server. These machines can be different, and we at InMobi are using >>> a >>> deployment on different machines, but let's go ahead with a single >>> machine >>> for simplicity. >>> 3. Find Hadoop installation location and export it as HADOOP_HOME. >>> This will be the path on the chosen machine's local file system, not >>> HDFS. >>> 4. Install hive, export the install location as HIVE_HOME. >>> 5. Create a configuration directory for hive metastore. The >>> directory will have hive-site.xml, and will have details on the port >>> metastore will listen on (hive.metastore.uris=thrift://localhost:9083), >>> and >>> the database metastore will be using >>> >>> (javax.jdo.option.ConnectionURL=jdbc:derby:;databaseName=/Users/rajat.khandelwal/conf/hive/metastore/metastore_db;create=true). >>> In your case, provide the mysql url. >>> 6. Start metastore using >>> >>> nohup $HIVE_HOME/bin/hive --config conf_location --service metastore >>> & >>> >>> 7. Create a configuration directory for hive server. This will have >>> details on metastore end-point to talk to. >>> 8. Start Hive server using >>> >>> nohup $HIVE_HOME/bin/hive --config conf_location --service >>> hiveserver2 & >>> >>> 9. Verify both have started using >>> >>> telnet localhost 9083 # for metastore >>> telnet localhost 10000 # for server >>> >>> 10. Create a lens configuration directory, which will have metastore >>> endpoint in lens-site.xml, and Hive Server endpoint in >>> drivers/hive/hive1/hivedriver-site.xml. Here `hive1` is the name of the >>> driver. You can have multiple hive drivers in drivers/hive/ directory. >>> 11. Start lens server using this configuration. >>> >>> bin/lens-ctl restart --conf conf_dir && tail -f logs/lensserver.log >>> >>> 12. Go to client directory and run the following commands: >>> >>> bin/run-examples.sh sample-metastore # Will create sample cubes, >>> dimensions etc. Basically the example schema. >>> bin/run-examples.sh populate-metastore # Will populate data in the >>> sample schema created by the previous command. >>> bin/run-examples.sh runqueries # Will run a bunch of queries over >>> the sample data created and populated by the previous commands. >>> >>> 13. The sample queries should work correctly. Here, the data is >>> present on local file system, and when the query is sent to Hive, Hive >>> would be able to execute it using the local files. >>> >>> Once this succeeds, you can add partitions manually where data is >>> present on HDFS and try running queries. >>> >>> Now, I'll provide some diagnosis for the issues you mentioned: >>> >>> a, b, c: Seems like HADOOP_HOME is not a local path, but an HDFS path. >>> Refer to point 3. >>> d: Partitions have not been added. If they were, then the properties >>> cube.storagetable.partition.timeline.cache.<UpdatePeriod>.<PartCol>.first >>> and latest would have values, instead of being empty. >>> e, f, g: Not sure why this class is required, but you can download the >>> jar via maven, transfer it to the machine and export its path as >>> LENSCPPATH. This will also be a local file system path, not HDFS. You >>> should be able to find the jar in your box in the directory >>> ~/.m2/repository/org/htrace/htrace-core/3.0.4/htrace-core-3.0.4.jar. >>> If not, download via maven. After exporting LENSCPPATH, restart lens server >>> and try again. >>> >>> >>> Let us know whether it's helpful. If not, let's pick from where you get >>> stuck. >>> >>> I'm also attaching configurations in my local lens-hive installation. >>> You can take help from there for sample configuration values. >>> >>> >>>
