hive 0.11 on MRv1 Vs. YARN
Hi, Is hive 0.11 faster on YARN? What am I missing out if I am using hive 0.11 on MRv1 ? Thanks a lot :) - Austin
Re: Unsubscribe
Fail ! You have to send a mail to user-unsubscr...@hive.apache.org to unsubscribe. Nothing is going to happen if you send unsubscribe here. Read this : http://hive.apache.org/mailing_lists.html On Mon, Jul 15, 2013 at 1:04 PM, Kasa V Varun kasa.va...@mu-sigma.comwrote: Unsubsribe -- This email message may contain proprietary, private and confidential information. The information transmitted is intended only for the person(s) or entities to which it is addressed. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited and may be illegal. If you received this in error, please contact the sender and delete the message from your system. Mu Sigma takes all reasonable steps to ensure that its electronic communications are free from viruses. However, given Internet accessibility, the Company cannot accept liability for any virus introduced by this e-mail or any attachment and you are advised to use up-to-date virus checking software.
Re: Who is the hive admin user?
Thanks a lot Owen Lefty. Sorry for the late reply, got a bit busy during the weekend. So... what I understand is: 1. To enable security in Hive you need minimum of Hive 0.10 and Kerberos enabled. 2. The best option is to protect the HDFS directories that the data is stored in. 3. The user roles in Hive are advisory only. Everyone is an admin so anyone can grant anyone additional permissions. But since permissions on the HDFS dir can't be changed by any user, the data will be secure. Will everyone being an admin be changed in a future version of Hive? Is it difficult to implement? I guess it will be more helpful if there was a single or a group of predefined admins. Thanks, Austin On Sat, May 11, 2013 at 12:50 AM, Lefty Leverenz le...@hortonworks.comwrote: Hive 0.10's metastore server security is documented in the revised Authorization wiki, which your link didn't go to because the name had been changed from auth to Authorization (sorry about that): https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Authorization . The jira is HIVE-3705 https://issues.apache.org/jira/browse/HIVE-3705. – Lefty On Fri, May 10, 2013 at 8:24 AM, Owen O'Malley omal...@apache.org wrote: Unfortunately, the roles in Hive are advisory only. Effectively everyone is an admin who can grant anyone (including themselves) additional permissions. If you need security, the best option is to protect the HDFS directories that the data is stored in. Set the HDFS owner, group, and permissions so that the users have read/write permission as desired. Don't forget to set things at both the database directory and table directory levels. Then you need to configure hive.security.metastore.authorization.manager with org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider, which uses HDFS permissions to control access to databases and tables. By setting it up this way, the HDFS permissions will be enforced by the NameNode and keep each group from reading each others data. The metastore will use the same HDFS permissions to prevent access to the other groups databases' metadata. Of course, all of this assumes you have Kerberos turned on for your Hadoop cluster. It also requires a minimum of Hive 0.10. Hope it helps, Owen On Fri, May 10, 2013 at 1:19 AM, Austin Chungath austi...@gmail.comwrote: Hi all, This seems silly but I couldn't get any reliable information even after a few minutes of googling. If I am creating user roles and groups in hive, which user should I be doing it with? Is there some configuration in hive-site.xml which sets a user as hive admin? Right now I see that any user can create user roles and groups or am I missing something? I am using the following link as reference https://cwiki.apache.org/Hive/languagemanual-auth.html Thanks, Austin
Who is the hive admin user?
Hi all, This seems silly but I couldn't get any reliable information even after a few minutes of googling. If I am creating user roles and groups in hive, which user should I be doing it with? Is there some configuration in hive-site.xml which sets a user as hive admin? Right now I see that any user can create user roles and groups or am I missing something? I am using the following link as reference https://cwiki.apache.org/Hive/languagemanual-auth.html Thanks, Austin
How to revert metastore of hive after HA has been disabled?
Hi, I am using Cloudera Manager 4.5 and I have enabled HA. This requires update of the hive metastore to accommodate the HA feature, which I did and it's working. But now I disabled HA and I want to revert the metastore of hive to the normal mode. How can I do this? Will I have to do a dump of the metastore db and manually change to the correct hostname of the namenode? Is there an easier workaround? Thanks, Austin
RE: Security for Hive
Thanks Bejoy. Sent from my Windows Phone -- From: bejoy...@yahoo.com Sent: 23-02-2013 08:29 PM To: user@hive.apache.org Subject: Re: Security for Hive Hi Austin AFAIK at the moment you can control permissions gracefully only on a data level not on the metadata level. ie you can play with the hdfs permissions . Regards Bejoy KS Sent from remote device, Please excuse typos -- *From: * Austin Chungath austi...@gmail.com *Date: *Fri, 22 Feb 2013 23:11:51 -0800 *To: *bejoy...@yahoo.combejoy...@yahoo.com; user@hive.apache.org user@hive.apache.org *ReplyTo: * user@hive.apache.org *Subject: *RE: Security for Hive So that means any user can revoke or give permissions to any user for any table in the metastore? Sent from my Phone, please ignore typos -- From: bejoy...@yahoo.com Sent: 22-02-2013 11:30 PM To: user@hive.apache.org Subject: Re: Security for Hive Hi Sachin Currently there is no such admin user concept in hive. Regards Bejoy KS Sent from remote device, Please excuse typos -- *From: * Sachin Sudarshana sachin.sudarsh...@gmail.com *Date: *Fri, 22 Feb 2013 16:40:49 +0530 *To: *user@hive.apache.org *ReplyTo: * user@hive.apache.org *Subject: *Re: Security for Hive Hi, I have read about roles, user privileges, group privileges etc. But these roles can be created by any user for any database/table. I would like to know if there is a specific 'administrator' for hive who can log on with his credentials and is the only one entitled to create roles, grant privileges etc. Thank you. On Fri, Feb 22, 2013 at 4:19 PM, Jagat Singh jagatsi...@gmail.com wrote: You might want to read this https://cwiki.apache.org/Hive/languagemanual-auth.html On Fri, Feb 22, 2013 at 9:44 PM, Sachin Sudarshana sachin.sudarsh...@gmail.com wrote: Hi, I have just started learning about hive. I have configured Hive to use mysql as the metastore instead of derby. If I wish to use GRANT and REVOKE commands, i can use it with any user. A user can issue GRANT or REVOKE commands to any other users' table since both the users' tables are present in the same warehouse. Isn't there a concept of superuser/admin in hive who alone has the authority to issue these commands ? Any answer is greatly appreciated! -- Thanks and Regards, Sachin Sudarshana -- Thanks and Regards, Sachin Sudarshana
RE: Security for Hive
So that means any user can revoke or give permissions to any user for any table in the metastore? Sent from my Phone, please ignore typos -- From: bejoy...@yahoo.com Sent: 22-02-2013 11:30 PM To: user@hive.apache.org Subject: Re: Security for Hive Hi Sachin Currently there is no such admin user concept in hive. Regards Bejoy KS Sent from remote device, Please excuse typos -- *From: * Sachin Sudarshana sachin.sudarsh...@gmail.com *Date: *Fri, 22 Feb 2013 16:40:49 +0530 *To: *user@hive.apache.org *ReplyTo: * user@hive.apache.org *Subject: *Re: Security for Hive Hi, I have read about roles, user privileges, group privileges etc. But these roles can be created by any user for any database/table. I would like to know if there is a specific 'administrator' for hive who can log on with his credentials and is the only one entitled to create roles, grant privileges etc. Thank you. On Fri, Feb 22, 2013 at 4:19 PM, Jagat Singh jagatsi...@gmail.com wrote: You might want to read this https://cwiki.apache.org/Hive/languagemanual-auth.html On Fri, Feb 22, 2013 at 9:44 PM, Sachin Sudarshana sachin.sudarsh...@gmail.com wrote: Hi, I have just started learning about hive. I have configured Hive to use mysql as the metastore instead of derby. If I wish to use GRANT and REVOKE commands, i can use it with any user. A user can issue GRANT or REVOKE commands to any other users' table since both the users' tables are present in the same warehouse. Isn't there a concept of superuser/admin in hive who alone has the authority to issue these commands ? Any answer is greatly appreciated! -- Thanks and Regards, Sachin Sudarshana -- Thanks and Regards, Sachin Sudarshana
Re: Multiuser setup on Hive
Shreepadam, So what do you recommend for this? What are the current best practices for deploying hive in a multi-user environment? Thanks, Austin On Thu, Nov 22, 2012 at 1:10 PM, Shreepadma Venugopalan shreepa...@cloudera.com wrote: Hi Austin, Hive authorization in its current form has a number of bugs and it is not recommended that you use it. We are planning to work on supporting authorization in a subsequent version of Hive. Thanks. Shreepadma On Wed, Nov 21, 2012 at 11:12 PM, Austin Chungath austi...@gmail.comwrote: Hi Bejoy, Thanks for the quick reply. I had been reading through hive authorization https://cwiki.apache.org/Hive/languagemanual-auth.html Is it any good. Can anyone explain what happens if I enable this? Will I be able to prevent users from deleting other user's tables? Regards, Austin On Thu, Nov 22, 2012 at 12:20 PM, Bejoy KS bejoy...@yahoo.com wrote: ** Hi Austin In hive currently you can have permissions only on the hdfs layer not on the metastore. The current hive metastore don't have multiuser permission support. Any user will be able to drop the metadata information now. Regards Bejoy KS Sent from handheld, please excuse typos. -- *From: * Austin Chungath austi...@gmail.com *Date: *Thu, 22 Nov 2012 12:16:24 +0530 *To: *user@hive.apache.org; u...@hadoop.apache.org *ReplyTo: * user@hive.apache.org *Subject: *Multiuser setup on Hive Hi, I had been trying to set up a multi user environment for hive. I have set up the hive metastore db in MySQL and hive works. Consider this scenario: user1 has created a database data1 user2 has created a database data2 Now user2 logs into hive and he is able to see and delete database data2 How do I prevent this? Regards, Austin
Re: Multiuser setup on Hive
Thanks Alex, But unfortunately I don't have kerberos implementation right now to try it out. I was wondering if we can create multiple metastore dbs in mysql and then for each user group make separate hive-site.xml which has the username and jdbc connection details. Do I make any sense? is something in these lines possible? Regards, Austin On Thu, Nov 22, 2012 at 2:11 PM, Alexander Alten-Lorenz wget.n...@gmail.com wrote: You could use SASL / kerberos implementation within HiveServer2. Depends on a kerberosized cluster, too. Hive's metastore server provides the same mechanism, but isn't fully multi connect ready. Here's a link: http://ben-tech.blogspot.de/2012/10/hive-server-2-in-cdh41.html - Alex On Nov 22, 2012, at 7:46 AM, Austin Chungath austi...@gmail.com wrote: Hi, I had been trying to set up a multi user environment for hive. I have set up the hive metastore db in MySQL and hive works. Consider this scenario: user1 has created a database data1 user2 has created a database data2 Now user2 logs into hive and he is able to see and delete database data2 How do I prevent this? Regards, Austin -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: Multiuser setup on Hive
Typo, I meant user2 logs into hive and he is able to see and delete database data1 On Thu, Nov 22, 2012 at 12:16 PM, Austin Chungath austi...@gmail.comwrote: Hi, I had been trying to set up a multi user environment for hive. I have set up the hive metastore db in MySQL and hive works. Consider this scenario: user1 has created a database data1 user2 has created a database data2 Now user2 logs into hive and he is able to see and delete database data2 How do I prevent this? Regards, Austin
Re: Multiuser setup on Hive
Thanks dean. On Thu, Nov 22, 2012 at 7:44 PM, Dean Wampler dean.wamp...@thinkbiganalytics.com wrote: If you go the route of locking down permissions at the HDFS level, then it will help if everyone works in his or her own database, since all the tables will be rooted at a directory for each db. dean On Thu, Nov 22, 2012 at 2:26 AM, Austin Chungath austi...@gmail.comwrote: Shreepadam, So what do you recommend for this? What are the current best practices for deploying hive in a multi-user environment? Thanks, Austin On Thu, Nov 22, 2012 at 1:10 PM, Shreepadma Venugopalan shreepa...@cloudera.com wrote: Hi Austin, Hive authorization in its current form has a number of bugs and it is not recommended that you use it. We are planning to work on supporting authorization in a subsequent version of Hive. Thanks. Shreepadma On Wed, Nov 21, 2012 at 11:12 PM, Austin Chungath austi...@gmail.comwrote: Hi Bejoy, Thanks for the quick reply. I had been reading through hive authorization https://cwiki.apache.org/Hive/languagemanual-auth.html Is it any good. Can anyone explain what happens if I enable this? Will I be able to prevent users from deleting other user's tables? Regards, Austin On Thu, Nov 22, 2012 at 12:20 PM, Bejoy KS bejoy...@yahoo.com wrote: ** Hi Austin In hive currently you can have permissions only on the hdfs layer not on the metastore. The current hive metastore don't have multiuser permission support. Any user will be able to drop the metadata information now. Regards Bejoy KS Sent from handheld, please excuse typos. -- *From: * Austin Chungath austi...@gmail.com *Date: *Thu, 22 Nov 2012 12:16:24 +0530 *To: *user@hive.apache.org; u...@hadoop.apache.org *ReplyTo: * user@hive.apache.org *Subject: *Multiuser setup on Hive Hi, I had been trying to set up a multi user environment for hive. I have set up the hive metastore db in MySQL and hive works. Consider this scenario: user1 has created a database data1 user2 has created a database data2 Now user2 logs into hive and he is able to see and delete database data2 How do I prevent this? Regards, Austin -- *Dean Wampler, Ph.D.* thinkbiganalytics.com +1-312-339-1330
Multiuser setup on Hive
Hi, I had been trying to set up a multi user environment for hive. I have set up the hive metastore db in MySQL and hive works. Consider this scenario: user1 has created a database data1 user2 has created a database data2 Now user2 logs into hive and he is able to see and delete database data2 How do I prevent this? Regards, Austin
Re: Multiuser setup on Hive
Hi Bejoy, Thanks for the quick reply. I had been reading through hive authorization https://cwiki.apache.org/Hive/languagemanual-auth.html Is it any good. Can anyone explain what happens if I enable this? Will I be able to prevent users from deleting other user's tables? Regards, Austin On Thu, Nov 22, 2012 at 12:20 PM, Bejoy KS bejoy...@yahoo.com wrote: ** Hi Austin In hive currently you can have permissions only on the hdfs layer not on the metastore. The current hive metastore don't have multiuser permission support. Any user will be able to drop the metadata information now. Regards Bejoy KS Sent from handheld, please excuse typos. -- *From: * Austin Chungath austi...@gmail.com *Date: *Thu, 22 Nov 2012 12:16:24 +0530 *To: *user@hive.apache.org; u...@hadoop.apache.org *ReplyTo: * user@hive.apache.org *Subject: *Multiuser setup on Hive Hi, I had been trying to set up a multi user environment for hive. I have set up the hive metastore db in MySQL and hive works. Consider this scenario: user1 has created a database data1 user2 has created a database data2 Now user2 logs into hive and he is able to see and delete database data2 How do I prevent this? Regards, Austin
Implementing a star schema (facts dimension model)
Hi, I am new to data warehousing in hadoop. This might be a trivial question but I was unable to find any answers in the mailing list. My questions are: A person has an existing data warehouse that uses a star schema (implemented in a mysql database).How to migrate it to Hadoop? I can use sqoop to copy my tables to hive, that much I know. But what happens to referential integrity? since there are no primary key / foreign key concepts. I have seen that I can use Hive Hbase together. Is there a method for storing facts and dimension tables in hadoop using Hive Hbase together? Does putting dimensions in Hbase facts in Hive make any sense? or should it be the other way around? Consider de-normalization is not an option. What is the best practice to port an existing data warehouse to hadoop, with minimum changes to the database model? Please let me know with whatever views you have on this. Thanks, Austin
