答复: Can I specify database name in hive metastore service?
Because I have many clusters and I want to manage all these clusters' meta data
with one metastore, and make the clusters' meta data isolated from each other.
Maybe I could create a hive JIRA. Thank you all the same.
发件人: Peter Vary <pv...@cloudera.com>
发送时间: 2016年10月27日 19:06:32
收件人: Huang Meilong
抄送: user@hive.apache.org
主题: Re: Can I specify database name in hive metastore service?
Hi Huang,
It is possible to use the same Metastore server for 2 HiveServer2 instances,
but then the HiveServer2 instances will share the metastore. They will have the
same databases and tables.
As far as I know it is not possible for a metastore to serve different data for
different HiveServers.
Here someone with more experience might help more.
Thanks,
Peter
On Oct 27, 2016, at 4:41 AM, Huang Meilong
<ims...@outlook.com<mailto:ims...@outlook.com>> wrote:
Thanks Peter for your detailed explanation, it works now.
BTW, can I start 2 HiveServer2 instances with ONLY ONE remote metastore service
and make the metastore db stored in the same RDBMS in a different database?
发件人: Peter Vary <pv...@cloudera.com<mailto:pv...@cloudera.com>>
发送时间: 2016年10月26日 21:11:26
收件人: Huang Meilong
抄送: user@hive.apache.org<mailto:user@hive.apache.org>
主题: Re: Can I specify database name in hive metastore service?
Hi Huang,
According to the picture you want to start 2 HiveServer2 instances both with
embedded metastore, where the metastore db is stored in the same RDBMS in a
different database.
This is certainly possible.
You have to set the database options according to this:
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-RemoteMetastoreDatabase
You have to set the metastore options according to this:
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-Local/EmbeddedMetastoreServer
Hope this helps,
Peter
On Oct 26, 2016, at 4:48 AM, Huang Meilong
<ims...@outlook.com<mailto:ims...@outlook.com>> wrote:
Look at this local metastore architecture:
If I set different database name in javax.jdo.option.ConnectionURL, say,
"jdbc:mysql://x/hivemeta_1?createDatabaseIfNotExist=truecharacterEncoding=UTF-8"
and
"jdbc:mysql://x/hivemeta_2?createDatabaseIfNotExist=truecharacterEncoding=UTF-8",
will the to metastore services work fine?
In short, I want to use the same RDBMS database for the two hive metastore
services, and the meta data is isolated form each other. How can I achieve that?
发件人: Peter Vary <pv...@cloudera.com<mailto:pv...@cloudera.com>>
发送时间: 2016年10月26日 0:49
收件人: user@hive.apache.org<mailto:user@hive.apache.org>
主题: Re: Can I specify database name in hive metastore service?
Hi Huang,
Hive metastore is a component of the "Hive database". See:
https://cwiki.apache.org/confluence/display/Hive/Design
The metastore uses traditional RDBMS to store "the structure information of the
various tables and partitions in the warehouse". The
javax.jdo.option.ConnectionURL and the javax.jdo.option.ConnectionDriverName
configuration options are used to access this RDBMS database. The
hive.metastore.uris is the endpoint where the metastore will communicate with
the other Hive components, like the HiveServer2.
So you can change the database name in the connectionUrl, which will change
only the database name where the metadata is stored in the relational database
and you can not add a database name to the thrift uri (metastore uri) since
HiveServer2 will use the same uri to access metadata regardless of which Hive
database is used by the client.
I hope this helps,
Peter
2016. okt. 25. 17:32 ezt írta ("Huang Meilong"
<ims...@outlook.com<mailto:ims...@outlook.com>>):
Hi,
To use hive metastore service, I must set `javax.jdo.option.ConnectionURL`,
`javax.jdo.option.ConnectionDriverName` and `hive.metastore.uris` in
hive-site.xml, like this:
javax.jdo.option.ConnectionURL
jdbc:mysql://x/hivemeta?createDatabaseIfNotExist=truecharacterEncoding=UTF-8
javax.jdo.option.ConnectionDriverName
com.mysql.jdbc.Driver
hive.metastore.uris
thrift://xxx:9083
I'm confused that can I change the database name (usually it's `hivemeta`) for
other names?
If I changed the database name from `hivemeta` to `my_hivemeta`, can hive
metastore work? We can not specify database name in `hive.metastore.uris`, we
can only specify hostname and port of metastore service.
Re: Can I specify database name in hive metastore service?
I am not familiar with MySQL. However, as long as login/user used in
Hive-site.xml is bound to given Hive database (metastore) on RDBMS it
should work.
In general you can have multiple databases/schema for multiple instances of
Hive each having its own metadata on the data server.
In Sybase or MSSQL you specify that in JDBC connection via
jdbc:sybase:Tds:rhes564:5200/hivedb1
assuming username hiveuser and you ensure that hievuser is the database
owner ofg hivedb1 or jhas its default database as hivedb1
In oracle the username maps to schema name. So hiveuser will own a database
called
jdbc:oracle:thin:@rhes564:1521:mydb12
within that Oracle instance mydb12, you will have schema called hiveuser
and the login either maps to that schema or has permissions on that schema.
It is pretty straight forward.
BTW in your diagram your metastores should reside inside your MySQL
HTH
Dr Mich Talebzadeh
LinkedIn *
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 26 October 2016 at 14:11, Peter Vary <pv...@cloudera.com> wrote:
> Hi Huang,
>
> According to the picture you want to start 2 HiveServer2 instances both
> with embedded metastore, where the metastore db is stored in the same RDBMS
> in a different database.
>
> This is certainly possible.
> You have to set the database options according to this:
> https://cwiki.apache.org/confluence/display/Hive/
> AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-
> RemoteMetastoreDatabase
>
> You have to set the metastore options according to this:
> https://cwiki.apache.org/confluence/display/Hive/
> AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-
> Local/EmbeddedMetastoreServer
>
>
> Hope this helps,
> Peter
>
>
> On Oct 26, 2016, at 4:48 AM, Huang Meilong <ims...@outlook.com> wrote:
>
> Look at this local metastore architecture:
>
>
>
> If I set different database name in javax.jdo.option.ConnectionURL, say,
> "jdbc:mysql://x/hivemeta_1?createDatabaseIfNotExist=true&
> amp;characterEncoding=UTF-8" and "jdbc:mysql://x/hivemeta_2?
> createDatabaseIfNotExist=truecharacterEncoding=UTF-8", will the
> to metastore services work fine?
>
> In short, I want to use the same RDBMS database for the two hive
> metastore services, and the meta data is isolated form each other. How
> can I achieve that?
>
>
>
> ----------
> *发件人:* Peter Vary <pv...@cloudera.com>
> *发送时间:* 2016年10月26日 0:49
> *收件人:* user@hive.apache.org
> *主题:* Re: Can I specify database name in hive metastore service?
>
> Hi Huang,
> Hive metastore is a component of the "Hive database". See:
> https://cwiki.apache.org/confluence/display/Hive/Design
> The metastore uses traditional RDBMS to store "the structure information
> of the various tables and partitions in the warehouse". The
> javax.jdo.option.ConnectionURL and the javax.jdo.option.ConnectionDriverName
> configuration options are used to access this RDBMS database. The
> hive.metastore.uris is the endpoint where the metastore will communicate
> with the other Hive components, like the HiveServer2.
> So you can change the database name in the connectionUrl, which will
> change only the database name where the metadata is stored in the
> relational database and you can not add a database name to the thrift uri
> (metastore uri) since HiveServer2 will use the same uri to access metadata
> regardless of which Hive database is used by the client.
> I hope this helps,
> Peter
>
> 2016. okt. 25. 17:32 ezt írta ("Huang Meilong" <ims...@outlook.com>):
>
>> Hi,
>>
>> To use hive metastore service, I must set `javax.jdo.option.ConnectionURL`,
>> `javax.jdo.option.ConnectionDriverName` and `hive.metastore.uris` in
>> hive-site.xml, like this:
>>
>>
>>
>> javax.jdo.option.ConnectionURL
>>
>> jdbc:mysql://x/hivemeta?createDatabaseIfNotExist=
>> truecharacterEncoding=UTF-8
>>
>>
>>
>>
>>
>> javax.jdo.option.ConnectionDriverName
>>
>> com.mysql.jdbc.Driver
>>
>>
>>
>>
>>
>> hive.metastore.uris
>>
>> thrift://xxx:9083
>>
>>
>>
>>
>>
>> I'm confused that can I change the database name (usually it's `hivemeta`)
>> for other names?
>>
>> If I changed the database name from `hivemeta` to `my_hivemeta`, can
>> hive metastore work? We can not specify database name in `
>> hive.metastore.uris`, we can only specify hostname and port of metastore
>> service.
>>
>
>
Re: Can I specify database name in hive metastore service?
Hi Huang,
According to the picture you want to start 2 HiveServer2 instances both with
embedded metastore, where the metastore db is stored in the same RDBMS in a
different database.
This is certainly possible.
You have to set the database options according to this:
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-RemoteMetastoreDatabase
<https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-RemoteMetastoreDatabase>
You have to set the metastore options according to this:
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-Local/EmbeddedMetastoreServer
<https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-Local/EmbeddedMetastoreServer>
Hope this helps,
Peter
> On Oct 26, 2016, at 4:48 AM, Huang Meilong <ims...@outlook.com> wrote:
>
> Look at this local metastore architecture:
>
>
>
> If I set different database name in javax.jdo.option.ConnectionURL, say,
> "jdbc:mysql://x/hivemeta_1?createDatabaseIfNotExist=truecharacterEncoding=UTF-8"
> and
> "jdbc:mysql://x/hivemeta_2?createDatabaseIfNotExist=truecharacterEncoding=UTF-8",
> will the to metastore services work fine?
>
> In short, I want to use the same RDBMS database for the two hive metastore
> services, and the meta data is isolated form each other. How can I achieve
> that?
>
>
>
> 发件人: Peter Vary <pv...@cloudera.com>
> 发送时间: 2016年10月26日 0:49
> 收件人: user@hive.apache.org
> 主题: Re: Can I specify database name in hive metastore service?
>
> Hi Huang,
> Hive metastore is a component of the "Hive database". See:
> https://cwiki.apache.org/confluence/display/Hive/Design
> <https://cwiki.apache.org/confluence/display/Hive/Design>
> The metastore uses traditional RDBMS to store "the structure information of
> the various tables and partitions in the warehouse". The
> javax.jdo.option.ConnectionURL and the javax.jdo.option.ConnectionDriverName
> configuration options are used to access this RDBMS database. The
> hive.metastore.uris is the endpoint where the metastore will communicate with
> the other Hive components, like the HiveServer2.
> So you can change the database name in the connectionUrl, which will change
> only the database name where the metadata is stored in the relational
> database and you can not add a database name to the thrift uri (metastore
> uri) since HiveServer2 will use the same uri to access metadata regardless of
> which Hive database is used by the client.
> I hope this helps,
> Peter
>
> 2016. okt. 25. 17:32 ezt írta ("Huang Meilong" <ims...@outlook.com
> <mailto:ims...@outlook.com>>):
> Hi,
>
> To use hive metastore service, I must set `javax.jdo.option.ConnectionURL`,
> `javax.jdo.option.ConnectionDriverName` and `hive.metastore.uris` in
> hive-site.xml, like this:
>
>
> javax.jdo.option.ConnectionURL
>
> jdbc:mysql://x/hivemeta?createDatabaseIfNotExist=truecharacterEncoding=UTF-8
>
>
> javax.jdo.option.ConnectionDriverName
> com.mysql.jdbc.Driver
>
>
> hive.metastore.uris
> thrift://xxx:9083
>
>
>
> I'm confused that can I change the database name (usually it's `hivemeta`)
> for other names?
>
> If I changed the database name from `hivemeta` to `my_hivemeta`, can hive
> metastore work? We can not specify database name in `hive.metastore.uris`, we
> can only specify hostname and port of metastore service.
答复: Can I specify database name in hive metastore service?
Look at this local metastore architecture:
[cid:87afe275-ff1d-4dc3-88ed-b2081163c512]
If I set different database name in javax.jdo.option.ConnectionURL, say,
"jdbc:mysql://x/hivemeta_1?createDatabaseIfNotExist=truecharacterEncoding=UTF-8"
and
"jdbc:mysql://x/hivemeta_2?createDatabaseIfNotExist=truecharacterEncoding=UTF-8",
will the to metastore services work fine?
In short, I want to use the same RDBMS database for the two hive metastore
services, and the meta data is isolated form each other. How can I achieve that?
发件人: Peter Vary <pv...@cloudera.com>
发送时间: 2016年10月26日 0:49
收件人: user@hive.apache.org
主题: Re: Can I specify database name in hive metastore service?
Hi Huang,
Hive metastore is a component of the "Hive database". See:
https://cwiki.apache.org/confluence/display/Hive/Design
The metastore uses traditional RDBMS to store "the structure information of the
various tables and partitions in the warehouse". The
javax.jdo.option.ConnectionURL and the javax.jdo.option.ConnectionDriverName
configuration options are used to access this RDBMS database. The
hive.metastore.uris is the endpoint where the metastore will communicate with
the other Hive components, like the HiveServer2.
So you can change the database name in the connectionUrl, which will change
only the database name where the metadata is stored in the relational database
and you can not add a database name to the thrift uri (metastore uri) since
HiveServer2 will use the same uri to access metadata regardless of which Hive
database is used by the client.
I hope this helps,
Peter
2016. okt. 25. 17:32 ezt írta ("Huang Meilong"
<ims...@outlook.com<mailto:ims...@outlook.com>>):
Hi,
To use hive metastore service, I must set `javax.jdo.option.ConnectionURL`,
`javax.jdo.option.ConnectionDriverName` and `hive.metastore.uris` in
hive-site.xml, like this:
javax.jdo.option.ConnectionURL
jdbc:mysql://x/hivemeta?createDatabaseIfNotExist=truecharacterEncoding=UTF-8
javax.jdo.option.ConnectionDriverName
com.mysql.jdbc.Driver
hive.metastore.uris
thrift://xxx:9083
I'm confused that can I change the database name (usually it's `hivemeta`) for
other names?
If I changed the database name from `hivemeta` to `my_hivemeta`, can hive
metastore work? We can not specify database name in `hive.metastore.uris`, we
can only specify hostname and port of metastore service.
Re: Can I specify database name in hive metastore service?
Hi Huang,
Hive metastore is a component of the "Hive database". See:
https://cwiki.apache.org/confluence/display/Hive/Design
The metastore uses traditional RDBMS to store "the structure information of
the various tables and partitions in the warehouse". The
javax.jdo.option.ConnectionURL and the
javax.jdo.option.ConnectionDriverName configuration options are used to
access this RDBMS database. The hive.metastore.uris is the endpoint where
the metastore will communicate with the other Hive components, like the
HiveServer2.
So you can change the database name in the connectionUrl, which will change
only the database name where the metadata is stored in the relational
database and you can not add a database name to the thrift uri (metastore
uri) since HiveServer2 will use the same uri to access metadata regardless
of which Hive database is used by the client.
I hope this helps,
Peter
2016. okt. 25. 17:32 ezt írta ("Huang Meilong" ):
> Hi,
>
>
> To use hive metastore service, I must set `javax.jdo.option.ConnectionURL`,
> `javax.jdo.option.ConnectionDriverName` and `hive.metastore.uris` in
> hive-site.xml, like this:
>
>
>
>
> javax.jdo.option.ConnectionURL
>
> jdbc:mysql://x/hivemeta?createDatabaseIfNotExist=true&
> amp;characterEncoding=UTF-8
>
>
>
>
>
> javax.jdo.option.ConnectionDriverName
>
> com.mysql.jdbc.Driver
>
>
>
>
>
> hive.metastore.uris
>
> thrift://xxx:9083
>
>
>
>
>
> I'm confused that can I change the database name (usually it's `hivemeta`)
> for other names?
>
>
> If I changed the database name from `hivemeta` to `my_hivemeta`, can hive
> metastore work? We can not specify database name in `hive.metastore.uris`,
> we can only specify hostname and port of metastore service.
>
Can I specify database name in hive metastore service?
Hi, To use hive metastore service, I must set `javax.jdo.option.ConnectionURL`, `javax.jdo.option.ConnectionDriverName` and `hive.metastore.uris` in hive-site.xml, like this: javax.jdo.option.ConnectionURL jdbc:mysql://x/hivemeta?createDatabaseIfNotExist=truecharacterEncoding=UTF-8 javax.jdo.option.ConnectionDriverName com.mysql.jdbc.Driver hive.metastore.uris thrift://xxx:9083 I'm confused that can I change the database name (usually it's `hivemeta`) for other names? If I changed the database name from `hivemeta` to `my_hivemeta`, can hive metastore work? We can not specify database name in `hive.metastore.uris`, we can only specify hostname and port of metastore service.
