how can I collect hive metastore metrics and hiveserver2 metrics separately

2018-08-22 Thread Huang Meilong 
Hi all,


I found the configuration of collecting hive metrics here: 
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Metrics

Configuration Properties - Apache Hive - Apache Software 
...
cwiki.apache.org
This document describes the Hive user configuration properties (sometimes 
called parameters, variables, or options), and notes which releases introduced 
new properties. The canonical list of configuration properties is managed in 
the HiveConf Java class, so refer to the HiveConf.java file for a ...




To enable metastore and hiveserver2 metrics collection, I need to set 
`hive.metastore.metrics.enabled=true` and 
`hive.hiveserver2.metrics.enabled=true`, specify 
`hive.service.metrics.file.location` to set the location where the metrics to 
dump, but I found that the json file content changed alternately, does 
metastore metrics and hiveserver2 metrics dump to the same file alternately? 
How can I specify tow locations for metastore metrics and hiveserver2 metrics 
separately, I just want to collect these metrics separately.


Any suggestions are appreciated!


Thank you,

Huang.

What's the 'hive.metastore.fastpath' in hive site for?

2017-01-09 Thread Huang Meilong 
Hi all,


What's the 'hive.metastore.fastpath' in hive site for?

HMS connections to meta db

2016-12-13 Thread Huang Meilong 
Hi all,


Will HMS keep the connection to meta db when HMS is up? Or will HMS build 
connection to meta db every time the query comes to HMS and release connection 
to meta db when query finished?

答复: Difference between MANAGED_TABLE and EXTERNAL_TABLE in org.apache.hadoop.hive.metastore.TableType

2016-12-01 Thread Huang Meilong 
Thanks for your detailed explanations.


I'm not asking what the concepts of INTERNAL TABLE and EXTERNAL TABLE in hive, 
I'm just confused what does the value "EXTERNAL_TABLE" in class 
org.apache.hadoop.hive.TableType mean?


I create a table setting table type TableType.EXTERNAL_TABLE by calling hive 
metastore api like this:


final HiveMetaStoreClient client = new HiveMetaStoreClient(c);

Table newTable = new Table();

newTable.setTableType(TableType.EXTERNAL_TABLE.toString());

...

client.createTable(newTable);


then I describe the created table with hive metastore api:

client.getTable("dbName", "tableName");


I can see that the table type is still "MANAGED_TABLE", is this a bug of 
metastore api?






发件人: Mich Talebzadeh 
发送时间: 2016年12月2日 5:29
收件人: user
主题: Re: Difference between MANAGED_TABLE and EXTERNAL_TABLE in 
org.apache.hadoop.hive.metastore.TableType

Adding to Alan's points external tables are used often as a staging area. For 
example, ingesting data from HDFS location on a daily basis and putting that 
data into Hive managed tables. That location of that external table can change 
pointing to a new HDFS directory created by say Flume etc through ALTER table 
like below

ALTER TABLE ${DATABASE}.EXTERNALMARKETDATA set location 
'hdfs://rhes564:9000/data/prices/${TODAY}';


HTH


Dr Mich Talebzadeh



LinkedIn  
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw



http://talebzadehmich.wordpress.com

[https://secure.gravatar.com/blavatar/eeed9e1445477d955b8dd2470c042649?s=200&ts=1480663659]<http://talebzadehmich.wordpress.com/>

Mich Talebzadeh<http://talebzadehmich.wordpress.com/>
talebzadehmich.wordpress.com
Technical Architecture, Big Data, Oracle, Sybase, CEP, IMDB and Data Grid




Disclaimer: Use it at your own risk. Any and all responsibility for any loss, 
damage or destruction of data or any other property which may arise from 
relying on this email's technical content is explicitly disclaimed. The author 
will in no case be liable for any monetary damages arising from such loss, 
damage or destruction.



On 1 December 2016 at 17:53, Alan Gates 
mailto:alanfga...@gmail.com>> wrote:
Hive does not assume that it owns the data for an external table.  Thus when an 
external table is dropped, the data is not deleted.  People often use this as a 
way to load data into a directory in HDFS and then “cast” a table structure 
over it by creating an external table with that directory as its location.

Alan.

> On Dec 1, 2016, at 06:15, Huang Meilong 
> mailto:ims...@outlook.com>> wrote:
>
> Hi all,
>
> I found an enum TableType in package org.apache.hadoop.hive.metastore. What's 
> the difference between MANAGED_TABLE and EXTERNAL_TABLE?
>
> Will the table be an EXTERNAL TABLE with setting table type EXTERNAL_TABLE 
> when creating table?
>
> I found the code to determine whether a table is an external table in 
> MetaStoreUtils.java
> https://github.com/apache/hive/blob/master/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java#L1425
>
> I'm confused what is EXTERNAL_TABLE in TableType for?

Difference between MANAGED_TABLE and EXTERNAL_TABLE in org.apache.hadoop.hive.metastore.TableType

2016-12-01 Thread Huang Meilong 
Hi all,


I found an enum TableType in package org.apache.hadoop.hive.metastore. What's 
the difference between MANAGED_TABLE and EXTERNAL_TABLE?


Will the table be an EXTERNAL TABLE with setting table type EXTERNAL_TABLE when 
creating table?


I found the code to determine whether a table is an external table in 
MetaStoreUtils.java

https://github.com/apache/hive/blob/master/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java#L1425


I'm confused what is EXTERNAL_TABLE in TableType for?

Got NullPointerException when create table specifying location url with hive metastore api

2016-11-25 Thread Huang Meilong 
Hi,


I'm using hive metastore api(org.apache.hive.hive-metastore-2.1.0.jar) to 
create table, when I do not specify the location uri, it works fine:


```

HiveConf c = new HiveConf();
c.set("hive.metastore.uris", "thrift://xx.xx.xx.xx:9083");
HiveMetaStoreClient client = new HiveMetaStoreClient(c);
Table table = new Table();
table.setDbName("default");
table.setTableName("dmx4");

StorageDescriptor sd = new StorageDescriptor();
List cols = new ArrayList();
FieldSchema f1 = new FieldSchema("id", "int", "id");
FieldSchema f2 = new FieldSchema("name", "string", "name");

cols.add(f1);
cols.add(f2);
sd.setCols(cols);

//sd.setLocation("hdfs://user/hive/warehouse/mx");

sd.setInputFormat("org.apache.hadoop.mapred.TextInputFormat");

sd.setOutputFormat("org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat");

SerDeInfo serdeInfo = new SerDeInfo();

serdeInfo.setSerializationLib("org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe");
Map serdeParameters = new HashMap();
serdeParameters.put("field.delim", ",");
serdeInfo.setParameters(serdeParameters);
sd.setSerdeInfo(serdeInfo);
table.setSd(sd);

table.setOwner("root");
table.setTableType("MANAGED_TABLE");
client.createTable(table);

```


But it throws NPE when I specify the location uri:

```


HiveConf c = new HiveConf();
c.set("hive.metastore.uris", "thrift://xx.xx.xx.xx:9083");
HiveMetaStoreClient client = new HiveMetaStoreClient(c);
Table table = new Table();
table.setDbName("default");
table.setTableName("dmx4");

StorageDescriptor sd = new StorageDescriptor();
List cols = new ArrayList();
FieldSchema f1 = new FieldSchema("id", "int", "id");
FieldSchema f2 = new FieldSchema("name", "string", "name");

cols.add(f1);
cols.add(f2);
sd.setCols(cols);

sd.setLocation("hdfs://user/hive/warehouse/mx");

sd.setInputFormat("org.apache.hadoop.mapred.TextInputFormat");

sd.setOutputFormat("org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat");

SerDeInfo serdeInfo = new SerDeInfo();

serdeInfo.setSerializationLib("org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe");
Map serdeParameters = new HashMap();
serdeParameters.put("field.delim", ",");
serdeInfo.setParameters(serdeParameters);
sd.setSerdeInfo(serdeInfo);
table.setSd(sd);

table.setOwner("root");
table.setTableType("MANAGED_TABLE");
client.createTable(table);

```


stack trace:

```

org.apache.hadoop.hive.metastore.api.MetaException: 
java.lang.NullPointerException
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result$create_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:41498)
 ~[org.apache.hive.hive-metastore-2.1.0.jar:2.1.0]
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result$create_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:41466)
 ~[org.apache.hive.hive-metastore-2.1.0.jar:2.1.0]
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result.read(ThriftHiveMetastore.java:41392)
 ~[org.apache.hive.hive-metastore-2.1.0.jar:2.1.0]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86) 
~[org.apache.thrift.libthrift-0.9.3.jar:0.9.3]
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table_with_environment_context(ThriftHiveMetastore.java:1183)
 ~[org.apache.hive.hive-metastore-2.1.0.jar:2.1.0]
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table_with_environment_context(ThriftHiveMetastore.java:1169)
 ~[org.apache.hive.hive-metastore-2.1.0.jar:2.1.0]
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:2323)
 ~[org.apache.hive.hive-metastore-2.1.0.jar:2.1.0]
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:736)
 ~[org.apache.hive.hive-metastore-2.1.0.jar:2.1.0]
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:724)
 ~[org.apache.hive.hive-metastore-2.1.0.jar:2.1.0]


```


Is this a bug? Do you have any idea?

答复: Can I specify database name in hive metastore service?

2016-10-27 Thread Huang Meilong 
Because I have many clusters and I want to manage all these clusters' meta data 
with one metastore, and make the clusters' meta data isolated from each other. 
Maybe I could create a hive JIRA. Thank you all the same.


发件人: Peter Vary 
发送时间: 2016年10月27日 19:06:32
收件人: Huang Meilong
抄送: user@hive.apache.org
主题: Re: Can I specify database name in hive metastore service?

Hi Huang,

It is possible to use the same Metastore server for 2 HiveServer2 instances, 
but then the HiveServer2 instances will share the metastore. They will have the 
same databases and tables.

As far as I know it is not possible for a metastore to serve different data for 
different HiveServers.
Here someone with more experience might help more.

Thanks,
Peter


On Oct 27, 2016, at 4:41 AM, Huang Meilong 
mailto:ims...@outlook.com>> wrote:

Thanks Peter for your detailed explanation, it works now.

BTW, can I start 2 HiveServer2 instances with ONLY ONE remote metastore service 
and make the metastore db stored in the same RDBMS in a different database?

发件人: Peter Vary mailto:pv...@cloudera.com>>
发送时间: 2016年10月26日 21:11:26
收件人: Huang Meilong
抄送: user@hive.apache.org<mailto:user@hive.apache.org>
主题: Re: Can I specify database name in hive metastore service?

Hi Huang,

According to the picture you want to start 2 HiveServer2 instances both with 
embedded metastore, where the metastore db is stored in the same RDBMS in a 
different database.

This is certainly possible.
You have to set the database options according to this:
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-RemoteMetastoreDatabase

You have to set the metastore options according to this:
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-Local/EmbeddedMetastoreServer


Hope this helps,
Peter


On Oct 26, 2016, at 4:48 AM, Huang Meilong 
mailto:ims...@outlook.com>> wrote:

Look at this local metastore architecture:



If I set different database name in javax.jdo.option.ConnectionURL, say,
"jdbc:mysql://x/hivemeta_1?createDatabaseIfNotExist=true&characterEncoding=UTF-8"
 and 
"jdbc:mysql://x/hivemeta_2?createDatabaseIfNotExist=true&characterEncoding=UTF-8",
 will the to metastore services work fine?

In short, I want to use the same RDBMS database for the two hive metastore 
services, and the meta data is isolated form each other. How can I achieve that?





发件人: Peter Vary mailto:pv...@cloudera.com>>
发送时间: 2016年10月26日 0:49
收件人: user@hive.apache.org<mailto:user@hive.apache.org>
主题: Re: Can I specify database name in hive metastore service?

Hi Huang,
Hive metastore is a component of the "Hive database". See: 
https://cwiki.apache.org/confluence/display/Hive/Design
The metastore uses traditional RDBMS to store "the structure information of the 
various tables and partitions in the warehouse". The 
javax.jdo.option.ConnectionURL and the javax.jdo.option.ConnectionDriverName 
configuration options are used to access this RDBMS database. The 
hive.metastore.uris is the endpoint where the metastore will communicate with 
the other Hive components, like the HiveServer2.
So you can change the database name in the connectionUrl, which will change 
only the database name where the metadata is stored in the relational database 
and you can not add a database name to the thrift uri (metastore uri) since 
HiveServer2 will use the same uri to access metadata regardless of which Hive 
database is used by the client.
I hope this helps,
Peter

2016. okt. 25. 17:32 ezt írta ("Huang Meilong" 
mailto:ims...@outlook.com>>):
Hi,

To use hive metastore service, I must set `javax.jdo.option.ConnectionURL`, 
`javax.jdo.option.ConnectionDriverName` and `hive.metastore.uris` in 
hive-site.xml, like this:


  

javax.jdo.option.ConnectionURL


jdbc:mysql://x/hivemeta?createDatabaseIfNotExist=true&characterEncoding=UTF-8

  

  

javax.jdo.option.ConnectionDriverName

com.mysql.jdbc.Driver

  



hive.metastore.uris

thrift://xxx:9083

  


I'm confused that can I change the database name (usually it's `hivemeta`) for 
other names?

If I changed the database name from `hivemeta` to `my_hivemeta`, can hive 
metastore work? We can not specify database name in `hive.metastore.uris`, we 
can only specify hostname and port of metastore service.

答复: Can I specify database name in hive metastore service?

2016-10-26 Thread Huang Meilong 
Thanks Peter for your detailed explanation, it works now.


BTW, can I start 2 HiveServer2 instances with ONLY ONE remote metastore service 
and make the metastore db stored in the same RDBMS in a different database?


发件人: Peter Vary 
发送时间: 2016年10月26日 21:11:26
收件人: Huang Meilong
抄送: user@hive.apache.org
主题: Re: Can I specify database name in hive metastore service?

Hi Huang,

According to the picture you want to start 2 HiveServer2 instances both with 
embedded metastore, where the metastore db is stored in the same RDBMS in a 
different database.

This is certainly possible.
You have to set the database options according to this:
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-RemoteMetastoreDatabase

You have to set the metastore options according to this:
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-Local/EmbeddedMetastoreServer


Hope this helps,
Peter


On Oct 26, 2016, at 4:48 AM, Huang Meilong 
mailto:ims...@outlook.com>> wrote:

Look at this local metastore architecture:



If I set different database name in javax.jdo.option.ConnectionURL, say,
"jdbc:mysql://x/hivemeta_1?createDatabaseIfNotExist=true&characterEncoding=UTF-8"
 and 
"jdbc:mysql://x/hivemeta_2?createDatabaseIfNotExist=true&characterEncoding=UTF-8",
 will the to metastore services work fine?

In short, I want to use the same RDBMS database for the two hive metastore 
services, and the meta data is isolated form each other. How can I achieve that?




发件人: Peter Vary mailto:pv...@cloudera.com>>
发送时间: 2016年10月26日 0:49
收件人: user@hive.apache.org<mailto:user@hive.apache.org>
主题: Re: Can I specify database name in hive metastore service?

Hi Huang,
Hive metastore is a component of the "Hive database". See: 
https://cwiki.apache.org/confluence/display/Hive/Design
The metastore uses traditional RDBMS to store "the structure information of the 
various tables and partitions in the warehouse". The 
javax.jdo.option.ConnectionURL and the javax.jdo.option.ConnectionDriverName 
configuration options are used to access this RDBMS database. The 
hive.metastore.uris is the endpoint where the metastore will communicate with 
the other Hive components, like the HiveServer2.
So you can change the database name in the connectionUrl, which will change 
only the database name where the metadata is stored in the relational database 
and you can not add a database name to the thrift uri (metastore uri) since 
HiveServer2 will use the same uri to access metadata regardless of which Hive 
database is used by the client.
I hope this helps,
Peter

2016. okt. 25. 17:32 ezt írta ("Huang Meilong" 
mailto:ims...@outlook.com>>):
Hi,

To use hive metastore service, I must set `javax.jdo.option.ConnectionURL`, 
`javax.jdo.option.ConnectionDriverName` and `hive.metastore.uris` in 
hive-site.xml, like this:


  

javax.jdo.option.ConnectionURL


jdbc:mysql://x/hivemeta?createDatabaseIfNotExist=true&characterEncoding=UTF-8

  

  

javax.jdo.option.ConnectionDriverName

com.mysql.jdbc.Driver

  



hive.metastore.uris

thrift://xxx:9083

  


I'm confused that can I change the database name (usually it's `hivemeta`) for 
other names?

If I changed the database name from `hivemeta` to `my_hivemeta`, can hive 
metastore work? We can not specify database name in `hive.metastore.uris`, we 
can only specify hostname and port of metastore service.

答复: Can I specify database name in hive metastore service?

2016-10-25 Thread Huang Meilong 
Look at this local metastore architecture:


[cid:87afe275-ff1d-4dc3-88ed-b2081163c512]


If I set different database name in javax.jdo.option.ConnectionURL, say,

"jdbc:mysql://x/hivemeta_1?createDatabaseIfNotExist=true&characterEncoding=UTF-8"
 and 
"jdbc:mysql://x/hivemeta_2?createDatabaseIfNotExist=true&characterEncoding=UTF-8",
 will the to metastore services work fine?


In short, I want to use the same RDBMS database for the two hive metastore 
services, and the meta data is isolated form each other. How can I achieve that?




发件人: Peter Vary 
发送时间: 2016年10月26日 0:49
收件人: user@hive.apache.org
主题: Re: Can I specify database name in hive metastore service?


Hi Huang,

Hive metastore is a component of the "Hive database". See: 
https://cwiki.apache.org/confluence/display/Hive/Design

The metastore uses traditional RDBMS to store "the structure information of the 
various tables and partitions in the warehouse". The 
javax.jdo.option.ConnectionURL and the javax.jdo.option.ConnectionDriverName 
configuration options are used to access this RDBMS database. The 
hive.metastore.uris is the endpoint where the metastore will communicate with 
the other Hive components, like the HiveServer2.
So you can change the database name in the connectionUrl, which will change 
only the database name where the metadata is stored in the relational database 
and you can not add a database name to the thrift uri (metastore uri) since 
HiveServer2 will use the same uri to access metadata regardless of which Hive 
database is used by the client.

I hope this helps,
Peter

2016. okt. 25. 17:32 ezt írta ("Huang Meilong" 
mailto:ims...@outlook.com>>):

Hi,


To use hive metastore service, I must set `javax.jdo.option.ConnectionURL`, 
`javax.jdo.option.ConnectionDriverName` and `hive.metastore.uris` in 
hive-site.xml, like this:


  

javax.jdo.option.ConnectionURL


jdbc:mysql://x/hivemeta?createDatabaseIfNotExist=true&characterEncoding=UTF-8

  

  

javax.jdo.option.ConnectionDriverName

com.mysql.jdbc.Driver

  



hive.metastore.uris

thrift://xxx:9083

  



I'm confused that can I change the database name (usually it's `hivemeta`) for 
other names?


If I changed the database name from `hivemeta` to `my_hivemeta`, can hive 
metastore work? We can not specify database name in `hive.metastore.uris`, we 
can only specify hostname and port of metastore service.

Can I specify database name in hive metastore service?

2016-10-25 Thread Huang Meilong 
Hi,


To use hive metastore service, I must set `javax.jdo.option.ConnectionURL`, 
`javax.jdo.option.ConnectionDriverName` and `hive.metastore.uris` in 
hive-site.xml, like this:


  

javax.jdo.option.ConnectionURL


jdbc:mysql://x/hivemeta?createDatabaseIfNotExist=true&characterEncoding=UTF-8

  

  

javax.jdo.option.ConnectionDriverName

com.mysql.jdbc.Driver

  



hive.metastore.uris

thrift://xxx:9083

  



I'm confused that can I change the database name (usually it's `hivemeta`) for 
other names?


If I changed the database name from `hivemeta` to `my_hivemeta`, can hive 
metastore work? We can not specify database name in `hive.metastore.uris`, we 
can only specify hostname and port of metastore service.

答复: Hive failed with "No files matching path"

2016-07-14 Thread Huang Meilong 
Why not output for "echo $sql;"?


发件人: gm yu 
发送时间: 2016年7月15日 9:59:39
收件人: user@hive.apache.org
主题: Hive failed with "No files matching path"

Hello

I have a question about Hive(2.0.0) and HDFS(2.7.2). I met many Hive failure 
with "No files matching path". The script like:

hadoop fs -test -e $hdfspath
if [ $? -ne 0 ]; then
echo "Directory not exists!"
hadoop fs -mkdir $hdfspath
else
echo "Directory exists!"
fi

hadoop distcp oss://${bucket}/${filename} ${hdfspath}
hdfsname = $hdfspath/$filename
hadoop fs -test -e $hdfsname
if [ $? -ne 0 ]; then
   echo "File not exists!"
else
echo "File exists!"
fi

sql=" LOAD DATA INPATH '${hdfspath}' OVERWRITE INTO TABLE {output} PARTITION 
(pt='${time}');"

echo $sql;
hive -e "$sql"

if [ $? -ne 0 ]; then
echo "Hive unsuccessful!"
else
echo "Hive successful!"
fi

The log is:

Directory not exists!
File exists!
Hive unsuccessful!
FAILED: SemanticException Line 2:19 Invalid path '': No files matching path 
''

But rerun this script, everything is OK.

Any idea what is this about?

Thanks & Regards