Are you running a stand alone metastore process? The compactor runs in a
stand alone metastore, not in the client.
Alan.
vic0777 <mailto:vic0...@163.com>
December 2, 2014 at 23:13
Hi Alan,
Thans for your help. I set the hive.compactor.initiator.on(= true) and
hive.compactor.worker.threads(=2) in hive-site.xml. After the
configuration, I started Hive for its first run. what do you mean by
"When you say you set hive.compactor.initiator.on (=true I hope) and
hive.compactor.worker.threads, did you did that in your metastore
process?" Do I need to config something in other place?
In order to verify the compaction feature, I executed a alter table
t1_txn compact 'major' command first. The request is enqueued but its
state is always initiated even after I restart hive. How to make the
request execute? Then I set hive.compactor.delta.num.threshold(=2) in
hive-site.xml. There supposed to be a minor compaction after two
update or delete operations. But after 5 UPDATEs, the compaction did
not happen. The show compactions command only lists the request of the
previous alter table command. Besides, I set
hive.compactor.delta.pct.threshold(=0.01), according to the document,
it specifies the percentage (fractional) size of the delta files
relative to the base that will trigger a major compaction. Since the
base does not exist in the beginning, how does the system know when to
trigger a major compaction? So, my question is how to make compaction
work? Is there any tutorial or help?
Following is my hive-site.xml:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hive.txn.manager</name>
<value>org.apache.hadoop.hive.ql.lockmgr.DbTxnManager</value>
</property>
<property>
<name>hive.txn.timeout</name>
<value>1000</value>
</property>
<property>
<name>hive.compactor.initiator.on</name>
<value>true</value>
</property>
<property>
<name>hive.compactor.worker.threads</name>
<value>2</value>
</property>
<property>
<name>hive.support.concurrency</name>
<value>true</value>
</property>
<property>
<name>hive.enforce.bucketing</name>
<value>true</value>
</property>
<property>
<name>hive.exec.dynamic.partition.mode</name>
<value>nonstrict</value>
</property>
<property>
<name>hive.in.test</name>
<value>true</value>
</property>
<property>
<name>hive.compactor.delta.num.threshold</name>
<value>2</value>
</property>
<property>
<name>hive.compactor.delta.pct.threshold</name>
<value>0.01</value>
</property>
</configuration>
At 2014-12-03 09:59:34, "Alan Gates" <ga...@hortonworks.com> wrote:
Alan Gates <mailto:ga...@hortonworks.com>
December 2, 2014 at 17:59
The base directories will only exist after compaction has run. When
you say you set hive.compactor.initiator.on (=true I hope) and
hive.compactor.worker.threads, did you did that in your metastore
process? If so, did you restart the metastore after changing the
config values?
Alan.
vic0777 <mailto:vic0...@163.com>
December 1, 2014 at 23:12
Hi All,
I am trying to use the new transaction feature in Hive-0.14. According
to its document, every transaction table have a base directory and one
delta directory for each transaction in HDFS for data storage. But I
can not find where the base directory is in HDFS, there is only delta
directories. Following is the commands I used.
create table test_txn (id int,name string ) clustered by (id) into
2buckets stored as orc TBLPROPERTIES('transactional'='true');
insert into table test_txn select * from test_text;
update test_txn set name="liu" where id = 10;
P.S. I have configured the parameters required by the transaction
feature:
hive.support.concurrency,
hive.enforce.bucketing,
hive.exec.dynamic.partition.mode,
hive.txn.manager,
hive.compactor.initiator.on
hive.compactor.worker.threads.
Although I cannot find the base directory in HDFS, all SELECT, UPDATE
and DELETE statements works fine and the data in the table is correct.
I am wondering where the base directory is.
Any help is appreciated.
Thanks,
Wantao
--
Sent with Postbox <http://www.getpostbox.com>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
