Sorry for the slow response, I missed the email in my inbox.
When you write the data directly using a storm topology, how are you
communicating to Hive that the new data exists? When streaming data in
via Hive Streaming the txn commits tells the system that new data is
arriving in that table or partition and thus it should watch for a need
to compact that table or partition. Are you doing txn commits via the
metastore thrift interface?
Regardless of this, when you've written data in and you manually request
a compaction, if there are delta files, the compaction should occur.
Can you share the arguments you are passing to the compact call and the
output of the SHOW COMPACTIONS you issued afterwards.
Alan.
Sachin Pasalkar <mailto:sachin_pasal...@symantec.com>
September 15, 2015 at 22:35
Yes below are the values set in hive. Initally I haven’t mentioned
NO_AUTO_COMPACTION in my table definition, which didn’t work so I have
put it with value as false.
hive.compactor.initiator.on
hive.compactor.worker.threads
hive.compactor.worker.timeout
hive.compactor.check.interval
hive.compactor.delta.num.threshold
hive.compactor.delta.pct.threshold
Thanks,
Sachin
From: Alan Gates <alanfga...@gmail.com <mailto:alanfga...@gmail.com>>
Reply-To: "dev@hive.apache.org <mailto:dev@hive.apache.org>"
<dev@hive.apache.org <mailto:dev@hive.apache.org>>
Date: Tuesday, 15 September 2015 10:30 pm
To: "dev@hive.apache.org <mailto:dev@hive.apache.org>"
<dev@hive.apache.org <mailto:dev@hive.apache.org>>
Subject: Re: How to enable compaction for table with external data?
If you want it to compact automatically you should not put
NO_AUTO_COMPACTION in the table properties.
First question, did you turn on the compactor on your metastore thrift
server? To do this you need to set a couple of values in the
metastore's hive-site.xml:
hive.compactor.initiator.on=true
hive.compactor.worker.threads=1 # or more
Alan.
Alan Gates <mailto:alanfga...@gmail.com>
September 15, 2015 at 10:00
If you want it to compact automatically you should not put
NO_AUTO_COMPACTION in the table properties.
First question, did you turn on the compactor on your metastore thrift
server? To do this you need to set a couple of values in the
metastore's hive-site.xml:
hive.compactor.initiator.on=true
hive.compactor.worker.threads=1 # or more
Alan.
Sachin Pasalkar <mailto:sachin_pasal...@symantec.com>
September 14, 2015 at 3:03
Hi,
We are writing direct orc file from storm topology instead of using
hive streaming (Due to performance issue with our data). However, we
want to compact the data. So we have added the
"NO_AUTO_COMPACTION"=“false” option in table which we created to read
data(1.6 GB scattered in multiple small files) in ORC file. Does
“NO_AUTO_COMPACTION” means it will not compact data while hive
streaming is used? If no, why it did not compact our data into 1 file?
We also tried manually calling compaction from java code using
org.apache.hadoop.hive.metastore.txn.TxnHandler’s compact API which
shows it has started compaction, when we execute command Show
compactions. But still does not work. I don’t want to execute the
manual commands from command line.
Is there any way?
PS: We are writing all files in one directory only.
Thanks,
Sachin
