Yes Peter, we're working on it. We try to make compaction work automatically. With crontab otherwise. Thanks for your help
David Le mar. 2 juin 2020 à 14:48, Peter Vary <pv...@cloudera.com> a écrit : > Hi David, > > Maybe this can help: > https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.5/bk_data-access/content/understanding-administering-compactions.html > > Also if everything else fails, you can still issue the ALTER TABLE command > periodically using crontab. Running extra compaction will not hurt that > much. > > Thanks, > Peter > > On Jun 2, 2020, at 14:25, David Morin <morin.david....@gmail.com> wrote: > > Yes Peter. We can wait a little. > We use streaming so we have to merge files regularly. > If we can reduce the duration for some big Merge queries and make auto > compaction works properly it should be ok. > Problem: at the moment, compactions are not triggered automatically. Have > you an idea ? > > Le mar. 2 juin 2020 à 12:57, Peter Vary <pv...@cloudera.com> a écrit : > >> Hi David, >> >> You do not really need to run compaction every time. >> Is it possible to wait for the compaction to start automatically next >> time? >> >> Thanks, >> Peter >> >> On Jun 2, 2020, at 12:51, David Morin <morin.david....@gmail.com> wrote: >> >> Thanks Peter, >> >> Any workaround on HDP 2.6.x with Hive 2 ? >> Otherwise, the only way is to reduce time it takes for this "merge" >> queries in order to cancel locks and related transactions. Am I right ? >> >> Le mar. 2 juin 2020 à 11:52, Peter Vary <pv...@cloudera.com> a écrit : >> >>> Hi David, >>> >>> I think this jira describes your situation: >>> https://issues.apache.org/jira/browse/HIVE-16360 >>> >>> "The reason is that compactor won't compact anything above the level of >>> an open transaction. So if there is a very long running txn, it may filter >>> out all deltas in a partition. This looks very confusing when looking at >>> the logs." >>> >>> Thanks, >>> Peter >>> >>> On Jun 2, 2020, at 11:44, David Morin <morin.david....@gmail.com> wrote: >>> >>> I don't get it. >>> The transaction id in the error message "No delta files or original >>> files found to compact in hdfs://... with min Open txnid:6704143" is >>> related to a current transaction (transaction + lock) for another user, >>> another database and table. But still it blocks the compaction for the >>> current database/table >>> >>> On 2020/06/01 20:13:08, David Morin <morin.david....@gmail.com> wrote: >>> >>> Hi, >>> >>> I have a compaction issue on my cluster. When I force a compaction >>> (major) on one table I get this error in Metastore logs: >>> >>> 2020-06-01 19:49:35,512 ERROR [XXXX-78]: compactor.CompactorMR >>> (CompactorMR.java:run(264)) - No delta files or original files found to >>> compact in hdfs://...hive/warehouse/my.db/test for >>> compactionId=57944651 with min Open txnid:6704143. Compaction cannot >>> compact above this txnid >>> >>> Any idea about the root cause and how to fix it ? >>> thx >>> >>> Current env: HDP 2.6.5 >>> Hive table in transactional mode (ORC) >>> >>> >>> >> >
