[
https://issues.apache.org/jira/browse/HIVE-22977?focusedWorklogId=838362&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-838362
]
ASF GitHub Bot logged work on HIVE-22977:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 10/Jan/23 15:47
Start Date: 10/Jan/23 15:47
Worklog Time Spent: 10m
Work Description: veghlaci05 commented on code in PR #3801:
URL: https://github.com/apache/hive/pull/3801#discussion_r1065947652
##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java:
##########
@@ -3189,11 +3302,42 @@ private void testCompactionWithMerge(CompactionType
compactionType, boolean isPa
// Verify delete delta
Assert.assertTrue(CompactorTestUtil.getBaseOrDeltaNames(fs,
AcidUtils.deleteEventDeltaDirFilter, table,
partitionName).isEmpty());
- // Run a compaction
- if (partitionName == null) {
- CompactorTestUtil.runCompaction(conf, dbName, tableName, compactionType,
true);
- } else {
- CompactorTestUtil.runCompaction(conf, dbName, tableName, compactionType,
true, partitionName);
+ // Run a compaction which uses only merge compactor / query-based
compaction
+ CompactorFactory compactorFactory = mock(CompactorFactory.class);
+ // Return mocked instance of CompactorFactory
+ try (MockedStatic mockedStatic = mockStatic(CompactorFactory.class)) {
+
mockedStatic.when(CompactorFactory::getInstance).thenReturn(compactorFactory);
+ Compactor compactor;
+ List<Compactor> compactorList = new ArrayList<>();
+ if (useFallback) {
+ // Create a mock of merge compactor
+ compactor = mock(MergeCompactor.class);
+ compactorList.add(compactor);
+ if (compactionType == CompactionType.MAJOR) {
+ compactorList.add(new MajorQueryCompactor());
+ } else {
+ compactorList.add(new MinorQueryCompactor());
+ }
+ } else {
+ // Create a mock of major/minor compactor
+ if (compactionType == CompactionType.MAJOR) {
+ compactor = mock(MajorQueryCompactor.class);
+ } else {
+ compactor = mock(MinorQueryCompactor.class);
+ }
+ compactorList.add(new MergeCompactor());
+ compactorList.add(compactor);
+ }
+ // Mocked compaction always returns false
+ when(compactor.run(any(), any(), any(), any(), any(), any(),
any())).thenReturn(false);
+ CompactorChain compactorChain = new CompactorChain(compactorList);
+ // Return mocked compactor chain
+ when(compactorFactory.getCompactor(any(), any(), any(),
any())).thenReturn(compactorChain);
+ if (partitionName == null) {
+ CompactorTestUtil.runCompaction(conf, dbName, tableName,
compactionType, true);
+ } else {
+ CompactorTestUtil.runCompaction(conf, dbName, tableName,
compactionType, true, partitionName);
+ }
Review Comment:
Sorry when I said you may need to mock something I was using bad wording.
You should not re-create the compactor chain and the compactors inside, because
in that case you are not testing the production code anymore. Instead, you
should only check if the right chain was created with the right compactors in
the right order. Also, you should check that the fallback Compactor was called
only when MergeCompactor was expected to fail.
Issue Time Tracking
-------------------
Worklog Id: (was: 838362)
Time Spent: 3h 40m (was: 3.5h)
> Merge delta files instead of running a query in major/minor compaction
> ----------------------------------------------------------------------
>
> Key: HIVE-22977
> URL: https://issues.apache.org/jira/browse/HIVE-22977
> Project: Hive
> Issue Type: Improvement
> Reporter: László Pintér
> Assignee: Sourabh Badhya
> Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22977.01.patch, HIVE-22977.02.patch
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> [Compaction Optimiziation]
> We should analyse the possibility to move a delta file instead of running a
> major/minor compaction query.
> Please consider the following use cases:
> - full acid table but only insert queries were run. This means that no
> delete delta directories were created. Is it possible to merge the delta
> directory contents without running a compaction query?
> - full acid table, initiating queries through the streaming API. If there
> are no abort transactions during the streaming, is it possible to merge the
> delta directory contents without running a compaction query?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
