[
https://issues.apache.org/jira/browse/HIVE-22977?focusedWorklogId=838782&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-838782
]
ASF GitHub Bot logged work on HIVE-22977:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 12/Jan/23 08:06
Start Date: 12/Jan/23 08:06
Worklog Time Spent: 10m
Work Description: SourabhBadhya commented on code in PR #3801:
URL: https://github.com/apache/hive/pull/3801#discussion_r1067811008
##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java:
##########
@@ -3189,11 +3302,42 @@ private void testCompactionWithMerge(CompactionType
compactionType, boolean isPa
// Verify delete delta
Assert.assertTrue(CompactorTestUtil.getBaseOrDeltaNames(fs,
AcidUtils.deleteEventDeltaDirFilter, table,
partitionName).isEmpty());
- // Run a compaction
- if (partitionName == null) {
- CompactorTestUtil.runCompaction(conf, dbName, tableName, compactionType,
true);
- } else {
- CompactorTestUtil.runCompaction(conf, dbName, tableName, compactionType,
true, partitionName);
+ // Run a compaction which uses only merge compactor / query-based
compaction
+ CompactorFactory compactorFactory = mock(CompactorFactory.class);
+ // Return mocked instance of CompactorFactory
+ try (MockedStatic mockedStatic = mockStatic(CompactorFactory.class)) {
+
mockedStatic.when(CompactorFactory::getInstance).thenReturn(compactorFactory);
+ Compactor compactor;
+ List<Compactor> compactorList = new ArrayList<>();
+ if (useFallback) {
+ // Create a mock of merge compactor
+ compactor = mock(MergeCompactor.class);
+ compactorList.add(compactor);
+ if (compactionType == CompactionType.MAJOR) {
+ compactorList.add(new MajorQueryCompactor());
+ } else {
+ compactorList.add(new MinorQueryCompactor());
+ }
+ } else {
+ // Create a mock of major/minor compactor
+ if (compactionType == CompactionType.MAJOR) {
+ compactor = mock(MajorQueryCompactor.class);
+ } else {
+ compactor = mock(MinorQueryCompactor.class);
+ }
+ compactorList.add(new MergeCompactor());
+ compactorList.add(compactor);
+ }
+ // Mocked compaction always returns false
+ when(compactor.run(any(), any(), any(), any(), any(), any(),
any())).thenReturn(false);
+ CompactorChain compactorChain = new CompactorChain(compactorList);
+ // Return mocked compactor chain
+ when(compactorFactory.getCompactor(any(), any(), any(),
any())).thenReturn(compactorChain);
+ if (partitionName == null) {
+ CompactorTestUtil.runCompaction(conf, dbName, tableName,
compactionType, true);
+ } else {
+ CompactorTestUtil.runCompaction(conf, dbName, tableName,
compactionType, true, partitionName);
+ }
Review Comment:
Implemented using spies and made sure we are not creating a compactor chain
again. Fetched the array of compactors using reflection and changed the
compactor objects to corresponding spies.
Issue Time Tracking
-------------------
Worklog Id: (was: 838782)
Time Spent: 4h (was: 3h 50m)
> Merge delta files instead of running a query in major/minor compaction
> ----------------------------------------------------------------------
>
> Key: HIVE-22977
> URL: https://issues.apache.org/jira/browse/HIVE-22977
> Project: Hive
> Issue Type: Improvement
> Reporter: László Pintér
> Assignee: Sourabh Badhya
> Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22977.01.patch, HIVE-22977.02.patch
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> [Compaction Optimiziation]
> We should analyse the possibility to move a delta file instead of running a
> major/minor compaction query.
> Please consider the following use cases:
> - full acid table but only insert queries were run. This means that no
> delete delta directories were created. Is it possible to merge the delta
> directory contents without running a compaction query?
> - full acid table, initiating queries through the streaming API. If there
> are no abort transactions during the streaming, is it possible to merge the
> delta directory contents without running a compaction query?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
