[
https://issues.apache.org/jira/browse/HIVE-26437?focusedWorklogId=822283&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-822283
]
ASF GitHub Bot logged work on HIVE-26437:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 01/Nov/22 12:08
Start Date: 01/Nov/22 12:08
Worklog Time Spent: 10m
Work Description: atsaonerk commented on PR #3644:
URL: https://github.com/apache/hive/pull/3644#issuecomment-1298419541
<meta charset="utf-8"><b style="font-weight:normal;"
id="docs-internal-guid-329ca8fe-7fff-05d0-74b7-0602c56ca25c"><ol
style="margin-top:0;margin-bottom:0;padding-inline-start:48px;"><li dir="ltr"
style="list-style-type:decimal;font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;"
aria-level="1"><p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"
role="presentation"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">Jmh
Performance Benchmark Test </span><span
style="font-size:10pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">(itests/hive-jmh/src/main/java/org/apache/hive/benchmark/ql/exec/TableAndPartitionExportBench.java)</span></p></li></ol><p
dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align:
justify;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:10pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">The
test runs in two modes, one is parallel and other is serial. The parallel mode
makes use of ExportService and creates 100 threads to execute a total of 500
tasks. The serial mode does not use ExportService and runs in single threaded
mode to execute 500 tasks. The comparison is made between both runs. The total
5 iterations of each mode is run by the test and time of run is measured in
milliseconds. The average time required for 5 iterations is then output as a
benchmark for the operations.</span></p><br /><br /><p dir="ltr"
style="line-height:1.38;margin-left: 36pt;text-align:
justify;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:10pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">Results:-</span></p><p
dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align:
justify;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:10pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"></span><span
style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">Results
of jmh performance benchmark test indicates an improvement in TableExport
operation. If the table is dumped in parallel instead of serial , the operation
completes much faster. Below is the result seen when 500 tableexport operations
are done both in serial and parallel manner.</span></p><p dir="ltr"
style="line-height:1.38;margin-left: 36pt;text-align:
justify;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;padding:0pt
0pt 12pt 0pt;"><span
style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">Result
"org.apache.hive.benchmark.ql.exec.TableAndPartitionExportBench.BaseBench.parallel":</span></p><p
dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align:
justify;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;padding:0pt
0pt 12pt 0pt;"><span
style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">N
= 5</span></p><p dir="ltr" style="line-height:1.38;margin-left:
36pt;text-align:
justify;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;padding:0pt
0pt 12pt 0pt;"><span
style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">mean
= 640.862 ?(99.9%) 113.354 ms/op</span></p><p dir="ltr"
style="line-height:1.38;margin-left: 36pt;text-align:
justify;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;padding:0pt
0pt 12pt 0pt;"><span
style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">Result
"org.apache.hive.benchmark.ql.exec.TableAndPartitionExportBench.BaseBench.serial":</span></p><p
dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align:
justify;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;padding:0pt
0pt 12pt 0pt;"><span
style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">N
= 5</span></p><p dir="ltr" style="line-height:1.38;margin-left:
36pt;text-align:
justify;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;padding:0pt
0pt 12pt 0pt;"><span
style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">mean
= 51697.322 ?(99.9%) 322.747 ms/op</span></p><p dir="ltr"
style="line-height:1.38;margin-left: 36pt;text-align:
justify;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">Benchmark
</span><span
style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"><span
class="Apple-tab-span" style="white-space:pre;"> </span></span><span
style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"><span
class="Apple-tab-span" style="white-space:pre;"> </span></span><span
style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"><span
class="Apple-tab-span" style="white-space:pre;"> </span></span><span
style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"><span
class="Apple-tab-span" style="white-space:pre;"> </span></span><span
style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"><span
class="Apple-tab-span" style="white-space:pre;"> </span></span><span
style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">
Mode Cnt Score Error
Units</span></p><p dir="ltr" style="line-height:1.38;margin-left:
36pt;text-align:
justify;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">TableAndPartitionExportBench.BaseBench.parallel
ss 5 640.862 ? 113.354 ms/op</span></p><p
dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align:
justify;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:10.5pt;font-family:Arial;color:#24292f;background-color:#ffff00;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">TableAndPartitionExportBench.BaseBench.serial
ss 5 51697.322 ? 322.747 ms/op</span></p><p
dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align:
justify;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;"> </p><p
dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align:
justify;background-color:#ffffff;margin-top:0pt;margin-bottom:0pt;"> </p><ol
style="margin-top:0;margin-bottom:0;padding-inline-start:48px;" start="2"><li
dir="ltr"
style="list-style-type:decimal;font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;"
aria-level="1"><p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"
role="presentation"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">End-End
Performance benchmark number </span></p></li></ol><p dir="ltr"
style="line-height:1.38;margin-left:
36pt;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;"> </span></p><p
dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align:
justify;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:10pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">A
database is created with 1k managed acid tables which are all unpartitioned
tables.</span></p><p dir="ltr" style="line-height:1.38;margin-left:
36pt;text-align: justify;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:10pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">The
config parameter REPL_TABLE_DUMP_PARALLELISM value of 100 is set before the
replication dump command is executed. It is found that replication dump takes 9
sec to complete table metadata dump with new Export service. </span></p><p
dir="ltr" style="line-height:1.38;margin-left: 36pt;text-align:
justify;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:10pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">When
export service is not used it is found that the same number of tables took
around 27 sec to complete the entire dump process. Hence it is seen that there
is 3x improvement in performance of replication dump command
execution.</span></p><br /><div dir="ltr" style="margin-left:36pt;"
align="left">
No. of Tables | REPL_TABLE_DUMP_PARALLELISM | Export Service used | Time
taken for REPL DUMP
Issue Time Tracking
-------------------
Worklog Id: (was: 822283)
Time Spent: 2h 50m (was: 2h 40m)
> dump unpartitioned managed table metadata in parallel
> -----------------------------------------------------
>
> Key: HIVE-26437
> URL: https://issues.apache.org/jira/browse/HIVE-26437
> Project: Hive
> Issue Type: Improvement
> Components: Hive
> Reporter: Amit Saonerkar
> Assignee: Amit Saonerkar
> Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
