[
https://issues.apache.org/jira/browse/KUDU-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841195#comment-17841195
]
Ashwani Raina commented on KUDU-3568:
-------------------------------------
Thank you reporting this [~aserbin]
Here the compaction went ahead even with high memory budget requirement
(atleast as per test expectations)
The test bases it memory requirements on number of factors, one of which is
{{{}rowset_compaction_delta_memory_factor{}}}.
It sets it to a very high value so that node on which test is running is not
able to satisfy the memory requirements and hence skipping of compaction
becomes necessary.
I think the failure is caused due to a bug in the test where we are relying on
{{rowset_compaction_delta_memory_factor}} without setting
{{rowset_compaction_enforce_preset_factor}} to true.
Since {{rowset_compaction_enforce_preset_factor}} is set to false by default,
which means compaction policy relies on metrics to come up with memory
budgeting numbers.
It used to pass because in most runs, the metrics (i.e.
compact_rs_mem_usage_to_deltas_size_ratio) that compaction policy relied on
used to be 0.
Due to this, memory budgeting logic chose the
{{rowset_compaction_delta_memory_factor}} (set to a very high value in this
test) as the factor and since the node doesn't have so much memory, it used to
skip compaction as expected.
Essentially, using {{rowset_compaction_delta_memory_factor}} in tests makes
sense only when {{rowset_compaction_enforce_preset_factor}} is true.
If it is set to false, test will rely on metrics instead of test defined memory
factor.
This is most likely the root-cause here. If you have a way to repro this, I can
try verifying it.
> TestRowSetCompactionSkipWithBudgetingConstraints fails when run on some nodes
> -----------------------------------------------------------------------------
>
> Key: KUDU-3568
> URL: https://issues.apache.org/jira/browse/KUDU-3568
> Project: Kudu
> Issue Type: Bug
> Affects Versions: 1.18.0
> Reporter: Alexey Serbin
> Assignee: Ashwani Raina
> Priority: Major
> Attachments: test-failure.log.xz
>
>
> The {{TestCompaction.TestRowSetCompactionSkipWithBudgetingConstraints}}
> scenario fails with the error like below when run on a machine with
> relatively high memory (it might be just a Docker instance with tiny actual
> memory allocated, but having the access to the {{/proc}} filesystem of the
> host machine). The full test log is attached.
> {noformat}
> src/kudu/tablet/compaction-test.cc:908: Failure
> Value of: JoinStrings(sink.logged_msgs(), "\n")
> Expected: has substring "removed from compaction input due to memory
> constraints"
> Actual: "I20240425 10:13:05.497732 3573764 compaction-test.cc:902]
> CompactRowSetsOp complete. Timing: real 0.673s\tuser 0.669s\tsys 0.004s
> Metrics:
> {\"bytes_written\":4817,\"cfile_cache_hit\":90,\"cfile_cache_hit_bytes\":4310,\"cfile_cache_miss\":330,\"cfile_cache_miss_bytes\":3794180,\"cfile_init\":41,\"delta_iterators_relevant\":40,\"dirs.queue_time_us\":503,\"dirs.run_cpu_time_us\":338,\"dirs.run_wall_time_us\":1780,\"drs_written\":1,\"lbm_read_time_us\":1951,\"lbm_reads_lt_1ms\":494,\"lbm_write_time_us\":1767,\"lbm_writes_lt_1ms\":132,\"mutex_wait_us\":189,\"num_input_rowsets\":10,\"peak_mem_usage\":2147727,\"rows_written\":20,\"thread_start_us\":242,\"threads_started\":5}"
> (of type std::string)
> {noformat}
> For extra information, below is 10 lines from {{/proc/meminfo}} file on a
> node where the test failed:
> {noformat}
> # cat /proc/meminfo | head -10
> MemTotal: 527417196 kB
> MemFree: 96640684 kB
> MemAvailable: 363590980 kB
> Buffers: 15352304 kB
> Cached: 246687576 kB
> SwapCached: 1294016 kB
> Active: 214889608 kB
> Inactive: 189745504 kB
> Active(anon): 133110648 kB
> Inactive(anon): 16977280 kB
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
