ninsmiracle opened a new issue, #1564:
URL: https://github.com/apache/incubator-pegasus/issues/1564
## Bug Report
In online usage, Pegasus users often use Spark for offline data bulk
loading. In the configuration of Spark tasks, users can set the
`manual_compact.periodic.trigger_time parameter` to define the desired time for
manual compaction, avoiding peak traffic periods. For example, a user generates
a batch of data during the day (2 PM) and imports it using Spark, expecting to
perform a manual compaction to reduce disk usage at 1 AM the next day.
However, in the existing code logic now, when a user performs the Bulkload
operation and sets the `manual_compact.periodic.trigger_time`, if the specified
time (e.g., 01:00) is earlier than the current time (14:00), an immediate
manual compaction is triggered. This unexpected behavior leads to two problems:
1. The user expects to perform two bulk loading operations in a short time.
The unexpected manual compaction may prevent the second bulk load from starting.
2. In pegasus duplication, when using full-app duplicate, all parameters of
the primary cluster's original app are copied to the corresponding app in the
backup cluster. If the primary cluster's table has undergone manual compaction
using Spark, the parameters related to manual compaction are also copied.
**This leads to the backup cluster's RocksDB instances performing manual
compaction before they are fully initialized, resulting in undefined behavior
and ultimately causing all nodes in the backup cluster to crash.** Although the
SRE can delete all manual compaction parameters of the original app before
enabling full duplicate, forgetting to do so will result in serious issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
