maytasm commented on a change in pull request #10371:
URL: https://github.com/apache/druid/pull/10371#discussion_r489217716
##########
File path:
server/src/main/java/org/apache/druid/server/coordinator/duty/NewestSegmentFirstIterator.java
##########
@@ -336,25 +353,72 @@ private boolean
needsCompaction(ClientCompactionTaskQueryTuningConfig tuningConf
* @return segments to compact
*/
private SegmentsToCompact findSegmentsToCompact(
+ final String dataSourceName,
final CompactibleTimelineObjectHolderCursor
compactibleTimelineObjectHolderCursor,
final DataSourceCompactionConfig config
)
{
- final long inputSegmentSize = config.getInputSegmentSizeBytes();
+ while (compactibleTimelineObjectHolderCursor.hasNext()) {
+ final SegmentsToCompact candidates = new
SegmentsToCompact(compactibleTimelineObjectHolderCursor.next());
+ if (isSegmentsNeedCompact(candidates, config, true)) {
+ return candidates;
+ } else {
+ collectSegmentStatistics(processedSegments, dataSourceName,
candidates);
+ }
+ }
+ log.info("All segments look good! Nothing to compact");
+ return new SegmentsToCompact();
+ }
+ /**
+ * Progressively iterates all remaining time intervals (latest first) in the
+ * timeline {@param compactibleTimelineObjectHolderCursor}. Note that the
timeline lookup duration is one day.
+ * The logic for checking if the segments can be compacted or not is then
perform on each iteration.
+ * This is repeated until no remaining time intervals in {@param
compactibleTimelineObjectHolderCursor}.
+ */
+ private void iterateAllSegments(
+ final String dataSourceName,
+ final CompactibleTimelineObjectHolderCursor
compactibleTimelineObjectHolderCursor,
+ final DataSourceCompactionConfig config
+ )
+ {
while (compactibleTimelineObjectHolderCursor.hasNext()) {
final SegmentsToCompact candidates = new
SegmentsToCompact(compactibleTimelineObjectHolderCursor.next());
+ if (isSegmentsNeedCompact(candidates, config, false)) {
+ // Collect statistic for segments that need compaction
+ collectSegmentStatistics(remainingSegments, dataSourceName,
candidates);
+ } else {
+ // Collect statistic for segments that does not need compaction
+ collectSegmentStatistics(processedSegments, dataSourceName,
candidates);
Review comment:
I agree. I split the processedSegments into compactedSegments and
skippedSegments. Also added new fields in autoCompactionSnapshot for the
skipped byte, skipped interval count and skipped segment counts
##########
File path:
server/src/main/java/org/apache/druid/server/coordinator/duty/EmitClusterStatsAndMetrics.java
##########
@@ -301,13 +301,82 @@ public DruidCoordinatorRuntimeParams
run(DruidCoordinatorRuntimeParams params)
)
);
+ emitter.emit(
+ new ServiceMetricEvent.Builder().build(
+ "compact/maxSlot/count",
+ stats.getGlobalStat(CompactSegments.MAX_COMPACTION_TASK_SLOT)
+ )
+ );
+
+ emitter.emit(
+ new ServiceMetricEvent.Builder().build(
+ "compact/availableSlot/count",
+ stats.getGlobalStat(CompactSegments.AVAILABLE_COMPACTION_TASK_SLOT)
+ )
+ );
+
+ stats.forEachDataSourceStat(
+ CompactSegments.TOTAL_SIZE_OF_SEGMENTS_AWAITING_COMPACTION,
+ (final String dataSource, final long count) -> {
+ emitter.emit(
+ new ServiceMetricEvent.Builder()
+ .setDimension(DruidMetrics.DATASOURCE, dataSource)
+ .build("segment/waitCompact/segmentByte", count)
+ );
+ }
+ );
+
+ stats.forEachDataSourceStat(
+ CompactSegments.TOTAL_COUNT_OF_SEGMENTS_AWAITING_COMPACTION,
+ (final String dataSource, final long count) -> {
+ emitter.emit(
+ new ServiceMetricEvent.Builder()
+ .setDimension(DruidMetrics.DATASOURCE, dataSource)
+ .build("segment/waitCompact/segmentCount", count)
+ );
+ }
+ );
+
+ stats.forEachDataSourceStat(
+ CompactSegments.TOTAL_INTERVAL_OF_SEGMENTS_AWAITING_COMPACTION,
+ (final String dataSource, final long count) -> {
+ emitter.emit(
+ new ServiceMetricEvent.Builder()
+ .setDimension(DruidMetrics.DATASOURCE, dataSource)
+ .build("segment/waitCompact/intervalCount", count)
+ );
+ }
+ );
+
+ stats.forEachDataSourceStat(
+ CompactSegments.TOTAL_SIZE_OF_SEGMENTS_COMPACTED,
+ (final String dataSource, final long count) -> {
+ emitter.emit(
+ new ServiceMetricEvent.Builder()
+ .setDimension(DruidMetrics.DATASOURCE, dataSource)
+ .build("segment/compacted/segmentByte", count)
+ );
+ }
+ );
+
+ stats.forEachDataSourceStat(
+ CompactSegments.TOTAL_COUNT_OF_SEGMENTS_COMPACTED,
+ (final String dataSource, final long count) -> {
+ emitter.emit(
+ new ServiceMetricEvent.Builder()
+ .setDimension(DruidMetrics.DATASOURCE, dataSource)
+ .build("segment/compacted/segmentCount", count)
Review comment:
Added metrics for skipped byte, skipped interval count and skipped
segment counts
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
