mihai-cazacu-adswizz edited a comment on issue #8063: The compaction task fails without errors URL: https://github.com/apache/incubator-druid/issues/8063#issuecomment-514917428 Another compaction task has just failed (after 16 hours of running). Again, no errors in its log or in the overlord's log. This time I'm using `Druid 0.13.0`. The last lines from the compaction's log: ``` 2019-07-25T03:46:08,296 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[1] in 1 millis. 2019-07-25T03:46:08,298 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[2] in 1 millis. 2019-07-25T03:46:08,299 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,299 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,300 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[1] in 1 millis. 2019-07-25T03:46:08,301 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 1 millis. 2019-07-25T03:46:08,301 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,301 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[1] in 0 millis. 2019-07-25T03:46:08,302 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 1 millis. 2019-07-25T03:46:08,302 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,302 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,302 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,302 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,302 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[1] in 0 millis. 2019-07-25T03:46:08,303 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,303 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,303 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,303 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,303 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,303 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,303 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,303 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,303 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,304 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[3] in 1 millis. 2019-07-25T03:46:08,305 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,305 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,306 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[1] in 1 millis. 2019-07-25T03:46:08,306 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,306 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,306 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,306 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,306 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,306 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,306 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,306 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,307 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 1 millis. 2019-07-25T03:46:08,307 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,307 INFO [some-ds-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[SOME_DIM] inverted with cardinality[0] in 0 millis. 2019-07-25T03:46:08,307 INFO [some-ds-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed index.drd in 0 millis. 2019-07-25T03:46:08,307 INFO [some-ds-incremental-persist] org.apache.druid.java.util.common.io.smoosh.FileSmoosher - Created smoosh file [/mnt/druid/persistent/task/compact_some-ds_2019-07-24T11:44:40.947Z/work/persist/some-ds_2018-10-01T00:00:00.000Z_2018-11-01T00:00:00.000Z_2019-07-24T11:44:40.976Z_1423/0/00000.smoosh] of size [59994686] bytes. 57688.578: [GC pause (G1 Evacuation Pause) (young), 0.1909283 secs] [Parallel Time: 183.8 ms, GC Workers: 8] [GC Worker Start (ms): Min: 57688577.9, Avg: 57688578.0, Max: 57688578.1, Diff: 0.2] [Ext Root Scanning (ms): Min: 0.9, Avg: 4.3, Max: 11.8, Diff: 10.9, Sum: 34.4] [Update RS (ms): Min: 17.6, Avg: 22.9, Max: 33.7, Diff: 16.1, Sum: 183.0] [Processed Buffers: Min: 7, Avg: 39.4, Max: 63, Diff: 56, Sum: 315] [Scan RS (ms): Min: 37.5, Avg: 47.9, Max: 53.3, Diff: 15.8, Sum: 383.2] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] [Object Copy (ms): Min: 79.6, Avg: 95.7, Max: 108.6, Diff: 29.0, Sum: 765.7] [Termination (ms): Min: 0.0, Avg: 10.0, Max: 12.4, Diff: 12.4, Sum: 80.2] [Termination Attempts: Min: 1, Avg: 761.1, Max: 1306, Diff: 1305, Sum: 6089] [GC Worker Other (ms): Min: 0.0, Avg: 1.1, Max: 2.9, Diff: 2.9, Sum: 8.7] [GC Worker Total (ms): Min: 180.7, Avg: 181.9, Max: 183.7, Diff: 2.9, Sum: 1455.2] [GC Worker End (ms): Min: 57688758.8, Avg: 57688759.9, Max: 57688761.7, Diff: 2.9] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.7 ms] [Other: 6.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 4.1 ms] [Ref Enq: 0.0 ms] [Redirty Cards: 1.0 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.2 ms] [Free CSet: 0.6 ms] [Eden: 294.0M(294.0M)->0.0B(474.0M) Survivors: 166.0M->30.0M Heap: 4307.0M(9216.0M)->4041.0M(9216.0M)] [Times: user=0.83 sys=0.00, real=0.19 secs] 57719.479: [GC pause (G1 Evacuation Pause) (young) (initial-mark), 0.1198657 secs] [Parallel Time: 115.9 ms, GC Workers: 8] [GC Worker Start (ms): Min: 57719479.6, Avg: 57719481.8, Max: 57719482.6, Diff: 3.0] [Ext Root Scanning (ms): Min: 0.1, Avg: 12.3, Max: 22.1, Diff: 22.0, Sum: 98.1] [Update RS (ms): Min: 0.0, Avg: 17.6, Max: 52.1, Diff: 52.1, Sum: 140.9] [Processed Buffers: Min: 0, Avg: 22.0, Max: 45, Diff: 45, Sum: 176] [Scan RS (ms): Min: 0.0, Avg: 6.2, Max: 16.9, Diff: 16.9, Sum: 49.9] [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] [Object Copy (ms): Min: 49.6, Avg: 67.5, Max: 76.0, Diff: 26.3, Sum: 539.7] [Termination (ms): Min: 0.0, Avg: 6.7, Max: 10.2, Diff: 10.2, Sum: 53.6] [Termination Attempts: Min: 1, Avg: 1.4, Max: 3, Diff: 2, Sum: 11] [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.1] [GC Worker Total (ms): Min: 108.8, Avg: 110.3, Max: 114.1, Diff: 5.3, Sum: 882.4] [GC Worker End (ms): Min: 57719591.4, Avg: 57719592.1, Max: 57719594.6, Diff: 3.3] [Code Root Fixup: 0.0 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.5 ms] [Other: 3.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 0.7 ms] [Ref Enq: 0.0 ms] [Redirty Cards: 0.4 ms] [Humongous Register: 0.1 ms] [Humongous Reclaim: 0.1 ms] [Free CSet: 0.5 ms] [Eden: 474.0M(474.0M)->0.0B(396.0M) Survivors: 30.0M->64.0M Heap: 4515.0M(9216.0M)->4123.0M(9216.0M)] [Times: user=0.34 sys=0.00, real=0.12 secs] 57719.599: [GC concurrent-root-region-scan-start] 57719.654: [GC concurrent-root-region-scan-end, 0.0549703 secs] 57719.654: [GC concurrent-mark-start] 57719.924: [GC concurrent-mark-end, 0.2701431 secs] 57720.062: [GC remark 57720.062: [Finalize Marking, 0.0006635 secs] 57720.062: [GC ref-proc, 0.0007011 secs] 57720.063: [Unloading, 0.0149653 secs], 0.0196835 secs] [Times: user=0.06 sys=0.00, real=0.02 secs] 57720.082: [GC cleanup 4127M->902M(9216M), 0.0081479 secs] [Times: user=0.03 sys=0.00, real=0.00 secs] 57720.091: [GC concurrent-cleanup-start] 57720.096: [GC concurrent-cleanup-end, 0.0053799 secs] ``` The spec: ``` { "type": "compact", "dataSource": "some-ds", "interval": "2018-10-01/2018-11-01", "keepSegmentGranularity": false, "segmentGranularity": "month", "queryGranularity": "day", "targetCompactionSizeBytes": 80000000, "tuningConfig": { "type": "index", "maxRowsInMemory": 25000, "maxBytesInMemory": 2147483647, "forceExtendableShardSpecs": true } } ``` The intervals: ``` 2018-10-29T00 | 67 | 55 GB (67 segments (55 GB) for 2018-10-29T00 to 2018-11-05T00) 2018-10-01T00 | 31 | 147 GB (31 segments (147 GB) for 2018-10-01T00 to 2018-11-01T00 ``` I've also added these JVM params to the Peons: ``` -Xmx9g -XX:MaxDirectMemorySize=12g ``` Is there anything that I can do in order to find the error cause?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
