[jira] [Commented] (KUDU-1736) kudu crash in debug build: unordered undo delta
[ https://issues.apache.org/jira/browse/KUDU-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17004358#comment-17004358 ] Alexey Serbin commented on KUDU-1736: - Another occurrence if pre-commit build (ASAN). Attaching the log. [^mt-tablet-test-20191227.txt.xz] > kudu crash in debug build: unordered undo delta > --- > > Key: KUDU-1736 > URL: https://issues.apache.org/jira/browse/KUDU-1736 > Project: Kudu > Issue Type: Bug > Components: tablet >Affects Versions: 1.7.0, 1.8.0, 1.7.1, 1.9.0, 1.10.0, 1.10.1, 1.12.0, > 1.11.1 >Reporter: zhangsong >Priority: Critical > Attachments: mt-tablet-test-20171123.txt.xz, > mt-tablet-test-20191227.txt.xz, mt-tablet-test.1.txt.xz, > mt-tablet-test.3.txt, mt-tablet-test.txt, mt-tablet-test.txt.gz > > > in jd cluster we met kudu-tserver crash with fatal messages described as > follow: > Check failed: last_key_.CompareTo(key) <= 0 must insert undo deltas in > sorted order (ascending key, then descending ts): got key (row > 1422@tx6052042821982183424) after (row 1422@tx6052042821953155072) > This is a dcheck which should not failed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KUDU-1736) kudu crash in debug build: unordered undo delta
[ https://issues.apache.org/jira/browse/KUDU-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854236#comment-16854236 ] Alexey Serbin commented on KUDU-1736: - Just another occurence. [^mt-tablet-test.1.txt.xz] > kudu crash in debug build: unordered undo delta > --- > > Key: KUDU-1736 > URL: https://issues.apache.org/jira/browse/KUDU-1736 > Project: Kudu > Issue Type: Bug > Components: tablet >Reporter: zhangsong >Priority: Critical > Attachments: mt-tablet-test-20171123.txt.xz, mt-tablet-test.1.txt.xz, > mt-tablet-test.3.txt, mt-tablet-test.txt, mt-tablet-test.txt.gz > > > in jd cluster we met kudu-tserver crash with fatal messages described as > follow: > Check failed: last_key_.CompareTo(key) <= 0 must insert undo deltas in > sorted order (ascending key, then descending ts): got key (row > 1422@tx6052042821982183424) after (row 1422@tx6052042821953155072) > This is a dcheck which should not failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KUDU-1736) kudu crash in debug build: unordered undo delta
[ https://issues.apache.org/jira/browse/KUDU-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754245#comment-16754245 ] Will Berkeley commented on KUDU-1736: - This still happens. [^mt-tablet-test.3.txt] > kudu crash in debug build: unordered undo delta > --- > > Key: KUDU-1736 > URL: https://issues.apache.org/jira/browse/KUDU-1736 > Project: Kudu > Issue Type: Bug > Components: tablet >Reporter: zhangsong >Priority: Critical > Attachments: mt-tablet-test-20171123.txt.xz, mt-tablet-test.3.txt, > mt-tablet-test.txt, mt-tablet-test.txt.gz > > > in jd cluster we met kudu-tserver crash with fatal messages described as > follow: > Check failed: last_key_.CompareTo(key) <= 0 must insert undo deltas in > sorted order (ascending key, then descending ts): got key (row > 1422@tx6052042821982183424) after (row 1422@tx6052042821953155072) > This is a dcheck which should not failed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KUDU-1736) kudu crash in debug build: unordered undo delta
[
https://issues.apache.org/jira/browse/KUDU-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263974#comment-16263974
]
Alexey Serbin commented on KUDU-1736:
-
And another instance of this bug (debug build, as well):
{noformat}
F1123 08:22:13.245149 9905 delta_tracker.cc:234] Check failed:
first_copy->delta_stats().min_timestamp() >=
second_copy->delta_stats().min_timestamp() (7227 vs. 7229) Found out-of-order
deltas: [{2351438349105711757 (ts range=[7227, 7366], delete_count=[0],
reinsert_count=[0], update_counts_by_col_id=[12:13))}, {2351438349105711740 (ts
range=[7229, 7300], delete_count=[0], reinsert_count=[0],
update_counts_by_col_id=[12:16))}]: type = 1
{noformat}
The full log is attached as mt-tablet-test-20171123.txt
> kudu crash in debug build: unordered undo delta
> ---
>
> Key: KUDU-1736
> URL: https://issues.apache.org/jira/browse/KUDU-1736
> Project: Kudu
> Issue Type: Bug
> Components: tablet
>Reporter: zhangsong
>Priority: Critical
> Attachments: mt-tablet-test.txt, mt-tablet-test.txt.gz
>
>
> in jd cluster we met kudu-tserver crash with fatal messages described as
> follow:
> Check failed: last_key_.CompareTo(key) <= 0 must insert undo deltas in
> sorted order (ascending key, then descending ts): got key (row
> 1422@tx6052042821982183424) after (row 1422@tx6052042821953155072)
> This is a dcheck which should not failed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KUDU-1736) kudu crash in debug build: unordered undo delta
[ https://issues.apache.org/jira/browse/KUDU-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16012757#comment-16012757 ] Will Berkeley commented on KUDU-1736: - Seen again in this debug jenkins build dist-test run: http://dist-test.cloudera.org/job?job_id=jenkins-slave.1494953882.31448 > kudu crash in debug build: unordered undo delta > --- > > Key: KUDU-1736 > URL: https://issues.apache.org/jira/browse/KUDU-1736 > Project: Kudu > Issue Type: Bug > Components: tablet >Reporter: zhangsong >Priority: Critical > Attachments: mt-tablet-test.txt.gz > > > in jd cluster we met kudu-tserver crash with fatal messages described as > follow: > Check failed: last_key_.CompareTo(key) <= 0 must insert undo deltas in > sorted order (ascending key, then descending ts): got key (row > 1422@tx6052042821982183424) after (row 1422@tx6052042821953155072) > This is a dcheck which should not failed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KUDU-1736) kudu crash in debug build: unordered undo delta
[
https://issues.apache.org/jira/browse/KUDU-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905847#comment-15905847
]
Andrew Wong commented on KUDU-1736:
---
Also saw this on Jenkins:
```
delta_tracker.cc:208] Check failed: first_copy->delta_stats().min_timestamp()
>= second_copy->delta_stats().min_timestamp() (5678 vs. 5681) Found
out-of-order deltas: [{11440216355432745334 (ts range=[5678, 5775],
delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[12:3))},
{11440216355432745315 (ts range=[5681, 5761], delete_count=[0],
reinsert_count=[0], update_counts_by_col_id=[12:18))}]: type = 1
@ 0x7f4f0ef11c37 gsignal at ??:0
@ 0x7f4f0ef15028 abort at ??:0
@ 0x7f4f1bc02ae4 kudu::tablet::DeltaTracker::ValidateDeltaOrder() at
??:0
@ 0x7f4f1bc031c5 kudu::tablet::DeltaTracker::AtomicUpdateStores() at
??:0
@ 0x7f4f1bbe8304
kudu::tablet::MajorDeltaCompaction::UpdateDeltaTracker() at ??:0
@ 0x7f4f1baf7e35
kudu::tablet::DiskRowSet::MajorCompactDeltaStoresWithColumnIds() at ??:0
@ 0x7f4f1baf7811 kudu::tablet::DiskRowSet::MajorCompactDeltaStores() at
??:0
@ 0x7f4f1b9d7687 kudu::tablet::Tablet::CompactWorstDeltas() at ??:0
@ 0x5e5cbc
kudu::tablet::MultiThreadedTabletTest<>::CompactDeltas() at
/home/jenkins-slave/workspace/kudu-master/2/src/kudu/tablet/mt-tablet-test.cc:307
(discriminator 17)
@ 0x5e48e8 boost::_bi::bind_t<>::operator()() at
/home/jenkins-slave/workspace/kudu-master/2/thirdparty/installed/common/include/boost/bind/bind.hpp:1223
@ 0x7f4f143a5d3f boost::function0<>::operator()() at ??:0
@ 0x7f4f121565e4 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4f17da4184 start_thread at ??:0
@ 0x7f4f0efd537d clone at ??:0
```
Failure log can be found here:
http://104.196.14.100/job/kudu-gerrit/6963/BUILD_TYPE=ASAN/
> kudu crash in debug build: unordered undo delta
> ---
>
> Key: KUDU-1736
> URL: https://issues.apache.org/jira/browse/KUDU-1736
> Project: Kudu
> Issue Type: Bug
> Components: tablet
>Reporter: zhangsong
>Priority: Critical
> Attachments: mt-tablet-test.txt.gz
>
>
> in jd cluster we met kudu-tserver crash with fatal messages described as
> follow:
> Check failed: last_key_.CompareTo(key) <= 0 must insert undo deltas in
> sorted order (ascending key, then descending ts): got key (row
> 1422@tx6052042821982183424) after (row 1422@tx6052042821953155072)
> This is a dcheck which should not failed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KUDU-1736) kudu crash in debug build: unordered undo delta
[
https://issues.apache.org/jira/browse/KUDU-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15895385#comment-15895385
]
Mike Percy commented on KUDU-1736:
--
I think I saw this on Jenkins again, now triggered by the additional validation
I added a few weeks ago:
{code}
delta_tracker.cc:208] Check failed: first_copy->delta_stats().min_timestamp()
>= second_copy->delta_stats().min_timestamp() (2931 vs. 2932) Found
out-of-order deltas: [{1644469961406533227 (ts range=[2931, 3022],
delete_count=[0], reinsert_count=[0], update_counts_by_col_id=[12:9))},
{1644469961406533202 (ts range=[2932, 2941], delete_count=[0],
reinsert_count=[0], update_counts_by_col_id=[12:10))}]: type = 1
@ 0x7f6832bfcc37 gsignal at ??:0
@ 0x7f6832c00028 abort at ??:0
@ 0x7f68385bf492 kudu::tablet::DeltaTracker::ValidateDeltaOrder() at
??:0
@ 0x7f68385bfc79 kudu::tablet::DeltaTracker::AtomicUpdateStores() at
??:0
@ 0x7f68385b150f
kudu::tablet::MajorDeltaCompaction::UpdateDeltaTracker() at ??:0
@ 0x7f6838530c2b
kudu::tablet::DiskRowSet::MajorCompactDeltaStoresWithColumnIds() at ??:0
@ 0x7f683853060e kudu::tablet::DiskRowSet::MajorCompactDeltaStores() at
??:0
@ 0x7f683849498f kudu::tablet::Tablet::CompactWorstDeltas() at ??:0
@ 0x495eaa
kudu::tablet::MultiThreadedTabletTest<>::CompactDeltas() at
/home/jenkins-slave/workspace/kudu-master/0/src/kudu/tablet/mt-tablet-test.cc:307
@ 0x47fe9e
kudu::tablet::MultiThreadedTabletTest<>::MajorCompactDeltasThread() at
/home/jenkins-slave/workspace/kudu-master/0/src/kudu/tablet/mt-tablet-test.cc:301
@ 0x4a7371 boost::_mfi::mf1<>::operator()() at
/home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/uninstrumented/include/boost/bind/mem_fn_template.hpp:166
(discriminator 3)
@ 0x4a6989 boost::_bi::list2<>::operator()<>() at
/home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/uninstrumented/include/boost/bind/bind.hpp:320
@ 0x4a5f1d boost::_bi::bind_t<>::operator()() at
/home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/uninstrumented/include/boost/bind/bind.hpp:1223
@ 0x4a5890
boost::detail::function::void_function_obj_invoker0<>::invoke() at
/home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/uninstrumented/include/boost/function/function_template.hpp:160
@ 0x7f6834855874 boost::function0<>::operator()() at ??:0
@ 0x7f68348a5239 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f68322e5184 start_thread at ??:0
@ 0x7f6832cc037d clone at ??:0
{code}
> kudu crash in debug build: unordered undo delta
> ---
>
> Key: KUDU-1736
> URL: https://issues.apache.org/jira/browse/KUDU-1736
> Project: Kudu
> Issue Type: Bug
>Reporter: zhangsong
>
> in jd cluster we met kudu-tserver crash with fatal messages described as
> follow:
> Check failed: last_key_.CompareTo(key) <= 0 must insert undo deltas in
> sorted order (ascending key, then descending ts): got key (row
> 1422@tx6052042821982183424) after (row 1422@tx6052042821953155072)
> This is a dcheck which should not failed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KUDU-1736) kudu crash in debug build: unordered undo delta
[ https://issues.apache.org/jira/browse/KUDU-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15634917#comment-15634917 ] zhangsong commented on KUDU-1736: - Sorry, i forgot to dump the log, will check whether the log still exist (depends on whether our regular log cleaner has delete it). > kudu crash in debug build: unordered undo delta > --- > > Key: KUDU-1736 > URL: https://issues.apache.org/jira/browse/KUDU-1736 > Project: Kudu > Issue Type: Bug >Reporter: zhangsong > > in jd cluster we met kudu-tserver crash with fatal messages described as > follow: > Check failed: last_key_.CompareTo(key) <= 0 must insert undo deltas in > sorted order (ascending key, then descending ts): got key (row > 1422@tx6052042821982183424) after (row 1422@tx6052042821953155072) > This is a dcheck which should not failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KUDU-1736) kudu crash in debug build: unordered undo delta
[ https://issues.apache.org/jira/browse/KUDU-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15626896#comment-15626896 ] Todd Lipcon commented on KUDU-1736: --- Do you still have INFO logs available from this tablet server around the timestamp '2016-10-26 23:25:17'? The two operations were assigned timestamps at 2016-10-26 23:25:17.085494 and 2016-10-26 23:25:17.078407, so would be interesting to know if the tablet was undergoing a compaction, etc, during that time range. > kudu crash in debug build: unordered undo delta > --- > > Key: KUDU-1736 > URL: https://issues.apache.org/jira/browse/KUDU-1736 > Project: Kudu > Issue Type: Bug >Reporter: zhangsong > > in jd cluster we met kudu-tserver crash with fatal messages described as > follow: > Check failed: last_key_.CompareTo(key) <= 0 must insert undo deltas in > sorted order (ascending key, then descending ts): got key (row > 1422@tx6052042821982183424) after (row 1422@tx6052042821953155072) > This is a dcheck which should not failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
