[
https://issues.apache.org/jira/browse/KUDU-1573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15430345#comment-15430345
]
zhangsong edited comment on KUDU-1573 at 8/22/16 9:05 AM:
----------------------------------------------------------
before log message in description, there are logs:
Log segment file Log segment file
/export/servers/kudu/tserver_wal_data_7052/wals/ed0d8b3a835e4c27afe695252ad0b8f5.recove
ry/wal-000000018 has 12 initial NULL bytes instead of magic and header length:
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 and will be treated a
s a blank segment.
W0819 14:16:07.722328 20997 log_util.cc:311] Could not read footer for segment:
/export/servers/kudu/tserver_wal_data_7052/wals/ed0d8b3a835e4c27afe69
5252ad0b8f5.recovery/wal-000000018: Not found: Footer not found. Footer magic
doesn't match
I0819 14:16:07.722486 20997 log_reader.cc:152] Log segment
/export/servers/kudu/tserver_wal_data_7052/wals/ed0d8b3a835e4c27afe695252ad0b8f5.recovery/
wal-000000018 was likely left in-progress after a previous crash. Will try to
rebuild footer by scanning data.
I0819 14:16:07.722496 20997 log_util.cc:359] Successfully rebuilt footer for
segment: /export/servers/kudu/tserver_wal_data_7052/wals/ed0d8b3a835e4c2
7afe695252ad0b8f5.recovery/wal-000000018 (valid entries through byte offset
13896955103899491)
It is possible that the "12 initial NULL bytes" make kudu-tserver think the
segment is blank , while it is not . those segments maybe the the segments with
Segment sequence numbers 1-15
was (Author: brucesz):
before log message in description, there are logs:
Log segment file Log segment file
/export/servers/kudu/tserver_wal_data_7052/wals/ed0d8b3a835e4c27afe695252ad0b8f5.recove
ry/wal-000000018 has 12 initial NULL bytes instead of magic and header length:
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 and will be treated a
s a blank segment.
W0819 14:16:07.722328 20997 log_util.cc:311] Could not read footer for segment:
/export/servers/kudu/tserver_wal_data_7052/wals/ed0d8b3a835e4c27afe69
5252ad0b8f5.recovery/wal-000000018: Not found: Footer not found. Footer magic
doesn't match
I0819 14:16:07.722486 20997 log_reader.cc:152] Log segment
/export/servers/kudu/tserver_wal_data_7052/wals/ed0d8b3a835e4c27afe695252ad0b8f5.recovery/
wal-000000018 was likely left in-progress after a previous crash. Will try to
rebuild footer by scanning data.
I0819 14:16:07.722496 20997 log_util.cc:359] Successfully rebuilt footer for
segment: /export/servers/kudu/tserver_wal_data_7052/wals/ed0d8b3a835e4c2
7afe695252ad0b8f5.recovery/wal-000000018 (valid entries through byte offset
13896955103899491)
> In corner cases , tablet could not recovery successfully from node crash.
> ---------------------------------------------------------------------------
>
> Key: KUDU-1573
> URL: https://issues.apache.org/jira/browse/KUDU-1573
> Project: Kudu
> Issue Type: Bug
> Reporter: zhangsong
>
> Last friday ,one of nodes of my kudu cluster crashed and , tablet can not
> recovery successfully after restart kudu-tserver, i observed error messages
> from log:
> (TABLET_DATA_READY): Corruption: Could not open LogReader. Reason: Unable to
> initialize log reader: Segment sequence numbers are not consecutive. Previous
> segment: seqno 0, path
> /export/servers/kudu/tserver_wal_data_7052/wals/ed0d8b3a835e4c27afe695252ad0b8f5.recovery/wal-000000018;
> Current segment: seqno 17, path
> /export/servers/kudu/tserver_wal_data_7052/wals/ed0d8b3a835e4c27afe695252ad0b8f5.recovery/wal-000000017
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
