[
https://issues.apache.org/jira/browse/TRAFODION-341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14721186#comment-14721186
]
Suresh Subbiah commented on TRAFODION-341:
------------------------------------------
Atanu Mishra (atanu-mishra) on 2014-06-02
Changed in trafodion:
assignee: nobody → Sean Broeder (sean-broeder)
status: New → In Progress
status: In Progress → New
Marvin Anderson (marvin-anderson) on 2014-06-02
description: updated
tags: added: dtm
Atanu Mishra (atanu-mishra) on 2014-06-02
Changed in trafodion:
status: New → In Progress
Atanu Mishra (atanu-mishra) wrote on 2014-06-03: #1
1) From hbase shell, you can scan MARVIN1, so implies the catalog is
damaged
2) From /var/log/hbase (ah4), we can see MARVIN1 able to process the
THLOG, find 10 in-doubt transactions in MARVIN1, but abandon them because
cleanat is set to 1 (due to hbase xml property tlog is false there is no TLOG
support, so throw them away).
The root cause likely resides in the catalog part (may be with no transactional
DDL support) on Trafodion.
Besides, we see hbase xml property tlog is NOT set to true (we need this one
for RS to resolve in-doubt txns after edits processing). Therefore, we have to
set 1) both recovery flags in ms.env, and 2) hbase xml property (tlog=true for
recovery) and (hlog=false for performance).
(from centos-ah4 /var/log/hbase)
2014-06-01 20:56:22,267 DEBUG
org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion:
Trafodion Recovery: WAL HLOG setting true
2014-06-01 20:56:22,267 DEBUG
org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion:
Trafodion Recovery: TM TLOG setting false
2014-06-01 20:56:28,451 DEBUG
org.apache.hadoop.hbase.regionserver.transactional.THLogRecoveryManager:
Trafodion Recovery:
TRAFODION.SEABASE.MARVIN1,,1401655718884.ff40e40a3eddd3142394befdb93f0300.
Edits replay read 3694 transactional operations (skipped 0 because sequence id
<= 0): 1848 commitRequests, 8 aborts, 1838 commits, and 0 flushes. this flush
means “extra” bytes to push the edits out, only used in local mode
2014-06-01 20:56:28,451 DEBUG
org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion:
Trafodion Recovery: TM clean AT mode 1 discards 10 in-doubt transaction
2014-06-01 20:56:28,451 DEBUG
org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion:
Trafodion Recovery: Region
TRAFODION.SEABASE.MARVIN1,,1401655718884.ff40e40a3eddd3142394befdb93f0300. is
in state RECOVERING
2014-06-01 20:56:28,451 DEBUG
org.apache.hadoop.hbase.regionserver.transactional.TransactionalRegion:
Trafodion Recovery: Region
TRAFODION.SEABASE.MARVIN1,,1401655718884.ff40e40a3eddd3142394befdb93f0300. has
no in-doubt transaction, set region START
Additional information
I was able to a “cleandb” and wipe all tables and metadata from HBase and
reinitialize Trafodion. I couldn’t just drop my two tables as the SQL metadata
tables were out of sync.
Both flags were already set in etc/ms.env (are these the proper flags or are
there some others?)
TM_ENABLE_TLOG_WRITES=1
TM_ENABLE_RECOV_CLIENT=1
I added these settings to my HBase (these were not set before):
hbase.regionserver.region.transactional.hlog = false
hbase.regionserver.region.transactional.tlog = true
And more ...
The command to flush the memstores is, through the hbase shell:
hbase> flush ‘TABLENAME’
This would only flush what’s in the memstore to disk, however. It looks like
the table MARVIN1 is already present in HBase and can be scanned so it’s
probably not that the DDL is being flushed to disk.
It almost looks to me like the trafodion metadata operations have not been
completed, the TRAFODION._MD_.* table has not been flushed or a row has not
been inserted to let SQL know that the table exists.
Stacey Johnson (sjohnson-w) on 2014-06-10
information type: Proprietary → Public
Atanu Mishra (atanu-mishra) wrote on 2014-07-12: #2
Unable to make progress on this with the available information. Leaving it open
in case this problem shows up again.
Changed in trafodion:
status: In Progress → Incomplete
Marvin Anderson (marvin-anderson) wrote on 2014-10-24: #3
Closing this bug as I haven't seen this behavior in a little while and it was
on a whole different code base.
Changed in trafodion:
status: Incomplete → Won't Fix
See full activity log
> LP Bug: 1325465 - All Trafodion SQL tables disappeared after recovery
> ---------------------------------------------------------------------
>
> Key: TRAFODION-341
> URL: https://issues.apache.org/jira/browse/TRAFODION-341
> Project: Apache Trafodion
> Issue Type: Bug
> Components: dtm
> Reporter: Marvin Anderson
> Assignee: Sean Broeder
> Priority: Blocker
>
> 1.Running on build 20140529_1530
> 2.Started 10 concurrent connections
> 3.Transactions were random singleton INSERT or singleton UPDATE statements
> followed by COMMIT WORK on table MARVIN1
> 4.Let them run for a little bit watching record count in table
> >>select count(*) from marvin1;
> (EXPR)
> --------------------
> 1004
> --- 1 row(s) selected.
> >>select count(*) from marvin1;
> (EXPR)
> --------------------
> 1051
> --- 1 row(s) selected.
> 5.Turned on test point
> [trafodion@centos-ah1 ~]$ dtm_stall
> Processing cluster.conf on local host centos-ah1
> [$Z000E4W] Shell/shell Version 1.0.1 Release 0.7.0 (Build debug
> [0.7.0rc0-7-g57e0bd3_Bld8], date 20140529_083000)
> [$Z000E4W] %set dtm_stall_phase_2=3
> [$Z000E4W] Configuration Change Notice for Group: CLUSTER Key:
> DTM_STALL_PHASE_2
> 6.waited for all 10 programs to hang on commit
> 7.Killed HMaster and all HBase RegionServers
> [andersma@centos-ah1 ~]$ ./kill_hbase
> ***INFO: killing HMaster and all HBase RegionServer processes
> ***WARNING: This may only work for Ambari/Hortonworks
> 8.Confirmed all HBase processes stopped
> 9.ckillall for Trafodion
> 10.restarted HBbase
> 11.restarted Trafodion after HBase was fully back up
> 12.All Trafodion SQL tables are gone. I had two tables (BIGHERO1 & MARVIN1)
> and only one, MARVIN1, was being used by the programs.
> >>select count(*) from marvin1;
> *** ERROR[4082] Object TRAFODION.SEABASE.MARVIN1 does not exist or is
> inaccessible.
> *** ERROR[8822] The statement was not prepared.
> >>get tables;
> --- SQL operation complete.
> >>
> 13.Although the tables are still in HBase
> hbase(main):001:0> list
> TABLE
>
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/lib/hbase/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> TRAFODION.SEABASE.BIGHERO1
>
> TRAFODION.SEABASE.MARVIN1
>
> TRAFODION._DTM_.TLOG0
>
> TRAFODION._DTM_.TLOG0_CONTROL_POINT
>
> TRAFODION._DTM_.TLOG1
>
> TRAFODION._DTM_.TLOG1_CONTROL_POINT
>
> TRAFODION._DTM_.TLOG2
>
> TRAFODION._DTM_.TLOG2_CONTROL_POINT
>
> TRAFODION._DTM_.TLOG3
>
> TRAFODION._DTM_.TLOG3_CONTROL_POINT
>
> TRAFODION._DTM_.TLOG4
>
> TRAFODION._DTM_.TLOG4_CONTROL_POINT
>
> TRAFODION._DTM_.TLOG5
>
> TRAFODION._DTM_.TLOG5_CONTROL_POINT
>
> TRAFODION._MD_.AUTHS
>
> TRAFODION._MD_.COLUMNS
>
> TRAFODION._MD_.DEFAULTS
>
> TRAFODION._MD_.INDEXES
>
> TRAFODION._MD_.KEYS
>
> TRAFODION._MD_.LIBRARIES
>
> TRAFODION._MD_.LIBRARIES_USAGE
>
> TRAFODION._MD_.OBJECTS
>
> TRAFODION._MD_.OBJECTS_UNIQ_IDX
>
> TRAFODION._MD_.REF_CONSTRAINTS
>
> TRAFODION._MD_.ROUTINES
>
> TRAFODION._MD_.TABLES
>
> TRAFODION._MD_.TABLE_CONSTRAINTS
>
> TRAFODION._MD_.TEXT
>
> TRAFODION._MD_.UNIQUE_REF_CONSTR_USAGE
>
> TRAFODION._MD_.VERSIONS
>
> TRAFODION._MD_.VIEWS
>
> TRAFODION._MD_.VIEWS_USAGE
>
> 32 row(s) in 0.7500 seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
