[jira] [Commented] (CASSANDRA-4741) Cassandra stops persisting data on 32 bit windows
[ https://issues.apache.org/jira/browse/CASSANDRA-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467775#comment-13467775 ] Mauri Melato commented on CASSANDRA-4741: - I could see data is actually persisted if I force flush, nodetool flush (or drain to shutdown). But the system does not recover anyway. I mean I have always to force the flush before closing the node to get data persisted. To quickly reproduce the bug you need a virtual machine with winXP32, Cassandra 1.1.5 and the attached directory tree to be configured into the cassandra.yaml. We could actually get this issue even on a Win7 64 bit, but at that time we didn't get the complete dataset directories to reproduce it. Thanks for the support. Cassandra stops persisting data on 32 bit windows - Key: CASSANDRA-4741 URL: https://issues.apache.org/jira/browse/CASSANDRA-4741 Project: Cassandra Issue Type: Bug Environment: Windows XP 32 bit Cassandra 1.1.4 or 1.1.5 Reporter: Mauri Melato Fix For: 1.1.4, 1.1.5 Attachments: cassandra-data.minimal.zip Hello, we have a Cassandra instance running on a single node with Windows XP 32bit. After some time of normal working and some restart, Cassandra seems to stop to persist data into the keyspace without raising any error. It's possible to update/insert rows or columns and you can query and see the updated keyspace but if you stop and start Cassandra, you get a keyspace 'from the past', always from the same point in the past. We were not able to understand how to trigger this inconsistency in the first place but once you get it, you can't get rid of it if not removing the data directories. Here in attachment you can find a complete set of Cassandra data directories: {code} cassandra-data/ |-- commitlog |-- data `-- saved_caches {code} with which it's possible test the inconsistent scenario. We tried this on another clean Win Xp 32 machine and we could get the same issue from the first run. On a Win XP 64 and the same dir set, Cassandra didn't get into this issue, but from the log we saw a difference in the disk access mode: * XP/64: {{DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap instead of 'standard'}} * XP/32: {{DiskAccessMode 'auto' determined to be standard, indexAccessMode is standard}} Not sure this is the key factor, nevertheless in another test bed we experimented the same issue even on a WinXP 64 node but we couldn't collect the dataset. Here a short cassandra-cli history to show the issue: {code} [default@NEUTRO_Keyspace] connect 192.168.1.33/9160; Connected to: NEUTRO Cluster on 192.168.1.33/9160 [default@unknown] use neutro_keyspace; Authenticated to keyspace: NEUTRO_Keyspace [default@NEUTRO_Keyspace] describe; Keyspace: NEUTRO_Keyspace: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:1] Column Families: ColumnFamily: TaskId Key Validation Class: org.apache.cassandra.db.marshal.AsciiType Default column value validator: org.apache.cassandra.db.marshal.UTF8Type Columns sorted by: org.apache.cassandra.db.marshal.AsciiType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 DC Local Read repair chance: 0.0 Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Column Metadata: Column Name: taskId Validation Class: org.apache.cassandra.db.marshal.LongType Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy Compression Options: sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor [default@NEUTRO_Keyspace] list TaskId; Using default limit of 100 Using default column limit of 100 --- RowKey: NE_MASTER_TASK_ID = (column=taskId, value=47, timestamp=1349090424494000) 1 Row Returned. Elapsed time: 93 msec(s). [default@NEUTRO_Keyspace] set taskid['NE_MASTER_TASK_ID']['taskId']='55'; Value inserted. Elapsed time: 31 msec(s). [default@NEUTRO_Keyspace] list TaskId; Using default limit of 100 Using default column limit of 100 --- RowKey: NE_MASTER_TASK_ID = (column=taskId, value=55, timestamp=1349108488164000) 1 Row Returned. Elapsed time: 31 msec(s). {code} ...Stop and Start Cassandra... {code} [default@NEUTRO_Keyspace] connect 192.168.1.33/9160; Connected to: NEUTRO Cluster on 192.168.1.33/9160 [default@unknown] use neutro_keyspace; Authenticated to keyspace: NEUTRO_Keyspace [default@NEUTRO_Keyspace] list TaskId; Using default limit of 100 Using default column
[jira] [Commented] (CASSANDRA-4741) Cassandra stops persisting data on 32 bit windows
[ https://issues.apache.org/jira/browse/CASSANDRA-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467805#comment-13467805 ] Jonathan Ellis commented on CASSANDRA-4741: --- So if you load this dataset onto win7 64bit, does it still exhibit the problem? Cassandra stops persisting data on 32 bit windows - Key: CASSANDRA-4741 URL: https://issues.apache.org/jira/browse/CASSANDRA-4741 Project: Cassandra Issue Type: Bug Environment: Windows XP 32 bit Cassandra 1.1.4 or 1.1.5 Reporter: Mauri Melato Fix For: 1.1.4, 1.1.5 Attachments: cassandra-data.minimal.zip Hello, we have a Cassandra instance running on a single node with Windows XP 32bit. After some time of normal working and some restart, Cassandra seems to stop to persist data into the keyspace without raising any error. It's possible to update/insert rows or columns and you can query and see the updated keyspace but if you stop and start Cassandra, you get a keyspace 'from the past', always from the same point in the past. We were not able to understand how to trigger this inconsistency in the first place but once you get it, you can't get rid of it if not removing the data directories. Here in attachment you can find a complete set of Cassandra data directories: {code} cassandra-data/ |-- commitlog |-- data `-- saved_caches {code} with which it's possible test the inconsistent scenario. We tried this on another clean Win Xp 32 machine and we could get the same issue from the first run. On a Win XP 64 and the same dir set, Cassandra didn't get into this issue, but from the log we saw a difference in the disk access mode: * XP/64: {{DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap instead of 'standard'}} * XP/32: {{DiskAccessMode 'auto' determined to be standard, indexAccessMode is standard}} Not sure this is the key factor, nevertheless in another test bed we experimented the same issue even on a WinXP 64 node but we couldn't collect the dataset. Here a short cassandra-cli history to show the issue: {code} [default@NEUTRO_Keyspace] connect 192.168.1.33/9160; Connected to: NEUTRO Cluster on 192.168.1.33/9160 [default@unknown] use neutro_keyspace; Authenticated to keyspace: NEUTRO_Keyspace [default@NEUTRO_Keyspace] describe; Keyspace: NEUTRO_Keyspace: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:1] Column Families: ColumnFamily: TaskId Key Validation Class: org.apache.cassandra.db.marshal.AsciiType Default column value validator: org.apache.cassandra.db.marshal.UTF8Type Columns sorted by: org.apache.cassandra.db.marshal.AsciiType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 DC Local Read repair chance: 0.0 Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Column Metadata: Column Name: taskId Validation Class: org.apache.cassandra.db.marshal.LongType Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy Compression Options: sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor [default@NEUTRO_Keyspace] list TaskId; Using default limit of 100 Using default column limit of 100 --- RowKey: NE_MASTER_TASK_ID = (column=taskId, value=47, timestamp=1349090424494000) 1 Row Returned. Elapsed time: 93 msec(s). [default@NEUTRO_Keyspace] set taskid['NE_MASTER_TASK_ID']['taskId']='55'; Value inserted. Elapsed time: 31 msec(s). [default@NEUTRO_Keyspace] list TaskId; Using default limit of 100 Using default column limit of 100 --- RowKey: NE_MASTER_TASK_ID = (column=taskId, value=55, timestamp=1349108488164000) 1 Row Returned. Elapsed time: 31 msec(s). {code} ...Stop and Start Cassandra... {code} [default@NEUTRO_Keyspace] connect 192.168.1.33/9160; Connected to: NEUTRO Cluster on 192.168.1.33/9160 [default@unknown] use neutro_keyspace; Authenticated to keyspace: NEUTRO_Keyspace [default@NEUTRO_Keyspace] list TaskId; Using default limit of 100 Using default column limit of 100 --- RowKey: NE_MASTER_TASK_ID = (column=taskId, value=47, timestamp=1349090424494000) 1 Row Returned. Elapsed time: 110 msec(s). {code} Thanks for your attention -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4741) Cassandra stops persisting data on 32 bit windows
[ https://issues.apache.org/jira/browse/CASSANDRA-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467857#comment-13467857 ] Mauri Melato commented on CASSANDRA-4741: - We've have just tested this dataset on linux and windows (both 64) and we got the same problem. I'm not sure if transferring the dataset directories commitlog, data and saved_caches, from a mono-node Cassandra to another it's an orthodox way to proceed for testing but it's anyway revealing the problem. Other normal dataset work properly across different Cassandra instances. Cassandra stops persisting data on 32 bit windows - Key: CASSANDRA-4741 URL: https://issues.apache.org/jira/browse/CASSANDRA-4741 Project: Cassandra Issue Type: Bug Environment: Windows XP 32 bit Cassandra 1.1.4 or 1.1.5 Reporter: Mauri Melato Fix For: 1.1.4, 1.1.5 Attachments: cassandra-data.minimal.zip Hello, we have a Cassandra instance running on a single node with Windows XP 32bit. After some time of normal working and some restart, Cassandra seems to stop to persist data into the keyspace without raising any error. It's possible to update/insert rows or columns and you can query and see the updated keyspace but if you stop and start Cassandra, you get a keyspace 'from the past', always from the same point in the past. We were not able to understand how to trigger this inconsistency in the first place but once you get it, you can't get rid of it if not removing the data directories. Here in attachment you can find a complete set of Cassandra data directories: {code} cassandra-data/ |-- commitlog |-- data `-- saved_caches {code} with which it's possible test the inconsistent scenario. We tried this on another clean Win Xp 32 machine and we could get the same issue from the first run. On a Win XP 64 and the same dir set, Cassandra didn't get into this issue, but from the log we saw a difference in the disk access mode: * XP/64: {{DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap instead of 'standard'}} * XP/32: {{DiskAccessMode 'auto' determined to be standard, indexAccessMode is standard}} Not sure this is the key factor, nevertheless in another test bed we experimented the same issue even on a WinXP 64 node but we couldn't collect the dataset. Here a short cassandra-cli history to show the issue: {code} [default@NEUTRO_Keyspace] connect 192.168.1.33/9160; Connected to: NEUTRO Cluster on 192.168.1.33/9160 [default@unknown] use neutro_keyspace; Authenticated to keyspace: NEUTRO_Keyspace [default@NEUTRO_Keyspace] describe; Keyspace: NEUTRO_Keyspace: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:1] Column Families: ColumnFamily: TaskId Key Validation Class: org.apache.cassandra.db.marshal.AsciiType Default column value validator: org.apache.cassandra.db.marshal.UTF8Type Columns sorted by: org.apache.cassandra.db.marshal.AsciiType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 DC Local Read repair chance: 0.0 Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Column Metadata: Column Name: taskId Validation Class: org.apache.cassandra.db.marshal.LongType Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy Compression Options: sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor [default@NEUTRO_Keyspace] list TaskId; Using default limit of 100 Using default column limit of 100 --- RowKey: NE_MASTER_TASK_ID = (column=taskId, value=47, timestamp=1349090424494000) 1 Row Returned. Elapsed time: 93 msec(s). [default@NEUTRO_Keyspace] set taskid['NE_MASTER_TASK_ID']['taskId']='55'; Value inserted. Elapsed time: 31 msec(s). [default@NEUTRO_Keyspace] list TaskId; Using default limit of 100 Using default column limit of 100 --- RowKey: NE_MASTER_TASK_ID = (column=taskId, value=55, timestamp=1349108488164000) 1 Row Returned. Elapsed time: 31 msec(s). {code} ...Stop and Start Cassandra... {code} [default@NEUTRO_Keyspace] connect 192.168.1.33/9160; Connected to: NEUTRO Cluster on 192.168.1.33/9160 [default@unknown] use neutro_keyspace; Authenticated to keyspace: NEUTRO_Keyspace [default@NEUTRO_Keyspace] list TaskId; Using default limit of 100 Using default column limit of 100 --- RowKey: NE_MASTER_TASK_ID = (column=taskId, value=47, timestamp=1349090424494000) 1 Row Returned. Elapsed time: 110 msec(s). {code}
[jira] [Commented] (CASSANDRA-4741) Cassandra stops persisting data on 32 bit windows
[ https://issues.apache.org/jira/browse/CASSANDRA-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467894#comment-13467894 ] Brandon Williams commented on CASSANDRA-4741: - Can you set your commit_log_sync to batch and see if it repros? Cassandra stops persisting data on 32 bit windows - Key: CASSANDRA-4741 URL: https://issues.apache.org/jira/browse/CASSANDRA-4741 Project: Cassandra Issue Type: Bug Environment: Windows XP 32 bit Cassandra 1.1.4 or 1.1.5 Reporter: Mauri Melato Fix For: 1.1.4, 1.1.5 Attachments: cassandra-data.minimal.zip Hello, we have a Cassandra instance running on a single node with Windows XP 32bit. After some time of normal working and some restart, Cassandra seems to stop to persist data into the keyspace without raising any error. It's possible to update/insert rows or columns and you can query and see the updated keyspace but if you stop and start Cassandra, you get a keyspace 'from the past', always from the same point in the past. We were not able to understand how to trigger this inconsistency in the first place but once you get it, you can't get rid of it if not removing the data directories. Here in attachment you can find a complete set of Cassandra data directories: {code} cassandra-data/ |-- commitlog |-- data `-- saved_caches {code} with which it's possible test the inconsistent scenario. We tried this on another clean Win Xp 32 machine and we could get the same issue from the first run. On a Win XP 64 and the same dir set, Cassandra didn't get into this issue, but from the log we saw a difference in the disk access mode: * XP/64: {{DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap instead of 'standard'}} * XP/32: {{DiskAccessMode 'auto' determined to be standard, indexAccessMode is standard}} Not sure this is the key factor, nevertheless in another test bed we experimented the same issue even on a WinXP 64 node but we couldn't collect the dataset. Here a short cassandra-cli history to show the issue: {code} [default@NEUTRO_Keyspace] connect 192.168.1.33/9160; Connected to: NEUTRO Cluster on 192.168.1.33/9160 [default@unknown] use neutro_keyspace; Authenticated to keyspace: NEUTRO_Keyspace [default@NEUTRO_Keyspace] describe; Keyspace: NEUTRO_Keyspace: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:1] Column Families: ColumnFamily: TaskId Key Validation Class: org.apache.cassandra.db.marshal.AsciiType Default column value validator: org.apache.cassandra.db.marshal.UTF8Type Columns sorted by: org.apache.cassandra.db.marshal.AsciiType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 DC Local Read repair chance: 0.0 Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Column Metadata: Column Name: taskId Validation Class: org.apache.cassandra.db.marshal.LongType Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy Compression Options: sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor [default@NEUTRO_Keyspace] list TaskId; Using default limit of 100 Using default column limit of 100 --- RowKey: NE_MASTER_TASK_ID = (column=taskId, value=47, timestamp=1349090424494000) 1 Row Returned. Elapsed time: 93 msec(s). [default@NEUTRO_Keyspace] set taskid['NE_MASTER_TASK_ID']['taskId']='55'; Value inserted. Elapsed time: 31 msec(s). [default@NEUTRO_Keyspace] list TaskId; Using default limit of 100 Using default column limit of 100 --- RowKey: NE_MASTER_TASK_ID = (column=taskId, value=55, timestamp=1349108488164000) 1 Row Returned. Elapsed time: 31 msec(s). {code} ...Stop and Start Cassandra... {code} [default@NEUTRO_Keyspace] connect 192.168.1.33/9160; Connected to: NEUTRO Cluster on 192.168.1.33/9160 [default@unknown] use neutro_keyspace; Authenticated to keyspace: NEUTRO_Keyspace [default@NEUTRO_Keyspace] list TaskId; Using default limit of 100 Using default column limit of 100 --- RowKey: NE_MASTER_TASK_ID = (column=taskId, value=47, timestamp=1349090424494000) 1 Row Returned. Elapsed time: 110 msec(s). {code} Thanks for your attention -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4741) Cassandra stops persisting data on 32 bit windows
[ https://issues.apache.org/jira/browse/CASSANDRA-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467924#comment-13467924 ] Mauri Melato commented on CASSANDRA-4741: - Changed {{cassandra.yaml}} to use {code} commitlog_sync: batch commitlog_sync_batch_window_in_ms: 50 {code} instead of periodic sync and the problem still persists Cassandra stops persisting data on 32 bit windows - Key: CASSANDRA-4741 URL: https://issues.apache.org/jira/browse/CASSANDRA-4741 Project: Cassandra Issue Type: Bug Environment: Windows XP 32 bit Cassandra 1.1.4 or 1.1.5 Reporter: Mauri Melato Fix For: 1.1.4, 1.1.5 Attachments: cassandra-data.minimal.zip Hello, we have a Cassandra instance running on a single node with Windows XP 32bit. After some time of normal working and some restart, Cassandra seems to stop to persist data into the keyspace without raising any error. It's possible to update/insert rows or columns and you can query and see the updated keyspace but if you stop and start Cassandra, you get a keyspace 'from the past', always from the same point in the past. We were not able to understand how to trigger this inconsistency in the first place but once you get it, you can't get rid of it if not removing the data directories. Here in attachment you can find a complete set of Cassandra data directories: {code} cassandra-data/ |-- commitlog |-- data `-- saved_caches {code} with which it's possible test the inconsistent scenario. We tried this on another clean Win Xp 32 machine and we could get the same issue from the first run. On a Win XP 64 and the same dir set, Cassandra didn't get into this issue, but from the log we saw a difference in the disk access mode: * XP/64: {{DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap instead of 'standard'}} * XP/32: {{DiskAccessMode 'auto' determined to be standard, indexAccessMode is standard}} Not sure this is the key factor, nevertheless in another test bed we experimented the same issue even on a WinXP 64 node but we couldn't collect the dataset. Here a short cassandra-cli history to show the issue: {code} [default@NEUTRO_Keyspace] connect 192.168.1.33/9160; Connected to: NEUTRO Cluster on 192.168.1.33/9160 [default@unknown] use neutro_keyspace; Authenticated to keyspace: NEUTRO_Keyspace [default@NEUTRO_Keyspace] describe; Keyspace: NEUTRO_Keyspace: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:1] Column Families: ColumnFamily: TaskId Key Validation Class: org.apache.cassandra.db.marshal.AsciiType Default column value validator: org.apache.cassandra.db.marshal.UTF8Type Columns sorted by: org.apache.cassandra.db.marshal.AsciiType GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 DC Local Read repair chance: 0.0 Replicate on write: true Caching: KEYS_ONLY Bloom Filter FP chance: default Built indexes: [] Column Metadata: Column Name: taskId Validation Class: org.apache.cassandra.db.marshal.LongType Compaction Strategy: org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy Compression Options: sstable_compression: org.apache.cassandra.io.compress.SnappyCompressor [default@NEUTRO_Keyspace] list TaskId; Using default limit of 100 Using default column limit of 100 --- RowKey: NE_MASTER_TASK_ID = (column=taskId, value=47, timestamp=1349090424494000) 1 Row Returned. Elapsed time: 93 msec(s). [default@NEUTRO_Keyspace] set taskid['NE_MASTER_TASK_ID']['taskId']='55'; Value inserted. Elapsed time: 31 msec(s). [default@NEUTRO_Keyspace] list TaskId; Using default limit of 100 Using default column limit of 100 --- RowKey: NE_MASTER_TASK_ID = (column=taskId, value=55, timestamp=1349108488164000) 1 Row Returned. Elapsed time: 31 msec(s). {code} ...Stop and Start Cassandra... {code} [default@NEUTRO_Keyspace] connect 192.168.1.33/9160; Connected to: NEUTRO Cluster on 192.168.1.33/9160 [default@unknown] use neutro_keyspace; Authenticated to keyspace: NEUTRO_Keyspace [default@NEUTRO_Keyspace] list TaskId; Using default limit of 100 Using default column limit of 100 --- RowKey: NE_MASTER_TASK_ID = (column=taskId, value=47, timestamp=1349090424494000) 1 Row Returned. Elapsed time: 110 msec(s). {code} Thanks for your attention -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: