date:20120814

[jira] [Commented] (CASSANDRA-4538) Strange CorruptedBlockException when massive insert binary data

2012-08-14 Thread Cathy Daw (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13433944#comment-13433944
 ] 

Cathy Daw commented on CASSANDRA-4538:
--

I tried lots of permutations and could not reproduce.
Can you verify if this consistently reproducible for you?
Here are my repro tests

{code}
// Test Setup
* Modify: InsertThread.java to change host IP address
* Run: mvn install
* Start: cassandra 1.1.4

// Test Run
* Test Setup:  create / modify KS and CF below
* Run test: mvn exec:java -Dexec.mainClass=com.test.CreateTestData

// *** cassandra-cli ***
create keyspace ST with
  placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy'
  and strategy_options = {replication_factor:1};
  
use ST;

// Test #1: SizeTieredCompactionStrategy
create column family company;

// Test #2: SizeTieredCompactionStrategy and 1mb sstables
drop column family company;
create column family company with 
and compaction_strategy=SizeTieredCompactionStrategy
and compaction_strategy_options={sstable_size_in_mb: 1};

// Test #3: SizeTieredCompactionStrategy and 100mb sstables
drop column family company;
create column family company with 
and compaction_strategy=SizeTieredCompactionStrategy
and compaction_strategy_options={sstable_size_in_mb: 100};


// Test #4: LeveledCompactionStrategy and 10mb sstables
drop column family company;
create column family company 
and compaction_strategy=LeveledCompactionStrategy
and compaction_strategy_options={sstable_size_in_mb: 10};

// Test #5: LeveledCompactionStrategy and 1mb sstables
drop column family company;
create column family company 
and compaction_strategy=LeveledCompactionStrategy
and compaction_strategy_options={sstable_size_in_mb: 1};

// Test #6: LeveledCompactionStrategy and 100mb sstables
drop column family company;
create column family company 
and compaction_strategy=LeveledCompactionStrategy
and compaction_strategy_options={sstable_size_in_mb: 100};

// ADDITIONAL TESTS VIA JAVA STRESS
[default@ST] drop keyspace Keyspace1;
./cassandra-stress --operation=INSERT --num-keys=10 
--num-different-keys=2 --columns=2 --threads=2 
--compression=SnappyCompressor --compaction-strategy=LeveledCompactionStrategy 
--column-size=2
./cassandra-stress --operation=READ --num-keys=10 
--num-different-keys=2 --columns=2 --threads=2 
--compression=SnappyCompressor --compaction-strategy=LeveledCompactionStrategy 
--column-size=2


// Distructive test: check nodetool -h localhost compactionstats and run the 
following while there are pending compactions
./cassandra-stress --operation=INSERT --num-keys=1000 --num-different-keys=100 
--columns=2 --threads=2 --compression=SnappyCompressor 
--compaction-strategy=LeveledCompactionStrategy --column-size=2

// Tried with SizeTieredCompactionStrategy
[default@ST] drop keyspace Keyspace1;
./cassandra-stress --operation=INSERT --num-keys=6 
--num-different-keys=2 --columns=2 --compression=SnappyCompressor 
--compaction-strategy=SizeTieredCompactionStrategy --column-size=2
./cassandra-stress --operation=READ --num-keys=6 --num-different-keys=2 
--columns=2 --compression=SnappyCompressor 
--compaction-strategy=SizeTieredCompactionStrategy --column-size=2

// Distructive test: check nodetool -h localhost compactionstats and kill the 
c* server while compactions are in progress and then restart

{code}

 Strange CorruptedBlockException when massive insert binary data
 ---

 Key: CASSANDRA-4538
 URL: https://issues.apache.org/jira/browse/CASSANDRA-4538
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 1.1.3
 Environment: Debian sequeeze 32bit
Reporter: Tommy Cheng
Priority: Critical
  Labels: CorruptedBlockException, binary, insert
 Attachments: cassandra-stresstest.zip


 After inserting ~ 1 records, here is the error log
  INFO 10:53:33,543 Compacted to 
 [/var/lib/cassandra/data/ST/company/ST-company.company_acct_no_idx-he-13-Data.db,].
   407,681 to 409,133 (~100% of original) bytes for 9,250 keys at 
 0.715926MB/s.  Time: 545ms.
 ERROR 10:53:35,445 Exception in thread Thread[CompactionExecutor:3,1,main]
 java.io.IOError: org.apache.cassandra.io.compress.CorruptedBlockException: 
 (/var/lib/cassandra/data/ST/company/ST-company-he-9-Data.db): corruption 
 detected, chunk at 7530128 of length 19575.
 at 
 org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:116)
 at 
 org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:99)
 at 
 org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:176)
 at

[jira] [Commented] (CASSANDRA-4538) Strange CorruptedBlockException when massive insert binary data

2012-08-14 Thread Tommy Cheng (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13433951#comment-13433951
 ] 

Tommy Cheng commented on CASSANDRA-4538:


Yes, consistently reproducible.
Funny thing is that another machine is okay.
I tried format the OS, health check the ram/harddisk, and all test pass.

What extra thing should i provide?
It is very important to find out the problem before really use cassandra for 
production.

 Strange CorruptedBlockException when massive insert binary data
 ---

 Key: CASSANDRA-4538
 URL: https://issues.apache.org/jira/browse/CASSANDRA-4538
 Project: Cassandra
  Issue Type: Bug
Affects Versions: 1.1.3
 Environment: Debian sequeeze 32bit
Reporter: Tommy Cheng
Priority: Critical
  Labels: CorruptedBlockException, binary, insert
 Attachments: cassandra-stresstest.zip


 After inserting ~ 1 records, here is the error log
  INFO 10:53:33,543 Compacted to 
 [/var/lib/cassandra/data/ST/company/ST-company.company_acct_no_idx-he-13-Data.db,].
   407,681 to 409,133 (~100% of original) bytes for 9,250 keys at 
 0.715926MB/s.  Time: 545ms.
 ERROR 10:53:35,445 Exception in thread Thread[CompactionExecutor:3,1,main]
 java.io.IOError: org.apache.cassandra.io.compress.CorruptedBlockException: 
 (/var/lib/cassandra/data/ST/company/ST-company-he-9-Data.db): corruption 
 detected, chunk at 7530128 of length 19575.
 at 
 org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:116)
 at 
 org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:99)
 at 
 org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:176)
 at 
 org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:83)
 at 
 org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:68)
 at 
 org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:118)
 at 
 org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:101)
 at 
 com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
 at 
 com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
 at 
 com.google.common.collect.Iterators$7.computeNext(Iterators.java:614)
 at 
 com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
 at 
 com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
 at 
 org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:173)
 at 
 org.apache.cassandra.db.compaction.CompactionManager$1.runMayThrow(CompactionManager.java:154)
 at 
 org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
 at 
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
 Caused by: org.apache.cassandra.io.compress.CorruptedBlockException: 
 (/var/lib/cassandra/data/ST/company/ST-company-he-9-Data.db): corruption 
 detected, chunk at 7530128 of length 19575.
 at 
 org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:98)
 at 
 org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:77)
 at 
 org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:302)
 at java.io.RandomAccessFile.readFully(RandomAccessFile.java:397)
 at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377)
 at 
 org.apache.cassandra.utils.BytesReadTracker.readFully(BytesReadTracker.java:95)
 at 
 org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:401)
 at 
 org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:363)
 at 
 org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:119)
 at 
 org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:36)
 at 
 org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:144)
 at

[jira] [Created] (CASSANDRA-4539) potential backwards incompatibility in native protocol

2012-08-14 Thread paul cannon (JIRA)

paul cannon created CASSANDRA-4539:
--

 Summary: potential backwards incompatibility in native protocol
 Key: CASSANDRA-4539
 URL: https://issues.apache.org/jira/browse/CASSANDRA-4539
 Project: Cassandra
  Issue Type: Improvement
  Components: API
Affects Versions: 1.2
Reporter: paul cannon
Assignee: Sylvain Lebresne
Priority: Minor
 Fix For: 1.2


In the text of the native_protocol.spec document, it explains the format for a 
notation called {{[option]}}, which should represent {{a pair of 
idvalue}}.

In doing a first-draft implementation of the protocol for the python driver, 
though, I found that I had a misunderstanding. I read that section as saying 
that {{value}} was a {{[value]}}, and that it could have a length of 0 (i.e., 
the {{[int]}} on the front of the {{[value]}} could be 0). However, it turns 
out that {{value}} might not be there at all, or might be *two* 
{{[value]}}'s, depending on the option id and message context.

I'm not a fan of this, since

 * A protocol parsing library can't simply implement a single function to read 
in {{[option]}}'s, since the length of the value part is dependent on the 
message context
 * If we add a new native data type (a new option id which could be used inside 
a {{col_spec_i}} in a RESULT message), then older clients will not know how 
to read past the value part. Of course they won't know how to interpret the 
data or deserialize later rows of that unknown type - that's not the problem - 
the problem is that the older client should be able just to mark that column as 
unparseable and still handle the rest of the columns.

Can we make {{value}} be a {{[value]}}, the contents of which can be 
re-interpreted as a {{[string]}}, an {{[option]}}, two {{[option]}}'s, or 
whatever?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (CASSANDRA-4540) nodetool clearsnapshot broken: gives java.io.IOException when trying to delete snapshot folder

2012-08-14 Thread JIRA

Christopher Lörken created CASSANDRA-4540:
-

 Summary: nodetool clearsnapshot broken: gives java.io.IOException 
when trying to delete snapshot folder
 Key: CASSANDRA-4540
 URL: https://issues.apache.org/jira/browse/CASSANDRA-4540
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
Affects Versions: 1.1.2
 Environment: Debian 6
Reporter: Christopher Lörken
Priority: Minor


nodetool clearsnapshots failes to delete snapshot directories and exits 
prematurely causing the exception at the bottom.
The actual snapshot files _within_ the directory are correctly deleted but the 
folder itself is not deleted.

I've chmodded all files and folders in the snapshots directory to 777 and rund 
the command as root to exclude file permissions as a cause. I also restarted 
cassandra which has no effect on the command.


---
root@server:/var/lib/cassandra/data/MyKeyspace/MyCf/snapshots# nodetool 
clearsnapshot MyKeyspace
Requested snapshot for: MyKeyspace
Exception in thread main java.io.IOException: Failed to delete 
/var/lib/cassandra/data/MyKeyspace/MyCf/snapshots/1344875270796
at 
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54)
at 
org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:220)
at 
org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:216)
at 
org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:371)
at 
org.apache.cassandra.db.ColumnFamilyStore.clearSnapshot(ColumnFamilyStore.java:1560)
at org.apache.cassandra.db.Table.clearSnapshot(Table.java:268)
at 
org.apache.cassandra.service.StorageService.clearSnapshot(StorageService.java:1866)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown 
Source)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown 
Source)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown 
Source)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown 
Source)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown 
Source)
at 
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown 
Source)
at 
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown 
Source)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
at sun.rmi.transport.Transport$1.run(Unknown Source)
at sun.rmi.transport.Transport$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown 
Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown 
Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-4482) In-memory merkle trees for repair

2012-08-14 Thread Nicolas Favre-Felix (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13434048#comment-13434048
]

Nicolas Favre-Felix commented on CASSANDRA-4482:

Jonathan,

I wrote the blog post linked in this ticket; the incremental repair process
we've implemented is not doing any random I/O on insert as you suggest.

Instead, we maintain a Merkle Tree (MT) in memory and update it with every
single column insert in ColumnFamilyStore.apply(). We use
column.updateDigest(digest) on all the changes in order to create a hash per
column update and then XOR this hash with the existing one in the Merkle Tree
bucket for the corresponding row.
This Merkle Tree is created with the column family (one per range), initialized
with zeros, and persisted to disk with regular snapshots.
The commutative properties of XOR make it possible to update the MT
incrementally without having to read on write.

When an incremental repair session starts, the CFS swap out their existing MTs
for new empty ones that will receive subsequent updates.

There are a few downsides to this approach:
* It is possible for the incremental MTs to miss a few inserts that happen when
the replicas involved swap out their MTs for new ones. An insert will be in the
previous MT for node A but in the fresh one for node B, for instance. This
leads to either a very small amount of extra streaming or some unrepaired
changes. For this reason, we still recommend that users run either a full
repair or a tombstone-only repair at least once every GCGraceSeconds.
* There is some overhead to keeping these MTs in memory. We actually maintain
only the leaves as a single ByteBuffer instead of creating all the intermediate
nodes like the MerkleTree class does. To avoid using too much RAM, we allocate
a fixed amount of memory per CF and divide it into a number of smaller buffers
(one per range) in order to give the same guarantees regardless of the number
of ranges per CF.
* There is a small cost in insert, about half of which is due to the hash
function (MD5).

We are looking into making our patch available to the community and would
welcome suggestions to solve or improve on these limitations.

In-memory merkle trees for repair
-

Key: CASSANDRA-4482
URL: https://issues.apache.org/jira/browse/CASSANDRA-4482
Project: Cassandra
Issue Type: New Feature
Reporter: Marcus Eriksson

this sounds cool, we should reimplement it in the open source cassandra;
http://www.acunu.com/2/post/2012/07/incremental-repair.html

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

76 matches

Mail list logo