date:20160215

cassandra git commit: Remove duplicate offline compaction tracking

2016-02-15 Thread marcuse

Repository: cassandra
Updated Branches:
  refs/heads/trunk f7d6ac7e4 -> f9a1a80af


Remove duplicate offline compaction tracking

Patch by marcuse; reviewed by Stefania for CASSANDRA-11148


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f9a1a80a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f9a1a80a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f9a1a80a

Branch: refs/heads/trunk
Commit: f9a1a80af181e568240bb8a005cd53af8de00648
Parents: f7d6ac7
Author: Marcus Eriksson 
Authored: Wed Feb 10 13:47:00 2016 +0100
Committer: Marcus Eriksson 
Committed: Tue Feb 16 07:22:40 2016 +0100

--
 CHANGES.txt |  1 +
 .../db/compaction/CompactionManager.java|  6 ++---
 .../cassandra/db/compaction/CompactionTask.java | 14 ++
 .../db/compaction/LeveledCompactionTask.java|  4 +--
 .../db/compaction/SSTableSplitter.java  |  4 +--
 .../cassandra/db/compaction/Scrubber.java   |  2 +-
 .../writers/CompactionAwareWriter.java  | 12 -
 .../writers/DefaultCompactionWriter.java| 12 ++---
 .../writers/MajorLeveledCompactionWriter.java   | 17 +---
 .../writers/MaxSSTableSizeWriter.java   | 18 ++---
 .../cassandra/io/sstable/SSTableRewriter.java   | 28 +---
 .../db/lifecycle/RealTransactionsTest.java  |  2 +-
 .../io/sstable/SSTableRewriterTest.java | 28 ++--
 13 files changed, 101 insertions(+), 47 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/f9a1a80a/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index c337825..c3bfdc3 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.4
+ * Remove duplicate offline compaction tracking (CASSANDRA-11148)
  * fix EQ semantics of analyzed SASI indexes (CASSANDRA-11130)
  * Support long name output for nodetool commands (CASSANDRA-7950)
  * Encrypted hints (CASSANDRA-11040)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f9a1a80a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java 
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index 9f7abf1..e9fa325 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -961,7 +961,7 @@ public class CompactionManager implements 
CompactionManagerMBean
 
 List finished;
 int nowInSec = FBUtilities.nowInSeconds();
-try (SSTableRewriter writer = new SSTableRewriter(txn, 
sstable.maxDataAge, false);
+try (SSTableRewriter writer = 
SSTableRewriter.constructKeepingOriginals(txn, false, sstable.maxDataAge);
  ISSTableScanner scanner = cleanupStrategy.getScanner(sstable, 
getRateLimiter());
  CompactionController controller = new CompactionController(cfs, 
txn.originals(), getDefaultGcBefore(cfs, nowInSec));
  CompactionIterator ci = new 
CompactionIterator(OperationType.CLEANUP, Collections.singletonList(scanner), 
controller, nowInSec, UUIDGen.getTimeUUID(), metrics))
@@ -1337,8 +1337,8 @@ public class CompactionManager implements 
CompactionManagerMBean
 int nowInSec = FBUtilities.nowInSeconds();
 
 CompactionStrategyManager strategy = 
cfs.getCompactionStrategyManager();
-try (SSTableRewriter repairedSSTableWriter = new 
SSTableRewriter(anticompactionGroup, groupMaxDataAge, false, false);
- SSTableRewriter unRepairedSSTableWriter = new 
SSTableRewriter(anticompactionGroup, groupMaxDataAge, false, false);
+try (SSTableRewriter repairedSSTableWriter = 
SSTableRewriter.constructWithoutEarlyOpening(anticompactionGroup, false, 
groupMaxDataAge);
+ SSTableRewriter unRepairedSSTableWriter = 
SSTableRewriter.constructWithoutEarlyOpening(anticompactionGroup, false, 
groupMaxDataAge);
  AbstractCompactionStrategy.ScannerList scanners = 
strategy.getScanners(anticompactionGroup.originals());
  CompactionController controller = new CompactionController(cfs, 
sstableAsSet, getDefaultGcBefore(cfs, nowInSec));
  CompactionIterator ci = new 
CompactionIterator(OperationType.ANTICOMPACTION, scanners.scanners, controller, 
nowInSec, UUIDGen.getTimeUUID(), metrics))

http://git-wip-us.apache.org/repos/asf/cassandra/blob/f9a1a80a/src/java/org/apache/cassandra/db/compaction/CompactionTask.java
--
diff --git

[jira] [Commented] (CASSANDRA-11171) conditional update without paxos

2016-02-15 Thread stuart (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147913#comment-15147913
 ] 

stuart commented on CASSANDRA-11171:


Sorry I should have elaborated a bit. I'm not by any means suggesting any form 
of performance improvement over paxos. Im developing a platform running on MS 
Project Orleans and so I have external means to control concurrency. (Infact 
I'm using cassandras lightweight transactions to guarantee single activation of 
actors). So having guarantees over actor isolation and single threaded 
execution already in place I'd like to be able to perform conditional updates 
for certain use cases. I do not have to worry about contention because I know 
these updates will be executed in a coordinated manner. Thererfore the 
atomicity of the conditional update operation currently provided by Cassandra, 
should not be required and I was wondering if there could be a means to turn it 
off? I'm not an expert in distributed programming so may well be missing 
something in my thinking here,

Thanks,
Stuart. 

> conditional update without paxos
> 
>
> Key: CASSANDRA-11171
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11171
> Project: Cassandra
>  Issue Type: Wish
>Reporter: stuart
>Priority: Minor
>
> I realise that currently conditional updates use lightweight transactions to 
> provide an atomic check and set operation but that this comes at a non 
> trivial performance cost. I have a solution where synchronised access is 
> ensured by an external mechanism therefore I don't think paxos would be 
> required. It would be nice to be able to run an update command or script that 
> could conditionally update without the performance hit. Currently I'd have to 
> retrieve each row first at the application level before deciding whether or 
> not to perform the update. Would it be possible to add a switch for the 
> conditional updates to turn paxos on or off? Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11158) AssertionError: null in Slice$Bound.create

2016-02-15 Thread Samu Kallio (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Samu Kallio updated CASSANDRA-11158:

Fix Version/s: 3.0.x

> AssertionError: null in Slice$Bound.create
> --
>
> Key: CASSANDRA-11158
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11158
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction, Local Write-Read Paths
>Reporter: Samu Kallio
> Fix For: 3.0.x
>
>
> We've been running Cassandra 3.0.2 for around a week now. Yesterday, we had a 
> network event that briefly isolated one node from others in a 3 node cluster. 
> Since then, we've been seeing a constant stream of "Finished hinted handoff" 
> messages, as well as:
> {noformat}
> WARN  16:34:39 Uncaught exception on thread 
> Thread[SharedPool-Worker-1,5,main]: {}
> java.lang.AssertionError: null
> at org.apache.cassandra.db.Slice$Bound.create(Slice.java:365) 
> ~[apache-cassandra-3.0.2.jar:3.0.2]
> at 
> org.apache.cassandra.db.Slice$Bound$Serializer.deserializeValues(Slice.java:553)
>  ~[apache-cassandra-3.0.2.jar:3.0.2]
> at 
> org.apache.cassandra.db.ClusteringPrefix$Serializer.deserialize(ClusteringPrefix.java:274)
>  ~[apache-cassandra-3.0.2.jar:3.0.2]
> at org.apache.cassandra.db.Serializers$2.deserialize(Serializers.java:115) 
> ~[apache-cassandra-3.0.2.jar:3.0.2]
> at org.apache.cassandra.db.Serializers$2.deserialize(Serializers.java:107) 
> ~[apache-cassandra-3.0.2.jar:3.0.2]
> at 
> org.apache.cassandra.io.sstable.IndexHelper$IndexInfo$Serializer.deserialize(IndexHelper.java:149)
>  ~[apache-cassandra-3.0.2.jar:3.0.2]
> at 
> org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:218)
>  ~[apache-cassandra-3.0.2.jar:3.0.2]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableReader.getPosition(BigTableReader.java:216)
>  ~[apache-cassandra-3.0.2.jar:3.0.2]
> at 
> org.apache.cassandra.io.sstable.format.SSTableReader.getPosition(SSTableReader.java:1568)
>  ~[apache-cassandra-3.0.2.jar:3.0.2]
> at 
> org.apache.cassandra.db.columniterator.SSTableIterator.(SSTableIterator.java:36)
>  ~[apache-cassandra-3.0.2.jar:3.0.2]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:62)
>  ~[apache-cassandra-3.0.2.jar:3.0.2]
> at 
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndSSTablesInTimestampOrder(SinglePartitionReadCommand.java:715)
>  ~[apache-cassandra-3.0.2.jar:3.0.2]
> at 
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:482)
>  ~[apache-cassandra-3.0.2.jar:3.0.2]
> at 
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:459)
>  ~[apache-cassandra-3.0.2.jar:3.0.2]
> at 
> org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:325)
>  ~[apache-cassandra-3.0.2.jar:3.0.2]
> at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:350) 
> ~[apache-cassandra-3.0.2.jar:3.0.2]
> at 
> org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:45)
>  ~[apache-cassandra-3.0.2.jar:3.0.2]
> at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) 
> ~[apache-cassandra-3.0.2.jar:3.0.2]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_72]
> at 
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>  ~[apache-cassandra-3.0.2.jar:3.0.2]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) 
> [apache-cassandra-3.0.2.jar:3.0.2]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
> {noformat}
> and also
> {noformat}
> ERROR 06:10:11 Exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.AssertionError: null
> at org.apache.cassandra.db.Slice$Bound.create(Slice.java:365) 
> ~[apache-cassandra-3.0.3.jar:3.0.3]
> at 
> org.apache.cassandra.db.Slice$Bound$Serializer.deserializeValues(Slice.java:553)
>  ~[apache-cassandra-3.0.3.jar:3.0.3]
> at 
> org.apache.cassandra.db.ClusteringPrefix$Serializer.deserialize(ClusteringPrefix.java:274)
>  ~[apache-cassandra-3.0.3.jar:3.0.3]
> at org.apache.cassandra.db.Serializers$2.deserialize(Serializers.java:115) 
> ~[apache-cassandra-3.0.3.jar:3.0.3]
> at org.apache.cassandra.db.Serializers$2.deserialize(Serializers.java:107) 
> ~[apache-cassandra-3.0.3.jar:3.0.3]
> at 
> org.apache.cassandra.io.sstable.IndexHelper$IndexInfo$Serializer.deserialize(IndexHelper.java:149)
>  ~[apache-cassandra-3.0.3.jar:3.0.3]
> at 
> org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:218)
>  ~[apache-cassandra-3.0.3.jar:3.0.3]
> at 
>

[jira] [Commented] (CASSANDRA-10625) Problem of year 10000: Dates too far in the future can be saved but not read back using cqlsh

2016-02-15 Thread Paulo Motta (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-10625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1514#comment-1514
 ] 

Paulo Motta commented on CASSANDRA-10625:
-

bq. How about a single warning, the first time it happens?

Sounds good! Will mark as ready to commit when CI is happy:

||3.0||trunk||
|[branch|https://github.com/apache/cassandra/compare/cassandra-3.0...pauloricardomg:3.0-10625]|[branch|https://github.com/apache/cassandra/compare/trunk...pauloricardomg:trunk-10625]|
|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-3.0-10625-testall/lastCompletedBuild/testReport/]|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-trunk-10625-testall/lastCompletedBuild/testReport/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-3.0-10625-dtest/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-trunk-10625-dtest/lastCompletedBuild/testReport/]|

committer: patch merges cleanly upwards.

> Problem of year 1: Dates too far in the future can be saved but not read 
> back using cqlsh
> -
>
> Key: CASSANDRA-10625
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10625
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Piotr Kołaczkowski
>Assignee: Adam Holmberg
>Priority: Minor
> Fix For: 3.0.x, 3.x
>
>
> {noformat}
> cqlsh> insert into test.timestamp_test (pkey, ts) VALUES (1, '-12-31 
> 23:59:59+');
> cqlsh> select * from test.timestamp_test ;
>  pkey | ts
> --+--
> 1 | -12-31 23:59:59+
> (1 rows)
> cqlsh> insert into test.timestamp_test (pkey, ts) VALUES (1, '1-01-01 
> 00:00:01+');
> cqlsh> select * from test.timestamp_test ;
> Traceback (most recent call last):
>   File "bin/../resources/cassandra/bin/cqlsh", line 1112, in 
> perform_simple_statement
> rows = self.session.execute(statement, trace=self.tracing_enabled)
>   File 
> "/home/pkolaczk/Projekty/DataStax/bdp/resources/cassandra/bin/../zipfiles/cassandra-driver-internal-only-2.7.2.zip/cassandra-driver-2.7.2/cassandra/cluster.py",
>  line 1602, in execute
> result = future.result()
>   File 
> "/home/pkolaczk/Projekty/DataStax/bdp/resources/cassandra/bin/../zipfiles/cassandra-driver-internal-only-2.7.2.zip/cassandra-driver-2.7.2/cassandra/cluster.py",
>  line 3347, in result
> raise self._final_exception
> OverflowError: date value out of range
> {noformat}
> The connection is broken afterwards:
> {noformat}
> cqlsh> insert into test.timestamp_test (pkey, ts) VALUES (1, '1-01-01 
> 00:00:01+');
> NoHostAvailable: ('Unable to complete the operation against any hosts', 
> {: ConnectionShutdown('Connection to 127.0.0.1 is 
> defunct',)})
> {noformat}
> Expected behaviors (one of):
> - don't allow to insert dates larger than -12-31 and document the 
> limitation
> - handle all dates up to Java Date(MAX_LONG) for writing and reading



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-10070) Automatic repair scheduling

2016-02-15 Thread Paulo Motta (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147751#comment-15147751
]

Paulo Motta edited comment on CASSANDRA-10070 at 2/15/16 8:07 PM:
--

Starting with a single repair per dc and adding support for parallel repair
sessions later sounds like a good idea.

bq. I agree and we could probably store the parent repair session id in an
extra column of the lock table and have a thread wake up periodically to see if
there are repair sessions without locks.

Do we intend to reuse the lock table for other maintenance tasks as well? If
so, we must add a generic "holder" column to the lock table so we can reuse to
identify resources other than the parent repair session in the future. We could
also add an "attributes" map in the lock table to store additional attributes
such as status, or have a separate table to maintain status to keep the lock
table simple.

bq. But then we must somehow be able to differentiate user-defined and
automatically scheduled repair sessions. It could be done by having all repairs
go through this scheduling interface, which also would reduce user mistakes
with multiple repairs in parallel. Another alternative is to have a custom flag
in the parent repair that makes the garbage collector ignore it if it's
user-defined. I think that the garbage collector/cancel repairs when unable to
lock feature is something that should be included in the first pass.

Ideally all repairs would go through this interface, but this would probably
add complexity at this stage. So we should probably just add a "lockResource"
attribute to each repair session object, and each node would go through all
repairs currently running checking if it still holds the lock in case the
"lockResource" field is set.

bq. The most basic failure scenarios should be covered by retrying a repair if
it fails and log a warning/error based on how many times it failed. Could the
retry behaviour cause some unexpected consequences?

It would probably be safe to abort ongoing validation and stream background
tasks and cleanup repair state on all involved nodes before starting a new
repair session in the same ranges. This doesn't seem to be done currently. As
far as I understood, if there are nodes A, B, C running repair, A is the
coordinator. If validation or streaming fails on node B, the coordinator (A) is
notified and fails the repair session, but node C will remain doing validation
and/or streaming, what could cause problems (or increased load) if we start
another repair session on the same range.

We will probably need to extend the repair protocol to perform this
cleanup/abort step on failure. We already have a legacy cleanup message that
doesn't seem to be used in the current protocol that we could maybe reuse to
cleanup repair state after a failure. This repair abortion will probably have
intersection with CASSANDRA-3486. In any case, this is a separate (but related)
issue and we should address it in an independent ticket, and make this ticket
dependent on that.

Another unrelated option that we should probably include in the future is a
timeout, and abort repair sessions running longer than that.

was (Author: pauloricardomg):
Starting with a single repair per dc and adding support for parallel repair
sessions later sounds like a good idea.

bq. I agree and we could probably store the parent repair session id in an
extra column of the lock table and have a thread wake up periodically to see if
there are repair sessions without locks.

Ideally all repairs would go through this interface, but this would probably
add complexity at this stage. So we should probably just add a "lockResource"
attribute to each local repair session object (as opposed to only the parent
repair object), and each node would go through all repairs currently running
checking if it still holds the lock if the "lockResource" field is

56 matches

Mail list logo