date:20150731


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14650055#comment-14650055
 ] 

Jason Brown commented on CASSANDRA-9945:


Updated code has been pushed to github branch, including the thread local cache 
of Cipher instances

 Add transparent data encryption core classes
 

 Key: CASSANDRA-9945
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9945
 Project: Cassandra
  Issue Type: Improvement
Reporter: Jason Brown
Assignee: Jason Brown
  Labels: encryption
 Fix For: 3.0 beta 1


 This patch will add the core infrastructure classes necessary for transparent 
 data encryption (file-level encryption), as required for CASSANDRA-6018 and 
 CASSANDRA-9633.  The phrase transparent data encryption, while not the most 
 aesthetically pleasing, seems to be used throughout the database industry 
 (Oracle, SQLQServer, Datastax Enterprise) to describe file level encryption, 
 so we'll go with that, as well. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7066) Simplify (and unify) cleanup of compaction leftovers

2015-07-31 Thread Benedict (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649193#comment-14649193
 ] 

Benedict commented on CASSANDRA-7066:
-

Thanks. Looks like great work as always. 

A few comments (I haven't done a full review):

* The last modified sort order was necessary for the same reason a full 
checksum of all of the old files is too heavy handed: restarting in the middle 
of deleting the files. We want to finish deleting them, and if we take the max 
timestamp we can say that will safely stay the same. We could keep the full 
checksum and include a count; if the count on disk is lower, and our max 
timestamp matches, we can proceed; if the count is the same we require a 
checksum match. This may or may not be overkill, but there's on harm in 
paranoia here.
* It looks like serialization of the sstables now assumes the ancestors won't 
be present, but this will break upgrade. We need to check the sstable version, 
and at least skip the ancestors if they're present
* The compaction metadata equality is probably not sensibly employed anywhere; 
we should just confirm this and remove it IMO since it makes very little sense, 
especially now that the ancestors have been removed
* I think you took my comments about panic warnings a little too literally. We 
should just use strong language to make it clear to the operator we've detected 
disk corruption of some kind, and have taken the most pessimistic course of 
action
** This warning should be less severe if only the last line is _incomplete_ 
(and all other lines are parsed correctly), as this would be expected if we 
crashed part way through serialization
** We should perhaps not worry at all, and continue if this last situation 
occurs, in the event that _all_ files logged are still present on the file 
system, and our other metadata all matches for them. In this case we can be 
very confident we just shutdown in the middle of writing a record, and can just 
clean up all of the new files. This should make us robust both to the scary 
situation, but also minimise the pessimism, hopefully giving us the best of 
both worlds.

WDYT?

 Simplify (and unify) cleanup of compaction leftovers
 

 Key: CASSANDRA-7066
 URL: https://issues.apache.org/jira/browse/CASSANDRA-7066
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Benedict
Assignee: Stefania
Priority: Minor
  Labels: benedict-to-commit, compaction
 Fix For: 3.0 alpha 1

 Attachments: 7066.txt


 Currently we manage a list of in-progress compactions in a system table, 
 which we use to cleanup incomplete compactions when we're done. The problem 
 with this is that 1) it's a bit clunky (and leaves us in positions where we 
 can unnecessarily cleanup completed files, or conversely not cleanup files 
 that have been superceded); and 2) it's only used for a regular compaction - 
 no other compaction types are guarded in the same way, so can result in 
 duplication if we fail before deleting the replacements.
 I'd like to see each sstable store in its metadata its direct ancestors, and 
 on startup we simply delete any sstables that occur in the union of all 
 ancestor sets. This way as soon as we finish writing we're capable of 
 cleaning up any leftovers, so we never get duplication. It's also much easier 
 to reason about.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9945) Add transparent data encryption core classes


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649208#comment-14649208
 ] 

T Jake Luciani commented on CASSANDRA-9945:
---

bq. you need to have the Java Cryptography Extension (JCE) Unlimited Strength 
Jurisdiction Policy Files 8 installed. It's a jar that can be downloaded from 
Oracle

Is this a requirement to run C*? Will you get a runtime exception if you don't 
have it and don't want to use any encryption? 

 Add transparent data encryption core classes
 

 Key: CASSANDRA-9945
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9945
 Project: Cassandra
  Issue Type: Improvement
Reporter: Jason Brown
Assignee: Jason Brown
  Labels: encryption
 Fix For: 3.0 beta 1


 This patch will add the core infrastructure classes necessary for transparent 
 data encryption (file-level encryption), as required for CASSANDRA-6018 and 
 CASSANDRA-9633.  The phrase transparent data encryption, while not the most 
 aesthetically pleasing, seems to be used throughout the database industry 
 (Oracle, SQLQServer, Datastax Enterprise) to describe file level encryption, 
 so we're go with that, as well. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-9947) nodetool verify is broken

Jonathan Ellis created CASSANDRA-9947:
-

 Summary: nodetool verify is broken
 Key: CASSANDRA-9947
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9947
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Jonathan Ellis
Priority: Critical
 Fix For: 2.2.x


Raised these issues on CASSANDRA-5791, but didn't revert/re-open, so they were 
ignored:

We mark sstables that fail verification as unrepaired, but that's not going to 
do what you think.  What it means is that the local node will use that sstable 
in the next repair, but other nodes will not. So all we'll end up doing is 
streaming whatever data we can read from it, to the other replicas.  If we 
could magically mark whatever sstables correspond on the remote nodes, to the 
data in the local sstable, that would work, but we can't.

IMO what we should do is:

*scrub, because it's quite likely we'll fail reading from the sstable 
otherwise and
*full repair across the data range covered by the sstable

Additionally,

* I'm not sure that keeping extended verify code around is worth it. Since 
the point is to work around not having a checksum, we could just scrub instead. 
This is slightly more heavyweight but it would be a one-time cost (scrub would 
build a new checksum) and we wouldn't have to worry about keeping two versions 
of almost-the-same-code in sync.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-9945) Add transparent data encryption core classes


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649209#comment-14649209
 ] 

Jason Brown edited comment on CASSANDRA-9945 at 7/31/15 1:39 PM:
-

[~tjake] no, JCE is only required if you enable transparent data encryption (of 
course, we'll ship with it disabled).


was (Author: jasobrown):
[~tjake] no, JCE is only required if you are using transparent data encryption.

 Add transparent data encryption core classes
 

 Key: CASSANDRA-9945
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9945
 Project: Cassandra
  Issue Type: Improvement
Reporter: Jason Brown
Assignee: Jason Brown
  Labels: encryption
 Fix For: 3.0 beta 1


 This patch will add the core infrastructure classes necessary for transparent 
 data encryption (file-level encryption), as required for CASSANDRA-6018 and 
 CASSANDRA-9633.  The phrase transparent data encryption, while not the most 
 aesthetically pleasing, seems to be used throughout the database industry 
 (Oracle, SQLQServer, Datastax Enterprise) to describe file level encryption, 
 so we're go with that, as well. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9945) Add transparent data encryption core classes


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649209#comment-14649209
 ] 

Jason Brown commented on CASSANDRA-9945:


[~tjake] no, JCE is only required if you are using transparent data encryption.

 Add transparent data encryption core classes
 

 Key: CASSANDRA-9945
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9945
 Project: Cassandra
  Issue Type: Improvement
Reporter: Jason Brown
Assignee: Jason Brown
  Labels: encryption
 Fix For: 3.0 beta 1


 This patch will add the core infrastructure classes necessary for transparent 
 data encryption (file-level encryption), as required for CASSANDRA-6018 and 
 CASSANDRA-9633.  The phrase transparent data encryption, while not the most 
 aesthetically pleasing, seems to be used throughout the database industry 
 (Oracle, SQLQServer, Datastax Enterprise) to describe file level encryption, 
 so we're go with that, as well. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7237) Optimize batchlog manager to avoid full scans


[ 
https://issues.apache.org/jira/browse/CASSANDRA-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649144#comment-14649144
 ] 

Aleksey Yeschenko commented on CASSANDRA-7237:
--

Could you please rebase - or, rather, reapply - on top of latest trukn, with 
CASSANDRA-8143 applied?

Thanks.

 Optimize batchlog manager to avoid full scans
 -

 Key: CASSANDRA-7237
 URL: https://issues.apache.org/jira/browse/CASSANDRA-7237
 Project: Cassandra
  Issue Type: Improvement
Reporter: Aleksey Yeschenko
Assignee: Branimir Lambov
Priority: Minor
 Fix For: 3.0.0 rc1


 Now that we use time-UUIDs for batchlog ids, and given that w/ local strategy 
 the partitions are ordered in time-order here, we can optimize the scanning 
 by limiting the range to replay taking the last replayed batch's id as the 
 beginning of the range, and uuid(now+timeout) as its end.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8143) Partitioner should not be accessed through StorageService


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649142#comment-14649142
 ] 

Aleksey Yeschenko commented on CASSANDRA-8143:
--

Thanks. The patch LGTM, with some nits. Committed to trunk as 
{{69f77cbddd4c74448f227e9aceef84d345118184}}.

Nits:
- redundant {{@Override}} for {{BOP.partitionOrdering()}}
- there is no point in checking for a partitioner in the legacy schema tables 
while migrating, this ticket being a 3.0 feature
- {{All.get(0).decorateKey}} in {{SchemaKeyspace.makeDropKeyspaceMutation()}} 
is potentially error-prone. Should just be {{Keyspaces.decorateKey()}} instead 
of relying on implicit order

Minor issue:
- we aren't going to store the partitioner in schema, since we aren't (yet? 
ever?) going to allow creating tables with non-default partitioner via CQL, 
externally. For now it's only 2i tables, the new batchlog table, and some 
hard-coded tables in one external project. All of them with hardcoded 
definitions, partitioner redefined explicitly in code.

Some [minor 
changes|https://github.com/iamaleksey/cassandra/commit/40b0d7bd74a10f183290c1f07411d8ea9ac67632]
 I took liberty to make on top (besides fixing the nits):
1. {{partitioner instanceof LocalPartitioner}} is not a sure-fire way to test 
for 2i after this patch (and I guess it's taken care of in CASSANDRA-7237), and 
scanning for the dot in the name has always been sort of ugly. I pulled 
{{isIndex}} into a {{boolean}} field in {{CFMetaData}} instead
2. As you've noticed, {{HHOM.listEndpointsPendingHints()}} logic is weird. It 
was first changed to this in CASSANDRA-4568, back when Ice Giants were roaming 
the Earth and the structure of the hints table was different. Even then, 
though, the change was wrong, the token of the node (pre-vnodes) was still 
being stored in the partition key. Fixed it in place to return the stringified 
UUID instead. Seems like nobody cares though, this being broken since 1.0.12


 Partitioner should not be accessed through StorageService
 -

 Key: CASSANDRA-8143
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8143
 Project: Cassandra
  Issue Type: Improvement
Reporter: Branimir Lambov
Assignee: Branimir Lambov
 Fix For: 3.0 beta 1


 The configured partitioner is no longer the only partitioner in use in the 
 database, as e.g. index tables use LocalPartitioner.
 To make sure the correct partitioner is used for each table, accesses of 
 StorageService.getPartitioner() should be replaced with retrieval of the 
 CFS-specific partitioner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9947) nodetool verify is broken


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649199#comment-14649199
 ] 

Jonathan Ellis commented on CASSANDRA-9947:
---

IMO we should disable verify for 2.2.1 until we can rearchitect it since this 
is a nontrivial change.

 nodetool verify is broken
 -

 Key: CASSANDRA-9947
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9947
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Jonathan Ellis
Priority: Critical
 Fix For: 2.2.x


 Raised these issues on CASSANDRA-5791, but didn't revert/re-open, so they 
 were ignored:
 We mark sstables that fail verification as unrepaired, but that's not going 
 to do what you think.  What it means is that the local node will use that 
 sstable in the next repair, but other nodes will not. So all we'll end up 
 doing is streaming whatever data we can read from it, to the other replicas.  
 If we could magically mark whatever sstables correspond on the remote nodes, 
 to the data in the local sstable, that would work, but we can't.
 IMO what we should do is:
 *scrub, because it's quite likely we'll fail reading from the sstable 
 otherwise and
 *full repair across the data range covered by the sstable
 Additionally,
 * I'm not sure that keeping extended verify code around is worth it. Since 
 the point is to work around not having a checksum, we could just scrub 
 instead. This is slightly more heavyweight but it would be a one-time cost 
 (scrub would build a new checksum) and we wouldn't have to worry about 
 keeping two versions of almost-the-same-code in sync.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-5791) A nodetool command to validate all sstables in a node


[ 
https://issues.apache.org/jira/browse/CASSANDRA-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649200#comment-14649200
 ] 

Jonathan Ellis commented on CASSANDRA-5791:
---

Created CASSANDRA-9947 to follow up.

 A nodetool command to validate all sstables in a node
 -

 Key: CASSANDRA-5791
 URL: https://issues.apache.org/jira/browse/CASSANDRA-5791
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: sankalp kohli
Assignee: Jeff Jirsa
Priority: Minor
 Fix For: 2.2.0 beta 1

 Attachments: cassandra-5791-20150319.diff, 
 cassandra-5791-patch-3.diff, cassandra-5791.patch-2


 CUrrently there is no nodetool command to validate all sstables on disk. The 
 only way to do this is to run a repair and see if it succeeds. But we cannot 
 repair the system keyspace. 
 Also we can run upgrade sstables but that re writes all the sstables. 
 This command should check the hash of all sstables and return whether all 
 data is readable all not. This should NOT care about consistency. 
 The compressed sstables do not have hash so not sure how it will work there.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8777) Streaming operations should log both endpoint and port associated with the operation

2015-07-31 Thread Ryan Magnusson (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649328#comment-14649328
 ] 

Ryan Magnusson commented on CASSANDRA-8777:
---

I would like to take this on, but have some questions to clarify the request. 
The description says that we currently log the endpoint for a streaming 
operation, but in the example provided I only see the planId being shown. I 
see in some parts of the code we do log out InetAddress, but it isn't in the 
example provided.
Is the goal of this task to also add the address to messages for stream 
operations where it currently is not shown? 

 Streaming operations should log both endpoint and port associated with the 
 operation
 

 Key: CASSANDRA-8777
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8777
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Jeremy Hanna
  Labels: lhf
 Fix For: 2.1.x


 Currently we log the endpoint for a streaming operation.  If the port has 
 been overridden, it would be valuable to know that that setting is getting 
 picked up.  Therefore, when logging the endpoint address, it would be nice to 
 also log the port it's trying to use.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (CASSANDRA-8368) Consider not using hints for batchlog replay, in any capacity

[
https://issues.apache.org/jira/browse/CASSANDRA-8368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

T Jake Luciani reopened CASSANDRA-8368:
---

This is now more of an issue with CASSANDRA-6477

To ensure consistency we shouldn't rely on hints.

Consider not using hints for batchlog replay, in any capacity
-

Key: CASSANDRA-8368
URL: https://issues.apache.org/jira/browse/CASSANDRA-8368
Project: Cassandra
Issue Type: Bug
Reporter: Aleksey Yeschenko

Currently, when replaying a batch, if a request times out, we simply write a
hint for it and call it a day.
It's simple, but it does tie us to hints, which some people prefer to disable
altogether (and some still will even after CASSANDRA-6230).
It also potentially violates the consistency level of the original request.
As an alternative, once CASSANDRA-7237 is complete, I suggest we stop relying
on hints at all, and do this instead:
1. Store the consistency level as batch metadata
2. On replay, hint in case of a timeout, but not if the node is down as per FD
3. If CL is met, consider the batch replayed and discard it, but not account
the hints towards CL (as per usual write patch), unless CL.ANY is being used
4. If CL is *not* met, write a new batch with contents of the current one,
but with timeuuid set in the future, for later replay (delayed by fixed
configurable time or exponentially backed off). With that new batch store the
list of nodes we've delivered the hint to, so that next time we replay it we
don't waste writes.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-8368) Consider not using hints for batchlog replay, in any capacity

[
https://issues.apache.org/jira/browse/CASSANDRA-8368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

T Jake Luciani updated CASSANDRA-8368:
--
Fix Version/s: 3.0.0 rc1

Consider not using hints for batchlog replay, in any capacity
-

Key: CASSANDRA-8368
URL: https://issues.apache.org/jira/browse/CASSANDRA-8368
Project: Cassandra
Issue Type: Bug
Reporter: Aleksey Yeschenko
Fix For: 3.0.0 rc1

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8368) Consider not using hints for batchlog replay, in any capacity

[
https://issues.apache.org/jira/browse/CASSANDRA-8368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649390#comment-14649390
]

Aleksey Yeschenko commented on CASSANDRA-8368:
--

I wouldn't mind, if only because it would allow to simplify CASSANDRA-6230
further, but what alternatives do you suggest?

Consider not using hints for batchlog replay, in any capacity
-

Key: CASSANDRA-8368
URL: https://issues.apache.org/jira/browse/CASSANDRA-8368
Project: Cassandra
Issue Type: Bug
Reporter: Aleksey Yeschenko
Fix For: 3.0.0 rc1

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-9941) SSTableVerify has wrong return codes on windows


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie reassigned CASSANDRA-9941:
--

Assignee: Joshua McKenzie

 SSTableVerify has wrong return codes on windows
 ---

 Key: CASSANDRA-9941
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9941
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
Reporter: Philip Thompson
Assignee: Joshua McKenzie
 Fix For: 2.2.x


 The dtest {{offline_tools_test.py:TestOfflineTools.sstableverify_test}} is 
 failing on windows. When running sstableverify.bat on a nonexistent keyspace, 
 the correct error message is printed, but an error code of 0 for success is 
 returned, which does not match the correct behavior on linux, which returns 1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8368) Consider not using hints for batchlog replay, in any capacity


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649460#comment-14649460
 ] 

T Jake Luciani commented on CASSANDRA-8368:
---

I don't have a clear design but sounds like you did?

{quote}
as an alternative, once CASSANDRA-7237 is complete, I suggest we stop relying 
on hints at all, and do this instead:
1. Store the consistency level as batch metadata
2. On replay, hint in case of a timeout, but not if the node is down as per FD
3. If CL is met, consider the batch replayed and discard it, but not account 
the hints towards CL (as per usual write patch), unless CL.ANY is being used
4. If CL is not met, write a new batch with contents of the current one, but 
with timeuuid set in the future, for later replay (delayed by fixed 
configurable time or exponentially backed off). With that new batch store the 
list of nodes we've delivered the hint to, so that next time we replay it we 
don't waste writes.
{quote}


My concern is we currently don't provide atomicity by defaulting to hints.  So 
if all replicas are down we end up CL.ANY which is not correct or safe since 
hints can be disabled or dropped after max_hint_window_in_ms

{code}
ReplayWriteResponseHandlerMutation handler = replayHandlers.get(i);
try
{
handler.get();
}
catch (WriteTimeoutException|WriteFailureException e)
{
logger.debug(Failed replaying a batched mutation to a 
node, will write a hint);
logger.debug(Failure was : {}, e.getMessage());
// writing hints for the rest to hints, starting from i
writeHintsForUndeliveredEndpoints(i);
return;
}
{code}

 Consider not using hints for batchlog replay, in any capacity
 -

 Key: CASSANDRA-8368
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8368
 Project: Cassandra
  Issue Type: Bug
Reporter: Aleksey Yeschenko
 Fix For: 3.0.0 rc1


 Currently, when replaying a batch, if a request times out, we simply write a 
 hint for it and call it a day.
 It's simple, but it does tie us to hints, which some people prefer to disable 
 altogether (and some still will even after CASSANDRA-6230).
 It also potentially violates the consistency level of the original request.
 As an alternative, once CASSANDRA-7237 is complete, I suggest we stop relying 
 on hints at all, and do this instead:
 1. Store the consistency level as batch metadata
 2. On replay, hint in case of a timeout, but not if the node is down as per FD
 3. If CL is met, consider the batch replayed and discard it, but not account 
 the hints towards CL (as per usual write patch), unless CL.ANY is being used
 4. If CL is *not* met, write a new batch with contents of the current one, 
 but with timeuuid set in the future, for later replay (delayed by fixed 
 configurable time or exponentially backed off). With that new batch store the 
 list of nodes we've delivered the hint to, so that next time we replay it we 
 don't waste writes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8368) Consider not using hints for batchlog replay, in any capacity

[
https://issues.apache.org/jira/browse/CASSANDRA-8368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649395#comment-14649395
]

T Jake Luciani commented on CASSANDRA-8368:
---

The batch itself should have a minimum CL level it must reach before it's
cleaned up.

Consider not using hints for batchlog replay, in any capacity
-

Key: CASSANDRA-8368
URL: https://issues.apache.org/jira/browse/CASSANDRA-8368
Project: Cassandra
Issue Type: Bug
Reporter: Aleksey Yeschenko
Fix For: 3.0.0 rc1

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9946) use ioprio_set on compaction threads by default instead of manually throttling


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649393#comment-14649393
 ] 

Joshua McKenzie commented on CASSANDRA-9946:


I think the right Windows analog would be 
[SetThreadPriority|https://msdn.microsoft.com/en-us/library/windows/desktop/ms686277(v=vs.85).aspx],
 specifically using THREAD_MODE_BACKGROUND_BEGIN. I'm assuming we can use 
something like 
[GetCurrentThread|https://msdn.microsoft.com/en-us/library/windows/desktop/ms683182(v=vs.85).aspx]
 via jna and use the HANDLE from there but I'd have to tinker with it to see 
how it behaves; for some reason I have a recollection of getting tid's being 
dicey w/regards to java's internal threading model.

 use ioprio_set on compaction threads by default instead of manually throttling
 --

 Key: CASSANDRA-9946
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9946
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Jonathan Ellis
Assignee: Ariel Weisberg
 Fix For: 3.x


 Compaction throttling works as designed, but it has two drawbacks:
 * it requires manual tuning to choose the right value for a given machine
 * it does not allow compaction to burst above its limit if there is 
 additional i/o capacity available while there are less application requests 
 to serve
 Using ioprio_set instead solves both of these problems.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-9924) Error when invoking csqlsh

2015-07-31 Thread Philip Thompson (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson updated CASSANDRA-9924:
---
Fix Version/s: (was: 2.1.x)

 Error when invoking csqlsh
 --

 Key: CASSANDRA-9924
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9924
 Project: Cassandra
  Issue Type: Bug
Reporter: philip
  Labels: cqlsh

 Getting Error as below.
 {code}
  bin]# ./cqlsh
 Traceback (most recent call last):
   File ./cqlsh, line 134, in module
 from cqlshlib import cqlhandling, cql3handling, pylexotron, sslhandling
   File 
 /u01/cassandra/usr/share/dse/resources/cassandra/bin/../pylib/cqlshlib/sslhandling.py,
  line 20, in module
 import ssl
   File /usr/local/lib/python2.6/ssl.py, line 60, in module
 import _ssl # if we can't import it, let the error propagate
 ImportError: No module named _ssl
 {code}
 Our versions are as below.
  cat /etc/redhat-release
 Red Hat Enterprise Linux Server release 6.6 (Santiago)
  bin]# python --version
 Python 2.6.9
 Pls help us resolve this issue.
 Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8368) Consider not using hints for batchlog replay, in any capacity

[
https://issues.apache.org/jira/browse/CASSANDRA-8368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649406#comment-14649406
]

Aleksey Yeschenko commented on CASSANDRA-8368:
--

CL level is something that BL explicitly isn't involved in (and you'll have to
be a bit more specific about what exactly your suggestion is - what the new
replay process would look like).

Consider not using hints for batchlog replay, in any capacity
-

Key: CASSANDRA-8368
URL: https://issues.apache.org/jira/browse/CASSANDRA-8368
Project: Cassandra
Issue Type: Bug
Reporter: Aleksey Yeschenko
Fix For: 3.0.0 rc1

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8368) Consider not using hints for batchlog replay, in any capacity

[
https://issues.apache.org/jira/browse/CASSANDRA-8368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649407#comment-14649407
]

Aleksey Yeschenko commented on CASSANDRA-8368:
--

And assume that CASSANDRA-7237 is committed, because it soon will be, while
devising a new procedure.

Consider not using hints for batchlog replay, in any capacity
-

Key: CASSANDRA-8368
URL: https://issues.apache.org/jira/browse/CASSANDRA-8368
Project: Cassandra
Issue Type: Bug
Reporter: Aleksey Yeschenko
Fix For: 3.0.0 rc1

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-9948) cql3.validation.entities.UFTest.testScriptParamReturnTypes timing out on Windows

Joshua McKenzie created CASSANDRA-9948:
--

 Summary: 
cql3.validation.entities.UFTest.testScriptParamReturnTypes timing out on Windows
 Key: CASSANDRA-9948
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9948
 Project: Cassandra
  Issue Type: Bug
Reporter: Joshua McKenzie
Priority: Minor
 Fix For: 2.2.x


Test times out in CI. Completes successfully locally so it might be a timing / 
horsepower / long-running test issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9940) ReadResponse serializes and then deserializes local responses

2015-07-31 Thread Sylvain Lebresne (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649189#comment-14649189
 ] 

Sylvain Lebresne commented on CASSANDRA-9940:
-

If it's just the serialization-deserialization cost we want to avoid, then it 
should be trivial to put everything in an {{ArrayBackedPartition}} (would be 
nice to benchmark it for good measure though). Avoiding pulling it all in 
memory is harder though, not only due to {{OpOrder}} reasons pointed out by 
Benedict, but also because in some case we need to both return the result and 
digest it. Both are likely solvable in some way, but will most probably require 
refactor that goes way beyond just {{ReadResponse}}. So I'd personally stick to 
just pulling it all in a {{Parttion}} object for now (instead of 
serializing-deserializing), but that's just my 2 cents. 

 ReadResponse serializes and then deserializes local responses
 -

 Key: CASSANDRA-9940
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9940
 Project: Cassandra
  Issue Type: Improvement
  Components: Core
Reporter: Ariel Weisberg
Assignee: Ariel Weisberg

 Noticed this reviewing CASSANDRA-9894. It would be nice to not have to do 
 this busy work. Benedict said it wasn't straightforward to avoid because it's 
 being done to allow the read op order group to close.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-9945) Add transparent data encryption core classes


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown updated CASSANDRA-9945:
---
Description: This patch will add the core infrastructure classes necessary 
for transparent data encryption (file-level encryption), as required for 
CASSANDRA-6018 and CASSANDRA-9633.  The phrase transparent data encryption, 
while not the most aesthetically pleasing, seems to be used throughout the 
database industry (Oracle, SQLQServer, Datastax Enterprise) to describe file 
level encryption, so we'll go with that, as well.   (was: This patch will add 
the core infrastructure classes necessary for transparent data encryption 
(file-level encryption), as required for CASSANDRA-6018 and CASSANDRA-9633.  
The phrase transparent data encryption, while not the most aesthetically 
pleasing, seems to be used throughout the database industry (Oracle, 
SQLQServer, Datastax Enterprise) to describe file level encryption, so we're go 
with that, as well. )

 Add transparent data encryption core classes
 

 Key: CASSANDRA-9945
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9945
 Project: Cassandra
  Issue Type: Improvement
Reporter: Jason Brown
Assignee: Jason Brown
  Labels: encryption
 Fix For: 3.0 beta 1


 This patch will add the core infrastructure classes necessary for transparent 
 data encryption (file-level encryption), as required for CASSANDRA-6018 and 
 CASSANDRA-9633.  The phrase transparent data encryption, while not the most 
 aesthetically pleasing, seems to be used throughout the database industry 
 (Oracle, SQLQServer, Datastax Enterprise) to describe file level encryption, 
 so we'll go with that, as well. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-9945) Add transparent data encryption core classes

2015-07-31 Thread Robert Stupp (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Stupp updated CASSANDRA-9945:

Reviewer: Robert Stupp

 Add transparent data encryption core classes
 

 Key: CASSANDRA-9945
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9945
 Project: Cassandra
  Issue Type: Improvement
Reporter: Jason Brown
Assignee: Jason Brown
  Labels: encryption
 Fix For: 3.0 beta 1


 This patch will add the core infrastructure classes necessary for transparent 
 data encryption (file-level encryption), as required for CASSANDRA-6018 and 
 CASSANDRA-9633.  The phrase transparent data encryption, while not the most 
 aesthetically pleasing, seems to be used throughout the database industry 
 (Oracle, SQLQServer, Datastax Enterprise) to describe file level encryption, 
 so we're go with that, as well. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9031) nodetool info -T throws ArrayOutOfBounds when the node has not joined the cluster

2015-07-31 Thread Yuki Morishita (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649498#comment-14649498
 ] 

Yuki Morishita commented on CASSANDRA-9031:
---

Sorry for the delay.
{{TokenMetadata#getToken}} throws AssertionException when the node is not yet a 
member of the cluster.
Changing that methods behavior is a bit scary since it is used in other parts.

Instead, we can use host ID to get endpoint from nodetool.
That way, we can display DC/Rack as well.

patch pushed to here: https://github.com/yukim/cassandra/tree/9031

 nodetool info -T throws ArrayOutOfBounds when the node has not joined the 
 cluster
 -

 Key: CASSANDRA-9031
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9031
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
Reporter: Ron Kuris
 Fix For: 2.1.x

 Attachments: patch.txt


 To reproduce, bring up a node that does not join the cluster, either using 
 -Dcassandra.write_survey=true or -Dcassandra.join_ring=false, then run 
 'nodetool info -T'. You'll get the following stack trace:
 {code}ID : e384209f-f7a9-4cff-8fd5-03adfaa0d846
 Gossip active  : true
 Thrift active  : true
 Native Transport active: true
 Load   : 76.69 KB
 Generation No  : 1427229938
 Uptime (seconds)   : 728
 Heap Memory (MB)   : 109.93 / 826.00
 Off Heap Memory (MB)   : 0.01
 Exception in thread main java.lang.IndexOutOfBoundsException: Index: 0, 
 Size: 0
   at java.util.ArrayList.rangeCheck(ArrayList.java:635)
   at java.util.ArrayList.get(ArrayList.java:411)
   at org.apache.cassandra.tools.NodeProbe.getEndpoint(NodeProbe.java:676)
   at 
 org.apache.cassandra.tools.NodeProbe.getDataCenter(NodeProbe.java:694)
   at org.apache.cassandra.tools.NodeCmd.printInfo(NodeCmd.java:666)
   at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:1277){code}
 After applying the attached patch, the new error is:
 {code}ID : a7d76a2a-82d2-4faa-94e1-a30df6663ebb
 Gossip active  : true
 Thrift active  : false
 Native Transport active: false
 Load   : 89.36 KB
 Generation No  : 1427231804
 Uptime (seconds)   : 12
 Heap Memory (MB)   : 135.49 / 826.00
 Off Heap Memory (MB)   : 0.01
 Exception in thread main java.lang.RuntimeException: This node does not 
 have any tokens. Perhaps it is not part of the ring?
   at org.apache.cassandra.tools.NodeProbe.getEndpoint(NodeProbe.java:678)
   at 
 org.apache.cassandra.tools.NodeProbe.getDataCenter(NodeProbe.java:698)
   at org.apache.cassandra.tools.NodeCmd.printInfo(NodeCmd.java:676)
   at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:1313){code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/4] cassandra git commit: Revert Stop accessing the partitioner directly via StorageService

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a22ce89e/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
--
diff --git a/src/java/org/apache/cassandra/db/index/SecondaryIndex.java 
b/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
index 9221090..94031ab 100644
--- a/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
+++ b/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
@@ -42,12 +42,15 @@ import org.apache.cassandra.db.index.keys.KeysIndex;
 import org.apache.cassandra.db.lifecycle.SSTableSet;
 import org.apache.cassandra.db.lifecycle.View;
 import org.apache.cassandra.db.marshal.AbstractType;
-import org.apache.cassandra.dht.LocalPartitioner;
+import org.apache.cassandra.db.marshal.BytesType;
+import org.apache.cassandra.db.marshal.LocalByPartionerType;
 import org.apache.cassandra.exceptions.ConfigurationException;
 import org.apache.cassandra.io.sstable.format.SSTableReader;
 import org.apache.cassandra.exceptions.InvalidRequestException;
 import org.apache.cassandra.io.sstable.ReducingKeyIterator;
+import org.apache.cassandra.service.StorageService;
 import org.apache.cassandra.utils.FBUtilities;
+
 import org.apache.cassandra.utils.concurrent.Refs;
 
 /**
@@ -76,6 +79,10 @@ public abstract class SecondaryIndex
  */
 public static final String INDEX_ENTRIES_OPTION_NAME = 
index_keys_and_values;
 
+public static final AbstractType? keyComparator = 
StorageService.getPartitioner().preservesOrder()
+  ? BytesType.instance
+  : new 
LocalByPartionerType(StorageService.getPartitioner());
+
 /**
  * Base CF that has many indexes
  */
@@ -296,7 +303,7 @@ public abstract class SecondaryIndex
  */
 public DecoratedKey getIndexKeyFor(ByteBuffer value)
 {
-return getIndexCfs().decorateKey(value);
+return getIndexCfs().partitioner.decorateKey(value);
 }
 
 /**
@@ -374,20 +381,11 @@ public abstract class SecondaryIndex
  */
 public static CFMetaData newIndexMetadata(CFMetaData baseMetadata, 
ColumnDefinition def)
 {
-return newIndexMetadata(baseMetadata, def, def.type);
-}
-
-/**
- * Create the index metadata for the index on a given column of a given 
table.
- */
-static CFMetaData newIndexMetadata(CFMetaData baseMetadata, 
ColumnDefinition def, AbstractType? comparator)
-{
 if (def.getIndexType() == IndexType.CUSTOM)
 return null;
 
 CFMetaData.Builder builder = 
CFMetaData.Builder.create(baseMetadata.ksName, 
baseMetadata.indexColumnFamilyName(def))

.withId(baseMetadata.cfId)
-   .withPartitioner(new 
LocalPartitioner(comparator))

.addPartitionKey(def.name, def.type);
 
 if (def.getIndexType() == IndexType.COMPOSITES)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a22ce89e/src/java/org/apache/cassandra/db/index/composites/CompositesIndex.java
--
diff --git 
a/src/java/org/apache/cassandra/db/index/composites/CompositesIndex.java 
b/src/java/org/apache/cassandra/db/index/composites/CompositesIndex.java
index 29f235c..42861c5 100644
--- a/src/java/org/apache/cassandra/db/index/composites/CompositesIndex.java
+++ b/src/java/org/apache/cassandra/db/index/composites/CompositesIndex.java
@@ -101,7 +101,7 @@ public abstract class CompositesIndex extends 
AbstractSimplePerColumnSecondaryIn
 
 protected static void addGenericClusteringColumns(CFMetaData.Builder 
indexMetadata, CFMetaData baseMetadata, ColumnDefinition columnDef)
 {
-indexMetadata.addClusteringColumn(partition_key, 
baseMetadata.partitioner.partitionOrdering());
+indexMetadata.addClusteringColumn(partition_key, 
SecondaryIndex.keyComparator);
 for (ColumnDefinition def : baseMetadata.clusteringColumns())
 indexMetadata.addClusteringColumn(def.name, def.type);
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a22ce89e/src/java/org/apache/cassandra/db/index/composites/CompositesIndexOnClusteringKey.java
--
diff --git 
a/src/java/org/apache/cassandra/db/index/composites/CompositesIndexOnClusteringKey.java
 
b/src/java/org/apache/cassandra/db/index/composites/CompositesIndexOnClusteringKey.java
index cd4aff9..6529ad9 100644
--- 
a/src/java/org/apache/cassandra/db/index/composites/CompositesIndexOnClusteringKey.java
+++ 
b/src/java/org/apache/cassandra/db/index/composites/CompositesIndexOnClusteringKey.java
@@ -24,6 +24,7 @@ import org.apache.cassandra.config.CFMetaData;
 import org.apache.cassandra.config.ColumnDefinition;
 import

[jira] [Commented] (CASSANDRA-9935) Repair fails with RuntimeException

2015-07-31 Thread mlowicki (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649575#comment-14649575
 ] 

mlowicki commented on CASSANDRA-9935:
-

Failed with the same error after ~13 hours:
{code}
[2015-07-31 16:57:43,909] Repair command #5 finished
error: nodetool failed, check server logs
-- StackTrace --
java.lang.RuntimeException: nodetool failed, check server logs
at 
org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:290)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:202)
{code}

Log file - 
https://drive.google.com/file/d/0B_8mc_afWmd2OV96RDZBclRNSFE/view?usp=sharing.

 Repair fails with RuntimeException
 --

 Key: CASSANDRA-9935
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9935
 Project: Cassandra
  Issue Type: Bug
 Environment: C* 2.1.8, Debian Wheezy
Reporter: mlowicki
Assignee: Yuki Morishita
 Attachments: db1.sync.lati.osa.cassandra.log, 
 db5.sync.lati.osa.cassandra.log


 We had problems with slow repair in 2.1.7 (CASSANDRA-9702) but after upgrade 
 to 2.1.8 it started to work faster but now it fails with:
 {code}
 ...
 [2015-07-29 20:44:03,956] Repair session 23a811b0-3632-11e5-a93e-4963524a8bde 
 for range (-5474076923322749342,-5468600594078911162] finished
 [2015-07-29 20:44:03,957] Repair session 336f8740-3632-11e5-a93e-4963524a8bde 
 for range (-8631877858109464676,-8624040066373718932] finished
 [2015-07-29 20:44:03,957] Repair session 4ccd8430-3632-11e5-a93e-4963524a8bde 
 for range (-5372806541854279315,-5369354119480076785] finished
 [2015-07-29 20:44:03,957] Repair session 59f129f0-3632-11e5-a93e-4963524a8bde 
 for range (8166489034383821955,8168408930184216281] finished
 [2015-07-29 20:44:03,957] Repair session 6ae7a9a0-3632-11e5-a93e-4963524a8bde 
 for range (6084602890817326921,6088328703025510057] finished
 [2015-07-29 20:44:03,957] Repair session 8938e4a0-3632-11e5-a93e-4963524a8bde 
 for range (-781874602493000830,-781745173070807746] finished
 [2015-07-29 20:44:03,957] Repair command #4 finished
 error: nodetool failed, check server logs
 -- StackTrace --
 java.lang.RuntimeException: nodetool failed, check server logs
 at 
 org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:290)
 at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:202)
 {code}
 After running:
 {code}
 nodetool repair --partitioner-range --parallel --in-local-dc sync
 {code}
 Last records in logs regarding repair are:
 {code}
 INFO  [Thread-173887] 2015-07-29 20:44:03,956 StorageService.java:2952 - 
 Repair session 09ff9e40-3632-11e5-a93e-4963524a8bde for range 
 (-7695808664784761779,-7693529816291585568] finished
 INFO  [Thread-173887] 2015-07-29 20:44:03,956 StorageService.java:2952 - 
 Repair session 17d8d860-3632-11e5-a93e-4963524a8bde for range 
 (806371695398849,8065203836608925992] finished
 INFO  [Thread-173887] 2015-07-29 20:44:03,956 StorageService.java:2952 - 
 Repair session 23a811b0-3632-11e5-a93e-4963524a8bde for range 
 (-5474076923322749342,-5468600594078911162] finished
 INFO  [Thread-173887] 2015-07-29 20:44:03,956 StorageService.java:2952 - 
 Repair session 336f8740-3632-11e5-a93e-4963524a8bde for range 
 (-8631877858109464676,-8624040066373718932] finished
 INFO  [Thread-173887] 2015-07-29 20:44:03,957 StorageService.java:2952 - 
 Repair session 4ccd8430-3632-11e5-a93e-4963524a8bde for range 
 (-5372806541854279315,-5369354119480076785] finished
 INFO  [Thread-173887] 2015-07-29 20:44:03,957 StorageService.java:2952 - 
 Repair session 59f129f0-3632-11e5-a93e-4963524a8bde for range 
 (8166489034383821955,8168408930184216281] finished
 INFO  [Thread-173887] 2015-07-29 20:44:03,957 StorageService.java:2952 - 
 Repair session 6ae7a9a0-3632-11e5-a93e-4963524a8bde for range 
 (6084602890817326921,6088328703025510057] finished
 INFO  [Thread-173887] 2015-07-29 20:44:03,957 StorageService.java:2952 - 
 Repair session 8938e4a0-3632-11e5-a93e-4963524a8bde for range 
 (-781874602493000830,-781745173070807746] finished
 {code}
 but a bit above I see (at least two times in attached log):
 {code}
 ERROR [Thread-173887] 2015-07-29 20:44:03,853 StorageService.java:2959 - 
 Repair session 1b07ea50-3608-11e5-a93e-4963524a8bde for range 
 (5765414319217852786,5781018794516851576] failed with error 
 org.apache.cassandra.exceptions.RepairException: [repair 
 #1b07ea50-3608-11e5-a93e-4963524a8bde on sync/entity_by_id2, 
 (5765414319217852786,5781018794516851576]] Validation failed in /10.195.15.162
 java.util.concurrent.ExecutionException: java.lang.RuntimeException: 
 org.apache.cassandra.exceptions.RepairException: [repair 
 #1b07ea50-3608-11e5-a93e-4963524a8bde on sync/entity_by_id2, 
 (5765414319217852786,5781018794516851576]] Validation failed in /10.195.15.162
 at

[jira] [Comment Edited] (CASSANDRA-9935) Repair fails with RuntimeException

2015-07-31 Thread mlowicki (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649575#comment-14649575
 ] 

mlowicki edited comment on CASSANDRA-9935 at 7/31/15 6:19 PM:
--

Failed with the same error after ~13 hours:
{code}
[2015-07-31 16:57:43,909] Repair command #5 finished
error: nodetool failed, check server logs
-- StackTrace --
java.lang.RuntimeException: nodetool failed, check server logs
at 
org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:290)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:202)
{code}

Log file - 
https://drive.google.com/file/d/0B_8mc_afWmd2OV96RDZBclRNSFE/view?usp=sharing.

Tried yesterday to run repair in another DC but got the same error.


was (Author: mlowicki):
Failed with the same error after ~13 hours:
{code}
[2015-07-31 16:57:43,909] Repair command #5 finished
error: nodetool failed, check server logs
-- StackTrace --
java.lang.RuntimeException: nodetool failed, check server logs
at 
org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:290)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:202)
{code}

Log file - 
https://drive.google.com/file/d/0B_8mc_afWmd2OV96RDZBclRNSFE/view?usp=sharing.

 Repair fails with RuntimeException
 --

 Key: CASSANDRA-9935
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9935
 Project: Cassandra
  Issue Type: Bug
 Environment: C* 2.1.8, Debian Wheezy
Reporter: mlowicki
Assignee: Yuki Morishita
 Attachments: db1.sync.lati.osa.cassandra.log, 
 db5.sync.lati.osa.cassandra.log


 We had problems with slow repair in 2.1.7 (CASSANDRA-9702) but after upgrade 
 to 2.1.8 it started to work faster but now it fails with:
 {code}
 ...
 [2015-07-29 20:44:03,956] Repair session 23a811b0-3632-11e5-a93e-4963524a8bde 
 for range (-5474076923322749342,-5468600594078911162] finished
 [2015-07-29 20:44:03,957] Repair session 336f8740-3632-11e5-a93e-4963524a8bde 
 for range (-8631877858109464676,-8624040066373718932] finished
 [2015-07-29 20:44:03,957] Repair session 4ccd8430-3632-11e5-a93e-4963524a8bde 
 for range (-5372806541854279315,-5369354119480076785] finished
 [2015-07-29 20:44:03,957] Repair session 59f129f0-3632-11e5-a93e-4963524a8bde 
 for range (8166489034383821955,8168408930184216281] finished
 [2015-07-29 20:44:03,957] Repair session 6ae7a9a0-3632-11e5-a93e-4963524a8bde 
 for range (6084602890817326921,6088328703025510057] finished
 [2015-07-29 20:44:03,957] Repair session 8938e4a0-3632-11e5-a93e-4963524a8bde 
 for range (-781874602493000830,-781745173070807746] finished
 [2015-07-29 20:44:03,957] Repair command #4 finished
 error: nodetool failed, check server logs
 -- StackTrace --
 java.lang.RuntimeException: nodetool failed, check server logs
 at 
 org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:290)
 at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:202)
 {code}
 After running:
 {code}
 nodetool repair --partitioner-range --parallel --in-local-dc sync
 {code}
 Last records in logs regarding repair are:
 {code}
 INFO  [Thread-173887] 2015-07-29 20:44:03,956 StorageService.java:2952 - 
 Repair session 09ff9e40-3632-11e5-a93e-4963524a8bde for range 
 (-7695808664784761779,-7693529816291585568] finished
 INFO  [Thread-173887] 2015-07-29 20:44:03,956 StorageService.java:2952 - 
 Repair session 17d8d860-3632-11e5-a93e-4963524a8bde for range 
 (806371695398849,8065203836608925992] finished
 INFO  [Thread-173887] 2015-07-29 20:44:03,956 StorageService.java:2952 - 
 Repair session 23a811b0-3632-11e5-a93e-4963524a8bde for range 
 (-5474076923322749342,-5468600594078911162] finished
 INFO  [Thread-173887] 2015-07-29 20:44:03,956 StorageService.java:2952 - 
 Repair session 336f8740-3632-11e5-a93e-4963524a8bde for range 
 (-8631877858109464676,-8624040066373718932] finished
 INFO  [Thread-173887] 2015-07-29 20:44:03,957 StorageService.java:2952 - 
 Repair session 4ccd8430-3632-11e5-a93e-4963524a8bde for range 
 (-5372806541854279315,-5369354119480076785] finished
 INFO  [Thread-173887] 2015-07-29 20:44:03,957 StorageService.java:2952 - 
 Repair session 59f129f0-3632-11e5-a93e-4963524a8bde for range 
 (8166489034383821955,8168408930184216281] finished
 INFO  [Thread-173887] 2015-07-29 20:44:03,957 StorageService.java:2952 - 
 Repair session 6ae7a9a0-3632-11e5-a93e-4963524a8bde for range 
 (6084602890817326921,6088328703025510057] finished
 INFO  [Thread-173887] 2015-07-29 20:44:03,957 StorageService.java:2952 - 
 Repair session 8938e4a0-3632-11e5-a93e-4963524a8bde for range 
 (-781874602493000830,-781745173070807746] finished
 {code}
 but a bit above I see (at least two times in attached log):
 {code}
 ERROR [Thread-173887] 2015-07-29 20:44:03,853 StorageService.java:2959 -

[jira] [Updated] (CASSANDRA-9483) Document incompatibilities with -XX:+PerfDisableSharedMem


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

T Jake Luciani updated CASSANDRA-9483:
--
Attachment: news_update.txt

Added some words in NEWS.TXT along with other JVM related news.

 Document incompatibilities with -XX:+PerfDisableSharedMem
 -

 Key: CASSANDRA-9483
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9483
 Project: Cassandra
  Issue Type: Task
  Components: Config, Documentation  website
Reporter: Tyler Hobbs
Assignee: T Jake Luciani
Priority: Minor
 Fix For: 3.0 beta 1

 Attachments: news_update.txt


 We recently discovered that [the Jolokia agent is incompatible with  the 
 -XX:+PerfDisableSharedMem JVM 
 option|https://github.com/rhuss/jolokia/issues/198].  I assume that this may 
 affect other monitoring tools as well.
 If we are going to leave this enabled by default, we should document the 
 potential problems with it.  A combination of a comment in 
 {{cassandra-env.sh}} (and the Windows equivalent) and a comment in NEWS.txt 
 should suffice, I think.
 If possible, it would be good to figure out what other tools are affected and 
 also mention them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8143) Partitioner should not be accessed through StorageService

2015-07-31 Thread Ariel Weisberg (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649501#comment-14649501
 ] 

Ariel Weisberg commented on CASSANDRA-8143:
---

This commit appears to cause the dtests to hang?
http://cassci.datastax.com/view/Dev/view/blambov/job/blambov-8143-partitioner-dtest/17/
http://cassci.datastax.com/view/trunk/job/trunk_dtest/414/

 Partitioner should not be accessed through StorageService
 -

 Key: CASSANDRA-8143
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8143
 Project: Cassandra
  Issue Type: Improvement
Reporter: Branimir Lambov
Assignee: Branimir Lambov
 Fix For: 3.0 beta 1


 The configured partitioner is no longer the only partitioner in use in the 
 database, as e.g. index tables use LocalPartitioner.
 To make sure the correct partitioner is used for each table, accesses of 
 StorageService.getPartitioner() should be replaced with retrieval of the 
 CFS-specific partitioner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-9031) nodetool info -T throws ArrayOutOfBounds when the node has not joined the cluster

2015-07-31 Thread Yuki Morishita (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuki Morishita updated CASSANDRA-9031:
--
Assignee: Yuki Morishita
Reviewer:   (was: Yuki Morishita)

 nodetool info -T throws ArrayOutOfBounds when the node has not joined the 
 cluster
 -

 Key: CASSANDRA-9031
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9031
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
Reporter: Ron Kuris
Assignee: Yuki Morishita
 Fix For: 2.1.x

 Attachments: patch.txt


 To reproduce, bring up a node that does not join the cluster, either using 
 -Dcassandra.write_survey=true or -Dcassandra.join_ring=false, then run 
 'nodetool info -T'. You'll get the following stack trace:
 {code}ID : e384209f-f7a9-4cff-8fd5-03adfaa0d846
 Gossip active  : true
 Thrift active  : true
 Native Transport active: true
 Load   : 76.69 KB
 Generation No  : 1427229938
 Uptime (seconds)   : 728
 Heap Memory (MB)   : 109.93 / 826.00
 Off Heap Memory (MB)   : 0.01
 Exception in thread main java.lang.IndexOutOfBoundsException: Index: 0, 
 Size: 0
   at java.util.ArrayList.rangeCheck(ArrayList.java:635)
   at java.util.ArrayList.get(ArrayList.java:411)
   at org.apache.cassandra.tools.NodeProbe.getEndpoint(NodeProbe.java:676)
   at 
 org.apache.cassandra.tools.NodeProbe.getDataCenter(NodeProbe.java:694)
   at org.apache.cassandra.tools.NodeCmd.printInfo(NodeCmd.java:666)
   at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:1277){code}
 After applying the attached patch, the new error is:
 {code}ID : a7d76a2a-82d2-4faa-94e1-a30df6663ebb
 Gossip active  : true
 Thrift active  : false
 Native Transport active: false
 Load   : 89.36 KB
 Generation No  : 1427231804
 Uptime (seconds)   : 12
 Heap Memory (MB)   : 135.49 / 826.00
 Off Heap Memory (MB)   : 0.01
 Exception in thread main java.lang.RuntimeException: This node does not 
 have any tokens. Perhaps it is not part of the ring?
   at org.apache.cassandra.tools.NodeProbe.getEndpoint(NodeProbe.java:678)
   at 
 org.apache.cassandra.tools.NodeProbe.getDataCenter(NodeProbe.java:698)
   at org.apache.cassandra.tools.NodeCmd.printInfo(NodeCmd.java:676)
   at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:1313){code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-6477) Materialized Views (was: Global Indexes)

2015-07-31 Thread Ariel Weisberg (JIRA)

[
https://issues.apache.org/jira/browse/CASSANDRA-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649520#comment-14649520
]

Ariel Weisberg commented on CASSANDRA-6477:
---

bq. Recommitted with fix. Looks like those errors are fixed in my test run
above. But we have so many flappy tests it's hard to tell at the moment. Went
from 38 to 22 vs 19 on trunk. Care to double check Ariel Weisberg?

I agree the flappy tests make it really annoying and time consuming. Right now
what I do is for every failing test in the build on my branch is look up the
test history on trunk. If it fails on trunk then I ignore it. That's how I got
to looking into cassci page load times...

This is why I am really bummed about all the regressions in utests and dtests.
It renders the entire CI process either very labor intensive or useless. But
once we work it off and have it clean it's MUCH easier to keep it clean.

I compared them. It did take 10 minutes :-( There are some things that are
passing on trunk due to fixes that came in after the branch was rebased, but
not things that look like regressions.

Materialized Views (was: Global Indexes)

Key: CASSANDRA-6477
URL: https://issues.apache.org/jira/browse/CASSANDRA-6477
Project: Cassandra
Issue Type: New Feature
Components: API, Core
Reporter: Jonathan Ellis
Assignee: Carl Yeksigian
Labels: cql
Fix For: 3.0 alpha 1

Attachments: test-view-data.sh, users.yaml

Local indexes are suitable for low-cardinality data, where spreading the
index across the cluster is a Good Thing. However, for high-cardinality
data, local indexes require querying most nodes in the cluster even if only a
handful of rows is returned.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-9917) MVs should validate gc grace seconds on the tables involved


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

T Jake Luciani updated CASSANDRA-9917:
--
Assignee: (was: T Jake Luciani)

 MVs should validate gc grace seconds on the tables involved
 ---

 Key: CASSANDRA-9917
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9917
 Project: Cassandra
  Issue Type: Bug
Reporter: Aleksey Yeschenko
 Fix For: 3.0 beta 1


 For correctness reasons (potential resurrection of dropped values), batchlog 
 entries are TTLs with the lowest gc grace second of all the tables involved 
 in a batch.
 It means that if gc gs is set to 0 in one of the tables, the batchlog entry 
 will be dead on arrival, and never replayed.
 We should probably warn against such LOGGED writes taking place, in general, 
 but for MVs, we must validate that gc gs on the base table (and on the MV 
 table, if we should allow altering gc gs there at all), is never set too low, 
 or else.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

cassandra git commit: Bytecode inspection for Java-UDFs

2015-07-31 Thread snazy

Repository: cassandra
Updated Branches:
  refs/heads/trunk a22ce89e8 - 1774eb9a8


Bytecode inspection for Java-UDFs

patch by Robert Stupp; reviewed by T Jake Luciani for CASSANDRA-9890


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1774eb9a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1774eb9a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1774eb9a

Branch: refs/heads/trunk
Commit: 1774eb9a8632fc68ec5e4ea0fd4ce237b74d0f51
Parents: a22ce89
Author: Robert Stupp sn...@snazy.de
Authored: Fri Jul 31 20:36:23 2015 +0200
Committer: Robert Stupp sn...@snazy.de
Committed: Fri Jul 31 20:36:23 2015 +0200

--
 CHANGES.txt |   2 +-
 NOTICE.txt  |   4 +
 lib/asm-5.0.4.jar   | Bin 0 - 53297 bytes
 lib/licenses/asm-5.0.4.txt  |  29 ++
 .../cql3/functions/JavaBasedUDFunction.java |  53 +++-
 .../cql3/functions/UDFByteCodeVerifier.java | 215 +
 .../cassandra/cql3/functions/UDFunction.java|   1 +
 .../cassandra/cql3/functions/JavaSourceUDF.txt  |   8 +-
 .../validation/entities/UFVerifierTest.java | 307 +++
 .../entities/udfverify/CallClone.java   |  49 +++
 .../entities/udfverify/CallComDatastax.java |  43 +++
 .../entities/udfverify/CallFinalize.java|  49 +++
 .../entities/udfverify/CallOrgApache.java   |  43 +++
 .../entities/udfverify/ClassWithField.java  |  43 +++
 .../udfverify/ClassWithInitializer.java |  47 +++
 .../udfverify/ClassWithInitializer2.java|  47 +++
 .../udfverify/ClassWithInitializer3.java|  45 +++
 .../udfverify/ClassWithStaticInitializer.java   |  46 +++
 .../entities/udfverify/GoodClass.java   |  41 +++
 .../entities/udfverify/UseOfSynchronized.java   |  45 +++
 .../udfverify/UseOfSynchronizedWithNotify.java  |  45 +++
 .../UseOfSynchronizedWithNotifyAll.java |  45 +++
 .../udfverify/UseOfSynchronizedWithWait.java|  52 
 .../udfverify/UseOfSynchronizedWithWaitL.java   |  52 
 .../udfverify/UseOfSynchronizedWithWaitLI.java  |  52 
 25 files changed, 1356 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1774eb9a/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index c92a771..5f0a52f 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,5 +1,5 @@
 3.0
- * Implement proper sandboxing for UDFs (CASSANDRA-9402)
+ * Implement proper sandboxing for UDFs (CASSANDRA-9402, 9890)
  * Simplify (and unify) cleanup of compaction leftovers (CASSANDRA-7066)
  * Allow extra schema definitions in cassandra-stress yaml (CASSANDRA-9850)
  * Metrics should use up to date nomenclature (CASSANDRA-9448)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1774eb9a/NOTICE.txt
--
diff --git a/NOTICE.txt b/NOTICE.txt
index 0ad792f..a20994f 100644
--- a/NOTICE.txt
+++ b/NOTICE.txt
@@ -79,3 +79,7 @@ Protocol buffers for varint encoding
 https://developers.google.com/protocol-buffers/
 Copyright 2008 Google Inc.  All rights reserved.
 BSD 3-clause
+
+ASM
+(http://asm.ow2.org/)
+Copyright (c) 2000-2011 INRIA, France Telecom

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1774eb9a/lib/asm-5.0.4.jar
--
diff --git a/lib/asm-5.0.4.jar b/lib/asm-5.0.4.jar
new file mode 100644
index 000..cdb283d
Binary files /dev/null and b/lib/asm-5.0.4.jar differ

http://git-wip-us.apache.org/repos/asf/cassandra/blob/1774eb9a/lib/licenses/asm-5.0.4.txt
--
diff --git a/lib/licenses/asm-5.0.4.txt b/lib/licenses/asm-5.0.4.txt
new file mode 100644
index 000..c5aba7b
--- /dev/null
+++ b/lib/licenses/asm-5.0.4.txt
@@ -0,0 +1,29 @@
+Copyright (c) 2000-2011 INRIA, France Telecom
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+
+1. Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright
+   notice, this list of conditions and the following disclaimer in the
+   documentation and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holders nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND

[jira] [Commented] (CASSANDRA-8368) Consider not using hints for batchlog replay, in any capacity

[
https://issues.apache.org/jira/browse/CASSANDRA-8368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649483#comment-14649483
]

Aleksey Yeschenko commented on CASSANDRA-8368:
--

FWIW BL ignores {{max_hint_window_in_ms}}, and always writes hints to the hints
table, no matter what, so there is that.

Consider not using hints for batchlog replay, in any capacity
-

Key: CASSANDRA-8368
URL: https://issues.apache.org/jira/browse/CASSANDRA-8368
Project: Cassandra
Issue Type: Bug
Reporter: Aleksey Yeschenko
Fix For: 3.0.0 rc1

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-6717) Modernize schema tables

2015-07-31 Thread Andrew Hust (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-6717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649507#comment-14649507
 ] 

Andrew Hust commented on CASSANDRA-6717:


I’ve started adding dtests for this and ran into a small issue.  With the 
schema_* tables being renamed some of them conflict with the cql describe 
command.  Example when scoped to the system_schema keyspace running `desc 
tables` gives a list of tables in the keyspace instead of the cql for the 
tables table.  Using `desc system_schema.tables` works as expected.

Does any documentation or ? need to be updated to explain that behavior?


 Modernize schema tables
 ---

 Key: CASSANDRA-6717
 URL: https://issues.apache.org/jira/browse/CASSANDRA-6717
 Project: Cassandra
  Issue Type: Sub-task
Reporter: Sylvain Lebresne
Assignee: Aleksey Yeschenko
  Labels: client-impacting, doc-impacting
 Fix For: 3.0 beta 1


 There is a few problems/improvements that can be done with the way we store 
 schema:
 # CASSANDRA-4988: as explained on the ticket, storing the comparator is now 
 redundant (or almost, we'd need to store whether the table is COMPACT or not 
 too, which we don't currently is easy and probably a good idea anyway), it 
 can be entirely reconstructed from the infos in schema_columns (the same is 
 true of key_validator and subcomparator, and replacing default_validator by a 
 COMPACT_VALUE column in all case is relatively simple). And storing the 
 comparator as an opaque string broke concurrent updates of sub-part of said 
 comparator (concurrent collection addition or altering 2 separate clustering 
 columns typically) so it's really worth removing it.
 # CASSANDRA-4603: it's time to get rid of those ugly json maps. I'll note 
 that schema_keyspaces is a problem due to its use of COMPACT STORAGE, but I 
 think we should fix it once and for-all nonetheless (see below).
 # For CASSANDRA-6382 and to allow indexing both map keys and values at the 
 same time, we'd need to be able to have more than one index definition for a 
 given column.
 # There is a few mismatches in table options between the one stored in the 
 schema and the one used when declaring/altering a table which would be nice 
 to fix. The compaction, compression and replication maps are one already 
 mentioned from CASSANDRA-4603, but also for some reason 
 'dclocal_read_repair_chance' in CQL is called just 'local_read_repair_chance' 
 in the schema table, and 'min/max_compaction_threshold' are column families 
 option in the schema but just compaction options for CQL (which makes more 
 sense).
 None of those issues are major, and we could probably deal with them 
 independently but it might be simpler to just fix them all in one shot so I 
 wanted to sum them all up here. In particular, the fact that 
 'schema_keyspaces' uses COMPACT STORAGE is annoying (for the replication map, 
 but it may limit future stuff too) which suggest we should migrate it to a 
 new, non COMPACT table. And while that's arguably a detail, it wouldn't hurt 
 to rename schema_columnfamilies to schema_tables for the years to come since 
 that's the prefered vernacular for CQL.
 Overall, what I would suggest is to move all schema tables to a new keyspace, 
 named 'schema' for instance (or 'system_schema' but I prefer the shorter 
 version), and fix all the issues above at once. Since we currently don't 
 exchange schema between nodes of different versions, all we'd need to do that 
 is a one shot startup migration, and overall, I think it could be simpler for 
 clients to deal with one clear migration than to have to handle minor 
 individual changes all over the place. I also think it's somewhat cleaner 
 conceptually to have schema tables in their own keyspace since they are 
 replicated through a different mechanism than other system tables.
 If we do that, we could, for instance, migrate to the following schema tables 
 (details up for discussion of course):
 {noformat}
 CREATE TYPE user_type (
   name text,
   column_names listtext,
   column_types listtext
 )
 CREATE TABLE keyspaces (
   name text PRIMARY KEY,
   durable_writes boolean,
   replication mapstring, string,
   user_types mapstring, user_type
 )
 CREATE TYPE trigger_definition (
   name text,
   options maptex, text
 )
 CREATE TABLE tables (
   keyspace text,
   name text,
   id uuid,
   table_type text, // COMPACT, CQL or SUPER
   dropped_columns maptext, bigint,
   triggers maptext, trigger_definition,
   // options
   comment text,
   compaction maptext, text,
   compression maptext, text,
   read_repair_chance double,
   dclocal_read_repair_chance double,
   gc_grace_seconds int,
   caching text,
   rows_per_partition_to_cache text,
   default_time_to_live int,
   min_index_interval int,
   max_index_interval int,

[jira] [Reopened] (CASSANDRA-8143) Partitioner should not be accessed through StorageService


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis reopened CASSANDRA-8143:
---

 Partitioner should not be accessed through StorageService
 -

 Key: CASSANDRA-8143
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8143
 Project: Cassandra
  Issue Type: Improvement
Reporter: Branimir Lambov
Assignee: Branimir Lambov
 Fix For: 3.0 beta 1


 The configured partitioner is no longer the only partitioner in use in the 
 database, as e.g. index tables use LocalPartitioner.
 To make sure the correct partitioner is used for each table, accesses of 
 StorageService.getPartitioner() should be replaced with retrieval of the 
 CFS-specific partitioner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-9031) nodetool info -T throws ArrayOutOfBounds when the node has not joined the cluster


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-9031:
--
Reviewer: Stefania

[~Stefania] to review

 nodetool info -T throws ArrayOutOfBounds when the node has not joined the 
 cluster
 -

 Key: CASSANDRA-9031
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9031
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
Reporter: Ron Kuris
Assignee: Yuki Morishita
 Fix For: 2.1.x

 Attachments: patch.txt


 To reproduce, bring up a node that does not join the cluster, either using 
 -Dcassandra.write_survey=true or -Dcassandra.join_ring=false, then run 
 'nodetool info -T'. You'll get the following stack trace:
 {code}ID : e384209f-f7a9-4cff-8fd5-03adfaa0d846
 Gossip active  : true
 Thrift active  : true
 Native Transport active: true
 Load   : 76.69 KB
 Generation No  : 1427229938
 Uptime (seconds)   : 728
 Heap Memory (MB)   : 109.93 / 826.00
 Off Heap Memory (MB)   : 0.01
 Exception in thread main java.lang.IndexOutOfBoundsException: Index: 0, 
 Size: 0
   at java.util.ArrayList.rangeCheck(ArrayList.java:635)
   at java.util.ArrayList.get(ArrayList.java:411)
   at org.apache.cassandra.tools.NodeProbe.getEndpoint(NodeProbe.java:676)
   at 
 org.apache.cassandra.tools.NodeProbe.getDataCenter(NodeProbe.java:694)
   at org.apache.cassandra.tools.NodeCmd.printInfo(NodeCmd.java:666)
   at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:1277){code}
 After applying the attached patch, the new error is:
 {code}ID : a7d76a2a-82d2-4faa-94e1-a30df6663ebb
 Gossip active  : true
 Thrift active  : false
 Native Transport active: false
 Load   : 89.36 KB
 Generation No  : 1427231804
 Uptime (seconds)   : 12
 Heap Memory (MB)   : 135.49 / 826.00
 Off Heap Memory (MB)   : 0.01
 Exception in thread main java.lang.RuntimeException: This node does not 
 have any tokens. Perhaps it is not part of the ring?
   at org.apache.cassandra.tools.NodeProbe.getEndpoint(NodeProbe.java:678)
   at 
 org.apache.cassandra.tools.NodeProbe.getDataCenter(NodeProbe.java:698)
   at org.apache.cassandra.tools.NodeCmd.printInfo(NodeCmd.java:676)
   at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:1313){code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-8368) Consider not using hints for batchlog replay, in any capacity

[
https://issues.apache.org/jira/browse/CASSANDRA-8368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Aleksey Yeschenko updated CASSANDRA-8368:
-
Fix Version/s: (was: 3.0.0 rc1)

Consider not using hints for batchlog replay, in any capacity
-

Key: CASSANDRA-8368
URL: https://issues.apache.org/jira/browse/CASSANDRA-8368
Project: Cassandra
Issue Type: Bug
Reporter: Aleksey Yeschenko

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8368) Consider not using hints for batchlog replay, in any capacity

[
https://issues.apache.org/jira/browse/CASSANDRA-8368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649492#comment-14649492
]

T Jake Luciani commented on CASSANDRA-8368:
---

bq. BL ignores max_hint_window_in_ms, and always writes hints to the hints
table, no matter what, so there is that.

So you mean the hints will eventually be replayed even after
max_hint_window_in_ms goes by? Or just that you will store a hint for it after
max_hint_window_in_ms?

Consider not using hints for batchlog replay, in any capacity
-

Key: CASSANDRA-8368
URL: https://issues.apache.org/jira/browse/CASSANDRA-8368
Project: Cassandra
Issue Type: Bug
Reporter: Aleksey Yeschenko
Fix For: 3.0.0 rc1

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/4] cassandra git commit: Revert Stop accessing the partitioner directly via StorageService

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a22ce89e/src/java/org/apache/cassandra/io/sstable/format/big/BigTableScanner.java
--
diff --git 
a/src/java/org/apache/cassandra/io/sstable/format/big/BigTableScanner.java 
b/src/java/org/apache/cassandra/io/sstable/format/big/BigTableScanner.java
index 1b3b407..0451a98 100644
--- a/src/java/org/apache/cassandra/io/sstable/format/big/BigTableScanner.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/big/BigTableScanner.java
@@ -172,7 +172,7 @@ public class BigTableScanner implements ISSTableScanner
 while (!ifile.isEOF())
 {
 indexPosition = ifile.getFilePointer();
-DecoratedKey indexDecoratedKey = 
sstable.decorateKey(ByteBufferUtil.readWithShortLength(ifile));
+DecoratedKey indexDecoratedKey = 
sstable.partitioner.decorateKey(ByteBufferUtil.readWithShortLength(ifile));
 if (indexDecoratedKey.compareTo(currentRange.left)  0 || 
currentRange.contains(indexDecoratedKey))
 {
 // Found, just read the dataPosition and seek into index 
and data files
@@ -282,7 +282,7 @@ public class BigTableScanner implements ISSTableScanner
 if (ifile.isEOF())
 return endOfData();
 
-currentKey = 
sstable.decorateKey(ByteBufferUtil.readWithShortLength(ifile));
+currentKey = 
sstable.partitioner.decorateKey(ByteBufferUtil.readWithShortLength(ifile));
 currentEntry = 
rowIndexEntrySerializer.deserialize(ifile);
 } while (!currentRange.contains(currentKey));
 }
@@ -301,7 +301,7 @@ public class BigTableScanner implements ISSTableScanner
 else
 {
 // we need the position of the start of the next key, 
regardless of whether it falls in the current range
-nextKey = 
sstable.decorateKey(ByteBufferUtil.readWithShortLength(ifile));
+nextKey = 
sstable.partitioner.decorateKey(ByteBufferUtil.readWithShortLength(ifile));
 nextEntry = rowIndexEntrySerializer.deserialize(ifile);
 
 if (!currentRange.contains(nextKey))

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a22ce89e/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
--
diff --git 
a/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java 
b/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
index 5607a7e..ff279a8 100644
--- a/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
@@ -19,6 +19,7 @@ package org.apache.cassandra.io.sstable.format.big;
 
 import java.io.*;
 import java.util.Map;
+import java.util.Set;
 
 import org.apache.cassandra.db.*;
 import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
@@ -31,6 +32,7 @@ import org.slf4j.LoggerFactory;
 import org.apache.cassandra.config.CFMetaData;
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.db.rows.*;
+import org.apache.cassandra.dht.IPartitioner;
 import org.apache.cassandra.io.FSWriteError;
 import org.apache.cassandra.io.compress.CompressedSequentialWriter;
 import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
@@ -60,11 +62,12 @@ public class BigTableWriter extends SSTableWriter
   Long keyCount, 
   Long repairedAt, 
   CFMetaData metadata, 
+  IPartitioner partitioner, 
   MetadataCollector metadataCollector, 
   SerializationHeader header,
   LifecycleTransaction txn)
 {
-super(descriptor, keyCount, repairedAt, metadata, metadataCollector, 
header);
+super(descriptor, keyCount, repairedAt, metadata, partitioner, 
metadataCollector, header);
 txn.trackNew(this); // must track before any files are created
 
 if (compression)
@@ -240,12 +243,13 @@ public class BigTableWriter extends SSTableWriter
 StatsMetadata stats = statsMetadata();
 assert boundary.indexLength  0  boundary.dataLength  0;
 // open the reader early
-IndexSummary indexSummary = 
iwriter.summary.build(metadata.partitioner, boundary);
+IndexSummary indexSummary = iwriter.summary.build(partitioner, 
boundary);
 SegmentedFile ifile = iwriter.builder.buildIndex(descriptor, 
indexSummary, boundary);
 SegmentedFile dfile = dbuilder.buildData(descriptor, stats, boundary);
 SSTableReader sstable = SSTableReader.internalOpen(descriptor,

[4/4] cassandra git commit: Revert Stop accessing the partitioner directly via StorageService

Revert Stop accessing the partitioner directly via StorageService

This reverts commit 69f77cbddd4c74448f227e9aceef84d345118184.


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a22ce89e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a22ce89e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a22ce89e

Branch: refs/heads/trunk
Commit: a22ce89e868644ea04f0f3dacec05fff1673a345
Parents: 69f77cb
Author: Jonathan Ellis jbel...@apache.org
Authored: Fri Jul 31 12:24:52 2015 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Fri Jul 31 12:24:52 2015 -0500

--
 .../org/apache/cassandra/config/CFMetaData.java |  64 +---
 .../cassandra/config/DatabaseDescriptor.java|   7 +-
 .../org/apache/cassandra/config/Schema.java |   2 +-
 .../apache/cassandra/cql3/TokenRelation.java|   7 +-
 .../cassandra/cql3/functions/TokenFct.java  |  10 +-
 .../restrictions/StatementRestrictions.java |   2 +-
 .../cql3/restrictions/TokenFilter.java  |  14 +-
 .../cql3/restrictions/TokenRestriction.java |  23 ++-
 .../cql3/statements/BatchStatement.java |   5 +-
 .../cql3/statements/ModificationStatement.java  |   7 +-
 .../cql3/statements/SelectStatement.java|   2 +-
 .../db/AbstractReadCommandBuilder.java  |  11 +-
 .../apache/cassandra/db/BatchlogManager.java|   7 +-
 .../apache/cassandra/db/ColumnFamilyStore.java  |  35 ++--
 src/java/org/apache/cassandra/db/DataRange.java |   3 +-
 .../cassandra/db/HintedHandOffManager.java  |  29 ++--
 src/java/org/apache/cassandra/db/Memtable.java  |   1 +
 src/java/org/apache/cassandra/db/Mutation.java  |  26 +--
 .../apache/cassandra/db/PartitionPosition.java  |   2 +-
 .../cassandra/db/PartitionRangeReadCommand.java |   2 +-
 .../apache/cassandra/db/RowUpdateBuilder.java   |   2 +-
 .../db/SinglePartitionNamesCommand.java |  12 --
 .../db/SinglePartitionReadCommand.java  |  18 +--
 .../db/SinglePartitionSliceCommand.java |  17 --
 .../org/apache/cassandra/db/SystemKeyspace.java |  18 ++-
 .../db/compaction/CompactionManager.java|   4 +-
 .../db/compaction/LeveledManifest.java  |   2 +-
 .../cassandra/db/compaction/Scrubber.java   |   4 +-
 .../cassandra/db/compaction/Upgrader.java   |   1 +
 .../cassandra/db/compaction/Verifier.java   |   2 +-
 .../writers/DefaultCompactionWriter.java|   1 +
 .../writers/MajorLeveledCompactionWriter.java   |   2 +
 .../writers/MaxSSTableSizeWriter.java   |   2 +
 .../SplittingSizeTieredCompactionWriter.java|   2 +
 .../AbstractSimplePerColumnSecondaryIndex.java  |   4 +-
 .../cassandra/db/index/SecondaryIndex.java  |  20 ++-
 .../db/index/composites/CompositesIndex.java|   2 +-
 .../CompositesIndexOnClusteringKey.java |   3 +-
 .../db/index/composites/CompositesSearcher.java |   2 +-
 .../cassandra/db/index/keys/KeysIndex.java  |   3 +-
 .../cassandra/db/index/keys/KeysSearcher.java   |   2 +-
 .../db/marshal/LocalByPartionerType.java|  97 
 .../db/marshal/PartitionerDefinedOrder.java |  91 ---
 .../db/partitions/AtomicBTreePartition.java |   4 +-
 .../db/partitions/PartitionUpdate.java  | 108 +++--
 .../rows/UnfilteredRowIteratorSerializer.java   |   3 +-
 .../cassandra/db/view/MaterializedView.java |   7 +-
 .../apache/cassandra/db/view/TemporalRow.java   |   2 +-
 .../org/apache/cassandra/dht/BootStrapper.java  |  12 +-
 .../cassandra/dht/ByteOrderedPartitioner.java   |   5 -
 .../org/apache/cassandra/dht/IPartitioner.java  |   6 -
 .../apache/cassandra/dht/LocalPartitioner.java  |   5 -
 .../cassandra/dht/Murmur3Partitioner.java   |   7 -
 .../dht/OrderPreservingPartitioner.java |   5 -
 .../apache/cassandra/dht/RandomPartitioner.java |   7 -
 .../org/apache/cassandra/dht/RangeStreamer.java |   2 +-
 .../dht/tokenallocator/TokenAllocation.java |   8 +-
 src/java/org/apache/cassandra/gms/Gossiper.java |   2 +-
 .../io/sstable/AbstractSSTableSimpleWriter.java |  10 +-
 .../cassandra/io/sstable/CQLSSTableWriter.java  |  15 +-
 .../cassandra/io/sstable/KeyIterator.java   |   8 +-
 .../io/sstable/ReducingKeyIterator.java |   2 +-
 .../apache/cassandra/io/sstable/SSTable.java|  21 +--
 .../cassandra/io/sstable/SSTableLoader.java |  20 ++-
 .../io/sstable/SSTableSimpleUnsortedWriter.java |   5 +-
 .../io/sstable/SSTableSimpleWriter.java |   4 +-
 .../io/sstable/format/SSTableReader.java|  74 +
 .../io/sstable/format/SSTableWriter.java|  16 +-
 .../io/sstable/format/big/BigFormat.java|   8 +-
 .../io/sstable/format/big/BigTableReader.java   |   8 +-
 .../io/sstable/format/big/BigTableScanner.java  |   6 +-
 .../io/sstable/format/big/BigTableWriter.java   |  15 +-

[1/4] cassandra git commit: Revert Stop accessing the partitioner directly via StorageService

Repository: cassandra
Updated Branches:
  refs/heads/trunk 69f77cbdd - a22ce89e8


http://git-wip-us.apache.org/repos/asf/cassandra/blob/a22ce89e/test/unit/org/apache/cassandra/db/ScrubTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/ScrubTest.java 
b/test/unit/org/apache/cassandra/db/ScrubTest.java
index 25b9cde..ee51a4d 100644
--- a/test/unit/org/apache/cassandra/db/ScrubTest.java
+++ b/test/unit/org/apache/cassandra/db/ScrubTest.java
@@ -22,6 +22,8 @@ import java.io.File;
 import java.io.IOError;
 import java.io.IOException;
 import java.io.RandomAccessFile;
+import java.lang.reflect.Field;
+import java.lang.reflect.Modifier;
 import java.nio.ByteBuffer;
 import java.util.*;
 import java.util.concurrent.ExecutionException;
@@ -42,9 +44,9 @@ import org.apache.cassandra.cql3.UntypedResultSet;
 import org.apache.cassandra.db.compaction.CompactionManager;
 import org.apache.cassandra.db.compaction.OperationType;
 import org.apache.cassandra.db.compaction.Scrubber;
+import org.apache.cassandra.db.index.SecondaryIndex;
 import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
 import org.apache.cassandra.db.marshal.*;
-import org.apache.cassandra.db.marshal.UUIDType;
 import org.apache.cassandra.db.partitions.Partition;
 import org.apache.cassandra.db.partitions.PartitionUpdate;
 import org.apache.cassandra.dht.ByteOrderedPartitioner;
@@ -53,10 +55,14 @@ import 
org.apache.cassandra.exceptions.ConfigurationException;
 import org.apache.cassandra.exceptions.RequestExecutionException;
 import org.apache.cassandra.exceptions.WriteTimeoutException;
 import org.apache.cassandra.io.compress.CompressionMetadata;
-import org.apache.cassandra.io.sstable.*;
+import org.apache.cassandra.io.sstable.Component;
+import org.apache.cassandra.io.sstable.Descriptor;
+import org.apache.cassandra.io.sstable.SSTableRewriter;
+import org.apache.cassandra.io.sstable.SSTableTxnWriter;
 import org.apache.cassandra.io.sstable.format.SSTableReader;
 import org.apache.cassandra.io.util.FileUtils;
 import org.apache.cassandra.schema.KeyspaceParams;
+import org.apache.cassandra.service.StorageService;
 import org.apache.cassandra.utils.ByteBufferUtil;
 
 import static org.junit.Assert.*;
@@ -73,8 +79,6 @@ public class ScrubTest
 public static final String CF_UUID = UUIDKeys;
 public static final String CF_INDEX1 = Indexed1;
 public static final String CF_INDEX2 = Indexed2;
-public static final String CF_INDEX1_BYTEORDERED = Indexed1_ordered;
-public static final String CF_INDEX2_BYTEORDERED = Indexed2_ordered;
 
 public static final String COL_INDEX = birthdate;
 public static final String COL_NON_INDEX = notbirthdate;
@@ -94,9 +98,7 @@ public class ScrubTest
 
.compressionParameters(SchemaLoader.getCompressionParameters(COMPRESSION_CHUNK_LENGTH)),
 SchemaLoader.standardCFMD(KEYSPACE, 
CF_UUID, 0, UUIDType.instance),
 SchemaLoader.keysIndexCFMD(KEYSPACE, 
CF_INDEX1, true),
-SchemaLoader.compositeIndexCFMD(KEYSPACE, 
CF_INDEX2, true),
-SchemaLoader.keysIndexCFMD(KEYSPACE, 
CF_INDEX1_BYTEORDERED, true).copy(ByteOrderedPartitioner.instance),
-SchemaLoader.compositeIndexCFMD(KEYSPACE, 
CF_INDEX2_BYTEORDERED, true).copy(ByteOrderedPartitioner.instance));
+SchemaLoader.compositeIndexCFMD(KEYSPACE, 
CF_INDEX2, true));
 }
 
 @Test
@@ -304,7 +306,7 @@ public class ScrubTest
 {
 // This test assumes ByteOrderPartitioner to create out-of-order 
SSTable
 IPartitioner oldPartitioner = DatabaseDescriptor.getPartitioner();
-DatabaseDescriptor.setPartitionerUnsafe(new ByteOrderedPartitioner());
+DatabaseDescriptor.setPartitioner(new ByteOrderedPartitioner());
 
 // Create out-of-order SSTable
 File tempDir = File.createTempFile(ScrubTest.testScrubOutOfOrder, 
).getParentFile();
@@ -378,7 +380,7 @@ public class ScrubTest
 {
 FileUtils.deleteRecursive(tempDataDir);
 // reset partitioner
-DatabaseDescriptor.setPartitionerUnsafe(oldPartitioner);
+DatabaseDescriptor.setPartitioner(oldPartitioner);
 }
 }
 
@@ -392,9 +394,9 @@ public class ScrubTest
 CompressionMetadata compData = 
CompressionMetadata.create(sstable.getFilename());
 
 CompressionMetadata.Chunk chunk1 = compData.chunkFor(
-sstable.getPosition(PartitionPosition.ForKey.get(key1, 
sstable.getPartitioner()), SSTableReader.Operator.EQ).position);
+sstable.getPosition(PartitionPosition.ForKey.get(key1, 
sstable.partitioner), SSTableReader.Operator.EQ).position);
 CompressionMetadata.Chunk chunk2 = compData.chunkFor(
-

[jira] [Commented] (CASSANDRA-9945) Add transparent data encryption core classes

2015-07-31 Thread Robert Stupp (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649594#comment-14649594
 ] 

Robert Stupp commented on CASSANDRA-9945:
-

The code looks good so far. Some comments on the patch.

{{CipherFactory#buildCipher}} calls {{Cipher.getInstance}} for every 
encryption. {{Cipher.getInstance}} is a somewhat expensive operation. Using a 
Cipher-per-thread (ThreadLocal) would be nicer IMO. I ran a [quick’n’dirty 
microbench|https://gist.github.com/snazy/7839a7fdcf25dabafd4b]. It’s not about 
ms (at least not for AES).

Instead of forcing users to modify the JRE/JDK, we could provide [bouncycastle 
JCE|https://www.bouncycastle.org/documentation.html] (MIT license) - or even 
allow people to use it. This would just require a slight extension to 
{{TransparentDataEncryptionOptions}} to add a {{String provider}} and pass it 
to {{Cipher.getInstance(transformation, provider)}}.

Can you elaborate a bit how the {{Key}} cache is to be used in 
{{EncryptionContext}}?
I guess you need it when the encryption key for commit logs and/or sstables is 
changed.
Will there be any way to change the encryption keys e.g. via ”nodetool 
refresh_tde_keystore“?

A unit test covering especially the {{getEncryptor}}+{{getDecryptor}} methods 
in {{EncryptionContext}} would be great. Maybe also comparing data going all 
the way (clear-text - encryption - decryption - clear-text).

CipherFactory.secureRandom seems to be used multi-threaded. I assume 
{{SecureRandom}} is not thread-safe. Maybe put a {{SecureRandom}} (and Cipher 
instances) in a {{ThreadLocal}}?

Nits:
* DatabaseDescriptor: typo in the comment of {{encryptionContext}} - seems you 
were missing a ”git add” in the terminal? ;)
* EncryptionOptions: the check {{if (tdeOptions.enabled)}} in the catch-clause 
is redundant and can be omitted
* Unused method {{EncryptionContext.toHeaderParameters}} (at least in this 
commit)
* CipherFactory.getDecryptor: the assertion (or the message) is wrong. Should 
it be iv.length  0 ?

 Add transparent data encryption core classes
 

 Key: CASSANDRA-9945
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9945
 Project: Cassandra
  Issue Type: Improvement
Reporter: Jason Brown
Assignee: Jason Brown
  Labels: encryption
 Fix For: 3.0 beta 1


 This patch will add the core infrastructure classes necessary for transparent 
 data encryption (file-level encryption), as required for CASSANDRA-6018 and 
 CASSANDRA-9633.  The phrase transparent data encryption, while not the most 
 aesthetically pleasing, seems to be used throughout the database industry 
 (Oracle, SQLQServer, Datastax Enterprise) to describe file level encryption, 
 so we'll go with that, as well. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-8368) Consider not using hints for batchlog replay, in any capacity

[
https://issues.apache.org/jira/browse/CASSANDRA-8368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

T Jake Luciani resolved CASSANDRA-8368.
---
Resolution: Won't Fix

Ok well that makes me feel better, kinda.

Consider not using hints for batchlog replay, in any capacity
-

Key: CASSANDRA-8368
URL: https://issues.apache.org/jira/browse/CASSANDRA-8368
Project: Cassandra
Issue Type: Bug
Reporter: Aleksey Yeschenko
Fix For: 3.0.0 rc1

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9902) Windows dtest: invalid jolokia agent path

2015-07-31 Thread Carl Yeksigian (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649675#comment-14649675
 ] 

Carl Yeksigian commented on CASSANDRA-9902:
---

I'm not actually sure what that last test is supposed to be for; [~mambocab] do 
you remember what that part of the test is for? I think we probably want to 
write to the table, check that the size is the same, then write again and 
ensure that the commitlog increases in size.

 Windows dtest: invalid jolokia agent path
 -

 Key: CASSANDRA-9902
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9902
 Project: Cassandra
  Issue Type: Sub-task
Reporter: Joshua McKenzie
Assignee: Paulo Motta
Priority: Minor
  Labels: Windows
 Fix For: 2.2.x

 Attachments: Not_Dep.jpg, VMAttach.java, win-crashlogs.zip


 A variety of tests are failing with similar:
 {noformat}
 05:39:07 
 ==
 05:39:07 ERROR: change_durable_writes_test 
 (configuration_test.TestConfiguration)
 05:39:07 
 --
 05:39:07 Traceback (most recent call last):
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\configuration_test.py,
  line 63, in change_durable_writes_test
 05:39:07 durable_init_size = commitlog_size(durable_node)
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\configuration_test.py,
  line 127, in commitlog_size
 05:39:07 with JolokiaAgent(node) as jmx:
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\jmxutils.py, 
 line 188, in __enter__
 05:39:07 self.start()
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\jmxutils.py, 
 line 80, in start
 05:39:07 subprocess.check_output(args, stderr=subprocess.STDOUT)
 05:39:07   File D:\Python27\lib\subprocess.py, line 573, in check_output
 05:39:07 raise CalledProcessError(retcode, cmd, output=output)
 05:39:07 CalledProcessError: Command '('java', '-jar', 
 'lib\\jolokia-jvm-1.2.3-agent.jar', '--host', '127.0.0.1', 'start', '3436')' 
 returned non-zero exit status 2
 05:39:07   begin captured logging  
 
 05:39:07 dtest: DEBUG: cluster ccm directory: d:\temp\dtest-kjkypw
 05:39:07 -  end captured logging  
 -
 {noformat}
 Likely related:
 {noformat}
 20:54:46 Failed to start jolokia agent (command was: java -jar 
 lib\jolokia-jvm-1.2.3-agent.jar --host 127.0.0.1 start 3436): Command 
 '('java', '-jar', 'lib\\jolokia-jvm-1.2.3-agent.jar', '--host', '127.0.0.1', 
 'start', '3436')' returned non-zero exit status 2
 20:54:46 Exit status was: 2
 20:54:46 Output was: Error: Registry key 'Software\JavaSoft\Java Runtime 
 Environment'\CurrentVersion'
 20:54:46 has value '1.8', but '1.7' is required.
 20:54:46 Error: could not find java.dll
 20:54:46 Error: Could not find Java SE Runtime Environment.
 {noformat}
 And at the top of jmxutils.py we see:
 {code}
 JOLOKIA_JAR = os.path.join('lib', 'jolokia-jvm-1.2.3-agent.jar')
 {code}
 So that's not going to work out too well on Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9902) Windows dtest: invalid jolokia agent path

2015-07-31 Thread Jim Witschey (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649694#comment-14649694
 ] 

Jim Witschey commented on CASSANDRA-9902:
-

[~carlyeks] Sorry, which test?

 Windows dtest: invalid jolokia agent path
 -

 Key: CASSANDRA-9902
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9902
 Project: Cassandra
  Issue Type: Sub-task
Reporter: Joshua McKenzie
Assignee: Paulo Motta
Priority: Minor
  Labels: Windows
 Fix For: 2.2.x

 Attachments: Not_Dep.jpg, VMAttach.java, win-crashlogs.zip


 A variety of tests are failing with similar:
 {noformat}
 05:39:07 
 ==
 05:39:07 ERROR: change_durable_writes_test 
 (configuration_test.TestConfiguration)
 05:39:07 
 --
 05:39:07 Traceback (most recent call last):
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\configuration_test.py,
  line 63, in change_durable_writes_test
 05:39:07 durable_init_size = commitlog_size(durable_node)
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\configuration_test.py,
  line 127, in commitlog_size
 05:39:07 with JolokiaAgent(node) as jmx:
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\jmxutils.py, 
 line 188, in __enter__
 05:39:07 self.start()
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\jmxutils.py, 
 line 80, in start
 05:39:07 subprocess.check_output(args, stderr=subprocess.STDOUT)
 05:39:07   File D:\Python27\lib\subprocess.py, line 573, in check_output
 05:39:07 raise CalledProcessError(retcode, cmd, output=output)
 05:39:07 CalledProcessError: Command '('java', '-jar', 
 'lib\\jolokia-jvm-1.2.3-agent.jar', '--host', '127.0.0.1', 'start', '3436')' 
 returned non-zero exit status 2
 05:39:07   begin captured logging  
 
 05:39:07 dtest: DEBUG: cluster ccm directory: d:\temp\dtest-kjkypw
 05:39:07 -  end captured logging  
 -
 {noformat}
 Likely related:
 {noformat}
 20:54:46 Failed to start jolokia agent (command was: java -jar 
 lib\jolokia-jvm-1.2.3-agent.jar --host 127.0.0.1 start 3436): Command 
 '('java', '-jar', 'lib\\jolokia-jvm-1.2.3-agent.jar', '--host', '127.0.0.1', 
 'start', '3436')' returned non-zero exit status 2
 20:54:46 Exit status was: 2
 20:54:46 Output was: Error: Registry key 'Software\JavaSoft\Java Runtime 
 Environment'\CurrentVersion'
 20:54:46 has value '1.8', but '1.7' is required.
 20:54:46 Error: could not find java.dll
 20:54:46 Error: Could not find Java SE Runtime Environment.
 {noformat}
 And at the top of jmxutils.py we see:
 {code}
 JOLOKIA_JAR = os.path.join('lib', 'jolokia-jvm-1.2.3-agent.jar')
 {code}
 So that's not going to work out too well on Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9902) Windows dtest: invalid jolokia agent path

2015-07-31 Thread Carl Yeksigian (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649702#comment-14649702
 ] 

Carl Yeksigian commented on CASSANDRA-9902:
---

This 
[dtest|https://github.com/riptano/cassandra-dtest/commit/ce417ff9f74305260ca1fd75b8f4e2296a865135],
 which was related to CASSANDRA-9560.


 Windows dtest: invalid jolokia agent path
 -

 Key: CASSANDRA-9902
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9902
 Project: Cassandra
  Issue Type: Sub-task
Reporter: Joshua McKenzie
Assignee: Paulo Motta
Priority: Minor
  Labels: Windows
 Fix For: 2.2.x

 Attachments: Not_Dep.jpg, VMAttach.java, win-crashlogs.zip


 A variety of tests are failing with similar:
 {noformat}
 05:39:07 
 ==
 05:39:07 ERROR: change_durable_writes_test 
 (configuration_test.TestConfiguration)
 05:39:07 
 --
 05:39:07 Traceback (most recent call last):
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\configuration_test.py,
  line 63, in change_durable_writes_test
 05:39:07 durable_init_size = commitlog_size(durable_node)
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\configuration_test.py,
  line 127, in commitlog_size
 05:39:07 with JolokiaAgent(node) as jmx:
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\jmxutils.py, 
 line 188, in __enter__
 05:39:07 self.start()
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\jmxutils.py, 
 line 80, in start
 05:39:07 subprocess.check_output(args, stderr=subprocess.STDOUT)
 05:39:07   File D:\Python27\lib\subprocess.py, line 573, in check_output
 05:39:07 raise CalledProcessError(retcode, cmd, output=output)
 05:39:07 CalledProcessError: Command '('java', '-jar', 
 'lib\\jolokia-jvm-1.2.3-agent.jar', '--host', '127.0.0.1', 'start', '3436')' 
 returned non-zero exit status 2
 05:39:07   begin captured logging  
 
 05:39:07 dtest: DEBUG: cluster ccm directory: d:\temp\dtest-kjkypw
 05:39:07 -  end captured logging  
 -
 {noformat}
 Likely related:
 {noformat}
 20:54:46 Failed to start jolokia agent (command was: java -jar 
 lib\jolokia-jvm-1.2.3-agent.jar --host 127.0.0.1 start 3436): Command 
 '('java', '-jar', 'lib\\jolokia-jvm-1.2.3-agent.jar', '--host', '127.0.0.1', 
 'start', '3436')' returned non-zero exit status 2
 20:54:46 Exit status was: 2
 20:54:46 Output was: Error: Registry key 'Software\JavaSoft\Java Runtime 
 Environment'\CurrentVersion'
 20:54:46 has value '1.8', but '1.7' is required.
 20:54:46 Error: could not find java.dll
 20:54:46 Error: Could not find Java SE Runtime Environment.
 {noformat}
 And at the top of jmxutils.py we see:
 {code}
 JOLOKIA_JAR = os.path.join('lib', 'jolokia-jvm-1.2.3-agent.jar')
 {code}
 So that's not going to work out too well on Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9902) Windows dtest: invalid jolokia agent path


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649779#comment-14649779
 ] 

Joshua McKenzie commented on CASSANDRA-9902:


Well, this is officially strike 1 against ConEmu for me.

Tests look good w/v3 when I run it locally via command-prompt and code looks 
good to me. You'll want to open a PR to the github [dtest 
repo|https://github.com/riptano/cassandra-dtest] and get a review from the 
owners over there.

Along with that, could we get confirmation that this re-jiggering of .jar / cp 
for jolokia works well on linux as well [~philipthompson]? I don't have my 
linux VM handy atm else I'd do so.

Great find on this and excellent detective work!

 Windows dtest: invalid jolokia agent path
 -

 Key: CASSANDRA-9902
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9902
 Project: Cassandra
  Issue Type: Sub-task
Reporter: Joshua McKenzie
Assignee: Paulo Motta
Priority: Minor
  Labels: Windows
 Fix For: 2.2.x

 Attachments: Not_Dep.jpg, VMAttach.java, win-crashlogs.zip


 A variety of tests are failing with similar:
 {noformat}
 05:39:07 
 ==
 05:39:07 ERROR: change_durable_writes_test 
 (configuration_test.TestConfiguration)
 05:39:07 
 --
 05:39:07 Traceback (most recent call last):
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\configuration_test.py,
  line 63, in change_durable_writes_test
 05:39:07 durable_init_size = commitlog_size(durable_node)
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\configuration_test.py,
  line 127, in commitlog_size
 05:39:07 with JolokiaAgent(node) as jmx:
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\jmxutils.py, 
 line 188, in __enter__
 05:39:07 self.start()
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\jmxutils.py, 
 line 80, in start
 05:39:07 subprocess.check_output(args, stderr=subprocess.STDOUT)
 05:39:07   File D:\Python27\lib\subprocess.py, line 573, in check_output
 05:39:07 raise CalledProcessError(retcode, cmd, output=output)
 05:39:07 CalledProcessError: Command '('java', '-jar', 
 'lib\\jolokia-jvm-1.2.3-agent.jar', '--host', '127.0.0.1', 'start', '3436')' 
 returned non-zero exit status 2
 05:39:07   begin captured logging  
 
 05:39:07 dtest: DEBUG: cluster ccm directory: d:\temp\dtest-kjkypw
 05:39:07 -  end captured logging  
 -
 {noformat}
 Likely related:
 {noformat}
 20:54:46 Failed to start jolokia agent (command was: java -jar 
 lib\jolokia-jvm-1.2.3-agent.jar --host 127.0.0.1 start 3436): Command 
 '('java', '-jar', 'lib\\jolokia-jvm-1.2.3-agent.jar', '--host', '127.0.0.1', 
 'start', '3436')' returned non-zero exit status 2
 20:54:46 Exit status was: 2
 20:54:46 Output was: Error: Registry key 'Software\JavaSoft\Java Runtime 
 Environment'\CurrentVersion'
 20:54:46 has value '1.8', but '1.7' is required.
 20:54:46 Error: could not find java.dll
 20:54:46 Error: Could not find Java SE Runtime Environment.
 {noformat}
 And at the top of jmxutils.py we see:
 {code}
 JOLOKIA_JAR = os.path.join('lib', 'jolokia-jvm-1.2.3-agent.jar')
 {code}
 So that's not going to work out too well on Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-9670) Cannot run CQL scripts on Windows AND having error Ubuntu Linux

2015-07-31 Thread Sanjay Patel (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649728#comment-14649728
 ] 

Sanjay Patel edited comment on CASSANDRA-9670 at 7/31/15 7:52 PM:
--

Hi Tyler,
any update on this ? Any planned date ? 
At present we are really stuck here, as we are unable to create application 
schema and migrate application to new tech stack.

We are facing same issue in 2.2.0 

Best Regards
Sanjay



was (Author: bholya):
Hi Tyler,
any update on this ? Any planned date ? 
At present we are really stuck here, as we are unable to create application 
schema and migrate application to new tech stack.

Best Regards
Sanjay


 Cannot run CQL scripts on Windows AND having error Ubuntu Linux
 ---

 Key: CASSANDRA-9670
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9670
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment: - Apache Cassandra 2.1.7 and 2.1.8 in Ubuntu
 - DataStax Community Edition  on Windows 7, 64 Bit and Ubuntu 
Reporter: Sanjay Patel
Assignee: Tyler Hobbs
  Labels: cqlsh
 Fix For: 2.1.x

 Attachments: cities.cql, germany_cities.cql, germany_cities.cql, 
 india_cities.csv, india_states.csv, sp_setup.cql


 After installation of 2.1.6 and 2.1.7 it is not possible to execute cql 
 scripts, which were earlier executed on windows + Linux environment 
 successfully.
 I have tried to install Python 2 latest version and try to execute, but 
 having same error.
 Attaching cities.cql for reference.
 ---
 {code}
 cqlsh source 'shoppoint_setup.cql' ;
 shoppoint_setup.cql:16:InvalidRequest: code=2200 [Invalid query] 
 message=Keyspace 'shopping' does not exist
 shoppoint_setup.cql:647:'ascii' codec can't decode byte 0xc3 in position 57: 
 ordinal not in range(128)
 cities.cql:9:'ascii' codec can't decode byte 0xc3 in position 51: ordinal not 
 in range(128)
 cities.cql:14:
 Error starting import process:
 cities.cql:14:Can't pickle type 'thread.lock': it's not found as thread.lock
 cities.cql:14:can only join a started process
 cities.cql:16:
 Error starting import process:
 cities.cql:16:Can't pickle type 'thread.lock': it's not found as thread.lock
 cities.cql:16:can only join a started process
 Traceback (most recent call last):
   File string, line 1, in module
   File I:\programm\python2710\lib\multiprocessing\forking.py, line 380, in 
 main
 prepare(preparation_data)
   File I:\programm\python2710\lib\multiprocessing\forking.py, line 489, in 
 prepare
 Traceback (most recent call last):
   File string, line 1, in module
 file, path_name, etc = imp.find_module(main_name, dirs)
 ImportError: No module named cqlsh
   File I:\programm\python2710\lib\multiprocessing\forking.py, line 380, in 
 main
 prepare(preparation_data)
   File I:\programm\python2710\lib\multiprocessing\forking.py, line 489, in 
 prepare
 file, path_name, etc = imp.find_module(main_name, dirs)
 ImportError: No module named cqlsh
 shoppoint_setup.cql:663:'ascii' codec can't decode byte 0xc3 in position 18: 
 ordinal not in range(128)
 ipcache.cql:28:ServerError: ErrorMessage code= [Server error] 
 message=java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
 java.lang.RuntimeException: java.io.FileNotFoundException: 
 I:\var\lib\cassandra\data\syste
 m\schema_columns-296e9c049bec3085827dc17d3df2122a\system-schema_columns-ka-300-Data.db
  (The process cannot access the file because it is being used by another 
 process)
 ccavn_bulkupdate.cql:75:ServerError: ErrorMessage code= [Server error] 
 message=java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
 java.lang.RuntimeException: java.io.FileNotFoundException: 
 I:\var\lib\cassandra\d
 ata\system\schema_columns-296e9c049bec3085827dc17d3df2122a\system-schema_columns-tmplink-ka-339-Data.db
  (The process cannot access the file because it is being used by another 
 process)
 shoppoint_setup.cql:680:'ascii' codec can't decode byte 0xe2 in position 14: 
 ordinal not in range(128){code}
 -
 In one of Ubuntu development environment we have similar errors.
 -
 {code}
 shoppoint_setup.cql:647:'ascii' codec can't decode byte 0xc3 in position 57: 
 ordinal not in range(128)
 cities.cql:9:'ascii' codec can't decode byte 0xc3 in position 51: ordinal not 
 in range(128)
 (corresponding line) COPY cities (city,country_code,state,isactive) FROM 
 'testdata/india_cities.csv' ;
 [19:53:18] j.basu: shoppoint_setup.cql:663:'ascii' codec can't decode byte 
 0xc3 in position 18: ordinal not in range(128)
 {code}
 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

cassandra git commit: Add a note about new JVM settings for 3.0 in NEWS.TXT

2015-07-31 Thread jake

Repository: cassandra
Updated Branches:
  refs/heads/trunk 1774eb9a8 - 43139f866


Add a note about new JVM settings for 3.0 in NEWS.TXT

patch by tjake; reviewed by jbellis for CASSANDRA-9483


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/43139f86
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/43139f86
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/43139f86

Branch: refs/heads/trunk
Commit: 43139f8668e0a524035507efc619cd0e0f06cea9
Parents: 1774eb9
Author: T Jake Luciani j...@apache.org
Authored: Fri Jul 31 16:07:28 2015 -0400
Committer: T Jake Luciani j...@apache.org
Committed: Fri Jul 31 16:08:50 2015 -0400

--
 NEWS.txt | 6 ++
 1 file changed, 6 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/43139f86/NEWS.txt
--
diff --git a/NEWS.txt b/NEWS.txt
index 392a581..0e0d7c4 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -23,9 +23,15 @@ New features
  for non-primary key queries, and perform much better for indexing high
  cardinality columns. 
  See 
http://www.datastax.com/dev/blog/new-in-cassandra-3-0-materialized-views 
+

 Upgrading
 -
+   - 3.0 requires Java 8u20 or later.
+   - The default JVM GC has been changed to G1GC.
+   - The default JVM flag -XX:+PerfDisableSharedMem will cause the following 
tools JVM
+ to stop working: jps, jstack, jinfo, jmc, jcmd as well as 3rd party tools 
like Jolokia.
+ If you wish to use these tools you can comment this flag out in 
cassandra-env.{sh,ps1}
- New write stages have been added for batchlog and materialized view 
mutations
  you can set their size in cassandra.yaml
- User defined functions are now executed in a sandbox.

[Cassandra Wiki] Update of FrontPage by PauloMotta

2015-07-31 Thread Apache Wiki

Dear Wiki user,

You have subscribed to a wiki page or wiki category on Cassandra Wiki for 
change notification.

The FrontPage page has been changed by PauloMotta:
https://wiki.apache.org/cassandra/FrontPage?action=diffrev1=105rev2=106

   * [[HowToContribute|How To Contribute]]?
   * [[How To Commit?|HowToCommit]]
   * [[HowToPublishReleases|How To Release]] (Note: currently a work in 
progress) (Note: only relevant to Cassandra Committers)
+  * [[Windows Development|WindowsDevelopment]]
  
  == Mailing lists ==

[jira] [Commented] (CASSANDRA-9902) Windows dtest: invalid jolokia agent path


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649796#comment-14649796
 ] 

Joshua McKenzie commented on CASSANDRA-9902:


On a side note, it looks like it's 
[ConEmuHk|https://code.google.com/p/conemu-maximus5/wiki/ConEmuFAQ#Q._What_is_the_purpose_of_ConEmuHk_.dll_(_ConEmuHk64_.dll)?]
 that's the culprit. If you uncheck the Inject ConEmuHk option in ConEmu the 
tests pass in ConEmu.

 Windows dtest: invalid jolokia agent path
 -

 Key: CASSANDRA-9902
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9902
 Project: Cassandra
  Issue Type: Sub-task
Reporter: Joshua McKenzie
Assignee: Paulo Motta
Priority: Minor
  Labels: Windows
 Fix For: 2.2.x

 Attachments: Not_Dep.jpg, VMAttach.java, win-crashlogs.zip


 A variety of tests are failing with similar:
 {noformat}
 05:39:07 
 ==
 05:39:07 ERROR: change_durable_writes_test 
 (configuration_test.TestConfiguration)
 05:39:07 
 --
 05:39:07 Traceback (most recent call last):
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\configuration_test.py,
  line 63, in change_durable_writes_test
 05:39:07 durable_init_size = commitlog_size(durable_node)
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\configuration_test.py,
  line 127, in commitlog_size
 05:39:07 with JolokiaAgent(node) as jmx:
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\jmxutils.py, 
 line 188, in __enter__
 05:39:07 self.start()
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\jmxutils.py, 
 line 80, in start
 05:39:07 subprocess.check_output(args, stderr=subprocess.STDOUT)
 05:39:07   File D:\Python27\lib\subprocess.py, line 573, in check_output
 05:39:07 raise CalledProcessError(retcode, cmd, output=output)
 05:39:07 CalledProcessError: Command '('java', '-jar', 
 'lib\\jolokia-jvm-1.2.3-agent.jar', '--host', '127.0.0.1', 'start', '3436')' 
 returned non-zero exit status 2
 05:39:07   begin captured logging  
 
 05:39:07 dtest: DEBUG: cluster ccm directory: d:\temp\dtest-kjkypw
 05:39:07 -  end captured logging  
 -
 {noformat}
 Likely related:
 {noformat}
 20:54:46 Failed to start jolokia agent (command was: java -jar 
 lib\jolokia-jvm-1.2.3-agent.jar --host 127.0.0.1 start 3436): Command 
 '('java', '-jar', 'lib\\jolokia-jvm-1.2.3-agent.jar', '--host', '127.0.0.1', 
 'start', '3436')' returned non-zero exit status 2
 20:54:46 Exit status was: 2
 20:54:46 Output was: Error: Registry key 'Software\JavaSoft\Java Runtime 
 Environment'\CurrentVersion'
 20:54:46 has value '1.8', but '1.7' is required.
 20:54:46 Error: could not find java.dll
 20:54:46 Error: Could not find Java SE Runtime Environment.
 {noformat}
 And at the top of jmxutils.py we see:
 {code}
 JOLOKIA_JAR = os.path.join('lib', 'jolokia-jvm-1.2.3-agent.jar')
 {code}
 So that's not going to work out too well on Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9483) Document incompatibilities with -XX:+PerfDisableSharedMem


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649730#comment-14649730
 ] 

Jonathan Ellis commented on CASSANDRA-9483:
---

{{working:}} instead of {{working.}}, otherwise +1

 Document incompatibilities with -XX:+PerfDisableSharedMem
 -

 Key: CASSANDRA-9483
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9483
 Project: Cassandra
  Issue Type: Task
  Components: Config, Documentation  website
Reporter: Tyler Hobbs
Assignee: T Jake Luciani
Priority: Minor
 Fix For: 3.0 beta 1

 Attachments: news_update.txt


 We recently discovered that [the Jolokia agent is incompatible with  the 
 -XX:+PerfDisableSharedMem JVM 
 option|https://github.com/rhuss/jolokia/issues/198].  I assume that this may 
 affect other monitoring tools as well.
 If we are going to leave this enabled by default, we should document the 
 potential problems with it.  A combination of a comment in 
 {{cassandra-env.sh}} (and the Windows equivalent) and a comment in NEWS.txt 
 should suffice, I think.
 If possible, it would be good to figure out what other tools are affected and 
 also mention them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9902) Windows dtest: invalid jolokia agent path


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649719#comment-14649719
 ] 

Paulo Motta commented on CASSANDRA-9902:


to make the context clearer, the 
{{TestConfiguration.change_durable_writes_test}} was failing because of the 
following assertion near the end of the test:

{noformat}
self.assertGreater(commitlog_size(node), init_size, msg='ALTER KEYSPACE was not 
respected')
{noformat}

because the highlighted part from the test description was actually missing 
from the test:

{noformat}
- using ALTER KEYSPACE to set its DURABLE_WRITES option to true,
***writing a dataset to this keyspace that is known to trigger a 
commitlog fsync***
- asserting that the commitlog has grown in size since the data was 
written.
{noformat}

So I basically add the following missing line before the assertion to make the 
test pass (in this 
[commit|https://github.com/pauloricardomg/cassandra-dtest/commit/dfca2e8918b89bc9b88a2ac4a950699d7a99baaa]):

{noformat}
write_to_trigger_fsync(session, 'ks', 'tab')
{noformat}

Does this seems OK?

 Windows dtest: invalid jolokia agent path
 -

 Key: CASSANDRA-9902
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9902
 Project: Cassandra
  Issue Type: Sub-task
Reporter: Joshua McKenzie
Assignee: Paulo Motta
Priority: Minor
  Labels: Windows
 Fix For: 2.2.x

 Attachments: Not_Dep.jpg, VMAttach.java, win-crashlogs.zip


 A variety of tests are failing with similar:
 {noformat}
 05:39:07 
 ==
 05:39:07 ERROR: change_durable_writes_test 
 (configuration_test.TestConfiguration)
 05:39:07 
 --
 05:39:07 Traceback (most recent call last):
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\configuration_test.py,
  line 63, in change_durable_writes_test
 05:39:07 durable_init_size = commitlog_size(durable_node)
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\configuration_test.py,
  line 127, in commitlog_size
 05:39:07 with JolokiaAgent(node) as jmx:
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\jmxutils.py, 
 line 188, in __enter__
 05:39:07 self.start()
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\jmxutils.py, 
 line 80, in start
 05:39:07 subprocess.check_output(args, stderr=subprocess.STDOUT)
 05:39:07   File D:\Python27\lib\subprocess.py, line 573, in check_output
 05:39:07 raise CalledProcessError(retcode, cmd, output=output)
 05:39:07 CalledProcessError: Command '('java', '-jar', 
 'lib\\jolokia-jvm-1.2.3-agent.jar', '--host', '127.0.0.1', 'start', '3436')' 
 returned non-zero exit status 2
 05:39:07   begin captured logging  
 
 05:39:07 dtest: DEBUG: cluster ccm directory: d:\temp\dtest-kjkypw
 05:39:07 -  end captured logging  
 -
 {noformat}
 Likely related:
 {noformat}
 20:54:46 Failed to start jolokia agent (command was: java -jar 
 lib\jolokia-jvm-1.2.3-agent.jar --host 127.0.0.1 start 3436): Command 
 '('java', '-jar', 'lib\\jolokia-jvm-1.2.3-agent.jar', '--host', '127.0.0.1', 
 'start', '3436')' returned non-zero exit status 2
 20:54:46 Exit status was: 2
 20:54:46 Output was: Error: Registry key 'Software\JavaSoft\Java Runtime 
 Environment'\CurrentVersion'
 20:54:46 has value '1.8', but '1.7' is required.
 20:54:46 Error: could not find java.dll
 20:54:46 Error: Could not find Java SE Runtime Environment.
 {noformat}
 And at the top of jmxutils.py we see:
 {code}
 JOLOKIA_JAR = os.path.join('lib', 'jolokia-jvm-1.2.3-agent.jar')
 {code}
 So that's not going to work out too well on Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9670) Cannot run CQL scripts on Windows AND having error Ubuntu Linux

2015-07-31 Thread Sanjay Patel (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649728#comment-14649728
 ] 

Sanjay Patel commented on CASSANDRA-9670:
-

Hi Tyler,
any update on this ? Any planned date ? 
At present we are really stuck here, as we are unable to create application 
schema and migrate application to new tech stack.

Best Regards
Sanjay


 Cannot run CQL scripts on Windows AND having error Ubuntu Linux
 ---

 Key: CASSANDRA-9670
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9670
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment: - Apache Cassandra 2.1.7 and 2.1.8 in Ubuntu
 - DataStax Community Edition  on Windows 7, 64 Bit and Ubuntu 
Reporter: Sanjay Patel
Assignee: Tyler Hobbs
  Labels: cqlsh
 Fix For: 2.1.x

 Attachments: cities.cql, germany_cities.cql, germany_cities.cql, 
 india_cities.csv, india_states.csv, sp_setup.cql


 After installation of 2.1.6 and 2.1.7 it is not possible to execute cql 
 scripts, which were earlier executed on windows + Linux environment 
 successfully.
 I have tried to install Python 2 latest version and try to execute, but 
 having same error.
 Attaching cities.cql for reference.
 ---
 {code}
 cqlsh source 'shoppoint_setup.cql' ;
 shoppoint_setup.cql:16:InvalidRequest: code=2200 [Invalid query] 
 message=Keyspace 'shopping' does not exist
 shoppoint_setup.cql:647:'ascii' codec can't decode byte 0xc3 in position 57: 
 ordinal not in range(128)
 cities.cql:9:'ascii' codec can't decode byte 0xc3 in position 51: ordinal not 
 in range(128)
 cities.cql:14:
 Error starting import process:
 cities.cql:14:Can't pickle type 'thread.lock': it's not found as thread.lock
 cities.cql:14:can only join a started process
 cities.cql:16:
 Error starting import process:
 cities.cql:16:Can't pickle type 'thread.lock': it's not found as thread.lock
 cities.cql:16:can only join a started process
 Traceback (most recent call last):
   File string, line 1, in module
   File I:\programm\python2710\lib\multiprocessing\forking.py, line 380, in 
 main
 prepare(preparation_data)
   File I:\programm\python2710\lib\multiprocessing\forking.py, line 489, in 
 prepare
 Traceback (most recent call last):
   File string, line 1, in module
 file, path_name, etc = imp.find_module(main_name, dirs)
 ImportError: No module named cqlsh
   File I:\programm\python2710\lib\multiprocessing\forking.py, line 380, in 
 main
 prepare(preparation_data)
   File I:\programm\python2710\lib\multiprocessing\forking.py, line 489, in 
 prepare
 file, path_name, etc = imp.find_module(main_name, dirs)
 ImportError: No module named cqlsh
 shoppoint_setup.cql:663:'ascii' codec can't decode byte 0xc3 in position 18: 
 ordinal not in range(128)
 ipcache.cql:28:ServerError: ErrorMessage code= [Server error] 
 message=java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
 java.lang.RuntimeException: java.io.FileNotFoundException: 
 I:\var\lib\cassandra\data\syste
 m\schema_columns-296e9c049bec3085827dc17d3df2122a\system-schema_columns-ka-300-Data.db
  (The process cannot access the file because it is being used by another 
 process)
 ccavn_bulkupdate.cql:75:ServerError: ErrorMessage code= [Server error] 
 message=java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
 java.lang.RuntimeException: java.io.FileNotFoundException: 
 I:\var\lib\cassandra\d
 ata\system\schema_columns-296e9c049bec3085827dc17d3df2122a\system-schema_columns-tmplink-ka-339-Data.db
  (The process cannot access the file because it is being used by another 
 process)
 shoppoint_setup.cql:680:'ascii' codec can't decode byte 0xe2 in position 14: 
 ordinal not in range(128){code}
 -
 In one of Ubuntu development environment we have similar errors.
 -
 {code}
 shoppoint_setup.cql:647:'ascii' codec can't decode byte 0xc3 in position 57: 
 ordinal not in range(128)
 cities.cql:9:'ascii' codec can't decode byte 0xc3 in position 51: ordinal not 
 in range(128)
 (corresponding line) COPY cities (city,country_code,state,isactive) FROM 
 'testdata/india_cities.csv' ;
 [19:53:18] j.basu: shoppoint_setup.cql:663:'ascii' codec can't decode byte 
 0xc3 in position 18: ordinal not in range(128)
 {code}
 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-9903) Windows dtest: read timeout in consistency_test.py:TestConsistency.short_read_quorum_delete_test


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

T Jake Luciani updated CASSANDRA-9903:
--
Fix Version/s: (was: 2.2.x)
   2.2.1

 Windows dtest: read timeout in 
 consistency_test.py:TestConsistency.short_read_quorum_delete_test
 

 Key: CASSANDRA-9903
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9903
 Project: Cassandra
  Issue Type: Sub-task
Reporter: Joshua McKenzie
Assignee: T Jake Luciani
Priority: Minor
  Labels: Windows
 Fix For: 2.2.1


 {noformat}
 05:39:07 
 ==
 05:39:07 ERROR: short_read_quorum_delete_test 
 (consistency_test.TestConsistency)
 05:39:07 
 --
 05:39:07 Traceback (most recent call last):
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\consistency_test.py,
  line 702, in short_read_quorum_delete_test
 05:39:07 assert_none(session, SELECT * FROM t WHERE id = 0 LIMIT 1, 
 cl=ConsistencyLevel.QUORUM)
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\assertions.py,
  line 42, in assert_none
 05:39:07 res = session.execute(simple_query)
 05:39:07   File D:\Python27\lib\site-packages\cassandra\cluster.py, line 
 1405, in execute
 05:39:07 result = future.result(timeout)
 05:39:07   File D:\Python27\lib\site-packages\cassandra\cluster.py, line 
 2976, in result
 05:39:07 raise self._final_exception
 05:39:07 ReadTimeout: code=1200 [Coordinator node timed out waiting for 
 replica nodes' responses] message=Operation timed out - received only 1 
 responses. info={'received_responses': 1, 'required_responses': 2, 
 'consistency': 'QUORUM'}
 05:39:07   begin captured logging  
 
 05:39:07 dtest: DEBUG: cluster ccm directory: d:\temp\dtest-l5euwj
 05:39:07 -  end captured logging  
 -
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[Cassandra Wiki] Update of ContributorsGroup by BrandonWilliams

2015-07-31 Thread Apache Wiki

Dear Wiki user,

You have subscribed to a wiki page or wiki category on Cassandra Wiki for 
change notification.

The ContributorsGroup page has been changed by BrandonWilliams:
https://wiki.apache.org/cassandra/ContributorsGroup?action=diffrev1=47rev2=48

   * wombat
   * AlicePorfirio
   * SamTunnicliffe
+  * PauloMotta

[jira] [Commented] (CASSANDRA-8143) Partitioner should not be accessed through StorageService


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649733#comment-14649733
 ] 

Aleksey Yeschenko commented on CASSANDRA-8143:
--

My apologies for not checking with dtests. Schema propagation was failing, 
ultimately because of a bug in {{Mutation.deserialize()}} for size  1. It 
wasn't being caught by any utests (which I had run).

Fix 
[here|https://github.com/iamaleksey/cassandra/commit/7aaf15ebd4807b107cf24f47529424102b7ac026],
 will revert and commit once 
[dtest|http://cassci.datastax.com/view/Dev/view/iamaleksey/job/iamaleksey-8143-dtest/]
 and 
[testall|http://cassci.datastax.com/view/Dev/view/iamaleksey/job/iamaleksey-8143-testall/]
 are both clear.

 Partitioner should not be accessed through StorageService
 -

 Key: CASSANDRA-8143
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8143
 Project: Cassandra
  Issue Type: Improvement
Reporter: Branimir Lambov
Assignee: Branimir Lambov
 Fix For: 3.0 beta 1


 The configured partitioner is no longer the only partitioner in use in the 
 database, as e.g. index tables use LocalPartitioner.
 To make sure the correct partitioner is used for each table, accesses of 
 StorageService.getPartitioner() should be replaced with retrieval of the 
 CFS-specific partitioner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9902) Windows dtest: invalid jolokia agent path

2015-07-31 Thread Jim Witschey (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649766#comment-14649766
 ] 

Jim Witschey commented on CASSANDRA-9902:
-

Oh, wow, I see now; thanks to both of you for finding and explaining this.

Yes, I believe your changes are correct; I'm not sure why this test failed 
before CASSANDRA-9560 and succeeded after. I'd appreciate a pull request on the 
dtest repo.

Great find, thank you.

 Windows dtest: invalid jolokia agent path
 -

 Key: CASSANDRA-9902
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9902
 Project: Cassandra
  Issue Type: Sub-task
Reporter: Joshua McKenzie
Assignee: Paulo Motta
Priority: Minor
  Labels: Windows
 Fix For: 2.2.x

 Attachments: Not_Dep.jpg, VMAttach.java, win-crashlogs.zip


 A variety of tests are failing with similar:
 {noformat}
 05:39:07 
 ==
 05:39:07 ERROR: change_durable_writes_test 
 (configuration_test.TestConfiguration)
 05:39:07 
 --
 05:39:07 Traceback (most recent call last):
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\configuration_test.py,
  line 63, in change_durable_writes_test
 05:39:07 durable_init_size = commitlog_size(durable_node)
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\configuration_test.py,
  line 127, in commitlog_size
 05:39:07 with JolokiaAgent(node) as jmx:
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\jmxutils.py, 
 line 188, in __enter__
 05:39:07 self.start()
 05:39:07   File 
 D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\jmxutils.py, 
 line 80, in start
 05:39:07 subprocess.check_output(args, stderr=subprocess.STDOUT)
 05:39:07   File D:\Python27\lib\subprocess.py, line 573, in check_output
 05:39:07 raise CalledProcessError(retcode, cmd, output=output)
 05:39:07 CalledProcessError: Command '('java', '-jar', 
 'lib\\jolokia-jvm-1.2.3-agent.jar', '--host', '127.0.0.1', 'start', '3436')' 
 returned non-zero exit status 2
 05:39:07   begin captured logging  
 
 05:39:07 dtest: DEBUG: cluster ccm directory: d:\temp\dtest-kjkypw
 05:39:07 -  end captured logging  
 -
 {noformat}
 Likely related:
 {noformat}
 20:54:46 Failed to start jolokia agent (command was: java -jar 
 lib\jolokia-jvm-1.2.3-agent.jar --host 127.0.0.1 start 3436): Command 
 '('java', '-jar', 'lib\\jolokia-jvm-1.2.3-agent.jar', '--host', '127.0.0.1', 
 'start', '3436')' returned non-zero exit status 2
 20:54:46 Exit status was: 2
 20:54:46 Output was: Error: Registry key 'Software\JavaSoft\Java Runtime 
 Environment'\CurrentVersion'
 20:54:46 has value '1.8', but '1.7' is required.
 20:54:46 Error: could not find java.dll
 20:54:46 Error: Could not find Java SE Runtime Environment.
 {noformat}
 And at the top of jmxutils.py we see:
 {code}
 JOLOKIA_JAR = os.path.join('lib', 'jolokia-jvm-1.2.3-agent.jar')
 {code}
 So that's not going to work out too well on Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9941) SSTableVerify has wrong return codes on windows


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649799#comment-14649799
 ] 

Joshua McKenzie commented on CASSANDRA-9941:


Testing on this brought up a different issue: the ccm call to run_sstableverify 
is actually checking the contents of the data\ folder on the *root* cassandra 
installation directory rather than the ccm nodes. The test was failing out for 
me on the assertion checking for the failure text as I was sourcing from a 
local 2.2 checkout that I'd happened to perform stress on yesterday which still 
had data files lying around and it had the KS/CF that this test was looking for.

It's a separate ccm-bug that I'm discussing w/[~philipthompson] offline that 
shouldn't hold up inspection of this ticket. I'm out for the next week so if 
anyone wants to grab this in the interim, feel free.

 SSTableVerify has wrong return codes on windows
 ---

 Key: CASSANDRA-9941
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9941
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
Reporter: Philip Thompson
Assignee: Joshua McKenzie
 Fix For: 2.2.x


 The dtest {{offline_tools_test.py:TestOfflineTools.sstableverify_test}} is 
 failing on windows. When running sstableverify.bat on a nonexistent keyspace, 
 the correct error message is printed, but an error code of 0 for success is 
 returned, which does not match the correct behavior on linux, which returns 1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-9949) maxPurgeableTimestamp needs to check memtables too


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-9949:
--
Assignee: Stefania

 maxPurgeableTimestamp needs to check memtables too
 --

 Key: CASSANDRA-9949
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9949
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Jonathan Ellis
Assignee: Stefania
 Fix For: 2.1.x, 2.2.x


 overlapIterator/maxPurgeableTimestamp don't include the memtables, so a 
 very-out-of-order write could be ignored



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8143) Partitioner should not be accessed through StorageService

2015-07-31 Thread Ariel Weisberg (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649936#comment-14649936
 ] 

Ariel Weisberg commented on CASSANDRA-8143:
---

No need to apologize. Cassci is not in a good state right now.

I do want to emphasize that it doesn't matter if the tests pass when you run 
them. The only thing that matters is that they pass in Cassci. Well we want 
them to pass on desktops as well, but if it passes in Cassci and not on a 
desktop I would never fault the developer. I stress this because it's common 
for things to only fail in Cassci.

 Partitioner should not be accessed through StorageService
 -

 Key: CASSANDRA-8143
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8143
 Project: Cassandra
  Issue Type: Improvement
Reporter: Branimir Lambov
Assignee: Branimir Lambov
 Fix For: 3.0 beta 1


 The configured partitioner is no longer the only partitioner in use in the 
 database, as e.g. index tables use LocalPartitioner.
 To make sure the correct partitioner is used for each table, accesses of 
 StorageService.getPartitioner() should be replaced with retrieval of the 
 CFS-specific partitioner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/4] cassandra git commit: Revert Revert Stop accessing the partitioner directly via StorageService

Repository: cassandra
Updated Branches:
  refs/heads/trunk 43139f866 - 0a08525ad


http://git-wip-us.apache.org/repos/asf/cassandra/blob/0a08525a/test/unit/org/apache/cassandra/db/ScrubTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/ScrubTest.java 
b/test/unit/org/apache/cassandra/db/ScrubTest.java
index ee51a4d..25b9cde 100644
--- a/test/unit/org/apache/cassandra/db/ScrubTest.java
+++ b/test/unit/org/apache/cassandra/db/ScrubTest.java
@@ -22,8 +22,6 @@ import java.io.File;
 import java.io.IOError;
 import java.io.IOException;
 import java.io.RandomAccessFile;
-import java.lang.reflect.Field;
-import java.lang.reflect.Modifier;
 import java.nio.ByteBuffer;
 import java.util.*;
 import java.util.concurrent.ExecutionException;
@@ -44,9 +42,9 @@ import org.apache.cassandra.cql3.UntypedResultSet;
 import org.apache.cassandra.db.compaction.CompactionManager;
 import org.apache.cassandra.db.compaction.OperationType;
 import org.apache.cassandra.db.compaction.Scrubber;
-import org.apache.cassandra.db.index.SecondaryIndex;
 import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
 import org.apache.cassandra.db.marshal.*;
+import org.apache.cassandra.db.marshal.UUIDType;
 import org.apache.cassandra.db.partitions.Partition;
 import org.apache.cassandra.db.partitions.PartitionUpdate;
 import org.apache.cassandra.dht.ByteOrderedPartitioner;
@@ -55,14 +53,10 @@ import 
org.apache.cassandra.exceptions.ConfigurationException;
 import org.apache.cassandra.exceptions.RequestExecutionException;
 import org.apache.cassandra.exceptions.WriteTimeoutException;
 import org.apache.cassandra.io.compress.CompressionMetadata;
-import org.apache.cassandra.io.sstable.Component;
-import org.apache.cassandra.io.sstable.Descriptor;
-import org.apache.cassandra.io.sstable.SSTableRewriter;
-import org.apache.cassandra.io.sstable.SSTableTxnWriter;
+import org.apache.cassandra.io.sstable.*;
 import org.apache.cassandra.io.sstable.format.SSTableReader;
 import org.apache.cassandra.io.util.FileUtils;
 import org.apache.cassandra.schema.KeyspaceParams;
-import org.apache.cassandra.service.StorageService;
 import org.apache.cassandra.utils.ByteBufferUtil;
 
 import static org.junit.Assert.*;
@@ -79,6 +73,8 @@ public class ScrubTest
 public static final String CF_UUID = UUIDKeys;
 public static final String CF_INDEX1 = Indexed1;
 public static final String CF_INDEX2 = Indexed2;
+public static final String CF_INDEX1_BYTEORDERED = Indexed1_ordered;
+public static final String CF_INDEX2_BYTEORDERED = Indexed2_ordered;
 
 public static final String COL_INDEX = birthdate;
 public static final String COL_NON_INDEX = notbirthdate;
@@ -98,7 +94,9 @@ public class ScrubTest
 
.compressionParameters(SchemaLoader.getCompressionParameters(COMPRESSION_CHUNK_LENGTH)),
 SchemaLoader.standardCFMD(KEYSPACE, 
CF_UUID, 0, UUIDType.instance),
 SchemaLoader.keysIndexCFMD(KEYSPACE, 
CF_INDEX1, true),
-SchemaLoader.compositeIndexCFMD(KEYSPACE, 
CF_INDEX2, true));
+SchemaLoader.compositeIndexCFMD(KEYSPACE, 
CF_INDEX2, true),
+SchemaLoader.keysIndexCFMD(KEYSPACE, 
CF_INDEX1_BYTEORDERED, true).copy(ByteOrderedPartitioner.instance),
+SchemaLoader.compositeIndexCFMD(KEYSPACE, 
CF_INDEX2_BYTEORDERED, true).copy(ByteOrderedPartitioner.instance));
 }
 
 @Test
@@ -306,7 +304,7 @@ public class ScrubTest
 {
 // This test assumes ByteOrderPartitioner to create out-of-order 
SSTable
 IPartitioner oldPartitioner = DatabaseDescriptor.getPartitioner();
-DatabaseDescriptor.setPartitioner(new ByteOrderedPartitioner());
+DatabaseDescriptor.setPartitionerUnsafe(new ByteOrderedPartitioner());
 
 // Create out-of-order SSTable
 File tempDir = File.createTempFile(ScrubTest.testScrubOutOfOrder, 
).getParentFile();
@@ -380,7 +378,7 @@ public class ScrubTest
 {
 FileUtils.deleteRecursive(tempDataDir);
 // reset partitioner
-DatabaseDescriptor.setPartitioner(oldPartitioner);
+DatabaseDescriptor.setPartitionerUnsafe(oldPartitioner);
 }
 }
 
@@ -394,9 +392,9 @@ public class ScrubTest
 CompressionMetadata compData = 
CompressionMetadata.create(sstable.getFilename());
 
 CompressionMetadata.Chunk chunk1 = compData.chunkFor(
-sstable.getPosition(PartitionPosition.ForKey.get(key1, 
sstable.partitioner), SSTableReader.Operator.EQ).position);
+sstable.getPosition(PartitionPosition.ForKey.get(key1, 
sstable.getPartitioner()), SSTableReader.Operator.EQ).position);
 CompressionMetadata.Chunk chunk2 = compData.chunkFor(
-

[3/4] cassandra git commit: Revert Revert Stop accessing the partitioner directly via StorageService

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0a08525a/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
--
diff --git a/src/java/org/apache/cassandra/db/index/SecondaryIndex.java 
b/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
index 94031ab..9221090 100644
--- a/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
+++ b/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
@@ -42,15 +42,12 @@ import org.apache.cassandra.db.index.keys.KeysIndex;
 import org.apache.cassandra.db.lifecycle.SSTableSet;
 import org.apache.cassandra.db.lifecycle.View;
 import org.apache.cassandra.db.marshal.AbstractType;
-import org.apache.cassandra.db.marshal.BytesType;
-import org.apache.cassandra.db.marshal.LocalByPartionerType;
+import org.apache.cassandra.dht.LocalPartitioner;
 import org.apache.cassandra.exceptions.ConfigurationException;
 import org.apache.cassandra.io.sstable.format.SSTableReader;
 import org.apache.cassandra.exceptions.InvalidRequestException;
 import org.apache.cassandra.io.sstable.ReducingKeyIterator;
-import org.apache.cassandra.service.StorageService;
 import org.apache.cassandra.utils.FBUtilities;
-
 import org.apache.cassandra.utils.concurrent.Refs;
 
 /**
@@ -79,10 +76,6 @@ public abstract class SecondaryIndex
  */
 public static final String INDEX_ENTRIES_OPTION_NAME = 
index_keys_and_values;
 
-public static final AbstractType? keyComparator = 
StorageService.getPartitioner().preservesOrder()
-  ? BytesType.instance
-  : new 
LocalByPartionerType(StorageService.getPartitioner());
-
 /**
  * Base CF that has many indexes
  */
@@ -303,7 +296,7 @@ public abstract class SecondaryIndex
  */
 public DecoratedKey getIndexKeyFor(ByteBuffer value)
 {
-return getIndexCfs().partitioner.decorateKey(value);
+return getIndexCfs().decorateKey(value);
 }
 
 /**
@@ -381,11 +374,20 @@ public abstract class SecondaryIndex
  */
 public static CFMetaData newIndexMetadata(CFMetaData baseMetadata, 
ColumnDefinition def)
 {
+return newIndexMetadata(baseMetadata, def, def.type);
+}
+
+/**
+ * Create the index metadata for the index on a given column of a given 
table.
+ */
+static CFMetaData newIndexMetadata(CFMetaData baseMetadata, 
ColumnDefinition def, AbstractType? comparator)
+{
 if (def.getIndexType() == IndexType.CUSTOM)
 return null;
 
 CFMetaData.Builder builder = 
CFMetaData.Builder.create(baseMetadata.ksName, 
baseMetadata.indexColumnFamilyName(def))

.withId(baseMetadata.cfId)
+   .withPartitioner(new 
LocalPartitioner(comparator))

.addPartitionKey(def.name, def.type);
 
 if (def.getIndexType() == IndexType.COMPOSITES)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0a08525a/src/java/org/apache/cassandra/db/index/composites/CompositesIndex.java
--
diff --git 
a/src/java/org/apache/cassandra/db/index/composites/CompositesIndex.java 
b/src/java/org/apache/cassandra/db/index/composites/CompositesIndex.java
index 42861c5..29f235c 100644
--- a/src/java/org/apache/cassandra/db/index/composites/CompositesIndex.java
+++ b/src/java/org/apache/cassandra/db/index/composites/CompositesIndex.java
@@ -101,7 +101,7 @@ public abstract class CompositesIndex extends 
AbstractSimplePerColumnSecondaryIn
 
 protected static void addGenericClusteringColumns(CFMetaData.Builder 
indexMetadata, CFMetaData baseMetadata, ColumnDefinition columnDef)
 {
-indexMetadata.addClusteringColumn(partition_key, 
SecondaryIndex.keyComparator);
+indexMetadata.addClusteringColumn(partition_key, 
baseMetadata.partitioner.partitionOrdering());
 for (ColumnDefinition def : baseMetadata.clusteringColumns())
 indexMetadata.addClusteringColumn(def.name, def.type);
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0a08525a/src/java/org/apache/cassandra/db/index/composites/CompositesIndexOnClusteringKey.java
--
diff --git 
a/src/java/org/apache/cassandra/db/index/composites/CompositesIndexOnClusteringKey.java
 
b/src/java/org/apache/cassandra/db/index/composites/CompositesIndexOnClusteringKey.java
index 6529ad9..cd4aff9 100644
--- 
a/src/java/org/apache/cassandra/db/index/composites/CompositesIndexOnClusteringKey.java
+++ 
b/src/java/org/apache/cassandra/db/index/composites/CompositesIndexOnClusteringKey.java
@@ -24,7 +24,6 @@ import org.apache.cassandra.config.CFMetaData;
 import org.apache.cassandra.config.ColumnDefinition;
 import

[2/4] cassandra git commit: Revert Revert Stop accessing the partitioner directly via StorageService

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0a08525a/src/java/org/apache/cassandra/io/sstable/format/big/BigTableScanner.java
--
diff --git 
a/src/java/org/apache/cassandra/io/sstable/format/big/BigTableScanner.java 
b/src/java/org/apache/cassandra/io/sstable/format/big/BigTableScanner.java
index 0451a98..1b3b407 100644
--- a/src/java/org/apache/cassandra/io/sstable/format/big/BigTableScanner.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/big/BigTableScanner.java
@@ -172,7 +172,7 @@ public class BigTableScanner implements ISSTableScanner
 while (!ifile.isEOF())
 {
 indexPosition = ifile.getFilePointer();
-DecoratedKey indexDecoratedKey = 
sstable.partitioner.decorateKey(ByteBufferUtil.readWithShortLength(ifile));
+DecoratedKey indexDecoratedKey = 
sstable.decorateKey(ByteBufferUtil.readWithShortLength(ifile));
 if (indexDecoratedKey.compareTo(currentRange.left)  0 || 
currentRange.contains(indexDecoratedKey))
 {
 // Found, just read the dataPosition and seek into index 
and data files
@@ -282,7 +282,7 @@ public class BigTableScanner implements ISSTableScanner
 if (ifile.isEOF())
 return endOfData();
 
-currentKey = 
sstable.partitioner.decorateKey(ByteBufferUtil.readWithShortLength(ifile));
+currentKey = 
sstable.decorateKey(ByteBufferUtil.readWithShortLength(ifile));
 currentEntry = 
rowIndexEntrySerializer.deserialize(ifile);
 } while (!currentRange.contains(currentKey));
 }
@@ -301,7 +301,7 @@ public class BigTableScanner implements ISSTableScanner
 else
 {
 // we need the position of the start of the next key, 
regardless of whether it falls in the current range
-nextKey = 
sstable.partitioner.decorateKey(ByteBufferUtil.readWithShortLength(ifile));
+nextKey = 
sstable.decorateKey(ByteBufferUtil.readWithShortLength(ifile));
 nextEntry = rowIndexEntrySerializer.deserialize(ifile);
 
 if (!currentRange.contains(nextKey))

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0a08525a/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
--
diff --git 
a/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java 
b/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
index ff279a8..5607a7e 100644
--- a/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
@@ -19,7 +19,6 @@ package org.apache.cassandra.io.sstable.format.big;
 
 import java.io.*;
 import java.util.Map;
-import java.util.Set;
 
 import org.apache.cassandra.db.*;
 import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
@@ -32,7 +31,6 @@ import org.slf4j.LoggerFactory;
 import org.apache.cassandra.config.CFMetaData;
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.db.rows.*;
-import org.apache.cassandra.dht.IPartitioner;
 import org.apache.cassandra.io.FSWriteError;
 import org.apache.cassandra.io.compress.CompressedSequentialWriter;
 import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
@@ -62,12 +60,11 @@ public class BigTableWriter extends SSTableWriter
   Long keyCount, 
   Long repairedAt, 
   CFMetaData metadata, 
-  IPartitioner partitioner, 
   MetadataCollector metadataCollector, 
   SerializationHeader header,
   LifecycleTransaction txn)
 {
-super(descriptor, keyCount, repairedAt, metadata, partitioner, 
metadataCollector, header);
+super(descriptor, keyCount, repairedAt, metadata, metadataCollector, 
header);
 txn.trackNew(this); // must track before any files are created
 
 if (compression)
@@ -243,13 +240,12 @@ public class BigTableWriter extends SSTableWriter
 StatsMetadata stats = statsMetadata();
 assert boundary.indexLength  0  boundary.dataLength  0;
 // open the reader early
-IndexSummary indexSummary = iwriter.summary.build(partitioner, 
boundary);
+IndexSummary indexSummary = 
iwriter.summary.build(metadata.partitioner, boundary);
 SegmentedFile ifile = iwriter.builder.buildIndex(descriptor, 
indexSummary, boundary);
 SegmentedFile dfile = dbuilder.buildData(descriptor, stats, boundary);
 SSTableReader sstable = SSTableReader.internalOpen(descriptor,

[4/4] cassandra git commit: Revert Revert Stop accessing the partitioner directly via StorageService

Revert Revert Stop accessing the partitioner directly via StorageService

This reverts commit a22ce89e868644ea04f0f3dacec05fff1673a345.


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0a08525a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0a08525a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0a08525a

Branch: refs/heads/trunk
Commit: 0a08525ad236f78df05c854dead62f300eae271d
Parents: 43139f8
Author: Aleksey Yeschenko alek...@apache.org
Authored: Sat Aug 1 01:33:20 2015 +0300
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Sat Aug 1 01:33:51 2015 +0300

--
 .../org/apache/cassandra/config/CFMetaData.java |  64 +++-
 .../cassandra/config/DatabaseDescriptor.java|   7 +-
 .../org/apache/cassandra/config/Schema.java |   2 +-
 .../apache/cassandra/cql3/TokenRelation.java|   7 +-
 .../cassandra/cql3/functions/TokenFct.java  |  10 +-
 .../restrictions/StatementRestrictions.java |   2 +-
 .../cql3/restrictions/TokenFilter.java  |  14 +-
 .../cql3/restrictions/TokenRestriction.java |  23 +--
 .../cql3/statements/BatchStatement.java |   5 +-
 .../cql3/statements/ModificationStatement.java  |   7 +-
 .../cql3/statements/SelectStatement.java|   2 +-
 .../db/AbstractReadCommandBuilder.java  |  11 +-
 .../apache/cassandra/db/BatchlogManager.java|   7 +-
 .../apache/cassandra/db/ColumnFamilyStore.java  |  35 ++--
 src/java/org/apache/cassandra/db/DataRange.java |   3 +-
 .../cassandra/db/HintedHandOffManager.java  |  29 ++--
 src/java/org/apache/cassandra/db/Memtable.java  |   1 -
 src/java/org/apache/cassandra/db/Mutation.java  |  28 ++--
 .../apache/cassandra/db/PartitionPosition.java  |   2 +-
 .../cassandra/db/PartitionRangeReadCommand.java |   2 +-
 .../apache/cassandra/db/RowUpdateBuilder.java   |   2 +-
 .../db/SinglePartitionNamesCommand.java |  12 ++
 .../db/SinglePartitionReadCommand.java  |  18 ++-
 .../db/SinglePartitionSliceCommand.java |  17 ++
 .../org/apache/cassandra/db/SystemKeyspace.java |  18 +--
 .../db/compaction/CompactionManager.java|   4 +-
 .../db/compaction/LeveledManifest.java  |   2 +-
 .../cassandra/db/compaction/Scrubber.java   |   4 +-
 .../cassandra/db/compaction/Upgrader.java   |   1 -
 .../cassandra/db/compaction/Verifier.java   |   2 +-
 .../writers/DefaultCompactionWriter.java|   1 -
 .../writers/MajorLeveledCompactionWriter.java   |   2 -
 .../writers/MaxSSTableSizeWriter.java   |   2 -
 .../SplittingSizeTieredCompactionWriter.java|   2 -
 .../AbstractSimplePerColumnSecondaryIndex.java  |   4 +-
 .../cassandra/db/index/SecondaryIndex.java  |  20 +--
 .../db/index/composites/CompositesIndex.java|   2 +-
 .../CompositesIndexOnClusteringKey.java |   3 +-
 .../db/index/composites/CompositesSearcher.java |   2 +-
 .../cassandra/db/index/keys/KeysIndex.java  |   3 +-
 .../cassandra/db/index/keys/KeysSearcher.java   |   2 +-
 .../db/marshal/LocalByPartionerType.java|  97 
 .../db/marshal/PartitionerDefinedOrder.java |  91 +++
 .../db/partitions/AtomicBTreePartition.java |   4 +-
 .../db/partitions/PartitionUpdate.java  | 108 ++---
 .../rows/UnfilteredRowIteratorSerializer.java   |   3 +-
 .../cassandra/db/view/MaterializedView.java |   7 +-
 .../apache/cassandra/db/view/TemporalRow.java   |   2 +-
 .../org/apache/cassandra/dht/BootStrapper.java  |  12 +-
 .../cassandra/dht/ByteOrderedPartitioner.java   |   5 +
 .../org/apache/cassandra/dht/IPartitioner.java  |   6 +
 .../apache/cassandra/dht/LocalPartitioner.java  |   5 +
 .../cassandra/dht/Murmur3Partitioner.java   |   7 +
 .../dht/OrderPreservingPartitioner.java |   5 +
 .../apache/cassandra/dht/RandomPartitioner.java |   7 +
 .../org/apache/cassandra/dht/RangeStreamer.java |   2 +-
 .../dht/tokenallocator/TokenAllocation.java |   8 +-
 src/java/org/apache/cassandra/gms/Gossiper.java |   2 +-
 .../io/sstable/AbstractSSTableSimpleWriter.java |  10 +-
 .../cassandra/io/sstable/CQLSSTableWriter.java  |  15 +-
 .../cassandra/io/sstable/KeyIterator.java   |   8 +-
 .../io/sstable/ReducingKeyIterator.java |   2 +-
 .../apache/cassandra/io/sstable/SSTable.java|  21 ++-
 .../cassandra/io/sstable/SSTableLoader.java |  20 +--
 .../io/sstable/SSTableSimpleUnsortedWriter.java |   5 +-
 .../io/sstable/SSTableSimpleWriter.java |   4 +-
 .../io/sstable/format/SSTableReader.java|  74 -
 .../io/sstable/format/SSTableWriter.java|  16 +-
 .../io/sstable/format/big/BigFormat.java|   8 +-
 .../io/sstable/format/big/BigTableReader.java   |   8 +-
 .../io/sstable/format/big/BigTableScanner.java  |   6 +-
 .../io/sstable/format/big/BigTableWriter.java   |  15 +-

[jira] [Commented] (CASSANDRA-8143) Partitioner should not be accessed through StorageService


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649505#comment-14649505
 ] 

Jonathan Ellis commented on CASSANDRA-8143:
---

From IRC:

{noformat}
exlt this is going to be a hold the presses moment - we're going to jack up 
jenkins really quickly with this not being fixed
exlt ... the last run (as well as several before that) of that branch job was 
aborted for running out of control - 
http://cassci.datastax.com/view/Dev/view/blambov/job/blambov-8143-partitioner-dtest/17/
exlt this shouldn't have been merged
{noformat}

reverted pending a fix

 Partitioner should not be accessed through StorageService
 -

 Key: CASSANDRA-8143
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8143
 Project: Cassandra
  Issue Type: Improvement
Reporter: Branimir Lambov
Assignee: Branimir Lambov
 Fix For: 3.0 beta 1


 The configured partitioner is no longer the only partitioner in use in the 
 database, as e.g. index tables use LocalPartitioner.
 To make sure the correct partitioner is used for each table, accesses of 
 StorageService.getPartitioner() should be replaced with retrieval of the 
 CFS-specific partitioner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8368) Consider not using hints for batchlog replay, in any capacity

[
https://issues.apache.org/jira/browse/CASSANDRA-8368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649534#comment-14649534
]

Aleksey Yeschenko commented on CASSANDRA-8368:
--

bq. So you mean the hints will eventually be replayed even after
max_hint_window_in_ms goes by? Or just that you will store a hint for it after
max_hint_window_in_ms?

You misunderstand what {{max_hint_window_in_ms}} means. It's not about replay,
it's only about storing or not storing a hint in the first place.
Batchlog-created hints will always be stored.

Whether or not they get replayed in the end depends solely on hint's TTL (but
that same semantics is in place for BL entires, they have the same TTL).

Consider not using hints for batchlog replay, in any capacity
-

Key: CASSANDRA-8368
URL: https://issues.apache.org/jira/browse/CASSANDRA-8368
Project: Cassandra
Issue Type: Bug
Reporter: Aleksey Yeschenko
Fix For: 3.0.0 rc1

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9890) Bytecode inspection for Java-UDFs


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649559#comment-14649559
 ] 

T Jake Luciani commented on CASSANDRA-9890:
---

+1 very nice!

 Bytecode inspection for Java-UDFs
 -

 Key: CASSANDRA-9890
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9890
 Project: Cassandra
  Issue Type: Improvement
Reporter: Robert Stupp
Assignee: Robert Stupp
 Fix For: 3.0.0 rc1


 (Follow-up to CASSANDRA-9402)
 For Java-UDFs we could inspect the compiled Java byte code to find usages of 
 the Java language that are forbidden to UDFs.
 These include usages of:
 * {{synchronized}} keyword
 * call to {{j.l.Object.wait}}
 * call to {{j.l.Object.notify}}
 * call to {{j.l.Object.notifyAll}}
 * call to {{j.l.Object.getClass}}
 * calls to specific methods of currently allowed classes in the driver (but 
 would need some investigation)
 By inspecting the byte code _before_ the class is actually used, even dirty 
 constructs like the following would be impossible:
 {noformat}
 CREATE OR REPLACE FUNCTION ... AS $$  return Math.sin(val);
 }
 {
   // anonymous initializer code
 }
 static {
   // static initializer code
 $$;
 {noformat}
 (inspired by [this blog 
 post|http://jordan-wright.com/blog/2015/03/08/elasticsearch-rce-vulnerability-cve-2015-1427/])



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[Cassandra Wiki] Update of Windows Development by PauloMotta

2015-07-31 Thread Apache Wiki

Dear Wiki user,

You have subscribed to a wiki page or wiki category on Cassandra Wiki for 
change notification.

The Windows Development page has been changed by PauloMotta:
https://wiki.apache.org/cassandra/Windows%20Development

New page:
== Setting Up Environment ==

Open !PowerShell as administrator (“run as administrator” on right-click), and 
peform the steps below:
 
  * Set the execution policy as remotesigned in order to install chocolatey:
* `set-executionpolicy remotesigned`
  * Install chocolatey package manager
* Install instructions available [[https://chocolatey.org/|here]]
  * Install the following packages with “chocolatey install packageName”
* Mandatory packages (running, compiling and installing)
  * jdk7 and/or jdk8 (depending on C* version to develop)
  * ant
  * git
  * python2
  * easy.install
  * pip
* Optional packages (productivity)
  * gnuwin
  * poshgit

== Running cassandra-dtests ==

  * Set $CASSANDRA_DIR environment variable (the environment variable will only 
be exported during next terminal session)
* `[Environment]::SetEnvironmentVariable(CASSANDRA_DIR, 
c:\Users\username\cassandra_dir, User)`
  * Download and install Microsoft Visual C++ Compiler for Python 2.7
* Available on 
[[https://www.microsoft.com/en-us/download/confirmation.aspx?id=44266|this 
link]]
  * Instal necessary python modules
* `pip install blist 
git+https://github.com/datastax/python-driver.git@cassandra-test 
git+https://github.com/pcmanus/ccm.git cql decorator enum34 flaky futures nose 
nose-test-select pycassa six thrift psutil`
* Check 
https://github.com/riptano/cassandra-dtest/blob/master/requirements.txt for new 
requirements

== Running CCM ==

   * You need to set the execution policy to unrestricted to run CCM on 
!PowerShell as administrator:
* `set-executionpolicy unrestricted`
   * Add .PY extension to environment variable $PATHEXT, to allow ccm to be 
executed from any location (run on !PowerShell as administrator):
* `[Environment]::SetEnvironmentVariable(PATHEXT, $env:PATHEXT;.PY, 
MACHINE)`
   * Create a symbolinc link (as administrator) to make chocolatey ant work 
with ccm, since ccm expects an ant.bat executable, while chocolatey creates and 
ant.exe file:
* `cmd /c mklink C:\ProgramData\chocolatey\bin\ant.bat 
C:\ProgramData\chocolatey\bin\ant.exe`

== Script for switching JDKs ==

In a mixed JDK7/JDK8 the following PowerShell functions might be useful for 
switching jdks (place them on 
`$ENV:HOME\Documents\WindowsPowerShell\profile.ps1`):

{{{
function updateJavaHome($newHome)
{
$env:JAVA_HOME=$newHome
[Environment]::SetEnvironmentVariable(JAVA_HOME, $env:JAVA_HOME, 
Machine)
}

function updateRegistry($newVersion)
{
Set-ItemProperty -Path 'Registry::HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java 
Runtime Environment\' CurrentVersion $newVersion
}

function updatePath
{
$PathWithoutJava=(($Env:Path).Split(;) | where {$_ -notmatch 'java'}) 
-join ;
$NewPath=$env:JAVA_HOME\bin;$pathWithoutJava
$Env:Path=$NewPath
[Environment]::SetEnvironmentVariable(PATH,$NewPath,Machine);
}

function java8
{
updateJavaHome(C:\Program Files\Java\jdk1.8.0_51)
updateRegistry(1.8)
updatePath
}

function java7
{
updateJavaHome(C:\Program Files\Java\jdk1.7.0_76)
updateRegistry(1.7)
updatePath
}
}}}

[jira] [Created] (CASSANDRA-9949) maxPurgeableTimestamp needs to check memtables too

Jonathan Ellis created CASSANDRA-9949:
-

 Summary: maxPurgeableTimestamp needs to check memtables too
 Key: CASSANDRA-9949
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9949
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Jonathan Ellis
 Fix For: 2.1.x, 2.2.x


overlapIterator/maxPurgeableTimestamp don't include the memtables, so a 
very-out-of-order write could be ignored



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9923) stress against counters hangs

2015-07-31 Thread Robert Stupp (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649861#comment-14649861
 ] 

Robert Stupp commented on CASSANDRA-9923:
-

The same behavior on 
[2.1.0|http://cstar.datastax.com/tests/id/245c62fe-37c3-11e5-aeea-42010af0688f] 
and 
[2.0.11|http://cstar.datastax.com/tests/id/bfe0ca32-37c6-11e5-a2bf-42010af0688f]
 after approx 15 minutes ({{counter_write n=10M -pop dist=gaussian\(1..5M\) 
-rate threads=300 limit=5000/s}}).

 stress against counters hangs
 -

 Key: CASSANDRA-9923
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9923
 Project: Cassandra
  Issue Type: Bug
Reporter: Robert Stupp

 (Sorry for the vague description)
 I tried some cstar tests against counter columns. But all these tests against 
 2.1 and 2.2 ended (hang) during with the following output:
 {noformat}
 Created keyspaces. Sleeping 3s for propagation.
 Sleeping 2s...
 Warming up COUNTER_WRITE with 15 iterations...
 Running COUNTER_WRITE with 300 threads for 1500 iteration
 type,  total ops,op/s,pk/s,   row/s,mean, med, .95,   
   .99,.999, max,   time,   stderr, errors,  gc: #,  max ms,  sum ms,  
 sdv ms,  mb
 total, 98073,   98054,   98054,   98054, 3.1, 1.7, 8.9,   
  23.2,89.9,   107.7,1.0,  0.0,  0,  0,   0,   0,  
  0,   0
 total,188586,   72492,   72492,   72492, 4.1, 1.5,10.0,   
  61.4,   202.8,   214.7,2.2,  0.13101,  0,  3, 564, 564,  
  6,3447
 total,363880,  137986,  137986,  137986, 2.2, 1.4, 4.1,   
   9.6,   207.1,   253.3,3.5,  0.18684,  0,  0,   0,   0,  
  0,   0
 total,460122,  105062,  105062,  105062, 2.8, 1.4, 4.6,   
  14.7,   225.6,   236.2,4.4,  0.13969,  0,  1, 214, 214,  
  0,1199
 total,600625,  111453,  111453,  111453, 2.7, 1.4, 3.8,   
  10.4,   231.5,   241.6,5.7,  0.11366,  0,  2, 442, 442,  
  1,2389
 total,745680,  149583,  149583,  149583, 2.0, 1.4, 3.6,   
   6.7,   155.8,   159.7,6.7,  0.11318,  0,  0,   0,   0,  
  0,   0
 total,828453,   63632,   63632,   63632, 4.7, 1.4, 4.8,   
 261.9,   274.5,   279.3,8.0,  0.12645,  0,  3, 782, 782,  
  1,3542
 total,   1009560,  172429,  172429,  172429, 1.7, 1.4, 3.7,   
   6.1,16.2,29.7,9.0,  0.11629,  0,  0,   0,   0,  
  0,   0
 total,   1062409,   53860,   53860,   53860, 5.5, 1.3, 8.6,   
 270.3,   293.4,   324.3,   10.0,  0.12738,  0,  2, 542, 542,  
  7,2354
 total,   1186672,   96540,   96540,   96540, 3.1, 1.5, 5.9,   
  14.5,   266.4,   277.6,   11.3,  0.11451,  0,  1, 260, 260,  
  0,1183
 {noformat}
 ...
 {noformat}
 total,   4977251, 238, 238, 238, 0.7, 0.6, 0.7,   
   1.3, 3.4,   158.5,  352.3,  0.11749,  0,  0,   0,   0,  
  0,   0
 total,   4979839, 214, 214, 214, 0.6, 0.6, 0.7,   
   1.3, 2.5, 2.8,  364.4,  0.11761,  0,  0,   0,   0,  
  0,   0
 total,   4981729, 191, 191, 191, 0.6, 0.6, 0.7,   
   1.3, 3.2, 3.3,  374.3,  0.11774,  0,  0,   0,   0,  
  0,   0
 total,   4983362, 167, 167, 167, 0.8, 0.7, 1.8,   
   2.7, 3.9, 5.8,  384.0,  0.11787,  0,  0,   0,   0,  
  0,   0
 total,   4985171, 153, 153, 153, 0.7, 0.6, 1.2,   
   1.4, 2.0, 3.3,  395.9,  0.11799,  0,  0,   0,   0,  
  0,   0
 total,   4986684, 137, 137, 137, 0.7, 0.6, 0.8,   
   1.3, 2.0, 2.0,  406.9,  0.11812,  0,  0,   0,   0,  
  0,   0
 total,   4988410, 121, 121, 121, 0.7, 0.7, 0.8,   
   1.3, 2.0, 2.8,  421.1,  0.11824,  0,  0,   0,   0,  
  0,   0
 total,   4990216,  99,  99,  99, 0.7, 0.7, 0.8,   
   1.4, 2.6, 2.8,  439.5,  0.11836,  0,  0,   0,   0,  
  0,   0
 total,   4991765,  81,  81,  81, 0.8, 0.7, 0.8,   
   1.4,30.1,81.6,  458.7,  0.11848,  0,  1, 159, 159,  
  0,1179
 total,   4993731,  67,  67,  67, 0.7, 0.7, 0.8,   
   1.4, 3.2, 3.2,  488.1,  0.11860,  0,  0,   0,   0,  
  0,   0
 total,   4996565,  45,  45,  45,

[jira] [Updated] (CASSANDRA-9754) Make index info heap friendly for large CQL partitions

2015-07-31 Thread sankalp kohli (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

sankalp kohli updated CASSANDRA-9754:
-
Assignee: Michael Kjellman

 Make index info heap friendly for large CQL partitions
 --

 Key: CASSANDRA-9754
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9754
 Project: Cassandra
  Issue Type: Improvement
Reporter: sankalp kohli
Assignee: Michael Kjellman
Priority: Minor

  Looking at a heap dump of 2.0 cluster, I found that majority of the objects 
 are IndexInfo and its ByteBuffers. This is specially bad in endpoints with 
 large CQL partitions. If a CQL partition is say 6,4GB, it will have 100K 
 IndexInfo objects and 200K ByteBuffers. This will create a lot of churn for 
 GC. Can this be improved by not creating so many objects?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9945) Add transparent data encryption core classes


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649918#comment-14649918
 ] 

Jason Brown commented on CASSANDRA-9945:


bq. Using a Cipher-per-thread (ThreadLocal) ...

Well, your jmh numbers are rather convincing ;) We can try that, but there's a 
couple of wrinkles: for encrypting (writing), we do *not* want to reuse an IVs 
across files, in which case we still need to call {{Cipher#init}}, which is 
where I suspect the cost is with new Cipher instances. On the decryprt 
(reading) side, we would have to make sure the alg/key/iv combo from the 
parameters matches with that in the thread local, so it might need some kind of 
a thread local map (not sure). I can look into this further...

bq. we could provide bouncycastle JCE 

True, although the performance was terrible compared to the JDK's encryption 
implementation. In my tests, it was sometimes 10x slower on the read path. 
IIRC, there was still some magic in registering BC as the encryption provider 
(security SPI-type thinng), which you can either do in code or through messing 
with the JVM, but then you're in the same boat as the current patch. I'm not 
against supporting BC, per se, but I'd prefer to pass until somebody actually 
requests it.

bq. Can you elaborate a bit how the {{Key cache is to be used}} in 
{{EncryptionContext}}?

Do you mean {{CipherFactory}} instead of {{EncryptionContext}}? If so, the goal 
is to avoid always needing to reading from the keystore when a key is 
requested, then keep it memoized. THis alos expects that you might have 
multiple keys (because you migrated keys and have files encrypted with the 
previous key(s) that still need to be read).

bq. Will there be any way to change the encryption keys e.g. via ”nodetool 
refresh_tde_keystore“?

I hadn't envisioned a nodetool, command necessarily, because that would 
necessitate the new key already being deployed to keystore on the machine - and 
since you need a bounce of the process to recognize the updated keystore (and 
the update key_alias in the yaml), I figured just wait until a bounce to 
actaully switch to a new key. This might be less trouble for an operator than 
invoking a nodetool command, since swapping keys is probably a rather unique 
occurance.

bq. A unit test covering especially the {{getEncryptor}}+{{getDecryptor}} 
methods in {{EncryptionContext}} would be great.

Once again, I think you mean {{CipherFactory}} here. Either way, this patch was 
previously part of the (soon-to-be-ready-for-review) patch for CASSANDRA-6018. 
In that one, you will find more than enough round trip encrypt/decrypt 
operations ;). Either way, I can add a simple one to test that flow.

bq. I assume SecureRandom is not thread-safe

It is :), or at least, {{SecureRandom#nextBytes}} is synchronized. As creating 
new encrypting Ciphers should be relatively uncontended in cassandra's use 
case, I think this is fine.

bq. DatabaseDescriptor: typo

D'oh! well, that's a bit emabrassing. thanks

bq. the check if (tdeOptions.enabled)

In {{EncryptionContext}}, you are correct. Fixed.

bq. Unused method {{EncryptionContext#toHeaderParameters}}

Sorry, meant to commit that on CASSANDRA-6018. can remove for this commit, as 
well as {{EncryptionContext.createFromMap}}

bq. the assertion (or the message) is wrong

improved the assert logic

Will post an updated branch soon

 Add transparent data encryption core classes
 

 Key: CASSANDRA-9945
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9945
 Project: Cassandra
  Issue Type: Improvement
Reporter: Jason Brown
Assignee: Jason Brown
  Labels: encryption
 Fix For: 3.0 beta 1


 This patch will add the core infrastructure classes necessary for transparent 
 data encryption (file-level encryption), as required for CASSANDRA-6018 and 
 CASSANDRA-9633.  The phrase transparent data encryption, while not the most 
 aesthetically pleasing, seems to be used throughout the database industry 
 (Oracle, SQLQServer, Datastax Enterprise) to describe file level encryption, 
 so we'll go with that, as well. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8143) Partitioner should not be accessed through StorageService


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649955#comment-14649955
 ] 

Aleksey Yeschenko commented on CASSANDRA-8143:
--

Committed to trunk as {{0a08525ad236f78df05c854dead62f300eae271d}} with the fix 
mended with the revert-revert commit.

 Partitioner should not be accessed through StorageService
 -

 Key: CASSANDRA-8143
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8143
 Project: Cassandra
  Issue Type: Improvement
Reporter: Branimir Lambov
Assignee: Branimir Lambov
 Fix For: 3.0 beta 1


 The configured partitioner is no longer the only partitioner in use in the 
 database, as e.g. index tables use LocalPartitioner.
 To make sure the correct partitioner is used for each table, accesses of 
 StorageService.getPartitioner() should be replaced with retrieval of the 
 CFS-specific partitioner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-8143) Partitioner should not be accessed through StorageService


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko resolved CASSANDRA-8143.
--
Resolution: Fixed

 Partitioner should not be accessed through StorageService
 -

 Key: CASSANDRA-8143
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8143
 Project: Cassandra
  Issue Type: Improvement
Reporter: Branimir Lambov
Assignee: Branimir Lambov
 Fix For: 3.0 beta 1


 The configured partitioner is no longer the only partitioner in use in the 
 database, as e.g. index tables use LocalPartitioner.
 To make sure the correct partitioner is used for each table, accesses of 
 StorageService.getPartitioner() should be replaced with retrieval of the 
 CFS-specific partitioner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9606) this query is not supported in new version

2015-07-31 Thread zhaoyan (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648872#comment-14648872
 ] 

zhaoyan commented on CASSANDRA-9606:


hello~  what's the Fix Version?

 this query is not supported in new version
 --

 Key: CASSANDRA-9606
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9606
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment: cassandra 2.1.6
 jdk 1.7.0_55
Reporter: zhaoyan
Assignee: Benjamin Lerer
 Attachments: 9606-2.0.txt, 9606-2.1.txt, 9606-2.2.txt


 Background:
 1、create a table:
 {code}
 CREATE TABLE test (
 a int,
 b int,
 c int,
   d int,
 PRIMARY KEY (a, b, c)
 );
 {code}
 2、query by a=1 and b6
 {code}
 select * from test where a=1 and b6;
  a | b | c | d
 ---+---+---+---
  1 | 3 | 1 | 2
  1 | 3 | 2 | 2
  1 | 3 | 4 | 2
  1 | 3 | 5 | 2
  1 | 4 | 4 | 2
  1 | 5 | 5 | 2
 (6 rows)
 {code}
 3、query by page
 first page：
 {code}
 select * from test where a=1 and b6 limit 2;
  a | b | c | d
 ---+---+---+---
  1 | 3 | 1 | 2
  1 | 3 | 2 | 2
 (2 rows)
 {code}
 second page：
 {code}
 select * from test where a=1 and b6 and (b,c)  (3,2) limit 2;
  a | b | c | d
 ---+---+---+---
  1 | 3 | 4 | 2
  1 | 3 | 5 | 2
 (2 rows)
 {code}
 last page：
 {code}
 select * from test where a=1 and b6 and (b,c)  (3,5) limit 2;
  a | b | c | d
 ---+---+---+---
  1 | 4 | 4 | 2
  1 | 5 | 5 | 2
 (2 rows)
 {code}
 question:
 this query by page is ok when cassandra 2.0.8.
 but is not supported in the latest version 2.1.6
 when execute：
 {code}
 select * from test where a=1 and b6 and (b,c)  (3,2) limit 2;
 {code}
 get one error message：
 InvalidRequest: code=2200 [Invalid query] message=Column b cannot have 
 both tuple-notation inequalities and single-column inequalities: (b, c)  (3, 
 2)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9606) this query is not supported in new version

2015-07-31 Thread Sam Tunnicliffe (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648884#comment-14648884
 ] 

Sam Tunnicliffe commented on CASSANDRA-9606:


Sorry, this is still waiting for review. I'll try and get to it in the next few 
days so it will probably make it into the next release on each target branch 
(2.0.17, 2.1.9, 2.2.1).

 this query is not supported in new version
 --

 Key: CASSANDRA-9606
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9606
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment: cassandra 2.1.6
 jdk 1.7.0_55
Reporter: zhaoyan
Assignee: Benjamin Lerer
 Attachments: 9606-2.0.txt, 9606-2.1.txt, 9606-2.2.txt


 Background:
 1、create a table:
 {code}
 CREATE TABLE test (
 a int,
 b int,
 c int,
   d int,
 PRIMARY KEY (a, b, c)
 );
 {code}
 2、query by a=1 and b6
 {code}
 select * from test where a=1 and b6;
  a | b | c | d
 ---+---+---+---
  1 | 3 | 1 | 2
  1 | 3 | 2 | 2
  1 | 3 | 4 | 2
  1 | 3 | 5 | 2
  1 | 4 | 4 | 2
  1 | 5 | 5 | 2
 (6 rows)
 {code}
 3、query by page
 first page：
 {code}
 select * from test where a=1 and b6 limit 2;
  a | b | c | d
 ---+---+---+---
  1 | 3 | 1 | 2
  1 | 3 | 2 | 2
 (2 rows)
 {code}
 second page：
 {code}
 select * from test where a=1 and b6 and (b,c)  (3,2) limit 2;
  a | b | c | d
 ---+---+---+---
  1 | 3 | 4 | 2
  1 | 3 | 5 | 2
 (2 rows)
 {code}
 last page：
 {code}
 select * from test where a=1 and b6 and (b,c)  (3,5) limit 2;
  a | b | c | d
 ---+---+---+---
  1 | 4 | 4 | 2
  1 | 5 | 5 | 2
 (2 rows)
 {code}
 question:
 this query by page is ok when cassandra 2.0.8.
 but is not supported in the latest version 2.1.6
 when execute：
 {code}
 select * from test where a=1 and b6 and (b,c)  (3,2) limit 2;
 {code}
 get one error message：
 InvalidRequest: code=2200 [Invalid query] message=Column b cannot have 
 both tuple-notation inequalities and single-column inequalities: (b, c)  (3, 
 2)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/4] cassandra git commit: Stop accessing the partitioner directly via StorageService

Repository: cassandra
Updated Branches:
  refs/heads/trunk df3b6027b - 69f77cbdd


http://git-wip-us.apache.org/repos/asf/cassandra/blob/69f77cbd/test/unit/org/apache/cassandra/db/ScrubTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/ScrubTest.java 
b/test/unit/org/apache/cassandra/db/ScrubTest.java
index ee51a4d..25b9cde 100644
--- a/test/unit/org/apache/cassandra/db/ScrubTest.java
+++ b/test/unit/org/apache/cassandra/db/ScrubTest.java
@@ -22,8 +22,6 @@ import java.io.File;
 import java.io.IOError;
 import java.io.IOException;
 import java.io.RandomAccessFile;
-import java.lang.reflect.Field;
-import java.lang.reflect.Modifier;
 import java.nio.ByteBuffer;
 import java.util.*;
 import java.util.concurrent.ExecutionException;
@@ -44,9 +42,9 @@ import org.apache.cassandra.cql3.UntypedResultSet;
 import org.apache.cassandra.db.compaction.CompactionManager;
 import org.apache.cassandra.db.compaction.OperationType;
 import org.apache.cassandra.db.compaction.Scrubber;
-import org.apache.cassandra.db.index.SecondaryIndex;
 import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
 import org.apache.cassandra.db.marshal.*;
+import org.apache.cassandra.db.marshal.UUIDType;
 import org.apache.cassandra.db.partitions.Partition;
 import org.apache.cassandra.db.partitions.PartitionUpdate;
 import org.apache.cassandra.dht.ByteOrderedPartitioner;
@@ -55,14 +53,10 @@ import 
org.apache.cassandra.exceptions.ConfigurationException;
 import org.apache.cassandra.exceptions.RequestExecutionException;
 import org.apache.cassandra.exceptions.WriteTimeoutException;
 import org.apache.cassandra.io.compress.CompressionMetadata;
-import org.apache.cassandra.io.sstable.Component;
-import org.apache.cassandra.io.sstable.Descriptor;
-import org.apache.cassandra.io.sstable.SSTableRewriter;
-import org.apache.cassandra.io.sstable.SSTableTxnWriter;
+import org.apache.cassandra.io.sstable.*;
 import org.apache.cassandra.io.sstable.format.SSTableReader;
 import org.apache.cassandra.io.util.FileUtils;
 import org.apache.cassandra.schema.KeyspaceParams;
-import org.apache.cassandra.service.StorageService;
 import org.apache.cassandra.utils.ByteBufferUtil;
 
 import static org.junit.Assert.*;
@@ -79,6 +73,8 @@ public class ScrubTest
 public static final String CF_UUID = UUIDKeys;
 public static final String CF_INDEX1 = Indexed1;
 public static final String CF_INDEX2 = Indexed2;
+public static final String CF_INDEX1_BYTEORDERED = Indexed1_ordered;
+public static final String CF_INDEX2_BYTEORDERED = Indexed2_ordered;
 
 public static final String COL_INDEX = birthdate;
 public static final String COL_NON_INDEX = notbirthdate;
@@ -98,7 +94,9 @@ public class ScrubTest
 
.compressionParameters(SchemaLoader.getCompressionParameters(COMPRESSION_CHUNK_LENGTH)),
 SchemaLoader.standardCFMD(KEYSPACE, 
CF_UUID, 0, UUIDType.instance),
 SchemaLoader.keysIndexCFMD(KEYSPACE, 
CF_INDEX1, true),
-SchemaLoader.compositeIndexCFMD(KEYSPACE, 
CF_INDEX2, true));
+SchemaLoader.compositeIndexCFMD(KEYSPACE, 
CF_INDEX2, true),
+SchemaLoader.keysIndexCFMD(KEYSPACE, 
CF_INDEX1_BYTEORDERED, true).copy(ByteOrderedPartitioner.instance),
+SchemaLoader.compositeIndexCFMD(KEYSPACE, 
CF_INDEX2_BYTEORDERED, true).copy(ByteOrderedPartitioner.instance));
 }
 
 @Test
@@ -306,7 +304,7 @@ public class ScrubTest
 {
 // This test assumes ByteOrderPartitioner to create out-of-order 
SSTable
 IPartitioner oldPartitioner = DatabaseDescriptor.getPartitioner();
-DatabaseDescriptor.setPartitioner(new ByteOrderedPartitioner());
+DatabaseDescriptor.setPartitionerUnsafe(new ByteOrderedPartitioner());
 
 // Create out-of-order SSTable
 File tempDir = File.createTempFile(ScrubTest.testScrubOutOfOrder, 
).getParentFile();
@@ -380,7 +378,7 @@ public class ScrubTest
 {
 FileUtils.deleteRecursive(tempDataDir);
 // reset partitioner
-DatabaseDescriptor.setPartitioner(oldPartitioner);
+DatabaseDescriptor.setPartitionerUnsafe(oldPartitioner);
 }
 }
 
@@ -394,9 +392,9 @@ public class ScrubTest
 CompressionMetadata compData = 
CompressionMetadata.create(sstable.getFilename());
 
 CompressionMetadata.Chunk chunk1 = compData.chunkFor(
-sstable.getPosition(PartitionPosition.ForKey.get(key1, 
sstable.partitioner), SSTableReader.Operator.EQ).position);
+sstable.getPosition(PartitionPosition.ForKey.get(key1, 
sstable.getPartitioner()), SSTableReader.Operator.EQ).position);
 CompressionMetadata.Chunk chunk2 = compData.chunkFor(
-

[4/4] cassandra git commit: Stop accessing the partitioner directly via StorageService

Stop accessing the partitioner directly via StorageService

Changes the way the partitioner is accessed. Partitioner is now stored
and accessed according to the reason for needing it:
* if it is to access a table / prepare a statement, the partitioner
relevant to this table can be found in its CFMetaData.
* if it is to route messages to the relevant member of the cluster, the
partitioner in TokenMetadata is to be used.

patch by Branimir Lambov; reviewed by Aleksey Yeschenko for
CASSANDRA-8143


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/69f77cbd
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/69f77cbd
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/69f77cbd

Branch: refs/heads/trunk
Commit: 69f77cbddd4c74448f227e9aceef84d345118184
Parents: df3b602
Author: Branimir Lambov branimir.lam...@datastax.com
Authored: Fri Jul 3 14:38:40 2015 +0100
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Fri Jul 31 14:42:47 2015 +0300

--
 .../org/apache/cassandra/config/CFMetaData.java |  64 +++-
 .../cassandra/config/DatabaseDescriptor.java|   7 +-
 .../org/apache/cassandra/config/Schema.java |   2 +-
 .../apache/cassandra/cql3/TokenRelation.java|   7 +-
 .../cassandra/cql3/functions/TokenFct.java  |  10 +-
 .../restrictions/StatementRestrictions.java |   2 +-
 .../cql3/restrictions/TokenFilter.java  |  14 +-
 .../cql3/restrictions/TokenRestriction.java |  23 +--
 .../cql3/statements/BatchStatement.java |   5 +-
 .../cql3/statements/ModificationStatement.java  |   7 +-
 .../cql3/statements/SelectStatement.java|   2 +-
 .../db/AbstractReadCommandBuilder.java  |  11 +-
 .../apache/cassandra/db/BatchlogManager.java|   7 +-
 .../apache/cassandra/db/ColumnFamilyStore.java  |  35 ++--
 src/java/org/apache/cassandra/db/DataRange.java |   3 +-
 .../cassandra/db/HintedHandOffManager.java  |  29 ++--
 src/java/org/apache/cassandra/db/Memtable.java  |   1 -
 src/java/org/apache/cassandra/db/Mutation.java  |  26 ++-
 .../apache/cassandra/db/PartitionPosition.java  |   2 +-
 .../cassandra/db/PartitionRangeReadCommand.java |   2 +-
 .../apache/cassandra/db/RowUpdateBuilder.java   |   2 +-
 .../db/SinglePartitionNamesCommand.java |  12 ++
 .../db/SinglePartitionReadCommand.java  |  18 ++-
 .../db/SinglePartitionSliceCommand.java |  17 ++
 .../org/apache/cassandra/db/SystemKeyspace.java |  18 +--
 .../db/compaction/CompactionManager.java|   4 +-
 .../db/compaction/LeveledManifest.java  |   2 +-
 .../cassandra/db/compaction/Scrubber.java   |   4 +-
 .../cassandra/db/compaction/Upgrader.java   |   1 -
 .../cassandra/db/compaction/Verifier.java   |   2 +-
 .../writers/DefaultCompactionWriter.java|   1 -
 .../writers/MajorLeveledCompactionWriter.java   |   2 -
 .../writers/MaxSSTableSizeWriter.java   |   2 -
 .../SplittingSizeTieredCompactionWriter.java|   2 -
 .../AbstractSimplePerColumnSecondaryIndex.java  |   4 +-
 .../cassandra/db/index/SecondaryIndex.java  |  20 +--
 .../db/index/composites/CompositesIndex.java|   2 +-
 .../CompositesIndexOnClusteringKey.java |   3 +-
 .../db/index/composites/CompositesSearcher.java |   2 +-
 .../cassandra/db/index/keys/KeysIndex.java  |   3 +-
 .../cassandra/db/index/keys/KeysSearcher.java   |   2 +-
 .../db/marshal/LocalByPartionerType.java|  97 
 .../db/marshal/PartitionerDefinedOrder.java |  91 +++
 .../db/partitions/AtomicBTreePartition.java |   4 +-
 .../db/partitions/PartitionUpdate.java  | 108 ++---
 .../rows/UnfilteredRowIteratorSerializer.java   |   3 +-
 .../cassandra/db/view/MaterializedView.java |   7 +-
 .../apache/cassandra/db/view/TemporalRow.java   |   2 +-
 .../org/apache/cassandra/dht/BootStrapper.java  |  12 +-
 .../cassandra/dht/ByteOrderedPartitioner.java   |   5 +
 .../org/apache/cassandra/dht/IPartitioner.java  |   6 +
 .../apache/cassandra/dht/LocalPartitioner.java  |   5 +
 .../cassandra/dht/Murmur3Partitioner.java   |   7 +
 .../dht/OrderPreservingPartitioner.java |   5 +
 .../apache/cassandra/dht/RandomPartitioner.java |   7 +
 .../org/apache/cassandra/dht/RangeStreamer.java |   2 +-
 .../dht/tokenallocator/TokenAllocation.java |   8 +-
 src/java/org/apache/cassandra/gms/Gossiper.java |   2 +-
 .../io/sstable/AbstractSSTableSimpleWriter.java |  10 +-
 .../cassandra/io/sstable/CQLSSTableWriter.java  |  15 +-
 .../cassandra/io/sstable/KeyIterator.java   |   8 +-
 .../io/sstable/ReducingKeyIterator.java |   2 +-
 .../apache/cassandra/io/sstable/SSTable.java|  21 ++-
 .../cassandra/io/sstable/SSTableLoader.java |  20 +--
 .../io/sstable/SSTableSimpleUnsortedWriter.java |   5 +-
 .../io/sstable/SSTableSimpleWriter.java |   4 +-

[3/4] cassandra git commit: Stop accessing the partitioner directly via StorageService

http://git-wip-us.apache.org/repos/asf/cassandra/blob/69f77cbd/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
--
diff --git a/src/java/org/apache/cassandra/db/index/SecondaryIndex.java 
b/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
index 94031ab..9221090 100644
--- a/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
+++ b/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
@@ -42,15 +42,12 @@ import org.apache.cassandra.db.index.keys.KeysIndex;
 import org.apache.cassandra.db.lifecycle.SSTableSet;
 import org.apache.cassandra.db.lifecycle.View;
 import org.apache.cassandra.db.marshal.AbstractType;
-import org.apache.cassandra.db.marshal.BytesType;
-import org.apache.cassandra.db.marshal.LocalByPartionerType;
+import org.apache.cassandra.dht.LocalPartitioner;
 import org.apache.cassandra.exceptions.ConfigurationException;
 import org.apache.cassandra.io.sstable.format.SSTableReader;
 import org.apache.cassandra.exceptions.InvalidRequestException;
 import org.apache.cassandra.io.sstable.ReducingKeyIterator;
-import org.apache.cassandra.service.StorageService;
 import org.apache.cassandra.utils.FBUtilities;
-
 import org.apache.cassandra.utils.concurrent.Refs;
 
 /**
@@ -79,10 +76,6 @@ public abstract class SecondaryIndex
  */
 public static final String INDEX_ENTRIES_OPTION_NAME = 
index_keys_and_values;
 
-public static final AbstractType? keyComparator = 
StorageService.getPartitioner().preservesOrder()
-  ? BytesType.instance
-  : new 
LocalByPartionerType(StorageService.getPartitioner());
-
 /**
  * Base CF that has many indexes
  */
@@ -303,7 +296,7 @@ public abstract class SecondaryIndex
  */
 public DecoratedKey getIndexKeyFor(ByteBuffer value)
 {
-return getIndexCfs().partitioner.decorateKey(value);
+return getIndexCfs().decorateKey(value);
 }
 
 /**
@@ -381,11 +374,20 @@ public abstract class SecondaryIndex
  */
 public static CFMetaData newIndexMetadata(CFMetaData baseMetadata, 
ColumnDefinition def)
 {
+return newIndexMetadata(baseMetadata, def, def.type);
+}
+
+/**
+ * Create the index metadata for the index on a given column of a given 
table.
+ */
+static CFMetaData newIndexMetadata(CFMetaData baseMetadata, 
ColumnDefinition def, AbstractType? comparator)
+{
 if (def.getIndexType() == IndexType.CUSTOM)
 return null;
 
 CFMetaData.Builder builder = 
CFMetaData.Builder.create(baseMetadata.ksName, 
baseMetadata.indexColumnFamilyName(def))

.withId(baseMetadata.cfId)
+   .withPartitioner(new 
LocalPartitioner(comparator))

.addPartitionKey(def.name, def.type);
 
 if (def.getIndexType() == IndexType.COMPOSITES)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/69f77cbd/src/java/org/apache/cassandra/db/index/composites/CompositesIndex.java
--
diff --git 
a/src/java/org/apache/cassandra/db/index/composites/CompositesIndex.java 
b/src/java/org/apache/cassandra/db/index/composites/CompositesIndex.java
index 42861c5..29f235c 100644
--- a/src/java/org/apache/cassandra/db/index/composites/CompositesIndex.java
+++ b/src/java/org/apache/cassandra/db/index/composites/CompositesIndex.java
@@ -101,7 +101,7 @@ public abstract class CompositesIndex extends 
AbstractSimplePerColumnSecondaryIn
 
 protected static void addGenericClusteringColumns(CFMetaData.Builder 
indexMetadata, CFMetaData baseMetadata, ColumnDefinition columnDef)
 {
-indexMetadata.addClusteringColumn(partition_key, 
SecondaryIndex.keyComparator);
+indexMetadata.addClusteringColumn(partition_key, 
baseMetadata.partitioner.partitionOrdering());
 for (ColumnDefinition def : baseMetadata.clusteringColumns())
 indexMetadata.addClusteringColumn(def.name, def.type);
 }

http://git-wip-us.apache.org/repos/asf/cassandra/blob/69f77cbd/src/java/org/apache/cassandra/db/index/composites/CompositesIndexOnClusteringKey.java
--
diff --git 
a/src/java/org/apache/cassandra/db/index/composites/CompositesIndexOnClusteringKey.java
 
b/src/java/org/apache/cassandra/db/index/composites/CompositesIndexOnClusteringKey.java
index 6529ad9..cd4aff9 100644
--- 
a/src/java/org/apache/cassandra/db/index/composites/CompositesIndexOnClusteringKey.java
+++ 
b/src/java/org/apache/cassandra/db/index/composites/CompositesIndexOnClusteringKey.java
@@ -24,7 +24,6 @@ import org.apache.cassandra.config.CFMetaData;
 import org.apache.cassandra.config.ColumnDefinition;
 import

[2/4] cassandra git commit: Stop accessing the partitioner directly via StorageService

http://git-wip-us.apache.org/repos/asf/cassandra/blob/69f77cbd/src/java/org/apache/cassandra/io/sstable/format/big/BigTableScanner.java
--
diff --git 
a/src/java/org/apache/cassandra/io/sstable/format/big/BigTableScanner.java 
b/src/java/org/apache/cassandra/io/sstable/format/big/BigTableScanner.java
index 0451a98..1b3b407 100644
--- a/src/java/org/apache/cassandra/io/sstable/format/big/BigTableScanner.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/big/BigTableScanner.java
@@ -172,7 +172,7 @@ public class BigTableScanner implements ISSTableScanner
 while (!ifile.isEOF())
 {
 indexPosition = ifile.getFilePointer();
-DecoratedKey indexDecoratedKey = 
sstable.partitioner.decorateKey(ByteBufferUtil.readWithShortLength(ifile));
+DecoratedKey indexDecoratedKey = 
sstable.decorateKey(ByteBufferUtil.readWithShortLength(ifile));
 if (indexDecoratedKey.compareTo(currentRange.left)  0 || 
currentRange.contains(indexDecoratedKey))
 {
 // Found, just read the dataPosition and seek into index 
and data files
@@ -282,7 +282,7 @@ public class BigTableScanner implements ISSTableScanner
 if (ifile.isEOF())
 return endOfData();
 
-currentKey = 
sstable.partitioner.decorateKey(ByteBufferUtil.readWithShortLength(ifile));
+currentKey = 
sstable.decorateKey(ByteBufferUtil.readWithShortLength(ifile));
 currentEntry = 
rowIndexEntrySerializer.deserialize(ifile);
 } while (!currentRange.contains(currentKey));
 }
@@ -301,7 +301,7 @@ public class BigTableScanner implements ISSTableScanner
 else
 {
 // we need the position of the start of the next key, 
regardless of whether it falls in the current range
-nextKey = 
sstable.partitioner.decorateKey(ByteBufferUtil.readWithShortLength(ifile));
+nextKey = 
sstable.decorateKey(ByteBufferUtil.readWithShortLength(ifile));
 nextEntry = rowIndexEntrySerializer.deserialize(ifile);
 
 if (!currentRange.contains(nextKey))

http://git-wip-us.apache.org/repos/asf/cassandra/blob/69f77cbd/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
--
diff --git 
a/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java 
b/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
index ff279a8..5607a7e 100644
--- a/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/big/BigTableWriter.java
@@ -19,7 +19,6 @@ package org.apache.cassandra.io.sstable.format.big;
 
 import java.io.*;
 import java.util.Map;
-import java.util.Set;
 
 import org.apache.cassandra.db.*;
 import org.apache.cassandra.db.lifecycle.LifecycleTransaction;
@@ -32,7 +31,6 @@ import org.slf4j.LoggerFactory;
 import org.apache.cassandra.config.CFMetaData;
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.db.rows.*;
-import org.apache.cassandra.dht.IPartitioner;
 import org.apache.cassandra.io.FSWriteError;
 import org.apache.cassandra.io.compress.CompressedSequentialWriter;
 import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
@@ -62,12 +60,11 @@ public class BigTableWriter extends SSTableWriter
   Long keyCount, 
   Long repairedAt, 
   CFMetaData metadata, 
-  IPartitioner partitioner, 
   MetadataCollector metadataCollector, 
   SerializationHeader header,
   LifecycleTransaction txn)
 {
-super(descriptor, keyCount, repairedAt, metadata, partitioner, 
metadataCollector, header);
+super(descriptor, keyCount, repairedAt, metadata, metadataCollector, 
header);
 txn.trackNew(this); // must track before any files are created
 
 if (compression)
@@ -243,13 +240,12 @@ public class BigTableWriter extends SSTableWriter
 StatsMetadata stats = statsMetadata();
 assert boundary.indexLength  0  boundary.dataLength  0;
 // open the reader early
-IndexSummary indexSummary = iwriter.summary.build(partitioner, 
boundary);
+IndexSummary indexSummary = 
iwriter.summary.build(metadata.partitioner, boundary);
 SegmentedFile ifile = iwriter.builder.buildIndex(descriptor, 
indexSummary, boundary);
 SegmentedFile dfile = dbuilder.buildData(descriptor, stats, boundary);
 SSTableReader sstable = SSTableReader.internalOpen(descriptor,

[jira] [Created] (CASSANDRA-9950) Windows dtest: offline_tools_test.py:sstableverify_test

Paulo Motta created CASSANDRA-9950:
--

 Summary: Windows dtest: offline_tools_test.py:sstableverify_test
 Key: CASSANDRA-9950
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9950
 Project: Cassandra
  Issue Type: Sub-task
Reporter: Paulo Motta
Assignee: Paulo Motta
Priority: Minor
 Fix For: 2.2.x


This test is currently failing on Windows. Will add more details later.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-9950) Windows dtest: offline_tools_test.py:TestOfflineTools.sstableverify_test


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-9950:
---
Summary: Windows dtest: 
offline_tools_test.py:TestOfflineTools.sstableverify_test  (was: Windows dtest: 
offline_tools_test.py:sstableverify_test)

 Windows dtest: offline_tools_test.py:TestOfflineTools.sstableverify_test
 

 Key: CASSANDRA-9950
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9950
 Project: Cassandra
  Issue Type: Sub-task
Reporter: Paulo Motta
Assignee: Paulo Motta
Priority: Minor
  Labels: WIndows
 Fix For: 2.2.x


 This test is currently failing on Windows. Will add more details later.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9942) SStableofflinerevel and sstablelevelreset don't have windows versions


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14650132#comment-14650132
 ] 

Paulo Motta commented on CASSANDRA-9942:


* Added Windows bat executables for sstableofflinerelevel and 
sstabelelevelreset: [2.2 patch 
here|https://github.com/pauloricardomg/cassandra/tree/9942-2.2]
* Fixed related dtests on Windows
** Minor changes on ccm to fix TestOfflineTools dtests on Windows: [ccm 
PR343|https://github.com/pcmanus/ccm/pull/343] | [ccm 
PR344|https://github.com/pcmanus/ccm/pull/344]
** Fixed offline_tools_test.py:TestOfflineTools.sstableofflinerelevel_test, 
which was not testing releveling sstables, but actually reset sstables
** Refactored offline_tools_test.py:TestOfflineTools to create a cluster with 
only one node and write less keys: tests were taking too long with 3 machines, 
RF=3 and millions of rows. Essence of tests was maintained while making them 
faster. [cassandra-dtest 
PR434|https://github.com/riptano/cassandra-dtest/pull/434]
** Tried to also fix TestOfflineTools.sstableverify_test to fit into this 
ticket, but was taking longer than expected, so I created a separate ticket: 
[CASANDRA-9950|https://issues.apache.org/jira/browse/CASSANDRA-9950]

 SStableofflinerevel and sstablelevelreset don't have windows versions
 -

 Key: CASSANDRA-9942
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9942
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
Reporter: Philip Thompson
Assignee: Paulo Motta
 Fix For: 2.2.x


 These two tools located in tools/bin do not have corresponding .bat versions, 
 so they do not run on windows. This is also breaking their related dtests on 
 windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9949) maxPurgeableTimestamp needs to check memtables too


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649971#comment-14649971
 ] 

Aleksey Yeschenko commented on CASSANDRA-9949:
--

Very out of order, or just with ridiculously low gc gs.

This issue was raised by [~benedict] during CASSANDRA-6230 review, originally.

 maxPurgeableTimestamp needs to check memtables too
 --

 Key: CASSANDRA-9949
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9949
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Jonathan Ellis
Assignee: Stefania
 Fix For: 2.1.x, 2.2.x


 overlapIterator/maxPurgeableTimestamp don't include the memtables, so a 
 very-out-of-order write could be ignored



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9945) Add transparent data encryption core classes


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649982#comment-14649982
 ] 

Jason Brown commented on CASSANDRA-9945:


wrt to the thread local, it looks like {{Cipher#init}} is the biggest hog, 
however, there is still much to be gained by not creating a new Cipher instance 
each time. Here's [my 
variation|https://gist.github.com/jasobrown/99daa06776b926bdf987] on [~snazy]'s 
benchmark class; this reflects a little closer the actual use pattern in my 
patch. The biggest difference is I force the Cipher to {{init}} on every call, 
for both thread local and non-thread local.

The results are as follows:

{code}
snazy:
Benchmark  (cipher)   Mode  CntScore
Error   Units
CipherBench.cipherGetInstance  AES/CBC/PKCS5Padding  thrpt50.636 ±  
0.031  ops/us
CipherBench.cipherThreadLocal  AES/CBC/PKCS5Padding  thrpt5  594.063 ± 
58.416  ops/us
CipherBench.cipherGetInstance  AES/CBC/PKCS5Padding   avgt54.873 ±  
0.211   us/op
CipherBench.cipherThreadLocal  AES/CBC/PKCS5Padding   avgt50.005 ±  
0.001   us/op

jasobrown:
Benchmark  (cipher)   Mode  Cnt   Score   Error 
  Units
CipherBench.cipherGetInstance  AES/CBC/PKCS5Padding  thrpt5   0.402 ± 0.133 
 ops/us
CipherBench.cipherThreadLocal  AES/CBC/PKCS5Padding  thrpt5  27.989 ± 7.341 
 ops/us
CipherBench.cipherGetInstance  AES/CBC/PKCS5Padding   avgt5   8.527 ± 1.828 
  us/op
CipherBench.cipherThreadLocal  AES/CBC/PKCS5Padding   avgt5   0.093 ± 0.001 
  us/op
{code}

Thus, I think there's still benefit to using the thread local, to avoid the 
{{Cipher#getInstance}} call, and if we check the cipher, alg, iv, and so on, 
match the input params (when we fish out the Cipher instance form the thread 
local), then we can avoid the {{Cipher#init}} call, as well.

 Add transparent data encryption core classes
 

 Key: CASSANDRA-9945
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9945
 Project: Cassandra
  Issue Type: Improvement
Reporter: Jason Brown
Assignee: Jason Brown
  Labels: encryption
 Fix For: 3.0 beta 1


 This patch will add the core infrastructure classes necessary for transparent 
 data encryption (file-level encryption), as required for CASSANDRA-6018 and 
 CASSANDRA-9633.  The phrase transparent data encryption, while not the most 
 aesthetically pleasing, seems to be used throughout the database industry 
 (Oracle, SQLQServer, Datastax Enterprise) to describe file level encryption, 
 so we'll go with that, as well. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-9945) Add transparent data encryption core classes


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649982#comment-14649982
 ] 

Jason Brown edited comment on CASSANDRA-9945 at 7/31/15 11:14 PM:
--

wrt to the thread local, it looks like {{Cipher#init}} is the biggest hog, 
however, there is still much to be gained by not creating a new Cipher instance 
each time. Here's [my 
variation|https://gist.github.com/jasobrown/99daa06776b926bdf987] on [~snazy]'s 
benchmark class; this reflects a little closer the actual use pattern in my 
patch. The biggest difference is I force the Cipher to {{init}} on every call, 
for both thread local and non-thread local.

The results are as follows (run on my laptop, in a VM):

{code}
snazy:
Benchmark  (cipher)   Mode  CntScore
Error   Units
CipherBench.cipherGetInstance  AES/CBC/PKCS5Padding  thrpt50.636 ±  
0.031  ops/us
CipherBench.cipherThreadLocal  AES/CBC/PKCS5Padding  thrpt5  594.063 ± 
58.416  ops/us
CipherBench.cipherGetInstance  AES/CBC/PKCS5Padding   avgt54.873 ±  
0.211   us/op
CipherBench.cipherThreadLocal  AES/CBC/PKCS5Padding   avgt50.005 ±  
0.001   us/op

jasobrown:
Benchmark  (cipher)   Mode  Cnt   Score   Error 
  Units
CipherBench.cipherGetInstance  AES/CBC/PKCS5Padding  thrpt5   0.402 ± 0.133 
 ops/us
CipherBench.cipherThreadLocal  AES/CBC/PKCS5Padding  thrpt5  27.989 ± 7.341 
 ops/us
CipherBench.cipherGetInstance  AES/CBC/PKCS5Padding   avgt5   8.527 ± 1.828 
  us/op
CipherBench.cipherThreadLocal  AES/CBC/PKCS5Padding   avgt5   0.093 ± 0.001 
  us/op
{code}

Thus, I think there's still benefit to using the thread local, to avoid the 
{{Cipher#getInstance}} call, and if we check the cipher, alg, iv, and so on, 
match the input params (when we fish out the Cipher instance form the thread 
local), then we can avoid the {{Cipher#init}} call, as well.


was (Author: jasobrown):
wrt to the thread local, it looks like {{Cipher#init}} is the biggest hog, 
however, there is still much to be gained by not creating a new Cipher instance 
each time. Here's [my 
variation|https://gist.github.com/jasobrown/99daa06776b926bdf987] on [~snazy]'s 
benchmark class; this reflects a little closer the actual use pattern in my 
patch. The biggest difference is I force the Cipher to {{init}} on every call, 
for both thread local and non-thread local.

The results are as follows:

{code}
snazy:
Benchmark  (cipher)   Mode  CntScore
Error   Units
CipherBench.cipherGetInstance  AES/CBC/PKCS5Padding  thrpt50.636 ±  
0.031  ops/us
CipherBench.cipherThreadLocal  AES/CBC/PKCS5Padding  thrpt5  594.063 ± 
58.416  ops/us
CipherBench.cipherGetInstance  AES/CBC/PKCS5Padding   avgt54.873 ±  
0.211   us/op
CipherBench.cipherThreadLocal  AES/CBC/PKCS5Padding   avgt50.005 ±  
0.001   us/op

jasobrown:
Benchmark  (cipher)   Mode  Cnt   Score   Error 
  Units
CipherBench.cipherGetInstance  AES/CBC/PKCS5Padding  thrpt5   0.402 ± 0.133 
 ops/us
CipherBench.cipherThreadLocal  AES/CBC/PKCS5Padding  thrpt5  27.989 ± 7.341 
 ops/us
CipherBench.cipherGetInstance  AES/CBC/PKCS5Padding   avgt5   8.527 ± 1.828 
  us/op
CipherBench.cipherThreadLocal  AES/CBC/PKCS5Padding   avgt5   0.093 ± 0.001 
  us/op
{code}

Thus, I think there's still benefit to using the thread local, to avoid the 
{{Cipher#getInstance}} call, and if we check the cipher, alg, iv, and so on, 
match the input params (when we fish out the Cipher instance form the thread 
local), then we can avoid the {{Cipher#init}} call, as well.

 Add transparent data encryption core classes
 

 Key: CASSANDRA-9945
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9945
 Project: Cassandra
  Issue Type: Improvement
Reporter: Jason Brown
Assignee: Jason Brown
  Labels: encryption
 Fix For: 3.0 beta 1


 This patch will add the core infrastructure classes necessary for transparent 
 data encryption (file-level encryption), as required for CASSANDRA-6018 and 
 CASSANDRA-9633.  The phrase transparent data encryption, while not the most 
 aesthetically pleasing, seems to be used throughout the database industry 
 (Oracle, SQLQServer, Datastax Enterprise) to describe file level encryption, 
 so we'll go with that, as well. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9712) Refactor CFMetaData


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649997#comment-14649997
 ] 

Aleksey Yeschenko commented on CASSANDRA-9712:
--

Pushed another commit that's another step to {{CFMetaData}} immutability to the 
same [branch|https://github.com/iamaleksey/cassandra/commits/9712].

It moves all the attributes into an immutable {{TableParams}} class, in a 
similar way that {{KeyspaceParams}} work. The reference to it in {{CFMetaData}} 
is mutable, for the time being.

One exception - currently - is {{CompactionParams}}, which is still mutable. 
CASSANDRA-7978 makes making it fully immutable a bit hard, so I'm leaving that 
work to CASSANDRA-9839 instead.

cassci dtest and testall to be available at the same urls as before.

 Refactor CFMetaData
 ---

 Key: CASSANDRA-9712
 URL: https://issues.apache.org/jira/browse/CASSANDRA-9712
 Project: Cassandra
  Issue Type: Improvement
Reporter: Aleksey Yeschenko
Assignee: Aleksey Yeschenko
 Fix For: 3.x


 As part of CASSANDRA-9425 and a follow-up to CASSANDRA-9665, and a 
 pre-requisite for new schema change protocol, this ticket will do the 
 following
 1. Make the triggers {{HashMap}} immutable (new {{Triggers}} class)
 2. Allow multiple 2i definitions per column in CFMetaData
 3. to be filled in
 4. Rename and move {{config.CFMetaData}} to {{schema.TableMetadata}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-9949) maxPurgeableTimestamp needs to check memtables too