[jira] [Commented] (CASSANDRA-14423) SSTables stop being compacted

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523635#comment-16523635
 ] 

Stefan Podkowinski commented on CASSANDRA-14423:


Took another closer look at the patch today and I think that this should work. 
I put the assert back in, as I still can't see why it should not hold anymore. 
Also did some minor wording changes to a log message, changelog and git log, if 
that's ok for you, [~KurtG].
 
* [2.2|https://github.com/spodkowinski/cassandra-dtest/tree/CASSANDRA-14423-2.2]
 
[circleci|https://circleci.com/gh/spodkowinski/cassandra/tree/CASSANDRA-14423-2.2]
[dtests|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/578/]
* [3.0|https://github.com/spodkowinski/cassandra-dtest/tree/CASSANDRA-14423-3.0]
[circleci|https://circleci.com/gh/spodkowinski/cassandra/tree/CASSANDRA-14423-3.0]
[dtests|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/579/]
* 
[3.11|https://github.com/spodkowinski/cassandra-dtest/tree/CASSANDRA-14423-3.11]
[circleci|https://circleci.com/gh/spodkowinski/cassandra/tree/CASSANDRA-14423-3.11]
[dtests|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/580/]

The 2.2 circleci test failures seem to be flaky. Dtests still running.

> SSTables stop being compacted
> -
>
> Key: CASSANDRA-14423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14423
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Blocker
> Fix For: 2.2.13, 3.0.17, 3.11.3
>
>
> So seeing a problem in 3.11.0 where SSTables are being lost from the view and 
> not being included in compactions/as candidates for compaction. It seems to 
> get progressively worse until there's only 1-2 SSTables in the view which 
> happen to be the most recent SSTables and thus compactions completely stop 
> for that table.
> The SSTables seem to still be included in reads, just not compactions.
> The issue can be fixed by restarting C*, as it will reload all SSTables into 
> the view, but this is only a temporary fix. User defined/major compactions 
> still work - not clear if they include the result back in the view but is not 
> a good work around.
> This also results in a discrepancy between SSTable count and SSTables in 
> levels for any table using LCS.
> {code:java}
> Keyspace : xxx
> Read Count: 57761088
> Read Latency: 0.10527088681224288 ms.
> Write Count: 2513164
> Write Latency: 0.018211106398149903 ms.
> Pending Flushes: 0
> Table: xxx
> SSTable count: 10
> SSTables in each level: [2, 0, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 894498746
> Space used (total): 894498746
> Space used by snapshots (total): 0
> Off heap memory used (total): 11576197
> SSTable Compression Ratio: 0.6956629530569777
> Number of keys (estimate): 3562207
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 87
> Local read count: 57761088
> Local read latency: 0.108 ms
> Local write count: 2513164
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 86.33
> Bloom filter false positives: 43
> Bloom filter false ratio: 0.0
> Bloom filter space used: 8046104
> Bloom filter off heap memory used: 8046024
> Index summary off heap memory used: 3449005
> Compression metadata off heap memory used: 81168
> Compacted partition minimum bytes: 104
> Compacted partition maximum bytes: 5722
> Compacted partition mean bytes: 175
> Average live cells per slice (last five minutes): 1.0
> Maximum live cells per slice (last five minutes): 1
> Average tombstones per slice (last five minutes): 1.0
> Maximum tombstones per slice (last five minutes): 1
> Dropped Mutations: 0
> {code}
> Also for STCS we've confirmed that SSTable count will be different to the 
> number of SSTables reported in the Compaction Bucket's. In the below example 
> there's only 3 SSTables in a single bucket - no more are listed for this 
> table. Compaction thresholds haven't been modified for this table and it's a 
> very basic KV schema.
> {code:java}
> Keyspace : yyy
> Read Count: 30485
> Read Latency: 0.06708991307200263 ms.
> Write Count: 57044
> Write Latency: 0.02204061776873992 ms.
> Pending Flushes: 0
> Table: yyy
> SSTable count: 19
> Space used (live): 18195482
> Space used (total): 18195482
> Space used by snapshots (total): 0
> Off heap memory used (total): 747376
> SSTable Compression Ratio: 0.7607394576769735
> Number of keys (estimate): 116074
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 39
> Local 

[jira] [Comment Edited] (CASSANDRA-14423) SSTables stop being compacted

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523635#comment-16523635
 ] 

Stefan Podkowinski edited comment on CASSANDRA-14423 at 6/26/18 12:09 PM:
--

Took another closer look at the patch today and I think that this should work. 
I put the assert back in, as I still can't see why it should not hold anymore. 
Also did some minor wording changes to a log message, changelog and git log, if 
that's ok for you, [~KurtG].
 
* [2.2|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-14423-2.2]
 
[circleci|https://circleci.com/gh/spodkowinski/cassandra/tree/CASSANDRA-14423-2.2]
[dtests|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/578/]
* [3.0|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-14423-3.0]
[circleci|https://circleci.com/gh/spodkowinski/cassandra/tree/CASSANDRA-14423-3.0]
[dtests|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/579/]
* [3.11|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-14423-3.11]
[circleci|https://circleci.com/gh/spodkowinski/cassandra/tree/CASSANDRA-14423-3.11]
[dtests|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/580/]

The 2.2 circleci test failures seem to be flaky. Dtests still running.


was (Author: spo...@gmail.com):
Took another closer look at the patch today and I think that this should work. 
I put the assert back in, as I still can't see why it should not hold anymore. 
Also did some minor wording changes to a log message, changelog and git log, if 
that's ok for you, [~KurtG].
 
* [2.2|https://github.com/spodkowinski/cassandra-dtest/tree/CASSANDRA-14423-2.2]
 
[circleci|https://circleci.com/gh/spodkowinski/cassandra/tree/CASSANDRA-14423-2.2]
[dtests|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/578/]
* [3.0|https://github.com/spodkowinski/cassandra-dtest/tree/CASSANDRA-14423-3.0]
[circleci|https://circleci.com/gh/spodkowinski/cassandra/tree/CASSANDRA-14423-3.0]
[dtests|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/579/]
* 
[3.11|https://github.com/spodkowinski/cassandra-dtest/tree/CASSANDRA-14423-3.11]
[circleci|https://circleci.com/gh/spodkowinski/cassandra/tree/CASSANDRA-14423-3.11]
[dtests|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/580/]

The 2.2 circleci test failures seem to be flaky. Dtests still running.

> SSTables stop being compacted
> -
>
> Key: CASSANDRA-14423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14423
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Blocker
> Fix For: 2.2.13, 3.0.17, 3.11.3
>
>
> So seeing a problem in 3.11.0 where SSTables are being lost from the view and 
> not being included in compactions/as candidates for compaction. It seems to 
> get progressively worse until there's only 1-2 SSTables in the view which 
> happen to be the most recent SSTables and thus compactions completely stop 
> for that table.
> The SSTables seem to still be included in reads, just not compactions.
> The issue can be fixed by restarting C*, as it will reload all SSTables into 
> the view, but this is only a temporary fix. User defined/major compactions 
> still work - not clear if they include the result back in the view but is not 
> a good work around.
> This also results in a discrepancy between SSTable count and SSTables in 
> levels for any table using LCS.
> {code:java}
> Keyspace : xxx
> Read Count: 57761088
> Read Latency: 0.10527088681224288 ms.
> Write Count: 2513164
> Write Latency: 0.018211106398149903 ms.
> Pending Flushes: 0
> Table: xxx
> SSTable count: 10
> SSTables in each level: [2, 0, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 894498746
> Space used (total): 894498746
> Space used by snapshots (total): 0
> Off heap memory used (total): 11576197
> SSTable Compression Ratio: 0.6956629530569777
> Number of keys (estimate): 3562207
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 87
> Local read count: 57761088
> Local read latency: 0.108 ms
> Local write count: 2513164
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 86.33
> Bloom filter false positives: 43
> Bloom filter false ratio: 0.0
> Bloom filter space used: 8046104
> Bloom filter off heap memory used: 8046024
> Index summary off heap memory used: 3449005
> Compression metadata off heap memory used: 81168
> Compacted partition minimum bytes: 104
> Compacted partition maximum bytes: 5722
> Compacted partition mean bytes: 175
> Average live cells per slice (last five minutes): 1.0
> Maximum live 

[jira] [Resolved] (CASSANDRA-13023) Add droppable tombstone metrics

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski resolved CASSANDRA-13023.

Resolution: Not A Problem

Thanks for letting me know, [~arodrime]! It's been a while since I created the 
ticket, so I'm not fully sure if I had something else in mind back then, or 
simply wasn't aware of the metric at table level, which seems to be sufficient.

> Add droppable tombstone metrics
> ---
>
> Key: CASSANDRA-13023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13023
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Stefan Podkowinski
>Priority: Major
> Attachments: droppableTombstoneRatio.png
>
>
> Currently it's only possible to inspect the ratio of droppable tombstones on 
> sstable basis using the {{sstablemetadata}} tool. It would be very useful to 
> have this information provided as part of Cassandra metrics as well, e.g. to 
> get an idea on the effectiveness of tombstone eviction during compactions 
> over time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14528) Provide stacktraces for various error logs

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-14528:
---
Status: Patch Available  (was: Open)

* [trunk|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-14528] 
* [circleci|https://circleci.com/gh/spodkowinski/cassandra/298]
* 
[dtests|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/575/#showFailuresLink]

> Provide stacktraces for various error logs
> --
>
> Key: CASSANDRA-14528
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14528
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
> Fix For: 4.x
>
>
> We should reintroduce some stack traces that have gone missing since 
> CASSANDRA-13723 (ba87ab4e954ad2). The cleanest way would probably to use 
> {{String.format}} for any custom messages, e.g. 
> {{logger.error(String.format("Error using param {}", param), e)}}, so we make 
> this more implicit and robust for coming api changes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14423) SSTables stop being compacted

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519010#comment-16519010
 ] 

Stefan Podkowinski commented on CASSANDRA-14423:


I think this is going in the right direction.

As for log messages, the "does not intersect repaired ranges" message seems to 
be now incorrectly shown for already repaired sstables.
{quote}Also, the assert at the end of performAntiCompaction had to be removed 
as we're removing nonAnticompacting from the transaction which violates the 
assert, this is fine however as the assert was effectively incorrect in the 
first place, and only valid due to the bug.
{quote}
Help me out, in which case do we end up removing an sstable in 
nonAnticompacting from txn.originals that would not also be removed from 
sstableIterator?

> SSTables stop being compacted
> -
>
> Key: CASSANDRA-14423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14423
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Major
> Fix For: 2.2.13, 3.0.17, 3.11.3
>
>
> So seeing a problem in 3.11.0 where SSTables are being lost from the view and 
> not being included in compactions/as candidates for compaction. It seems to 
> get progressively worse until there's only 1-2 SSTables in the view which 
> happen to be the most recent SSTables and thus compactions completely stop 
> for that table.
> The SSTables seem to still be included in reads, just not compactions.
> The issue can be fixed by restarting C*, as it will reload all SSTables into 
> the view, but this is only a temporary fix. User defined/major compactions 
> still work - not clear if they include the result back in the view but is not 
> a good work around.
> This also results in a discrepancy between SSTable count and SSTables in 
> levels for any table using LCS.
> {code:java}
> Keyspace : xxx
> Read Count: 57761088
> Read Latency: 0.10527088681224288 ms.
> Write Count: 2513164
> Write Latency: 0.018211106398149903 ms.
> Pending Flushes: 0
> Table: xxx
> SSTable count: 10
> SSTables in each level: [2, 0, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 894498746
> Space used (total): 894498746
> Space used by snapshots (total): 0
> Off heap memory used (total): 11576197
> SSTable Compression Ratio: 0.6956629530569777
> Number of keys (estimate): 3562207
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 87
> Local read count: 57761088
> Local read latency: 0.108 ms
> Local write count: 2513164
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 86.33
> Bloom filter false positives: 43
> Bloom filter false ratio: 0.0
> Bloom filter space used: 8046104
> Bloom filter off heap memory used: 8046024
> Index summary off heap memory used: 3449005
> Compression metadata off heap memory used: 81168
> Compacted partition minimum bytes: 104
> Compacted partition maximum bytes: 5722
> Compacted partition mean bytes: 175
> Average live cells per slice (last five minutes): 1.0
> Maximum live cells per slice (last five minutes): 1
> Average tombstones per slice (last five minutes): 1.0
> Maximum tombstones per slice (last five minutes): 1
> Dropped Mutations: 0
> {code}
> Also for STCS we've confirmed that SSTable count will be different to the 
> number of SSTables reported in the Compaction Bucket's. In the below example 
> there's only 3 SSTables in a single bucket - no more are listed for this 
> table. Compaction thresholds haven't been modified for this table and it's a 
> very basic KV schema.
> {code:java}
> Keyspace : yyy
> Read Count: 30485
> Read Latency: 0.06708991307200263 ms.
> Write Count: 57044
> Write Latency: 0.02204061776873992 ms.
> Pending Flushes: 0
> Table: yyy
> SSTable count: 19
> Space used (live): 18195482
> Space used (total): 18195482
> Space used by snapshots (total): 0
> Off heap memory used (total): 747376
> SSTable Compression Ratio: 0.7607394576769735
> Number of keys (estimate): 116074
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 39
> Local read count: 30485
> Local read latency: NaN ms
> Local write count: 57044
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 79.76
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 690912
> Bloom filter off heap memory used: 690760
> Index summary off heap memory used: 54736
> Compression metadata off heap memory used: 1880
> Compacted 

[jira] [Created] (CASSANDRA-14545) dtests: fix pytest.raises argument names

[Stefan Podkowinski (JIRA)]

Stefan Podkowinski created CASSANDRA-14545:
--

 Summary: dtests: fix pytest.raises argument names
 Key: CASSANDRA-14545
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14545
 Project: Cassandra
  Issue Type: Bug
  Components: Testing
Reporter: Stefan Podkowinski
Assignee: Stefan Podkowinski


I've been through a couple of [dtest 
results|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/580/#showFailuresLink]
 lately and notices some interpreter errors regarding how we call 
pytest.raises. The 
[reference|https://docs.pytest.org/en/latest/assert.html#assertions-about-expected-exceptions]
 is pretty clear on what would be the correct arguments, but still want to make 
sure we're not working on different pytest versions. 
 [~bdeggleston], can you quickly check the following inconsistencies and look 
at my patch (msg->message, matches->match)?
{noformat}
git show 49b2dda4 |egrep 'raises.*, m' {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14545) dtests: fix pytest.raises argument names

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-14545:
---
Attachment: CASSANDRA-14545.patch

> dtests: fix pytest.raises argument names
> 
>
> Key: CASSANDRA-14545
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14545
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
> Attachments: CASSANDRA-14545.patch
>
>
> I've been through a couple of [dtest 
> results|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/580/#showFailuresLink]
>  lately and notices some interpreter errors regarding how we call 
> pytest.raises. The 
> [reference|https://docs.pytest.org/en/latest/assert.html#assertions-about-expected-exceptions]
>  is pretty clear on what would be the correct arguments, but still want to 
> make sure we're not working on different pytest versions. 
>  [~bdeggleston], can you quickly check the following inconsistencies and look 
> at my patch (msg->message, matches->match)?
> {noformat}
> git show 49b2dda4 |egrep 'raises.*, m' {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14545) dtests: fix pytest.raises argument names

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-14545:
---
Status: Patch Available  (was: Open)

> dtests: fix pytest.raises argument names
> 
>
> Key: CASSANDRA-14545
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14545
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
> Attachments: CASSANDRA-14545.patch
>
>
> I've been through a couple of [dtest 
> results|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/580/#showFailuresLink]
>  lately and notices some interpreter errors regarding how we call 
> pytest.raises. The 
> [reference|https://docs.pytest.org/en/latest/assert.html#assertions-about-expected-exceptions]
>  is pretty clear on what would be the correct arguments, but still want to 
> make sure we're not working on different pytest versions. 
> [~mkjellman] can you quickly check the following inconsistencies and look at 
> my patch (msg->message, matches->match)?
> {noformat}
> git show 49b2dda4 |egrep 'raises.*, m' {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14545) dtests: fix pytest.raises argument names

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-14545:
---
Description: 
I've been through a couple of [dtest 
results|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/580/#showFailuresLink]
 lately and notices some interpreter errors regarding how we call 
pytest.raises. The 
[reference|https://docs.pytest.org/en/latest/assert.html#assertions-about-expected-exceptions]
 is pretty clear on what would be the correct arguments, but still want to make 
sure we're not working on different pytest versions. 
[~mkjellman] can you quickly check the following inconsistencies and look at my 
patch (msg->message, matches->match)?
{noformat}
git show 49b2dda4 |egrep 'raises.*, m' {noformat}

  was:
I've been through a couple of [dtest 
results|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/580/#showFailuresLink]
 lately and notices some interpreter errors regarding how we call 
pytest.raises. The 
[reference|https://docs.pytest.org/en/latest/assert.html#assertions-about-expected-exceptions]
 is pretty clear on what would be the correct arguments, but still want to make 
sure we're not working on different pytest versions. 
 [~bdeggleston], can you quickly check the following inconsistencies and look 
at my patch (msg->message, matches->match)?
{noformat}
git show 49b2dda4 |egrep 'raises.*, m' {noformat}


> dtests: fix pytest.raises argument names
> 
>
> Key: CASSANDRA-14545
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14545
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
> Attachments: CASSANDRA-14545.patch
>
>
> I've been through a couple of [dtest 
> results|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/580/#showFailuresLink]
>  lately and notices some interpreter errors regarding how we call 
> pytest.raises. The 
> [reference|https://docs.pytest.org/en/latest/assert.html#assertions-about-expected-exceptions]
>  is pretty clear on what would be the correct arguments, but still want to 
> make sure we're not working on different pytest versions. 
> [~mkjellman] can you quickly check the following inconsistencies and look at 
> my patch (msg->message, matches->match)?
> {noformat}
> git show 49b2dda4 |egrep 'raises.*, m' {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13910) Remove read_repair_chance/dclocal_read_repair_chance

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-13910:
---
Fix Version/s: 3.11.3
   3.0.17

> Remove read_repair_chance/dclocal_read_repair_chance
> 
>
> Key: CASSANDRA-13910
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13910
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Sylvain Lebresne
>Assignee: Aleksey Yeschenko
>Priority: Minor
> Fix For: 3.0.17, 3.11.3, 4.0
>
>
> First, let me clarify so this is not misunderstood that I'm not *at all* 
> suggesting to remove the read-repair mechanism of detecting and repairing 
> inconsistencies between read responses: that mechanism is imo fine and 
> useful.  But the {{read_repair_chance}} and {{dclocal_read_repair_chance}} 
> have never been about _enabling_ that mechanism, they are about querying all 
> replicas (even when this is not required by the consistency level) for the 
> sole purpose of maybe read-repairing some of the replica that wouldn't have 
> been queried otherwise. Which btw, bring me to reason 1 for considering their 
> removal: their naming/behavior is super confusing. Over the years, I've seen 
> countless users (and not only newbies) misunderstanding what those options 
> do, and as a consequence misunderstand when read-repair itself was happening.
> But my 2nd reason for suggesting this is that I suspect 
> {{read_repair_chance}}/{{dclocal_read_repair_chance}} are, especially 
> nowadays, more harmful than anything else when enabled. When those option 
> kick in, what you trade-off is additional resources consumption (all nodes 
> have to execute the read) for a _fairly remote chance_ of having some 
> inconsistencies repaired on _some_ replica _a bit faster_ than they would 
> otherwise be. To justify that last part, let's recall that:
> # most inconsistencies are actually fixed by hints in practice; and in the 
> case where a node stay dead for a long time so that hints ends up timing-out, 
> you really should repair the node when it comes back (if not simply 
> re-bootstrapping it).  Read-repair probably don't fix _that_ much stuff in 
> the first place.
> # again, read-repair do happen without those options kicking in. If you do 
> reads at {{QUORUM}}, inconsistencies will eventually get read-repaired all 
> the same.  Just a tiny bit less quickly.
> # I suspect almost everyone use a low "chance" for those options at best 
> (because the extra resources consumption is real), so at the end of the day, 
> it's up to chance how much faster this fixes inconsistencies.
> Overall, I'm having a hard time imagining real cases where that trade-off 
> really make sense. Don't get me wrong, those options had their places a long 
> time ago when hints weren't working all that well, but I think they bring 
> more confusion than benefits now.
> And I think it's sane to reconsider stuffs every once in a while, and to 
> clean up anything that may not make all that much sense anymore, which I 
> think is the case here.
> Tl;dr, I feel the benefits brought by those options are very slim at best and 
> well overshadowed by the confusion they bring, and not worth maintaining the 
> code that supports them (which, to be fair, isn't huge, but getting rid of 
> {{ReadCallback.AsyncRepairRunner}} wouldn't hurt for instance).
> Lastly, if the consensus here ends up being that they can have their use in 
> weird case and that we fill supporting those cases is worth confusing 
> everyone else and maintaining that code, I would still suggest disabling them 
> totally by default.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14298) cqlshlib tests broken on b.a.o

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460720#comment-16460720
 ] 

Stefan Podkowinski commented on CASSANDRA-14298:


Nice work! Just found a single change to clarify, before spinning up tests runs.
{quote} 
 "Many tests were failing because they were asserting that some string was 
equal to content in stdout or stderr piped from subprocess, which was a bytes"
{quote}
Having both a local {{run_cqlsh}} method returning bytes and ccm's 
{{node.run_cqlsh}} returning strings, is rather confusing and error prone. 
Can't we convert the local method to return strings as well, just to keep 
things consistent and spare us from carefully b' prefixing string literals in 
our assertions?

> cqlshlib tests broken on b.a.o
> --
>
> Key: CASSANDRA-14298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14298
> Project: Cassandra
>  Issue Type: Bug
>  Components: Build, Testing
>Reporter: Stefan Podkowinski
>Assignee: Patrick Bannister
>Priority: Major
>  Labels: cqlsh, dtest
> Attachments: CASSANDRA-14298.txt, cqlsh_tests_notes.md
>
>
> It appears that cqlsh-tests on builds.apache.org on all branches stopped 
> working since we removed nosetests from the system environment. See e.g. 
> [here|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-cqlsh-tests/458/cython=no,jdk=JDK%201.8%20(latest),label=cassandra/console].
>  Looks like we either have to make nosetests available again or migrate to 
> pytest as we did with dtests. Giving pytest a quick try resulted in many 
> errors locally, but I haven't inspected them in detail yet. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14435) Diag. Events: JMX events

[Stefan Podkowinski (JIRA)]

Stefan Podkowinski created CASSANDRA-14435:
--

 Summary: Diag. Events: JMX events
 Key: CASSANDRA-14435
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14435
 Project: Cassandra
  Issue Type: New Feature
Reporter: Stefan Podkowinski
Assignee: Stefan Podkowinski
 Fix For: 4.x


Nodes currently use JMX events for progress reporting on bootstrap and repairs. 
This might also be an option to expose diagnostic events to external 
subscribers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14435) Diag. Events: JMX events

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460990#comment-16460990
 ] 

Stefan Podkowinski commented on CASSANDRA-14435:


Quick way to give this a local test:

* compile and start
* jconsole localhost:7199
* Enable events if disabled in yaml: {{o.a.c.diag DiagnosticEventService}} -> 
{{resumePublishing()}}
* Start emit dummy events: {{o.a.c.diag DummyEventEmitter}} -> 
{{dummyEventEmitIntervalMillis(1000)}}
* Enable listening to dummy events: {{o.a.c.diag DiagnosticEvents}} -> 
{{enableEvents(org.apache.cassandra.diag.DummyEvent)}}
* Go to {{o.a.c.diag DiagnosticEvents}} Notifications and subscribe 



> Diag. Events: JMX events
> 
>
> Key: CASSANDRA-14435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14435
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
> Fix For: 4.x
>
>
> Nodes currently use JMX events for progress reporting on bootstrap and 
> repairs. This might also be an option to expose diagnostic events to external 
> subscribers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14435) Diag. Events: JMX events

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-14435:
---
Status: Patch Available  (was: In Progress)

> Diag. Events: JMX events
> 
>
> Key: CASSANDRA-14435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14435
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
> Fix For: 4.x
>
>
> Nodes currently use JMX events for progress reporting on bootstrap and 
> repairs. This might also be an option to expose diagnostic events to external 
> subscribers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14435) Diag. Events: JMX events

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462475#comment-16462475
 ] 

Stefan Podkowinski commented on CASSANDRA-14435:


{quote}While I think its a good idea here, we should probably at least have a 
note in yaml about enabling it may impact operational tooling if using 
broadcaster. With the shared event buffer (1000), the more we use it (even if 
no one is listening to that mbean's events) the more lost notifications will 
occur. On an active node we already end up losing a lot of events if the client 
is anywhere with relevant latency from the node. Increasing the buffer isn't 
really a good option as it puts massive pressure on the heap as the composite 
data objects (particularly streaming ones) are huge.
{quote}
JMX surely isn't the most scalable and robust eventing solution. But any diag. 
event consumers would also fall into the "operational tooling" category and 
tool creators and users should be aware of latency and contention based 
limitations. It's not ideal, but selectively sending infrequent events should 
hurt that much either.

We can always improve form here and work on a more scalable long term solution. 
Maybe something based on chronicle queue with a CQL streaming extension and/or 
virtual tables on top. But that's not strictly related to diagnostic events and 
shouldn't prevent us from continue to use JMX until we have another solution.
{quote}Once of the issues I can see is that events are sent on the current 
thread, ref NotificationBroadcasterSupport.defaultExecutor.
{quote}
I've pushed a commit 
[here|https://github.com/spodkowinski/cassandra/commit/c7df1333e84f5b91ebe61161ab4d669fe8da9b32]
 to share the same executor introduced in CASSANDRA-12146.

> Diag. Events: JMX events
> 
>
> Key: CASSANDRA-14435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14435
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
> Fix For: 4.x
>
>
> Nodes currently use JMX events for progress reporting on bootstrap and 
> repairs. This might also be an option to expose diagnostic events to external 
> subscribers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-13971) Automatic certificate management using Vault

[Stefan Podkowinski (JIRA)]

Stefan Podkowinski created CASSANDRA-13971:
--

 Summary: Automatic certificate management using Vault
 Key: CASSANDRA-13971
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13971
 Project: Cassandra
  Issue Type: Improvement
  Components: Streaming and Messaging
Reporter: Stefan Podkowinski
Assignee: Stefan Podkowinski
 Fix For: 4.x


We've been adding security features during the last years to enable users to 
secure their clusters, if they are willing to use them and do so correctly. 
Some features are powerful and easy to work with, such as role based 
authorization. Other features that require to manage a local keystore are 
rather painful to deal with. Think about setting up SSL..

To be fair, keystore related issues and certificate handling hasn't been 
invented by us. We're just following Java standards there. But that doesn't 
mean that we absolutely have to, if there are better options. I'd like to give 
it a shoot and find out if we can automate certificate/key handling (PKI) by 
using external APIs. In this case, the implementation will be based on 
[Vault|https://vaultproject.io]. But certificate management services offered by 
cloud providers may also be able to handle the use-case and I intend to create 
a generic, pluggable API for that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-10404) Node to Node encryption transitional mode

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222036#comment-16222036
 ] 

Stefan Podkowinski commented on CASSANDRA-10404:


Latest commits look good. I like that the {{enable_legacy_ssl_storage_port}} 
option. Makes things more obvious.

Although users should follow advice in NEWS.txt, I'd suggest to better add a 
simple config validation in DatabaseDescriptor and throw a 
ConfigurationException in case cassandra.yaml hasn't been updated correctly 
during the upgrade. Cassandra should not start and switch from encrypted to 
unencrypted after upgrade in case you just keep your old config with 
{{internode_encryption}} != {{none}}, but the new {{enabled}} flag not 
specified and thus set to false by default. 



> Node to Node encryption transitional mode
> -
>
> Key: CASSANDRA-10404
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10404
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Tom Lewis
>Assignee: Jason Brown
> Fix For: 4.x
>
>
> Create a transitional mode for encryption that allows encrypted and 
> unencrypted traffic node-to-node during a change over to encryption from 
> unencrypted. This alleviates downtime during the switch.
>  This is similar to CASSANDRA-10559 which is intended for client-to-node



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13971) Automatic certificate management using Vault

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213045#comment-16213045
 ] 

Stefan Podkowinski commented on CASSANDRA-13971:


The first step is to create a pluggable API that would abstract away how 
certificates and keystore instances are acquired. The default implementation 
would be compatible with the current way we configure and handle an existing 
keystore in the filesystem. The alternative Vault implementation will be 
implemented based on the Vault [REST 
API|https://www.vaultproject.io/api/index.html] (using Netty's HTTP support). 
We won't have to integrate any Vault code or library for that.

> Automatic certificate management using Vault
> 
>
> Key: CASSANDRA-13971
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13971
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Streaming and Messaging
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
> Fix For: 4.x
>
>
> We've been adding security features during the last years to enable users to 
> secure their clusters, if they are willing to use them and do so correctly. 
> Some features are powerful and easy to work with, such as role based 
> authorization. Other features that require to manage a local keystore are 
> rather painful to deal with. Think about setting up SSL..
> To be fair, keystore related issues and certificate handling hasn't been 
> invented by us. We're just following Java standards there. But that doesn't 
> mean that we absolutely have to, if there are better options. I'd like to 
> give it a shoot and find out if we can automate certificate/key handling 
> (PKI) by using external APIs. In this case, the implementation will be based 
> on [Vault|https://vaultproject.io]. But certificate management services 
> offered by cloud providers may also be able to handle the use-case and I 
> intend to create a generic, pluggable API for that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-10404) Node to Node encryption transitional mode

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222363#comment-16222363
 ] 

Stefan Podkowinski commented on CASSANDRA-10404:


Sounds good.

> Node to Node encryption transitional mode
> -
>
> Key: CASSANDRA-10404
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10404
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Tom Lewis
>Assignee: Jason Brown
> Fix For: 4.x
>
>
> Create a transitional mode for encryption that allows encrypted and 
> unencrypted traffic node-to-node during a change over to encryption from 
> unencrypted. This alleviates downtime during the switch.
>  This is similar to CASSANDRA-10559 which is intended for client-to-node



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322253#comment-16322253
 ] 

Stefan Podkowinski commented on CASSANDRA-12151:


I took a different approach with CASSANDRA-13668 by proposing to send auditing 
events to external subscribers that would be able to store or analyze them 
further. It's a different approach from storing auditing events in a local or 
distributed keyspace, as implemented in this ticket. My assumption was 
basically that raising alerts on certain events would be easier and more 
efficient to implement, if consumed by an external client, compared to polling 
a table. But I'm curious how any discussion as part of this ticket will turn 
out and what the different requirements are.

> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: stefan setyadi
> Fix For: 4.x
>
> Attachments: 12151.txt
>
>
> we would like a way to enable cassandra to log database activity being done 
> on our server.
> It should show username, remote address, timestamp, action type, keyspace, 
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the 
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every 
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting 
> a specific keyspace and column family.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14134) Migrate dtests to use pytest and python3

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16321969#comment-16321969
 ] 

Stefan Podkowinski commented on CASSANDRA-14134:


My answer on how to coordinate test execution is still CASSANDRA-12944. Once 
we'd be able to raise e.g. a {{SecondaryIndexCreated(ks, table, col)}} event 
and propagate it to subscribed (dtest-)clients, we can get rid of sleep calls 
and brittle log file parsing. 

> Migrate dtests to use pytest and python3
> 
>
> Key: CASSANDRA-14134
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14134
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Testing
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
>
> h4. Get the C* dtests running on the pytest framework.
> C* DTests currently run using the python test framework nosetest. This 
> framework has been largely abandoned with no releases since 2015 and a 
> general strong consensus in the python community that pytest is the future.
> h4. Why should we do this.
> Currently (and historically) dtests have always been difficult to run, flaky 
> and unpredictable in CI environments, and almost impossible to debug.
> On November 28th, 2017, I proposed on the dev@ list that we move the dtests 
> from nosetests to pytests. I got replies from Jon Haddad, Philip Thompson, 
> and kurt greaves with really only "+1" like replies to the proposal.
> Since then I've been working pretty much non stop to complete the large 
> refactor of dtests to pytests. As part of this effort (and due to the 
> migration tools that exist require it) I also ported the code to python3 
> (from the current python 2.7 based code-base).
> h4. High-level summary of key changes, improvements, and new features.
> * Migrate dtests from executing using the nosetest framework to pytest
> * Port the entire code base from Python 2.7 to Python 3.6
> * Update run_dtests.py to work with pytest
> * Add --dtest-print-tests-only option to run_dtests.py to get easily parsable 
> list of all available collected tests
> * Update README.md for executing the dtests with pytest
> * Add new debugging tips section to README.md to help with some basics of 
> debugging python3 and pytest
> * Migrate all existing Enviornment Variable usage as a means to control dtest 
> operation modes to argparse command line options with documented help on each 
> toggles intended usage
> * Migration of old unitTest and nose based test structure to modern pytest 
> fixture approach
> * Automatic detection of physical system resources to automatically determine 
> if @pytest.mark.resource_intensive annotated tests should be collected and 
> run on the system where they are being executed
> * new pytest fixture replacements for @since and @pytest.mark.upgrade_test 
> annotations
> * Migration to python logging framework
> * Upgrade thrift bindings to latest version with full python3 compatibility
> * Remove deprecated cql and pycassa dependencies and migrate any remaining 
> tests to fully remove those dependencies
> * Fixed dozens of tests that would hang the pytest framework forever when run 
> in CI enviornments
> * Ran code nearly 300 times in CircleCI during the migration and to find, 
> identify, and fix any tests capable of hanging CI
> * Upgrade Tests do not yet run in CI and still need additional migration work 
> (although all upgrade test classes compile successfully)
> I started with the *nose2pytest* [https://github.com/pytest-dev/nose2pytest] 
> migration tool. As this required python 3 language support I found myself 
> down the 2to3 python migration path. While painful to do this, the benefits 
> of python3 over python2.7 are numerous and moving to python3 for the 
> additional debugging tools now available to use when fixing dtests makes the 
> effort worth it for that reason alone!
> After the automated tools did their thing I began what was a much longer and 
> tedious manual process than I ever could have expected due to the custom many 
> ways we did things in dtests (frequently to work around nosetest limitations 
> of missing features that thankfully are now all included with the pytest 
> framework). I've done nearly 300 test runs of my migration branch with 
> circleci.
> The latest CircleCI runs can be found at:
> (dtests without vnodes) [https://circleci.com/gh/mkjellman/cassandra/277]
> (dtests with vnodes) [https://circleci.com/gh/mkjellman/cassandra/278]
> With vnodes, there are currently only 6 remaining dtest test failures.
> Without vnodes, there are 12 remaining dtest failures.
> It turns out that after the dtests were moved to ASF Jenkins from cassci, the 
> jobs were misconfigured and we actually haven't been running the dtests in 
> the non-vnodes configuration. The current most recent trunk dtest 

[jira] [Commented] (CASSANDRA-14107) Introduce simple key alias versioning scheme for TDE

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320193#comment-16320193
 ] 

Stefan Podkowinski commented on CASSANDRA-14107:


Yes. The basic idea is that you just need to add a new key with an incremented 
version number (could also be a timestamp I guess) and Cassandra will pick it 
up automatically and uses it from that point. Any data written to disk will 
still refer to the original key alias (e.g. mykey:1) that has been used for 
encryption. 

I've also added a periodical job that will every 10 mins reload the local 
JKCKeyStore and check for new keys, so you wouldn't have to restart or touch 
the config after adding a new key. The Vault implementation supports this as 
well of course.

> Introduce simple key alias versioning scheme for TDE
> 
>
> Key: CASSANDRA-14107
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14107
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: encryption
> Fix For: 4.x
>
>
> Handling of encryption keys as introduced in CASSANDRA-9945 takes place by 
> referencing a key alias in either cassandra.yaml, or the header of the 
> (commitlog/hints) file that has been encrypted. Using the alias as literal 
> value will work, but requires some attention when rotating keys.
> Currently each time a key is rotated (i.e. adding a new key to the keystore 
> while preserving the previous version), the alias in cassandra.yaml has to be 
> update as well and the node needs to be restarted. It would be more 
> convenient to use a symbolic reference instead. My suggestion here would be 
> to use ":latest" for referring to the latest version. In this case 
> Cassandra always picks the key with the highest version in 
> ":".
> The non-trivial part of this suggestion is how the "latest" key is referenced 
> in the file header. If we use "latest", e.g. for the commit log header, and 
> the key gets rotated, we'd now try do decrypt the file with the new key, 
> instead of the key it has been created with. Therefor we'd have to introduce 
> an extra step that will resolve the canonical version for "latest" and refer 
> to that one during any encrypt operation. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14107) Dynamic key rotation support for transparent data encryption

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-14107:
---
Summary: Dynamic key rotation support for transparent data encryption  
(was: Introduce simple key alias versioning scheme for TDE)

> Dynamic key rotation support for transparent data encryption
> 
>
> Key: CASSANDRA-14107
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14107
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: encryption
> Fix For: 4.x
>
>
> Handling of encryption keys as introduced in CASSANDRA-9945 takes place by 
> referencing a key alias in either cassandra.yaml, or the header of the 
> (commitlog/hints) file that has been encrypted. Using the alias as literal 
> value will work, but requires some attention when rotating keys.
> Currently each time a key is rotated (i.e. adding a new key to the keystore 
> while preserving the previous version), the alias in cassandra.yaml has to be 
> update as well and the node needs to be restarted. It would be more 
> convenient to use a symbolic reference instead. My suggestion here would be 
> to use ":latest" for referring to the latest version. In this case 
> Cassandra always picks the key with the highest version in 
> ":".
> The non-trivial part of this suggestion is how the "latest" key is referenced 
> in the file header. If we use "latest", e.g. for the commit log header, and 
> the key gets rotated, we'd now try do decrypt the file with the new key, 
> instead of the key it has been created with. Therefor we'd have to introduce 
> an extra step that will resolve the canonical version for "latest" and refer 
> to that one during any encrypt operation. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14102) Vault support for transparent data encryption

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320298#comment-16320298
 ] 

Stefan Podkowinski commented on CASSANDRA-14102:


Performance impact by using the Vault key provider instead of a local keystore? 
Or the optimization of EncryptionContext use? 

Each key for any key alias is immutable and will be cached infinitely. The 
interaction with Vault should thus be minimal and in most cases we will only 
call Vault once on startup, to retrieve the key. With CASSANDRA-14107 we also 
call Vault to check for new keys and to fetch them. But also only sporadically. 

As for the optimization of  EncryptionContext use it probably depends on how 
many encrypted committlog segments and hint files are being generated. Not so 
trivial to create a good benchmark for that.

> Vault support for transparent data encryption
> -
>
> Key: CASSANDRA-14102
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14102
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>  Labels: encryption
> Fix For: 4.x
>
>
> Transparent data encryption provided by CASSANDRA-9945 can currently be used 
> for commitlog and hints. The default {{KeyProvider}} implementation that we 
> ship allows to use a local keystore for storing and retrieving keys. Thanks 
> to the pluggable handling of the {{KeyStore}} provider and basic Vault 
> related classes introduced in CASSANDRA-13971, a Vault based implementation 
> can be provided as {{KeyProvider}} as well. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-14107) Introduce simple key alias versioning scheme for TDE

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski reassigned CASSANDRA-14107:
--

Assignee: Stefan Podkowinski

> Introduce simple key alias versioning scheme for TDE
> 
>
> Key: CASSANDRA-14107
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14107
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: encryption
> Fix For: 4.x
>
>
> Handling of encryption keys as introduced in CASSANDRA-9945 takes place by 
> referencing a key alias in either cassandra.yaml, or the header of the 
> (commitlog/hints) file that has been encrypted. Using the alias as literal 
> value will work, but requires some attention when rotating keys.
> Currently each time a key is rotated (i.e. adding a new key to the keystore 
> while preserving the previous version), the alias in cassandra.yaml has to be 
> update as well and the node needs to be restarted. It would be more 
> convenient to use a symbolic reference instead. My suggestion here would be 
> to use ":last" for referring to the latest version. Omitting the 
> ":" part altogether would have the same effect. In this case 
> Cassandra always picks the latest key when the key cache has been expired.
> The non-trivial part of this suggestion is how the "latest" key is referenced 
> in the file header. If we use "latest", e.g. for the commit log header, and 
> the key gets rotated, we'd now try do decrypt the file with the new key, 
> instead of the key it has been created with. Therefor we'd have to introduce 
> an extra step that will resolve the canonical version for "latest" and refer 
> to that one during any encrypt operation. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14107) Introduce simple key alias versioning scheme for TDE

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-14107:
---
Description: 
Handling of encryption keys as introduced in CASSANDRA-9945 takes place by 
referencing a key alias in either cassandra.yaml, or the header of the 
(commitlog/hints) file that has been encrypted. Using the alias as literal 
value will work, but requires some attention when rotating keys.

Currently each time a key is rotated (i.e. adding a new key to the keystore 
while preserving the previous version), the alias in cassandra.yaml has to be 
update as well and the node needs to be restarted. It would be more convenient 
to use a symbolic reference instead. My suggestion here would be to use 
":latest" for referring to the latest version. In this case Cassandra 
always picks the key with the highest version in ":".

The non-trivial part of this suggestion is how the "latest" key is referenced 
in the file header. If we use "latest", e.g. for the commit log header, and the 
key gets rotated, we'd now try do decrypt the file with the new key, instead of 
the key it has been created with. Therefor we'd have to introduce an extra step 
that will resolve the canonical version for "latest" and refer to that one 
during any encrypt operation. 


  was:
Handling of encryption keys as introduced in CASSANDRA-9945 takes place by 
referencing a key alias in either cassandra.yaml, or the header of the 
(commitlog/hints) file that has been encrypted. Using the alias as literal 
value will work, but requires some attention when rotating keys.

Currently each time a key is rotated (i.e. adding a new key to the keystore 
while preserving the previous version), the alias in cassandra.yaml has to be 
update as well and the node needs to be restarted. It would be more convenient 
to use a symbolic reference instead. My suggestion here would be to use 
":last" for referring to the latest version. Omitting the ":" 
part altogether would have the same effect. In this case Cassandra always picks 
the latest key when the key cache has been expired.

The non-trivial part of this suggestion is how the "latest" key is referenced 
in the file header. If we use "latest", e.g. for the commit log header, and the 
key gets rotated, we'd now try do decrypt the file with the new key, instead of 
the key it has been created with. Therefor we'd have to introduce an extra step 
that will resolve the canonical version for "latest" and refer to that one 
during any encrypt operation. 



> Introduce simple key alias versioning scheme for TDE
> 
>
> Key: CASSANDRA-14107
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14107
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: encryption
> Fix For: 4.x
>
>
> Handling of encryption keys as introduced in CASSANDRA-9945 takes place by 
> referencing a key alias in either cassandra.yaml, or the header of the 
> (commitlog/hints) file that has been encrypted. Using the alias as literal 
> value will work, but requires some attention when rotating keys.
> Currently each time a key is rotated (i.e. adding a new key to the keystore 
> while preserving the previous version), the alias in cassandra.yaml has to be 
> update as well and the node needs to be restarted. It would be more 
> convenient to use a symbolic reference instead. My suggestion here would be 
> to use ":latest" for referring to the latest version. In this case 
> Cassandra always picks the key with the highest version in 
> ":".
> The non-trivial part of this suggestion is how the "latest" key is referenced 
> in the file header. If we use "latest", e.g. for the commit log header, and 
> the key gets rotated, we'd now try do decrypt the file with the new key, 
> instead of the key it has been created with. Therefor we'd have to introduce 
> an extra step that will resolve the canonical version for "latest" and refer 
> to that one during any encrypt operation. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14145) Detecting data resurrection during read

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316132#comment-16316132
 ] 

Stefan Podkowinski commented on CASSANDRA-14145:


I agree with the problem description and that we should detect inconsistencies 
in the repaired set when reading data. But, as described in CASSANDRA-13912, 
I'm not convinced that correcting data as part of read repairs is the way to go 
for already repaired data. 

There could be many different reasons for inconsistencies. Corrupted or missing 
sstables are relatively easy to reason about in this context. What I'm more 
concerned of are internal consistency issues due to rare edge cases, e.g. 
during upgrade paths or race conditions. Returning inconsistent results in rare 
cases might be less of an issue, compared to risking greater harm by running 
repair code which was created with other situations in mind.

I'm wondering if we shouldn't just start by logging inconsistencies, allowing 
operators to further investigate. If there's some serious issue with a node, it 
needs to be replaced anyways. If the hardware is fine and there are no external 
causes that lead to inconsistencies, we may have a bigger problem to fix.


>  Detecting data resurrection during read
> 
>
> Key: CASSANDRA-14145
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14145
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: sankalp kohli
>Priority: Minor
>
> We have seen several bugs in which deleted data gets resurrected. We should 
> try to see if we can detect this on the read path and possibly fix it. Here 
> are a few examples which brought back data
> A replica lost an sstable on startup which caused one replica to lose the 
> tombstone and not the data. This tombstone was past gc grace which means this 
> could resurrect data. We can detect such invalid states by looking at other 
> replicas. 
> If we are running incremental repair, Cassandra will keep repaired and 
> non-repaired data separate. Every-time incremental repair will run, it will 
> move the data from non-repaired to repaired. Repaired data across all 
> replicas should be 100% consistent. 
> Here is an example of how we can detect and mitigate the issue in most cases. 
> Say we have 3 machines, A,B and C. All these machines will have data split 
> b/w repaired and non-repaired. 
> 1. Machine A due to some bug bring backs data D. This data D is in repaired 
> dataset. All other replicas will have data D and tombstone T 
> 2. Read for data D comes from application which involve replicas A and B. The 
> data being read involves data which is in repaired state.  A will respond 
> back to co-ordinator with data D and B will send nothing as tombstone is past 
> gc grace. This will cause digest mismatch. 
> 3. This patch will only kick in when there is a digest mismatch. Co-ordinator 
> will ask both replicas to send back all data like we do today but with this 
> patch, replicas will respond back what data it is returning is coming from 
> repaired vs non-repaired. If data coming from repaired does not match, we 
> know there is a something wrong!! At this time, co-ordinator cannot determine 
> if replica A has resurrected some data or replica B has lost some data. We 
> can still log error in the logs saying we hit an invalid state.
> 4. Besides the log, we can take this further and even correct the response to 
> the query. After logging an invalid state, we can ask replica A and B (and 
> also C if alive) to send back all data for this including gcable tombstones. 
> If any machine returns a tombstone which is after this data, we know we 
> cannot return this data. This way we can avoid returning data which has been 
> deleted. 
> Some Challenges with this 
> 1. When data will be moved from non-repaired to repaired, there could be a 
> race here. We can look at which incremental repairs have promoted things on 
> which replica to avoid false positives.  
> 2. If the third replica is down and live replica does not have any tombstone, 
> we wont be able to break the tie in deciding whether data was actually 
> deleted or resurrected. 
> 3. If the read is for latest data only, we wont be able to detect it as the 
> read will be served from non-repaired data. 
> 4. If the replica where we lose a tombstone is the last replica to compact 
> the tombstone, we wont be able to decide if data is coming back or rest of 
> the replicas has lost that data. But we will still detect something is wrong. 
> 5. We wont affect 99.9% of the read queries as we only do extra work during 
> digest mismatch.
> 6. CL.ONE reads will not be able to detect this. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To 

[jira] [Updated] (CASSANDRA-14107) Dynamic key rotation support for transparent data encryption

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-14107:
---
Status: Patch Available  (was: In Progress)

My approach with the latest patch was to leave as much of the existing TDE code 
as is and instead only modify the way the key alias is accessed in the global 
config. We have to make sure to use the most recent key each time new data is 
to be encrypted. Decrypting existing data must take place based on the 
serialized encryption settings. The patch will therefor
 * Retrieve the most recent key alias in case the ':latest' suffix is specified
 * In that case also start a periodical task to retrieve and update it
 * Store the latest alias in the active EncryptionContext instance
 * Make sure to have existing code always pull the currently active 
EncryptionContext for new encryption operations, so we always use the latest key
 * Make EncryptionContext immutable, so we never change the alias for already 
in-flight or pending encryption operations

I've also added some code to use the active EncryptionContext to enable 
encryption for hints as well, which seems to be missing on trunk.

> Dynamic key rotation support for transparent data encryption
> 
>
> Key: CASSANDRA-14107
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14107
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: encryption
> Fix For: 4.x
>
>
> Handling of encryption keys as introduced in CASSANDRA-9945 takes place by 
> referencing a key alias in either cassandra.yaml, or the header of the 
> (commitlog/hints) file that has been encrypted. Using the alias as literal 
> value will work, but requires some attention when rotating keys.
> Currently each time a key is rotated (i.e. adding a new key to the keystore 
> while preserving the previous version), the alias in cassandra.yaml has to be 
> update as well and the node needs to be restarted. It would be more 
> convenient to use a symbolic reference instead. My suggestion here would be 
> to use ":latest" for referring to the latest version. In this case 
> Cassandra always picks the key with the highest version in 
> ":".
> The non-trivial part of this suggestion is how the "latest" key is referenced 
> in the file header. If we use "latest", e.g. for the commit log header, and 
> the key gets rotated, we'd now try do decrypt the file with the new key, 
> instead of the key it has been created with. Therefor we'd have to introduce 
> an extra step that will resolve the canonical version for "latest" and refer 
> to that one during any encrypt operation. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14180) cassandra.spec needs to require ant-junit

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16334081#comment-16334081
 ] 

Stefan Podkowinski commented on CASSANDRA-14180:


I think you're right. But how did you manage to build the source package at 
all? Mine was missing a version and revision macro declaration before rpmbuild 
was able to finish successfully.

We use to create these values dynamically and [pass 
them|https://github.com/apache/cassandra-builds/blob/master/docker/build-rpms.sh#L69]
 as {{rpmbuild --define="version .."}} during our release process. Maybe we 
should turn that into a string substitution in the spec instead.

> cassandra.spec needs to require ant-junit
> -
>
> Key: CASSANDRA-14180
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14180
> Project: Cassandra
>  Issue Type: Bug
>  Components: Packaging
>Reporter: Troels Arvin
>Priority: Minor
>
> I tried rebuilding cassandra-3.11.1-1.src.rpm on a Centos 7 host which had 
> ant installed, but not the "ant-junit" package; that failed with a somewhat 
> cryptic error message. It turnout out I needed to have the "ant-junit" 
> package installed, as well. So for the cassandra.spec file, I suggest that 
> the following line be added just below the existing BuildRequires line:
> {{BuildRequires: ant-junit >= 1.9}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Resolved] (CASSANDRA-14180) cassandra.spec needs to require ant-junit

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski resolved CASSANDRA-14180.

   Resolution: Fixed
 Assignee: Troels Arvin
 Reviewer: Stefan Podkowinski
Fix Version/s: 4.0
   3.11.2
   3.0.16

Thanks for reporting!

Committed as 0628520a9be69bb42a0ba73859888a5a8af83b27 to cassandra-3.0 and 
merged upwards.

> cassandra.spec needs to require ant-junit
> -
>
> Key: CASSANDRA-14180
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14180
> Project: Cassandra
>  Issue Type: Bug
>  Components: Packaging
>Reporter: Troels Arvin
>Assignee: Troels Arvin
>Priority: Minor
> Fix For: 3.0.16, 3.11.2, 4.0
>
>
> I tried rebuilding cassandra-3.11.1-1.src.rpm on a Centos 7 host which had 
> ant installed, but not the "ant-junit" package; that failed with a somewhat 
> cryptic error message. It turnout out I needed to have the "ant-junit" 
> package installed, as well. So for the cassandra.spec file, I suggest that 
> the following line be added just below the existing BuildRequires line:
> {{BuildRequires: ant-junit >= 1.9}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14222) Add hot reloading of SSL Certificates capability to Cassandra

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356640#comment-16356640
 ] 

Stefan Podkowinski commented on CASSANDRA-14222:


This has already been implemented in CASSANDRA-13971 (specifically 
[7e076c42|https://github.com/spodkowinski/cassandra/commit/7e076c42846dff72ed10c6b5e0d7ab0024284fcf]).
 

> Add hot reloading of SSL Certificates capability to Cassandra
> -
>
> Key: CASSANDRA-14222
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14222
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Dinesh Joshi
>Assignee: Dinesh Joshi
>Priority: Major
> Fix For: 4.x
>
>
> Cassandra does not currently hot reload SSL certificates. For ease of 
> operation it would be useful if we add this capability. This patch would 
> watch changes to the truststore & keystore and would reload them when they're 
> changed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13259) Use platform specific X.509 default algorithm

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362330#comment-16362330
 ] 

Stefan Podkowinski commented on CASSANDRA-13259:


Rebased the first patch submitted when creating this ticket:

||trunk||
|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13259-trunk]|
|[dtest|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/494/]|
|[unit_tests|https://circleci.com/gh/spodkowinski/cassandra/223]|


> Use platform specific X.509 default algorithm
> -
>
> Key: CASSANDRA-13259
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13259
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
> Fix For: 4.x
>
>
> We should replace the hardcoded "SunX509" default algorithm and use the JRE 
> default instead. This implementation will currently not work on less popular 
> platforms (e.g. IBM) and won't get any further updates.
> See also:
> https://bugs.openjdk.java.net/browse/JDK-8169745



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Resolved] (CASSANDRA-13546) Getting error while adding a node in existing cluster

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski resolved CASSANDRA-13546.

Resolution: Cannot Reproduce

> Getting error while adding a node in existing cluster
> -
>
> Key: CASSANDRA-13546
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13546
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
> Environment: Unix
>Reporter: Sonu Gupta
>Priority: Major
> Fix For: 2.1.x
>
>
> Getting error after adding a node in existing cluster, error are coming after 
> 1 hour after started service on seed node and due to this load got increased 
> vastly on seed node not on new node.
> Below errors from system.log on seed node:
> {code}
> ERROR [NonPeriodicTasks:1] 2017-05-18 10:03:01,819 CassandraDaemon.java:153 - 
> Exception in thread Thread[NonPeriodicTasks:1,5,main]
> java.lang.AssertionError: null
> at org.apache.cassandra.io.util.Memory.free(Memory.java:300) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.utils.obs.OffHeapBitSet.close(OffHeapBitSet.java:143) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.utils.BloomFilter.close(BloomFilter.java:116) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.io.sstable.SSTableReader$6.run(SSTableReader.java:645) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown 
> Source) ~[na:1.7.0_71]
> at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_71]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown
>  Source) ~[na:1.7.0_71]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>  Source) ~[na:1.7.0_71]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 
> [na:1.7.0_71]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
> [na:1.7.0_71]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
> INFO  [CompactionExecutor:162] 2017-05-18 10:03:01,820 
> ColumnFamilyStore.java:840 - Enqueuing flush of compactions_in_progress: 164 
> (0%) on-heap, 0 (0%) off-heap
> INFO  [MemtableFlushWriter:152] 2017-05-18 10:03:01,821 Memtable.java:325 - 
> Writing Memtable-compactions_in_progress@731506998(0 serialized bytes, 1 ops, 
> 0%/0% of on/off-heap limit)
> INFO  [MemtableFlushWriter:152] 2017-05-18 10:03:01,825 Memtable.java:364 - 
> Completed flushing 
> /var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-52434-Data.db
>  (42 bytes) for commitlog position ReplayPosition(segmentId=1495106933696, 
> position=9156004)
> ERROR [CompactionExecutor:162] 2017-05-18 10:03:01,829 
> CassandraDaemon.java:153 -
>  Exception in thread Thread[CompactionExecutor:162,1,main]
> java.lang.AssertionError: null
> at org.apache.cassandra.io.util.Memory.free(Memory.java:300) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.utils.obs.OffHeapBitSet.close(OffHeapBitSet.java:143) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.utils.BloomFilter.close(BloomFilter.java:116) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:345) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.io.sstable.SSTableRewriter.abort(SSTableRewriter.java:198)
>  ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:204)
>  ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>  ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75)
>  ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
>  ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:232)
>  ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown 
> Source) ~[na:1.7.0_71]
> at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_71]
>

[jira] [Updated] (CASSANDRA-12793) invalid jvm type and architecture [cassandra-env.sh]

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-12793:
---
Attachment: 12793.patch

> invalid jvm type and architecture [cassandra-env.sh]
> 
>
> Key: CASSANDRA-12793
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12793
> Project: Cassandra
>  Issue Type: Bug
>  Components: Configuration
> Environment: ubuntu 16.04, openjdk 1.8.0_91
>Reporter: Ali Ebrahiminejad
>Priority: Minor
> Fix For: 3.11.x
>
> Attachments: 12793.patch
>
>
> In cassandra-env.sh the part that determines the type of JVM we'll be running 
> on doesn't provide the right answer for openjdk 1.8.0_91.
> value of java_ver_output is "openjdk version "1.8.0_91" OpenJDK Runtime 
> Environment (build 1.8.0_91-8u91-b14-3ubuntu1~16.04.1-b14) OpenJDK 64-Bit 
> Server VM (build 25.91-b14, mixed mode)", yet the command looks for "java 
> version" (jvm=`echo "$java_ver_output" | grep -A 1 'java version' ...) which 
> does not exist.
> I guess it should be replaced with jvm=`echo "$java_ver_output" | grep -A 1 
> '[openjdk|java] version' | awk 'NR==2 {print $1}'`



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-12793) invalid jvm type and architecture [cassandra-env.sh]

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski reassigned CASSANDRA-12793:
--

Assignee: Stefan Podkowinski

> invalid jvm type and architecture [cassandra-env.sh]
> 
>
> Key: CASSANDRA-12793
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12793
> Project: Cassandra
>  Issue Type: Bug
>  Components: Configuration
> Environment: ubuntu 16.04, openjdk 1.8.0_91
>Reporter: Ali Ebrahiminejad
>Assignee: Stefan Podkowinski
>Priority: Minor
> Fix For: 3.11.x
>
> Attachments: 12793.patch
>
>
> In cassandra-env.sh the part that determines the type of JVM we'll be running 
> on doesn't provide the right answer for openjdk 1.8.0_91.
> value of java_ver_output is "openjdk version "1.8.0_91" OpenJDK Runtime 
> Environment (build 1.8.0_91-8u91-b14-3ubuntu1~16.04.1-b14) OpenJDK 64-Bit 
> Server VM (build 25.91-b14, mixed mode)", yet the command looks for "java 
> version" (jvm=`echo "$java_ver_output" | grep -A 1 'java version' ...) which 
> does not exist.
> I guess it should be replaced with jvm=`echo "$java_ver_output" | grep -A 1 
> '[openjdk|java] version' | awk 'NR==2 {print $1}'`



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Resolved] (CASSANDRA-13330) /var/run/cassandra directory not created on Centos7

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski resolved CASSANDRA-13330.

   Resolution: Won't Fix
Fix Version/s: (was: 3.11.x)

Affected version is no longer supported, neither are datastax packages. Please 
open a new ticket if this issue can be reproduced in a recent Apache release.

> /var/run/cassandra directory not created on Centos7
> ---
>
> Key: CASSANDRA-13330
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13330
> Project: Cassandra
>  Issue Type: Bug
>  Components: Configuration
> Environment: CentOS Linux release 7.3.1611 (Core) (x86_64)
> datastax-ddc.noarch 3.9.0-1  
> datastax-ddc-tools.noarch   3.9.0-1   
> java-1.8.0-openjdk.x86_64   1:1.8.0.121-0.b13.el7_3  
> java-1.8.0-openjdk-headless.x86_64  1:1.8.0.121-0.b13.el7_3 
>Reporter: Tobi
>Priority: Minor
>
> After updating cassandra from 3.4 to 3.9 via the datastax repo the startup 
> script /etc/init.d/cassandra was unable to stop cassandra. After checking the 
> startscript we found that it's tried to read pid file from 
> /var/run/cassandra/cassandra.pid
> But as this path does not exist, the cassandra process could not be stopped. 
> As a fix we created /usr/lib/tmpfiles.d/cassandra.conf with this content
> d /var/run/cassandra  0750  cassandra cassandra   -   
> -
> After that the directory was created on boot and the pid file could we 
> written in it



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-12793) invalid jvm type and architecture [cassandra-env.sh]

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-12793:
---
Status: Patch Available  (was: Open)

Thanks for reporting and sorry it took so long to respond. 

The issue is easy to reproduce and the suggested change does fix the issue. 
Eventually the broken OpenJDK detection will currently prevent the 
"-XX:+UseCondCardMark" flag from being set automatically.

OpenJDK unpatched:

{noformat}
+ java -version
+ java_ver_output=openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
+ grep [openjdk|java] version
+ cut -d- -f1
+ echo openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
+ awk -F" NR==1 {print $2}
+ jvmver=1.8.0_151
+ JVM_VERSION=1.8.0
+ JVM_PATCH_VERSION=151
+ [ 1.8.0 < 1.8 ]
+ [ 1.8.0 < 1.8 ]
+ echo openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
+ + awk NR==2 {print $1}
grep -A 1 java version
+ jvm=
+ JVM_VENDOR=other
+ JVM_ARCH=unknown
{noformat}


Oracle JDK unpatched:

{noformat}
+ java -version
+ java_ver_output=java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
+ echo java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
+ grep [openjdk|java] version
+ cut -d- -f1
+ awk -F" NR==1 {print $2}
+ jvmver=1.8.0_161
+ JVM_VERSION=1.8.0
+ JVM_PATCH_VERSION=161
+ [ 1.8.0 < 1.8 ]
+ [ 1.8.0 < 1.8 ]
+ echo java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
+ + awkgrep NR==2 {print $1}
 -A 1 java version
+ jvm=Java(TM)
+ JVM_VENDOR=Oracle
+ echo java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
+ awk NR==3 {print $3}
+ JVM_ARCH=64-Bit
{noformat}

OpenJDK patched:

{noformat}
+ java -version
+ java_ver_output=openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
+ echo openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
+ grep [openjdk|java] version
+ cut -d- -f1
+ awk -F" NR==1 {print $2}
+ jvmver=1.8.0_151
+ JVM_VERSION=1.8.0
+ JVM_PATCH_VERSION=151
+ [ 1.8.0 < 1.8 ]
+ [ 1.8.0 < 1.8 ]
+ grep -A 1 [openjdk|java] version
+ awk NR==2 {print $1}
+ echo openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
+ jvm=OpenJDK
+ JVM_VENDOR=OpenJDK
+ + awk NR==3 {print $2}
echo openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
+ JVM_ARCH=64-Bit
{noformat}

Oracle patched:

{noformat}
+ java -version
+ java_ver_output=java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
+ echo java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
+ grep [openjdk|java] version+ cut -d- -f1
+ awk -F" NR==1 {print $2}
+ jvmver=1.8.0_161
+ JVM_VERSION=1.8.0
+ JVM_PATCH_VERSION=161
+ [ 1.8.0 < 1.8 ]
+ [ 1.8.0 < 1.8 ]
+ echo java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
+ grep -A 1 [openjdk|java] version
+ awk NR==2 {print $1}
+ jvm=Java(TM)
+ JVM_VENDOR=Oracle
+ echo+ awk NR==3 {print $3}
 java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
+ JVM_ARCH=64-Bit
{noformat}


> invalid jvm type and architecture [cassandra-env.sh]
> 
>
> Key: CASSANDRA-12793
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12793
> Project: Cassandra
>  Issue Type: Bug
>  Components: Configuration
> Environment: ubuntu 16.04, openjdk 1.8.0_91
>Reporter: Ali Ebrahiminejad
>Priority: Minor
> Fix For: 3.11.x
>
> Attachments: 12793.patch
>
>
> In cassandra-env.sh the part that determines the type of JVM we'll be running 
> on doesn't provide the right answer for 

[jira] [Commented] (CASSANDRA-13259) Use platform specific X.509 default algorithm

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362506#comment-16362506
 ] 

Stefan Podkowinski commented on CASSANDRA-13259:


Lets then remove it from the yaml but keep it in EncryptionOptions. 

I'd suggest to commit this to trunk only, as this is not a bug fix and the 
value can be explicitly set in the yaml, in the hypothetical case SunX509 is 
getting pulled from future JVM releases.

> Use platform specific X.509 default algorithm
> -
>
> Key: CASSANDRA-13259
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13259
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
> Fix For: 4.x
>
>
> We should replace the hardcoded "SunX509" default algorithm and use the JRE 
> default instead. This implementation will currently not work on less popular 
> platforms (e.g. IBM) and won't get any further updates.
> See also:
> https://bugs.openjdk.java.net/browse/JDK-8169745



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13259) Use platform specific X.509 default algorithm

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-13259:
---
   Resolution: Fixed
Fix Version/s: (was: 4.x)
   4.0
   Status: Resolved  (was: Ready to Commit)

Merged as bb9aa098813b7f047f450086e18a78b149bb5349 on trunk

> Use platform specific X.509 default algorithm
> -
>
> Key: CASSANDRA-13259
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13259
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
> Fix For: 4.0
>
>
> We should replace the hardcoded "SunX509" default algorithm and use the JRE 
> default instead. This implementation will currently not work on less popular 
> platforms (e.g. IBM) and won't get any further updates.
> See also:
> https://bugs.openjdk.java.net/browse/JDK-8169745



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14169) Trivial intellij junit run fix

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-14169:
---
   Resolution: Fixed
 Reviewer: Stefan Podkowinski
Fix Version/s: 4.0
   Status: Resolved  (was: Patch Available)

Thanks Jay!
Merged as 7a424bc2a79dce98817054057dd3a479b202f09e to trunk.


> Trivial intellij junit run fix
> --
>
> Key: CASSANDRA-14169
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14169
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Trivial
> Fix For: 4.0
>
>
> Unable to run 
> {{[LegacySSTableTest|https://github.com/apache/cassandra/blob/trunk/test/unit/org/apache/cassandra/io/sstable/LegacySSTableTest.java#L63]}}
>  in the Intellij, because the 
> {{[legacy-sstable-root|https://github.com/apache/cassandra/blob/trunk/test/unit/org/apache/cassandra/io/sstable/LegacySSTableTest.java#L96]}}
>  is not defined.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14223) Provide ability to do custom certificate validations (e.g. hostname validation, certificate revocation checks)

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16371205#comment-16371205
 ] 

Stefan Podkowinski commented on CASSANDRA-14223:


It is already possible to use your own trust manager implementation that will 
validate certificates using your custom validation logic. You'll have to 
register your own security provider in {{java.security}}. The provider needs to 
specify your TrustManagerFactorySpi implementation using a specific algorithm 
name that you afterwards have to set as 
{{server_encryption_options.algorithms}} in {{cassandra.yaml}}. But this should 
not involve any SSLSocket code related changes and is not really Cassandra 
specific at all.

> Provide ability to do custom certificate validations (e.g. hostname 
> validation, certificate revocation checks)
> --
>
> Key: CASSANDRA-14223
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14223
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Ron Blechman
>Priority: Major
>  Labels: security
> Fix For: 4.x
>
>
> Cassandra server should be to be able do additional certificate validations, 
> such as hostname validatation and certificate revocation checking against 
> CRLs and/or using OCSP. 
> One approach couild be to have SSLFactory use SSLContext.getDefault() instead 
> of forcing the creation of a new SSLContext using SSLContext.getInstance().  
> Using the default SSLContext would allow a user to plug in their own custom 
> SSLSocketFactory via the java.security properties file. The custom 
> SSLSocketFactory could create a default SSLContext  that was customized to do 
> any extra validation such as certificate revocation, host name validation, 
> etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13459) Diag. Events: Native transport integration

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-13459:
---
Status: Patch Available  (was: In Progress)

> Diag. Events: Native transport integration
> --
>
> Key: CASSANDRA-13459
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13459
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: CQL
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>  Labels: client-impacting
>
> Events should be consumable by clients that would received subscribed events 
> from the connected node. This functionality is designed to work on top of 
> native transport with minor modifications to the protocol standard (see 
> [original 
> proposal|https://docs.google.com/document/d/1uEk7KYgxjNA0ybC9fOuegHTcK3Yi0hCQN5nTp5cNFyQ/edit?usp=sharing]
>  for further considered options). First we have to add another value for 
> existing event types. Also, we have to extend the protocol a bit to be able 
> to specify a sub-class and sub-type value. E.g. 
> {{DIAGNOSTIC_EVENT(GossiperEvent, MAJOR_STATE_CHANGE_HANDLED)}}. This still 
> has to be worked out and I'd appreciate any feedback.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14241) Apache dtests not passing after pytest/python 3

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368936#comment-16368936
 ] 

Stefan Podkowinski commented on CASSANDRA-14241:


I can't see anything wrong with Jolokia and the provided process dump. The 
dtest tools/jmxutils.py script will disable the PerfDisableSharedMem flag in 
jvm.options on node1, as work around for the mentioned Jolokia issue. The 
process listing does confirm that the JVM is actually started without the 
PerfDisableSharedMem flag, which should allow the agent to attach just fine. 
I've been testing this locally with 1.8.0_151; what java version is used by 
Jenkins?

> Apache dtests not passing after pytest/python 3
> ---
>
> Key: CASSANDRA-14241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14241
> Project: Cassandra
>  Issue Type: Task
>  Components: Testing
>Reporter: Ariel Weisberg
>Priority: Major
>
> Apache dtests are still not running correctly yet with pytest. Most of the 
> tests are running and passing but a solid chunk are still failing and these 
> are tests that don't fail in CircleCI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-12793) invalid jvm type and architecture [cassandra-env.sh]

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski reassigned CASSANDRA-12793:
--

Assignee: (was: Stefan Podkowinski)

> invalid jvm type and architecture [cassandra-env.sh]
> 
>
> Key: CASSANDRA-12793
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12793
> Project: Cassandra
>  Issue Type: Bug
>  Components: Configuration
> Environment: ubuntu 16.04, openjdk 1.8.0_91
>Reporter: Ali Ebrahiminejad
>Priority: Minor
> Fix For: 3.11.x
>
> Attachments: 12793.patch
>
>
> In cassandra-env.sh the part that determines the type of JVM we'll be running 
> on doesn't provide the right answer for openjdk 1.8.0_91.
> value of java_ver_output is "openjdk version "1.8.0_91" OpenJDK Runtime 
> Environment (build 1.8.0_91-8u91-b14-3ubuntu1~16.04.1-b14) OpenJDK 64-Bit 
> Server VM (build 25.91-b14, mixed mode)", yet the command looks for "java 
> version" (jvm=`echo "$java_ver_output" | grep -A 1 'java version' ...) which 
> does not exist.
> I guess it should be replaced with jvm=`echo "$java_ver_output" | grep -A 1 
> '[openjdk|java] version' | awk 'NR==2 {print $1}'`



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13569) Schedule schema pulls just once per endpoint

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-13569:
---
Fix Version/s: (was: 3.11.x)
   (was: 4.x)
   (was: 3.0.x)

> Schedule schema pulls just once per endpoint
> 
>
> Key: CASSANDRA-13569
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13569
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>
> Schema mismatches detected through gossip will get resolved by calling 
> {{MigrationManager.maybeScheduleSchemaPull}}. This method may decide to 
> schedule execution of {{MigrationTask}}, but only after using a 
> {{MIGRATION_DELAY_IN_MS = 6}} delay (for reasons unclear to me). 
> Meanwhile, as long as the migration task hasn't been executed, we'll continue 
> to have schema mismatches reported by gossip and will have corresponding 
> {{maybeScheduleSchemaPull}} calls, which will schedule further tasks with the 
> mentioned delay. Some local testing shows that dozens of tasks for the same 
> endpoint will eventually be executed and causing the same, stormy behavior 
> for this very endpoints.
> My proposal would be to simply not schedule new tasks for the same endpoint, 
> in case we still have pending tasks waiting for execution after 
> {{MIGRATION_DELAY_IN_MS}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13569) Schedule schema pulls just once per endpoint

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-13569:
---
Status: Open  (was: Patch Available)

> Schedule schema pulls just once per endpoint
> 
>
> Key: CASSANDRA-13569
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13569
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> Schema mismatches detected through gossip will get resolved by calling 
> {{MigrationManager.maybeScheduleSchemaPull}}. This method may decide to 
> schedule execution of {{MigrationTask}}, but only after using a 
> {{MIGRATION_DELAY_IN_MS = 6}} delay (for reasons unclear to me). 
> Meanwhile, as long as the migration task hasn't been executed, we'll continue 
> to have schema mismatches reported by gossip and will have corresponding 
> {{maybeScheduleSchemaPull}} calls, which will schedule further tasks with the 
> mentioned delay. Some local testing shows that dozens of tasks for the same 
> endpoint will eventually be executed and causing the same, stormy behavior 
> for this very endpoints.
> My proposal would be to simply not schedule new tasks for the same endpoint, 
> in case we still have pending tasks waiting for execution after 
> {{MIGRATION_DELAY_IN_MS}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-13569) Schedule schema pulls just once per endpoint

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski reassigned CASSANDRA-13569:
--

Assignee: (was: Stefan Podkowinski)

> Schedule schema pulls just once per endpoint
> 
>
> Key: CASSANDRA-13569
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13569
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Stefan Podkowinski
>Priority: Major
>
> Schema mismatches detected through gossip will get resolved by calling 
> {{MigrationManager.maybeScheduleSchemaPull}}. This method may decide to 
> schedule execution of {{MigrationTask}}, but only after using a 
> {{MIGRATION_DELAY_IN_MS = 6}} delay (for reasons unclear to me). 
> Meanwhile, as long as the migration task hasn't been executed, we'll continue 
> to have schema mismatches reported by gossip and will have corresponding 
> {{maybeScheduleSchemaPull}} calls, which will schedule further tasks with the 
> mentioned delay. Some local testing shows that dozens of tasks for the same 
> endpoint will eventually be executed and causing the same, stormy behavior 
> for this very endpoints.
> My proposal would be to simply not schedule new tasks for the same endpoint, 
> in case we still have pending tasks waiting for execution after 
> {{MIGRATION_DELAY_IN_MS}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13487) Generate snapshot packages through Jenkins

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-13487:
---
Attachment: (was: 13487.patch)

> Generate snapshot packages through Jenkins
> --
>
> Key: CASSANDRA-13487
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13487
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Build
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>
> Creating packages through the new docker based build scripts now work pretty 
> much independent from any local environment, as long as docker is available, 
> e.g. also on Jenkins. Having daily snapshots available for deb/rpm artifacts 
> would enable us to provide users dev-releases for testing and validating 
> fixes.
> I've created a branch for the Jenkins integration, which can be found here:
> https://github.com/spodkowinski/cassandra-builds/tree/jenkins_debrpm
> The major issue I'm currently struggling with is the handling of the actual 
> version value. We need to find a way to have Jenkins recognize the correct 
> version for the branch being build. Also we must create $version-SNAPSHOT 
> packages, as builds are not official releases and we should not have any 
> packages for versions that aren't published yet.
> The Debian build process will use the version defined in 
> {{debian/changelog}}. Adding a -SNAPSHOT suffix for the version should work, 
> but this has to be handled manually and care must be taken to change the 
> value back again for a regular release.
> With RPMs, the version must be set for {{cassandra.spec}}, which is currently 
> done by running {noformat}rpmbuild --define="version ${CASSANDRA_VERSION}" 
> -ba ./redhat/cassandra.spec{noformat}, where the version is passed as a 
> parameter by {{build-scripts/cassandra-rpm-packaging.sh}}. Maybe we could 
> grep the version from build.xml here?
> So I wonder if there any way we can keep track of the version in a single 
> place, such as build.xml or CHANGES. Afterwards we also need to enable a 
> SNAPSHOT mode, maybe by setting a Jenkins environment value.
> /cc [~mshuler]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13487) Generate snapshot packages through Jenkins

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-13487:
---
   Resolution: Won't Fix
 Assignee: (was: Stefan Podkowinski)
Fix Version/s: (was: 3.11.x)
   (was: 4.x)
   (was: 3.0.x)
   (was: 2.2.x)
   Status: Resolved  (was: Patch Available)

> Generate snapshot packages through Jenkins
> --
>
> Key: CASSANDRA-13487
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13487
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Build
>Reporter: Stefan Podkowinski
>Priority: Major
>
> Creating packages through the new docker based build scripts now work pretty 
> much independent from any local environment, as long as docker is available, 
> e.g. also on Jenkins. Having daily snapshots available for deb/rpm artifacts 
> would enable us to provide users dev-releases for testing and validating 
> fixes.
> I've created a branch for the Jenkins integration, which can be found here:
> https://github.com/spodkowinski/cassandra-builds/tree/jenkins_debrpm
> The major issue I'm currently struggling with is the handling of the actual 
> version value. We need to find a way to have Jenkins recognize the correct 
> version for the branch being build. Also we must create $version-SNAPSHOT 
> packages, as builds are not official releases and we should not have any 
> packages for versions that aren't published yet.
> The Debian build process will use the version defined in 
> {{debian/changelog}}. Adding a -SNAPSHOT suffix for the version should work, 
> but this has to be handled manually and care must be taken to change the 
> value back again for a regular release.
> With RPMs, the version must be set for {{cassandra.spec}}, which is currently 
> done by running {noformat}rpmbuild --define="version ${CASSANDRA_VERSION}" 
> -ba ./redhat/cassandra.spec{noformat}, where the version is passed as a 
> parameter by {{build-scripts/cassandra-rpm-packaging.sh}}. Maybe we could 
> grep the version from build.xml here?
> So I wonder if there any way we can keep track of the version in a single 
> place, such as build.xml or CHANGES. Afterwards we also need to enable a 
> SNAPSHOT mode, maybe by setting a Jenkins environment value.
> /cc [~mshuler]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-13175) Integrate "Error Prone" Code Analyzer

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski reassigned CASSANDRA-13175:
--

Assignee: (was: Stefan Podkowinski)

> Integrate "Error Prone" Code Analyzer
> -
>
> Key: CASSANDRA-13175
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13175
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Stefan Podkowinski
>Priority: Major
> Attachments: 0001-Add-Error-Prone-code-analyzer.patch, 
> checks-2_2.out, checks-3_0.out, checks-trunk.out
>
>
> I've been playing with [Error Prone|http://errorprone.info/] by integrating 
> it into the build process and to see what kind of warnings it would produce. 
> So far I'm positively impressed by the coverage and usefulness of some of the 
> implemented checks. See attachments for results.
> Unfortunately there are still some issues on how the analyzer is effecting 
> generated code and used guava versions, see 
> [#492|https://github.com/google/error-prone/issues/492]. In case those issues 
> have been solved and the resulting code isn't affected by the analyzer, I'd 
> suggest to add it to trunk with warn only behaviour and some less useful 
> checks disabled. Alternatively a new ant target could be added, maybe with 
> build breaking checks and CI integration.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13724) Externalize and cleanup build.xml dependencies

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-13724:
---
Attachment: 13724-1st-take.patch

> Externalize and cleanup build.xml dependencies
> --
>
> Key: CASSANDRA-13724
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13724
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Build
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
> Fix For: 4.x
>
> Attachments: 13724-1st-take.patch
>
>
> As discussed in CASSANDRA-12996, the build.xml has become pretty messy and 
> could use some cleanup and additional inline comments. Maybe parts of it 
> could be even split up or externalized.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13724) Externalize and cleanup build.xml dependencies

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-13724:
---
Fix Version/s: (was: 4.x)
   Status: Open  (was: Patch Available)

> Externalize and cleanup build.xml dependencies
> --
>
> Key: CASSANDRA-13724
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13724
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Build
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
> Attachments: 13724-1st-take.patch
>
>
> As discussed in CASSANDRA-12996, the build.xml has become pretty messy and 
> could use some cleanup and additional inline comments. Maybe parts of it 
> could be even split up or externalized.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13724) Externalize and cleanup build.xml dependencies

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-13724:
---
Status: Awaiting Feedback  (was: Open)

> Externalize and cleanup build.xml dependencies
> --
>
> Key: CASSANDRA-13724
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13724
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Build
>Reporter: Stefan Podkowinski
>Priority: Minor
> Attachments: 13724-1st-take.patch
>
>
> As discussed in CASSANDRA-12996, the build.xml has become pretty messy and 
> could use some cleanup and additional inline comments. Maybe parts of it 
> could be even split up or externalized.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-13724) Externalize and cleanup build.xml dependencies

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski reassigned CASSANDRA-13724:
--

Assignee: (was: Stefan Podkowinski)

> Externalize and cleanup build.xml dependencies
> --
>
> Key: CASSANDRA-13724
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13724
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Build
>Reporter: Stefan Podkowinski
>Priority: Minor
> Attachments: 13724-1st-take.patch
>
>
> As discussed in CASSANDRA-12996, the build.xml has become pretty messy and 
> could use some cleanup and additional inline comments. Maybe parts of it 
> could be even split up or externalized.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-12870) Calculate pending ranges for identical KS settings just once

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-12870:
---
Attachment: (was: 12870-trunk.patch)

> Calculate pending ranges for identical KS settings just once
> 
>
> Key: CASSANDRA-12870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12870
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>
> The {{TokenMetadata.calculatePendingRanges()}} operation can be very 
> expensive and already has been subject to micro optimization in 
> CASSANDRA-9258. Instead of further optimizing the calculation itself, I'd 
> like to prevent it from being executed as often by only calling it just once 
> for all keyspaces sharing identical replication settings. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-12870) Calculate pending ranges for identical KS settings just once

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-12870:
---
 Assignee: (was: Stefan Podkowinski)
Fix Version/s: (was: 4.x)
   Status: Open  (was: Patch Available)

> Calculate pending ranges for identical KS settings just once
> 
>
> Key: CASSANDRA-12870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12870
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Stefan Podkowinski
>Priority: Major
>
> The {{TokenMetadata.calculatePendingRanges()}} operation can be very 
> expensive and already has been subject to micro optimization in 
> CASSANDRA-9258. Instead of further optimizing the calculation itself, I'd 
> like to prevent it from being executed as often by only calling it just once 
> for all keyspaces sharing identical replication settings. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-14065) Docs: Fix page width exceeding the viewport

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski reassigned CASSANDRA-14065:
--

Assignee: (was: Stefan Podkowinski)

> Docs: Fix page width exceeding the viewport
> ---
>
> Key: CASSANDRA-14065
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14065
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Documentation and Website
>Reporter: Stefan Podkowinski
>Priority: Major
> Fix For: 4.x
>
> Attachments: 0001-Docs-optimize-css-layouting.patch, 
> 0002-site_svn-css.diff, 0003-site_svn-fixnavigation.diff, 14065-trunk.patch
>
>
> Ticket for [#175|https://github.com/apache/cassandra/pull/175] / 
> [#176|https://github.com/apache/cassandra/pull/176].
> The layout seems to adapt more natural after applying the patch with less 
> overlapping content. Seems to fix a real issue with our template.
> However, I'm not really sure about the extra.css changes, as the compile 
> website (build via jekyll) doesn't seem to reference the css file anywhere..



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13569) Schedule schema pulls just once per endpoint

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-13569:
---
Attachment: 13569-trunk.patch

> Schedule schema pulls just once per endpoint
> 
>
> Key: CASSANDRA-13569
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13569
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Stefan Podkowinski
>Priority: Major
> Attachments: 13569-trunk.patch
>
>
> Schema mismatches detected through gossip will get resolved by calling 
> {{MigrationManager.maybeScheduleSchemaPull}}. This method may decide to 
> schedule execution of {{MigrationTask}}, but only after using a 
> {{MIGRATION_DELAY_IN_MS = 6}} delay (for reasons unclear to me). 
> Meanwhile, as long as the migration task hasn't been executed, we'll continue 
> to have schema mismatches reported by gossip and will have corresponding 
> {{maybeScheduleSchemaPull}} calls, which will schedule further tasks with the 
> mentioned delay. Some local testing shows that dozens of tasks for the same 
> endpoint will eventually be executed and causing the same, stormy behavior 
> for this very endpoints.
> My proposal would be to simply not schedule new tasks for the same endpoint, 
> in case we still have pending tasks waiting for execution after 
> {{MIGRATION_DELAY_IN_MS}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-12126) CAS Reads Inconsistencies

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-12126:
---
Status: Open  (was: Patch Available)

> CAS Reads Inconsistencies 
> --
>
> Key: CASSANDRA-12126
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12126
> Project: Cassandra
>  Issue Type: Bug
>  Components: Coordination
>Reporter: sankalp kohli
>Priority: Major
>
> While looking at the CAS code in Cassandra, I found a potential issue with 
> CAS Reads. Here is how it can happen with RF=3
> 1) You issue a CAS Write and it fails in the propose phase. A machine replies 
> true to a propose and saves the commit in accepted filed. The other two 
> machines B and C does not get to the accept phase. 
> Current state is that machine A has this commit in paxos table as accepted 
> but not committed and B and C does not. 
> 2) Issue a CAS Read and it goes to only B and C. You wont be able to read the 
> value written in step 1. This step is as if nothing is inflight. 
> 3) Issue another CAS Read and it goes to A and B. Now we will discover that 
> there is something inflight from A and will propose and commit it with the 
> current ballot. Now we can read the value written in step 1 as part of this 
> CAS read.
> If we skip step 3 and instead run step 4, we will never learn about value 
> written in step 1. 
> 4. Issue a CAS Write and it involves only B and C. This will succeed and 
> commit a different value than step 1. Step 1 value will never be seen again 
> and was never seen before. 
> If you read the Lamport “paxos made simple” paper and read section 2.3. It 
> talks about this issue which is how learners can find out if majority of the 
> acceptors have accepted the proposal. 
> In step 3, it is correct that we propose the value again since we dont know 
> if it was accepted by majority of acceptors. When we ask majority of 
> acceptors, and more than one acceptors but not majority has something in 
> flight, we have no way of knowing if it is accepted by majority of acceptors. 
> So this behavior is correct. 
> However we need to fix step 2, since it caused reads to not be linearizable 
> with respect to writes and other reads. In this case, we know that majority 
> of acceptors have no inflight commit which means we have majority that 
> nothing was accepted by majority. I think we should run a propose step here 
> with empty commit and that will cause write written in step 1 to not be 
> visible ever after. 
> With this fix, we will either see data written in step 1 on next serial read 
> or will never see it which is what we want. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-12126) CAS Reads Inconsistencies

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski reassigned CASSANDRA-12126:
--

Assignee: (was: Stefan Podkowinski)

> CAS Reads Inconsistencies 
> --
>
> Key: CASSANDRA-12126
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12126
> Project: Cassandra
>  Issue Type: Bug
>  Components: Coordination
>Reporter: sankalp kohli
>Priority: Major
>
> While looking at the CAS code in Cassandra, I found a potential issue with 
> CAS Reads. Here is how it can happen with RF=3
> 1) You issue a CAS Write and it fails in the propose phase. A machine replies 
> true to a propose and saves the commit in accepted filed. The other two 
> machines B and C does not get to the accept phase. 
> Current state is that machine A has this commit in paxos table as accepted 
> but not committed and B and C does not. 
> 2) Issue a CAS Read and it goes to only B and C. You wont be able to read the 
> value written in step 1. This step is as if nothing is inflight. 
> 3) Issue another CAS Read and it goes to A and B. Now we will discover that 
> there is something inflight from A and will propose and commit it with the 
> current ballot. Now we can read the value written in step 1 as part of this 
> CAS read.
> If we skip step 3 and instead run step 4, we will never learn about value 
> written in step 1. 
> 4. Issue a CAS Write and it involves only B and C. This will succeed and 
> commit a different value than step 1. Step 1 value will never be seen again 
> and was never seen before. 
> If you read the Lamport “paxos made simple” paper and read section 2.3. It 
> talks about this issue which is how learners can find out if majority of the 
> acceptors have accepted the proposal. 
> In step 3, it is correct that we propose the value again since we dont know 
> if it was accepted by majority of acceptors. When we ask majority of 
> acceptors, and more than one acceptors but not majority has something in 
> flight, we have no way of knowing if it is accepted by majority of acceptors. 
> So this behavior is correct. 
> However we need to fix step 2, since it caused reads to not be linearizable 
> with respect to writes and other reads. In this case, we know that majority 
> of acceptors have no inflight commit which means we have majority that 
> nothing was accepted by majority. I think we should run a propose step here 
> with empty commit and that will cause write written in step 1 to not be 
> visible ever after. 
> With this fix, we will either see data written in step 1 on next serial read 
> or will never see it which is what we want. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Resolved] (CASSANDRA-13314) Config file based SSL settings

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski resolved CASSANDRA-13314.

   Resolution: Won't Fix
Fix Version/s: (was: 4.x)

Closing this ticket for now, as we haven't moved on in the discussion for 
months and the idea is likely obsolete by now, with the current netty based 
stack.

> Config file based SSL settings
> --
>
> Key: CASSANDRA-13314
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13314
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration, Tools
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
>
> As follow up of CASSANDRA-13259, I'd like to continue discussing how we can 
> make SSL less awkward to use and further move SSL related code out of our 
> code base. Currently we construct our own SSLContext in SSLFactory based on 
> EncryptionOptions passed by the MessagingService or any individual tool where 
> we need to offer SSL support. This leads to a situation where the user has 
> not only to learn how to enable the correct settings in cassandra.yaml, but 
> these settings must also be reflected in each tool's own command line 
> options. As argued in CASSANDRA-13259, these settings could be done as well 
> by setting the appropriate system and security properties 
> ([overview|http://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#InstallationAndCustomization])
>  and we should just point the user to the right files to do that (jvm.options 
> and java.security) and make sure that daemon and all affected tools will 
> source them. 
> Since giving this a quick try on my WIP branch, I've noticed the following 
> issues in doing so:
> * Keystore passwords will show up in process list 
> (-Djavax.net.ssl.keyStorePassword=..). We should keep the password setting in 
> cassandra.yaml and clis and do a System.setProperty() if they have been 
> provided. 
> * It's only possible to configure settings for a single default 
> key-/truststore. Since we currently allow configuring both 
> ServerEncryptionOptions and ClientEncryptionOptions with different settings, 
> we'd have to make this a breaking change. I don't really see why you would 
> want to use different stores for node-to-node and node-to-client, but that 
> wouldn't be possible anymore. 
> * This would probably only make sense if we really remove the affected CLI 
> options, or we'll end up with just another way to configure this stuff. This 
> will break existing scripts and obsolete existing documentation.
> Any opinions?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-11684) Cleanup key ranges during compaction

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski reassigned CASSANDRA-11684:
--

Assignee: (was: Stefan Podkowinski)

> Cleanup key ranges during compaction
> 
>
> Key: CASSANDRA-11684
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11684
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Stefan Podkowinski
>Priority: Major
>
> Currently cleanup is considered an optional, manual operation that users are 
> told to run to free disk space after a node was affected by topology changes. 
> However, unmanaged key ranges could also end up on a node through other ways, 
> e.g. manual added sstable files by an admin. 
> I'm also not sure unmanaged data is really that harmless and cleanup should 
> really be optional, if you don't need to reclaim the disk space. When it 
> comes to repairs, users are expected to purge a node after downtime in case 
> it was not fully covered by a repair within gc_grace afterwards, in order to 
> avoid re-introducing deleted data. But the same could happen with unmanaged 
> data, e.g. after topology changes activate unmanaged ranges again or after 
> restoring backups.
> I'd therefor suggest to avoid rewriting key ranges no longer belonging to a 
> node and older than gc_grace during compactions. 
> Maybe we could also introduce another CLEANUP_COMPACTION operation to find 
> candidates based on SSTable.first/last in case we don't have pending regular 
> or tombstone compactions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13023) Add droppable tombstone metrics

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-13023:
---
Assignee: (was: Stefan Podkowinski)
  Status: Open  (was: Patch Available)

> Add droppable tombstone metrics
> ---
>
> Key: CASSANDRA-13023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13023
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Stefan Podkowinski
>Priority: Major
>
> Currently it's only possible to inspect the ratio of droppable tombstones on 
> sstable basis using the {{sstablemetadata}} tool. It would be very useful to 
> have this information provided as part of Cassandra metrics as well, e.g. to 
> get an idea on the effectiveness of tombstone eviction during compactions 
> over time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13023) Add droppable tombstone metrics

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-13023:
---
Attachment: (was: 13023-3.0.patch)

> Add droppable tombstone metrics
> ---
>
> Key: CASSANDRA-13023
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13023
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>
> Currently it's only possible to inspect the ratio of droppable tombstones on 
> sstable basis using the {{sstablemetadata}} tool. It would be very useful to 
> have this information provided as part of Cassandra metrics as well, e.g. to 
> get an idea on the effectiveness of tombstone eviction during compactions 
> over time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14241) Apache dtests not passing after pytest/python 3

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367891#comment-16367891
 ] 

Stefan Podkowinski commented on CASSANDRA-14241:


I stumbled upon this through the failed dtest linked in the 
[PR|https://github.com/riptano/ccm/pull/664]. My assumption was that it would 
be preferable to restore the old behaviour, by having 
`handle_external_tool_process` return a string instead of binary value (which 
would be of little use). 

> Apache dtests not passing after pytest/python 3
> ---
>
> Key: CASSANDRA-14241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14241
> Project: Cassandra
>  Issue Type: Task
>  Components: Testing
>Reporter: Ariel Weisberg
>Priority: Major
>
> Apache dtests are still not running correctly yet with pytest. Most of the 
> tests are running and passing but a solid chunk are still failing and these 
> are tests that don't fail in CircleCI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14064) Allow using file based certificates instead of keystores

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-14064:
---
Status: Patch Available  (was: In Progress)

> Allow using file based certificates instead of keystores
> 
>
> Key: CASSANDRA-14064
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14064
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Streaming and Messaging
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>  Labels: security
> Fix For: 4.x
>
>
> The requirement of having to use a secure archive (keystore) for your 
> certificates and keys is not very common outside the Java ecosystem. Most 
> servers will accept individual certificate and key files and will come with 
> instructions how to generate those using openssl. This should also be an 
> option for Cassandra for users who see no reason in additionally having to 
> deal with keystores.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14134) Migrate dtests to use pytest and python3

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16309483#comment-16309483
 ] 

Stefan Podkowinski commented on CASSANDRA-14134:


There are a couple of tests left that need renaming to get picked up by pytest. 
Class names need to start with {{Test}} now.

{noformat}
> egrep '^class [^(]+\(Tester\)' *.py | grep -v 'class Test'
cql_test.py:class CQLTester(Tester):
delete_insert_test.py:class DeleteInsertTest(Tester):
json_test.py:class ToJsonSelectTests(Tester):
json_test.py:class FromJsonUpdateTests(Tester):
json_test.py:class FromJsonSelectTests(Tester):
json_test.py:class FromJsonInsertTests(Tester):
json_test.py:class FromJsonDeleteTests(Tester):
json_test.py:class JsonFullRowInsertSelect(Tester):
native_transport_ssl_test.py:class NativeTransportSSL(Tester):
paging_test.py:class BasePagingTester(Tester):
replace_address_test.py:class BaseReplaceAddressTest(Tester):
replication_test.py:class ReplicationTest(Tester):
replication_test.py:class SnitchConfigurationUpdateTest(Tester):
snapshot_test.py:class SnapshotTester(Tester):
sstable_generation_loading_test.py:class BaseSStableLoaderTest(Tester):
sstableutil_test.py:class SSTableUtilTest(Tester):
thrift_hsha_test.py:class ThriftHSHATest(Tester):
thrift_test.py:class ThriftTester(Tester):
{noformat}


> Migrate dtests to use pytest and python3
> 
>
> Key: CASSANDRA-14134
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14134
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Testing
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
>
> h4. Get the C* dtests running on the pytest framework.
> C* DTests currently run using the python test framework nosetest. This 
> framework has been largely abandoned with no releases since 2015 and a 
> general strong consensus in the python community that pytest is the future.
> h4. Why should we do this.
> Currently (and historically) dtests have always been difficult to run, flaky 
> and unpredictable in CI environments, and almost impossible to debug.
> On November 28th, 2017, I proposed on the dev@ list that we move the dtests 
> from nosetests to pytests. I got replies from Jon Haddad, Philip Thompson, 
> and kurt greaves with really only "+1" like replies to the proposal.
> Since then I've been working pretty much non stop to complete the large 
> refactor of dtests to pytests. As part of this effort (and due to the 
> migration tools that exist require it) I also ported the code to python3 
> (from the current python 2.7 based code-base).
> h4. High-level summary of key changes, improvements, and new features.
> * Migrate dtests from executing using the nosetest framework to pytest
> * Port the entire code base from Python 2.7 to Python 3.6
> * Update run_dtests.py to work with pytest
> * Add --dtest-print-tests-only option to run_dtests.py to get easily parsable 
> list of all available collected tests
> * Update README.md for executing the dtests with pytest
> * Add new debugging tips section to README.md to help with some basics of 
> debugging python3 and pytest
> * Migrate all existing Enviornment Variable usage as a means to control dtest 
> operation modes to argparse command line options with documented help on each 
> toggles intended usage
> * Migration of old unitTest and nose based test structure to modern pytest 
> fixture approach
> * Automatic detection of physical system resources to automatically determine 
> if @pytest.mark.resource_intensive annotated tests should be collected and 
> run on the system where they are being executed
> * new pytest fixture replacements for @since and @pytest.mark.upgrade_test 
> annotations
> * Migration to python logging framework
> * Upgrade thrift bindings to latest version with full python3 compatibility
> * Remove deprecated cql and pycassa dependencies and migrate any remaining 
> tests to fully remove those dependencies
> * Fixed dozens of tests that would hang the pytest framework forever when run 
> in CI enviornments
> * Ran code nearly 300 times in CircleCI during the migration and to find, 
> identify, and fix any tests capable of hanging CI
> * Upgrade Tests do not yet run in CI and still need additional migration work 
> (although all upgrade test classes compile successfully)
> I started with the *nose2pytest* [https://github.com/pytest-dev/nose2pytest] 
> migration tool. As this required python 3 language support I found myself 
> down the 2to3 python migration path. While painful to do this, the benefits 
> of python3 over python2.7 are numerous and moving to python3 for the 
> additional debugging tools now available to use when fixing dtests makes the 
> effort worth it for that reason alone!
> After the automated tools did their thing I began what was a much longer and 

[jira] [Comment Edited] (CASSANDRA-14134) Migrate dtests to use pytest and python3

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16309483#comment-16309483
 ] 

Stefan Podkowinski edited comment on CASSANDRA-14134 at 1/3/18 11:10 AM:
-

There are a couple of tests left that need renaming to get picked up by pytest. 
Class names need to start with {{Test}} now if they have {{test_}} methods that 
should be run and are not purely used as superclass.

{noformat}
> find . -name \*.py |xargs egrep '^class [^(]+\(Tester\)' | grep -v 'class 
> Test'
./thrift_hsha_test.py:class ThriftHSHATest(Tester):
./cql_test.py:class CQLTester(Tester):
./delete_insert_test.py:class DeleteInsertTest(Tester):
./sstable_generation_loading_test.py:class BaseSStableLoaderTest(Tester):
./replace_address_test.py:class BaseReplaceAddressTest(Tester):
./snapshot_test.py:class SnapshotTester(Tester):
./paging_test.py:class BasePagingTester(Tester):
./replication_test.py:class ReplicationTest(Tester):
./replication_test.py:class SnitchConfigurationUpdateTest(Tester):
./thrift_test.py:class ThriftTester(Tester):
./cqlsh_tests/cqlsh_copy_tests.py:class CqlshCopyTest(Tester):
./cqlsh_tests/cqlsh_tests.py:class CqlshSmokeTest(Tester):
./cqlsh_tests/cqlsh_tests.py:class CqlLoginTest(Tester):
./sstableutil_test.py:class SSTableUtilTest(Tester):
./upgrade_tests/compatibility_flag_test.py:class CompatibilityFlagTest(Tester):
./upgrade_tests/thrift_upgrade_test.py:class UpgradeSuperColumnsThrough(Tester):
./upgrade_tests/upgrade_through_versions_test.py:class UpgradeTester(Tester):
./upgrade_tests/upgrade_compact_storage.py:class 
UpgradeSuperColumnsThrough(Tester):
./json_test.py:class ToJsonSelectTests(Tester):
./json_test.py:class FromJsonUpdateTests(Tester):
./json_test.py:class FromJsonSelectTests(Tester):
./json_test.py:class FromJsonInsertTests(Tester):
./json_test.py:class FromJsonDeleteTests(Tester):
./json_test.py:class JsonFullRowInsertSelect(Tester):
./native_transport_ssl_test.py:class NativeTransportSSL(Tester):
./repair_tests/preview_repair_test.py:class PreviewRepairTest(Tester):
./repair_tests/repair_test.py:class BaseRepairTest(Tester):
{noformat}




was (Author: spo...@gmail.com):
There are a couple of tests left that need renaming to get picked up by pytest. 
Class names need to start with {{Test}} now.

{noformat}
> egrep '^class [^(]+\(Tester\)' *.py | grep -v 'class Test'
cql_test.py:class CQLTester(Tester):
delete_insert_test.py:class DeleteInsertTest(Tester):
json_test.py:class ToJsonSelectTests(Tester):
json_test.py:class FromJsonUpdateTests(Tester):
json_test.py:class FromJsonSelectTests(Tester):
json_test.py:class FromJsonInsertTests(Tester):
json_test.py:class FromJsonDeleteTests(Tester):
json_test.py:class JsonFullRowInsertSelect(Tester):
native_transport_ssl_test.py:class NativeTransportSSL(Tester):
paging_test.py:class BasePagingTester(Tester):
replace_address_test.py:class BaseReplaceAddressTest(Tester):
replication_test.py:class ReplicationTest(Tester):
replication_test.py:class SnitchConfigurationUpdateTest(Tester):
snapshot_test.py:class SnapshotTester(Tester):
sstable_generation_loading_test.py:class BaseSStableLoaderTest(Tester):
sstableutil_test.py:class SSTableUtilTest(Tester):
thrift_hsha_test.py:class ThriftHSHATest(Tester):
thrift_test.py:class ThriftTester(Tester):
{noformat}


> Migrate dtests to use pytest and python3
> 
>
> Key: CASSANDRA-14134
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14134
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Testing
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
>
> h4. Get the C* dtests running on the pytest framework.
> C* DTests currently run using the python test framework nosetest. This 
> framework has been largely abandoned with no releases since 2015 and a 
> general strong consensus in the python community that pytest is the future.
> h4. Why should we do this.
> Currently (and historically) dtests have always been difficult to run, flaky 
> and unpredictable in CI environments, and almost impossible to debug.
> On November 28th, 2017, I proposed on the dev@ list that we move the dtests 
> from nosetests to pytests. I got replies from Jon Haddad, Philip Thompson, 
> and kurt greaves with really only "+1" like replies to the proposal.
> Since then I've been working pretty much non stop to complete the large 
> refactor of dtests to pytests. As part of this effort (and due to the 
> migration tools that exist require it) I also ported the code to python3 
> (from the current python 2.7 based code-base).
> h4. High-level summary of key changes, improvements, and new features.
> * Migrate dtests from executing using the nosetest framework to pytest
> * Port the entire code base from Python 2.7 to Python 3.6
> * Update run_dtests.py 

[jira] [Commented] (CASSANDRA-14102) Vault support for transparent data encryption

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311410#comment-16311410
 ] 

Stefan Podkowinski commented on CASSANDRA-14102:


I've now pushed a few commits to address the concerns expressed in my previous 
comment. Creating new {{EncryptionContext}} instances will now use the same 
cipher factory and key caches with less generated garbage.

> Vault support for transparent data encryption
> -
>
> Key: CASSANDRA-14102
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14102
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>  Labels: encryption
> Fix For: 4.x
>
>
> Transparent data encryption provided by CASSANDRA-9945 can currently be used 
> for commitlog and hints. The default {{KeyProvider}} implementation that we 
> ship allows to use a local keystore for storing and retrieving keys. Thanks 
> to the pluggable handling of the {{KeyStore}} provider and basic Vault 
> related classes introduced in CASSANDRA-13971, a Vault based implementation 
> can be provided as {{KeyProvider}} as well. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14528) Provide stacktraces for various error logs

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-14528:
---
   Resolution: Fixed
Fix Version/s: (was: 4.x)
   4.0
   Status: Resolved  (was: Patch Available)

Committed as 0f5443d9c

> Provide stacktraces for various error logs
> --
>
> Key: CASSANDRA-14528
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14528
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
> Fix For: 4.0
>
>
> We should reintroduce some stack traces that have gone missing since 
> CASSANDRA-13723 (ba87ab4e954ad2). The cleanest way would probably to use 
> {{String.format}} for any custom messages, e.g. 
> {{logger.error(String.format("Error using param {}", param), e)}}, so we make 
> this more implicit and robust for coming api changes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14298) cqlshlib tests broken on b.a.o

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567970#comment-16567970
 ] 

Stefan Podkowinski commented on CASSANDRA-14298:


I've been able to get a dtest run to finish, after getting back to this issue.

https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/597/testReport/

Looks like there are still encoding issues left and {{TestCqlsh.setUpClass}} 
isn't executed at all without an {{@pytest.fixture}} annotation. Even when it 
does, it doesn't fix the error for me locally.

> cqlshlib tests broken on b.a.o
> --
>
> Key: CASSANDRA-14298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14298
> Project: Cassandra
>  Issue Type: Bug
>  Components: Build, Testing
>Reporter: Stefan Podkowinski
>Assignee: Patrick Bannister
>Priority: Major
>  Labels: cqlsh, dtest
> Attachments: CASSANDRA-14298-old.txt, CASSANDRA-14298.txt, 
> cqlsh_tests_notes.md
>
>
> It appears that cqlsh-tests on builds.apache.org on all branches stopped 
> working since we removed nosetests from the system environment. See e.g. 
> [here|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-cqlsh-tests/458/cython=no,jdk=JDK%201.8%20(latest),label=cassandra/console].
>  Looks like we either have to make nosetests available again or migrate to 
> pytest as we did with dtests. Giving pytest a quick try resulted in many 
> errors locally, but I haven't inspected them in detail yet. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14597) [dtest] snapshot_test.TestArchiveCommitlog

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-14597:
---
Attachment: 3_11_3.txt
3_0.txt
2_2.txt

> [dtest] snapshot_test.TestArchiveCommitlog
> --
>
> Key: CASSANDRA-14597
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14597
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jason Brown
>Priority: Minor
>  Labels: dtest
> Attachments: 2_2.txt, 3_0.txt, 3_11_3.txt
>
>
> All TestArchiveCommitLog dtests fail on 3.0, but no other branches. Output 
> from pytest error:
> {noformat}
> assert (
> time.time() <= stop_time), "It's been over a {s}s and we 
> haven't written a new " + \
> >   "commitlog segment. Something is wrong.".format(s=timeout)
> E   AssertionError: It's been over a {s}s and we haven't written a 
> new commitlog segment. Something is wrong.
> tools/hacks.py:61: AssertionError
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14597) [dtest] snapshot_test.TestArchiveCommitlog

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563537#comment-16563537
 ] 

Stefan Podkowinski commented on CASSANDRA-14597:


Looks like the test itself works fine, but the log allocation behaviour is 
changed across different versions and 3.0 doesn't allocate a new log to make 
the test pass.

> [dtest] snapshot_test.TestArchiveCommitlog
> --
>
> Key: CASSANDRA-14597
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14597
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jason Brown
>Priority: Minor
>  Labels: dtest
> Attachments: 2_2.txt, 3_0.txt, 3_11_3.txt
>
>
> All TestArchiveCommitLog dtests fail on 3.0, but no other branches. Output 
> from pytest error:
> {noformat}
> assert (
> time.time() <= stop_time), "It's been over a {s}s and we 
> haven't written a new " + \
> >   "commitlog segment. Something is wrong.".format(s=timeout)
> E   AssertionError: It's been over a {s}s and we haven't written a 
> new commitlog segment. Something is wrong.
> tools/hacks.py:61: AssertionError
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14597) [dtest] snapshot_test.TestArchiveCommitlog

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-14597:
---
Status: Patch Available  (was: Open)

> [dtest] snapshot_test.TestArchiveCommitlog
> --
>
> Key: CASSANDRA-14597
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14597
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jason Brown
>Priority: Minor
>  Labels: dtest
> Attachments: 2_2.txt, 3_0.txt, 3_11_3.txt
>
>
> All TestArchiveCommitLog dtests fail on 3.0, but no other branches. Output 
> from pytest error:
> {noformat}
> assert (
> time.time() <= stop_time), "It's been over a {s}s and we 
> haven't written a new " + \
> >   "commitlog segment. Something is wrong.".format(s=timeout)
> E   AssertionError: It's been over a {s}s and we haven't written a 
> new commitlog segment. Something is wrong.
> tools/hacks.py:61: AssertionError
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14597) [dtest] snapshot_test.TestArchiveCommitlog

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563594#comment-16563594
 ] 

Stefan Podkowinski commented on CASSANDRA-14597:


2.2 seems to consume space in segments much faster compared to 3.x, so it's 
allocating a new segment within shorter time and the test passes.

3.x allocates a new segment early, while still writing into an existing one. 
Maybe this behaviour has been changed in CASSANDRA-10202?

3.0 will slowly continue to write to the segment and eventually will allocate a 
new segment if the timeout is sufficiently long. I'd therefor propose to change 
the timeout to 120 seconds to make the test pass.

> [dtest] snapshot_test.TestArchiveCommitlog
> --
>
> Key: CASSANDRA-14597
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14597
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jason Brown
>Priority: Minor
>  Labels: dtest
> Attachments: 2_2.txt, 3_0.txt, 3_11_3.txt
>
>
> All TestArchiveCommitLog dtests fail on 3.0, but no other branches. Output 
> from pytest error:
> {noformat}
> assert (
> time.time() <= stop_time), "It's been over a {s}s and we 
> haven't written a new " + \
> >   "commitlog segment. Something is wrong.".format(s=timeout)
> E   AssertionError: It's been over a {s}s and we haven't written a 
> new commitlog segment. Something is wrong.
> tools/hacks.py:61: AssertionError
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14597) [dtest] snapshot_test.TestArchiveCommitlog

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563537#comment-16563537
 ] 

Stefan Podkowinski edited comment on CASSANDRA-14597 at 7/31/18 12:36 PM:
--

Looks like the test itself works fine, but the segment allocation behaviour is 
changed across different versions and 3.0 doesn't allocate a new segment within 
the timeout to make the test pass.


was (Author: spo...@gmail.com):
Looks like the test itself works fine, but the log allocation behaviour is 
changed across different versions and 3.0 doesn't allocate a new log to make 
the test pass.

> [dtest] snapshot_test.TestArchiveCommitlog
> --
>
> Key: CASSANDRA-14597
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14597
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jason Brown
>Priority: Minor
>  Labels: dtest
> Attachments: 2_2.txt, 3_0.txt, 3_11_3.txt
>
>
> All TestArchiveCommitLog dtests fail on 3.0, but no other branches. Output 
> from pytest error:
> {noformat}
> assert (
> time.time() <= stop_time), "It's been over a {s}s and we 
> haven't written a new " + \
> >   "commitlog segment. Something is wrong.".format(s=timeout)
> E   AssertionError: It's been over a {s}s and we haven't written a 
> new commitlog segment. Something is wrong.
> tools/hacks.py:61: AssertionError
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14598) [dtest] flakey test: test_decommissioned_node_cant_rejoin - topology_test.TestTopology

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563460#comment-16563460
 ] 

Stefan Podkowinski commented on CASSANDRA-14598:


Passes on b.a.o:

[https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-3.0-dtest/lastCompletedBuild/testReport/topology_test/TestTopology/]

 

> [dtest] flakey test: test_decommissioned_node_cant_rejoin - 
> topology_test.TestTopology
> --
>
> Key: CASSANDRA-14598
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14598
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jason Brown
>Priority: Minor
>  Labels: dtest
>
> Only saw this fail on 3.0, but looks like a problem with the dtest itself 
> (ubder some failure scenario). Output from pytest error:
> {noformat}
> > assert re.search(rejoin_err,
>  '\n'.join(['\n'.join(err_list) for err_list in 
> node3.grep_log_for_errors()]), re.MULTILINE)
> E AssertionError: assert None
> E + where None = ('This node was 
> decommissioned and will not rejoin the ring', '', )
> E + where  = re.search
> E + and '' = ([])
> E + where  = '\n'.join
> E + and  = re.MULTILINE
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13639) SSTableLoader always uses hostname to stream files from

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566565#comment-16566565
 ] 

Stefan Podkowinski commented on CASSANDRA-13639:


The {{BulkLoadConnectionFactory}} behaviour changed significantly after 
CASSANDRA-12229. Is the described issue still reproducable with latest trunk?

> SSTableLoader always uses hostname to stream files from
> ---
>
> Key: CASSANDRA-13639
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13639
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Jan Karlsson
>Assignee: Jan Karlsson
>Priority: Major
> Fix For: 4.x
>
> Attachments: 13639-trunk
>
>
> I stumbled upon an issue where SSTableLoader was ignoring our routing by 
> using the wrong interface to send the SSTables to the other nodes. Looking at 
> the code, it seems that we are using FBUtilities.getLocalAddress() to fetch 
> out the hostname, even if the yaml file specifies a different host. I am not 
> sure why we call this function instead of using the routing by leaving it 
> blank, perhaps someone could enlighten me.
> This behaviour comes from the fact that we use a default created 
> DatabaseDescriptor which does not set the values for listenAddress and 
> listenInterface. This causes the aforementioned function to retrieve the 
> hostname at all times, even if it is not the interface used in the yaml file.
> I propose we break out the function that handles listenAddress and 
> listenInterface and call it so that listenAddress or listenInterface is 
> getting populated in the DatabaseDescriptor.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13457) Diag. Events: Add base classes

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574508#comment-16574508
 ] 

Stefan Podkowinski commented on CASSANDRA-13457:


"The toString() changes should not have been part of the commit for this 
ticket, but for CASSANDRA-13668 instead." - I have to correct myself there, the 
changes are related to the SchemaAnnouncementEvent and are just fine as part of 
the commit. Doesn't have to do anything with the auditing events.

> Diag. Events: Add base classes
> --
>
> Key: CASSANDRA-13457
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13457
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core, Observability
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>
> Base ticket for adding classes that will allow you to implement and subscribe 
> to events.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Resolved] (CASSANDRA-13459) Diag. Events: Native transport integration

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski resolved CASSANDRA-13459.

Resolution: Won't Fix

Obsoleted for now by CASSANDRA-14435.

> Diag. Events: Native transport integration
> --
>
> Key: CASSANDRA-13459
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13459
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: CQL
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>  Labels: client-impacting
>
> Events should be consumable by clients that would received subscribed events 
> from the connected node. This functionality is designed to work on top of 
> native transport with minor modifications to the protocol standard (see 
> [original 
> proposal|https://docs.google.com/document/d/1uEk7KYgxjNA0ybC9fOuegHTcK3Yi0hCQN5nTp5cNFyQ/edit?usp=sharing]
>  for further considered options). First we have to add another value for 
> existing event types. Also, we have to extend the protocol a bit to be able 
> to specify a sub-class and sub-type value. E.g. 
> {{DIAGNOSTIC_EVENT(GossiperEvent, MAJOR_STATE_CHANGE_HANDLED)}}. This still 
> has to be worked out and I'd appreciate any feedback.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13668) Diag. events for user audit logging

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-13668:
---
Status: Patch Available  (was: In Progress)

Patch squashed and tested.

* [github|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13668]
* [circleci|https://circleci.com/gh/spodkowinski/cassandra/380]
* 
[dtests|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/609/]


> Diag. events for user audit logging
> ---
>
> Key: CASSANDRA-13668
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13668
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
> Fix For: 4.x
>
>
> With the availability of CASSANDRA-13459, any native transport enabled client 
> will be able to subscribe to internal Cassandra events. External tools can 
> take advantage by monitoring these events in various ways. Use-cases for this 
> can be e.g. auditing tools for compliance and security purposes.
> The scope of this ticket is to add diagnostic events that are raised around 
> authentication and CQL operations. These events can then be consumed and used 
> by external tools to implement a Cassandra user auditing solution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13472) Diag. Events: Create event viewer tool

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569132#comment-16569132
 ] 

Stefan Podkowinski commented on CASSANDRA-13472:


I'd still love to have a CLI tool for just creating a dump of all collected 
events locally. But since CASSANDRA-14435, we have to use JMX for that, so a 
small Python tool will probably not be enough. 

> Diag. Events: Create event viewer tool
> --
>
> Key: CASSANDRA-13472
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13472
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Tools
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
> Attachments: diag.log
>
>
> Starting with CASSANDRA-13459 we'll be able to subscribe to events for 
> debugging and troubleshooting purposes. It would be nice to ship a basic CLI  
> tool with Cassandra that enables users to dump events as JSON to stdout. This 
> very simple tool will make use of a patched Python client driver that will 
> work with the new diagnostic events native transport support.
> Invocation will simply look something like this ./diagview event_class ... 
> See attached diag.log for example json output produced by starting and 
> stopping another node.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13457) Diag. Events: Add base classes

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569122#comment-16569122
 ] 

Stefan Podkowinski commented on CASSANDRA-13457:


All changes have now been squashed on a separate branch and tested. 

* [github|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13457]
* [circleci|https://circleci.com/gh/spodkowinski/cassandra/381]
* 
[dtests|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/607/]

[~jasobrown], do you want to take the chance to verify the impact of these 
changes, while having the feature disabled, or any other aspect of the code for 
that matter?


> Diag. Events: Add base classes
> --
>
> Key: CASSANDRA-13457
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13457
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core, Observability
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>
> Base ticket for adding classes that will allow you to implement and subscribe 
> to events.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14435) Diag. Events: JMX events

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569129#comment-16569129
 ] 

Stefan Podkowinski commented on CASSANDRA-14435:


The latest version of the code has been squashed and tested. It now basically 
follows the design as described in my previous post.

* [github|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-14435]
* [circleci|https://circleci.com/gh/spodkowinski/cassandra/382]
* 
[dtests|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/602/]


> Diag. Events: JMX events
> 
>
> Key: CASSANDRA-14435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14435
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Observability
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
> Fix For: 4.x
>
>
> Nodes currently use JMX events for progress reporting on bootstrap and 
> repairs. This might also be an option to expose diagnostic events to external 
> subscribers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13472) Diag. Events: Create event viewer tool

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-13472:
---
Priority: Minor  (was: Major)

> Diag. Events: Create event viewer tool
> --
>
> Key: CASSANDRA-13472
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13472
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Tools
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
> Attachments: diag.log
>
>
> Starting with CASSANDRA-13459 we'll be able to subscribe to events for 
> debugging and troubleshooting purposes. It would be nice to ship a basic CLI  
> tool with Cassandra that enables users to dump events as JSON to stdout. This 
> very simple tool will make use of a patched Python client driver that will 
> work with the new diagnostic events native transport support.
> Invocation will simply look something like this ./diagview event_class ... 
> See attached diag.log for example json output produced by starting and 
> stopping another node.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13457) Diag. Events: Add base classes

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573293#comment-16573293
 ] 

Stefan Podkowinski commented on CASSANDRA-13457:


"Config.diagnostic_events_enabled needs to be volatile if you will modify it at 
runtime via JMX."

Will amend before committing, thx!

"nit: TokenAllocatorEvent's ctor has a huge parameter list."

One of the requirements for raising events was to generate as little garbage as 
possible. Some events, such as SchemaEvents, are rare enough to justify 
additional object allocations, but I'd also like to keep class design 
consistent and not having to use a constructor based approach for more frequent 
events again.

"DiagnosticEventService.publish() is invoked on the same thread as the caller."

I'd expect devs to be aware of this when creating new consumers and not block 
the caller. E.g. the DiagnosticEventMemoryStore uses a concurrent map for this. 
This is a non-pluggable part of the code (at least before CASSANDRA-13460) and 
I can't think of any way we suddenly would have consumers that would be 
implemented without seeing this issue.

"Please document why the map's value type of the return from 
DiagnosticEvent.toMap() is Serializable."

Will add some more docs, but really this method is only supposed to return 
details on the event that can be used by external consumers. Best to just look 
at some of the existing implementations to get a better idea on that.

"I'm not sure what to think about DiagnosticEvent.getSource()."

It's currently used in CASSANDRA-13458 to correlate events. Ideally this would 
be a session or any other transaction that you like to correlate events on.

"Gossiper - I would rather have callers ask for immutable copies, via getter 
methods ... "

GossiperEvent will create an immutable copies in the ctor. According to our 
style guide, getters should be avoided in favor of accessing members directly.

"You've added toString() implementations to about 10 classes."

The toString() changes should not have been part of the commit for this ticket, 
but for CASSANDRA-13668 instead. Good catch. It has been implemented to provide 
a short textual representation about the kind of operation that happened, as 
part of an auditing event. It should be short an concise, but I don't know what 
else to put there that would totally avoid breaking any potential consumers 
relying on it. It's intended to be used for information purposes for users at 
this point. That being said, there's no guarantees any events provided to 
external consumers will never be subject to change. They will most likely do 
and it's explicitly not the goal of diagnostic events to consume them for 
executing control logic, but just use them for debugging and troubleshooting 
instead.

"DiagnosticEvent uses millis for for the timestamp, and in particular, 
System.currentTimeMillis()."

This is the event creation time, determined on best effort basis with no formal 
guarantees to it whatsoever. It's really just to tell users when the event 
happened.

> Diag. Events: Add base classes
> --
>
> Key: CASSANDRA-13457
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13457
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core, Observability
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>
> Base ticket for adding classes that will allow you to implement and subscribe 
> to events.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13458) Diag. Events: Add unit testing support

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-13458:
---
Status: Open  (was: Patch Available)

> Diag. Events: Add unit testing support
> --
>
> Key: CASSANDRA-13458
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13458
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Testing
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>
> Diagnostic events will improve unit testing by
> * providing test execution control instances based on CompletableFutures (see 
> [PendingRangeCalculatorServiceTest.java|https://github.com/spodkowinski/cassandra/blob/WIP-13458/test/unit/org/apache/cassandra/gms/PendingRangeCalculatorServiceTest.java])
>  
> * validate state and behavior by allowing you to inspect generated events 
> (see 
> [HintsServiceEventsTest.java|https://github.com/spodkowinski/cassandra/blob/WIP-13458/test/unit/org/apache/cassandra/hints/HintsServiceEventsTest.java])
> See included 
> [testing.rst|https://github.com/spodkowinski/cassandra/blob/WIP-13458/doc/source/development/testing.rst#diagnostic-events-40]
>  draft for more details. Let me know if this would be useful for you as a 
> developer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13457) Diag. Events: Add base classes

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582478#comment-16582478
 ] 

Stefan Podkowinski commented on CASSANDRA-13457:


Rebased and pushed some changes for CASSANDRA-13457 
([a5be8dc|https://github.com/spodkowinski/cassandra/commit/a5be8dc0214dff4974f5d299c9eebc69f04b650b]),
 CASSANDRA-14435 
([2d3549c|https://github.com/spodkowinski/cassandra/commit/2d3549c53ce11d5d94951f093b5db15a44f71325]),
 CASSANDRA-13668 
([c26219d|https://github.com/apache/cassandra/commit/c26219d36e611f0c72c1e82fc3d6dd9753571b2b])
 (last one rebase only).

* Removed {{enableDiagnostics()}} in {{DiagnosticEventServiceMBean}} to enforce 
explicit opt-in via cassandra yaml, as exposed data could contain security 
sensitive details. The {{disableDiagnostics()}} method still exists to stop 
events without having to restart a node.
* {{SchemaAnnouncementEvent}} will no longer depend on 
{{SchemaTransformation.toString()}} output 
([a5be8dc#L|https://github.com/spodkowinski/cassandra/commit/a5be8dc0214dff4974f5d299c9eebc69f04b650b#diff-4e80da086bdbd68b1295161b00719a10R66])
** The added {{toString()}} methods have been left as before, but can also be 
removed if anyone minds having them
* Added threadName to returned events 
([2d3549c#L|https://github.com/spodkowinski/cassandra/commit/2d3549c53ce11d5d94951f093b5db15a44f71325#diff-734577dc8c504d46702efa2262dfaac6R89])
* Moved getSource() to CASSANDRA-13458, so we don't have to discuss it now

Also
* Expose Gossiper state by calling getters instead of inspecting members 
directly 
([a5be8dc#L|https://github.com/spodkowinski/cassandra/commit/a5be8dc0214dff4974f5d299c9eebc69f04b650b#diff-8be666c70553b1f0017a01458c490f47R903])

There are several other ways to solve this (making a snapshot of the internal 
state of a service like Gossiper). 
* calling getters from the event class (as in latest version)
* have gossiper return an event or part of the event details (merged in 
{{toMap()}})
* pass a builder object to gossip and have gossiper add any internal state to 
that

I'd still prefer the first approach due to separations of concerns and to avoid 
introducing leaky abstractions between services, such as Gossiper, and actual 
event handling and persistence. We'd first have to agree on a more precise 
contract for the kind of data to expose in events. The current {{Map}} take is probably already too ambiguous, but I don't have any 
better ideas for that, without opening the discussion again on how to persist 
and remotely access events in general.




> Diag. Events: Add base classes
> --
>
> Key: CASSANDRA-13457
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13457
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core, Observability
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>
> Base ticket for adding classes that will allow you to implement and subscribe 
> to events.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14346) Scheduled Repair in Cassandra

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583811#comment-16583811
 ] 

Stefan Podkowinski commented on CASSANDRA-14346:


Please include license files like we do in {{lib/licenses}}. Adding 
{{javax.servlet-api}} (assuming CDDL1.1) requires special handling (category 
b), so it would be preferable not having to use that dependency.



> Scheduled Repair in Cassandra
> -
>
> Key: CASSANDRA-14346
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14346
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Repair
>Reporter: Joseph Lynch
>Assignee: Joseph Lynch
>Priority: Major
>  Labels: 4.0-feature-freeze-review-requested, 
> CommunityFeedbackRequested
> Fix For: 4.0
>
> Attachments: ScheduledRepairV1_20180327.pdf
>
>
> There have been many attempts to automate repair in Cassandra, which makes 
> sense given that it is necessary to give our users eventual consistency. Most 
> recently CASSANDRA-10070, CASSANDRA-8911 and CASSANDRA-13924 have all looked 
> for ways to solve this problem.
> At Netflix we've built a scheduled repair service within Priam (our sidecar), 
> which we spoke about last year at NGCC. Given the positive feedback at NGCC 
> we focussed on getting it production ready and have now been using it in 
> production to repair hundreds of clusters, tens of thousands of nodes, and 
> petabytes of data for the past six months. Also based on feedback at NGCC we 
> have invested effort in figuring out how to integrate this natively into 
> Cassandra rather than open sourcing it as an external service (e.g. in Priam).
> As such, [~vinaykumarcse] and I would like to re-work and merge our 
> implementation into Cassandra, and have created a [design 
> document|https://docs.google.com/document/d/1RV4rOrG1gwlD5IljmrIq_t45rz7H3xs9GbFSEyGzEtM/edit?usp=sharing]
>  showing how we plan to make it happen, including the the user interface.
> As we work on the code migration from Priam to Cassandra, any feedback would 
> be greatly appreciated about the interface or v1 implementation features. I 
> have tried to call out in the document features which we explicitly consider 
> future work (as well as a path forward to implement them in the future) 
> because I would very much like to get this done before the 4.0 merge window 
> closes, and to do that I think aggressively pruning scope is going to be a 
> necessity.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-14346) Scheduled Repair in Cassandra

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583811#comment-16583811
 ] 

Stefan Podkowinski edited comment on CASSANDRA-14346 at 8/17/18 11:52 AM:
--

Please include license files like we do in {{lib/licenses}}. Adding 
{{javax.servlet-api}} (assuming CDDL1.1) requires special handling ([category 
b|https://www.apache.org/legal/resolved.html#category-x]), so it would be 
preferable not having to use that dependency.




was (Author: spo...@gmail.com):
Please include license files like we do in {{lib/licenses}}. Adding 
{{javax.servlet-api}} (assuming CDDL1.1) requires special handling (category 
b), so it would be preferable not having to use that dependency.



> Scheduled Repair in Cassandra
> -
>
> Key: CASSANDRA-14346
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14346
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Repair
>Reporter: Joseph Lynch
>Assignee: Joseph Lynch
>Priority: Major
>  Labels: 4.0-feature-freeze-review-requested, 
> CommunityFeedbackRequested
> Fix For: 4.0
>
> Attachments: ScheduledRepairV1_20180327.pdf
>
>
> There have been many attempts to automate repair in Cassandra, which makes 
> sense given that it is necessary to give our users eventual consistency. Most 
> recently CASSANDRA-10070, CASSANDRA-8911 and CASSANDRA-13924 have all looked 
> for ways to solve this problem.
> At Netflix we've built a scheduled repair service within Priam (our sidecar), 
> which we spoke about last year at NGCC. Given the positive feedback at NGCC 
> we focussed on getting it production ready and have now been using it in 
> production to repair hundreds of clusters, tens of thousands of nodes, and 
> petabytes of data for the past six months. Also based on feedback at NGCC we 
> have invested effort in figuring out how to integrate this natively into 
> Cassandra rather than open sourcing it as an external service (e.g. in Priam).
> As such, [~vinaykumarcse] and I would like to re-work and merge our 
> implementation into Cassandra, and have created a [design 
> document|https://docs.google.com/document/d/1RV4rOrG1gwlD5IljmrIq_t45rz7H3xs9GbFSEyGzEtM/edit?usp=sharing]
>  showing how we plan to make it happen, including the the user interface.
> As we work on the code migration from Priam to Cassandra, any feedback would 
> be greatly appreciated about the interface or v1 implementation features. I 
> have tried to call out in the document features which we explicitly consider 
> future work (as well as a path forward to implement them in the future) 
> because I would very much like to get this done before the 4.0 merge window 
> closes, and to do that I think aggressively pruning scope is going to be a 
> necessity.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13457) Diag. Events: Add base classes

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-13457:
---
   Resolution: Fixed
Fix Version/s: 4.0
   Status: Resolved  (was: Patch Available)

Committed as 2846b22a70d48bae.

Thanks [~michaelsembwever] and [~jasobrown] for reviewing and sharing your 
feedback!

> Diag. Events: Add base classes
> --
>
> Key: CASSANDRA-13457
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13457
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Core, Observability
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
> Fix For: 4.0
>
>
> Base ticket for adding classes that will allow you to implement and subscribe 
> to events.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13668) Diag. events for user audit logging

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-13668:
---
   Resolution: Fixed
Fix Version/s: (was: 4.x)
   4.0
   Status: Resolved  (was: Patch Available)

Committed as d8c45192318584!

> Diag. events for user audit logging
> ---
>
> Key: CASSANDRA-13668
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13668
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
> Fix For: 4.0
>
>
> With the availability of CASSANDRA-13459, any native transport enabled client 
> will be able to subscribe to internal Cassandra events. External tools can 
> take advantage by monitoring these events in various ways. Use-cases for this 
> can be e.g. auditing tools for compliance and security purposes.
> The scope of this ticket is to add diagnostic events that are raised around 
> authentication and CQL operations. These events can then be consumed and used 
> by external tools to implement a Cassandra user auditing solution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14435) Diag. Events: JMX events

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-14435:
---
   Resolution: Fixed
Fix Version/s: (was: 4.x)
   4.0
   Status: Resolved  (was: Patch Available)

Committed as a79e5903b552e40f77c!

> Diag. Events: JMX events
> 
>
> Key: CASSANDRA-14435
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14435
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Observability
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
> Fix For: 4.0
>
>
> Nodes currently use JMX events for progress reporting on bootstrap and 
> repairs. This might also be an option to expose diagnostic events to external 
> subscribers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14597) [dtest] snapshot_test.TestArchiveCommitlog

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-14597:
---
Resolution: Fixed
Status: Resolved  (was: Ready to Commit)

Merged as bd419a7ae

> [dtest] snapshot_test.TestArchiveCommitlog
> --
>
> Key: CASSANDRA-14597
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14597
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jason Brown
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: dtest
> Attachments: 2_2.txt, 3_0.txt, 3_11_3.txt
>
>
> All TestArchiveCommitLog dtests fail on 3.0, but no other branches. Output 
> from pytest error:
> {noformat}
> assert (
> time.time() <= stop_time), "It's been over a {s}s and we 
> haven't written a new " + \
> >   "commitlog segment. Something is wrong.".format(s=timeout)
> E   AssertionError: It's been over a {s}s and we haven't written a 
> new commitlog segment. Something is wrong.
> tools/hacks.py:61: AssertionError
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14346) Scheduled Repair in Cassandra

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584032#comment-16584032
 ] 

Stefan Podkowinski commented on CASSANDRA-14346:


I'd add a license file per jar to keep things consistent. Adding the 
servlet-api (dual CDDL/GPL) needs some more careful handling compared to 
permissive licensed deps, but shouldn't be a blocker per se, if we decide we 
really want to get this committed.

> Scheduled Repair in Cassandra
> -
>
> Key: CASSANDRA-14346
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14346
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Repair
>Reporter: Joseph Lynch
>Assignee: Joseph Lynch
>Priority: Major
>  Labels: 4.0-feature-freeze-review-requested, 
> CommunityFeedbackRequested
> Fix For: 4.0
>
> Attachments: ScheduledRepairV1_20180327.pdf
>
>
> There have been many attempts to automate repair in Cassandra, which makes 
> sense given that it is necessary to give our users eventual consistency. Most 
> recently CASSANDRA-10070, CASSANDRA-8911 and CASSANDRA-13924 have all looked 
> for ways to solve this problem.
> At Netflix we've built a scheduled repair service within Priam (our sidecar), 
> which we spoke about last year at NGCC. Given the positive feedback at NGCC 
> we focussed on getting it production ready and have now been using it in 
> production to repair hundreds of clusters, tens of thousands of nodes, and 
> petabytes of data for the past six months. Also based on feedback at NGCC we 
> have invested effort in figuring out how to integrate this natively into 
> Cassandra rather than open sourcing it as an external service (e.g. in Priam).
> As such, [~vinaykumarcse] and I would like to re-work and merge our 
> implementation into Cassandra, and have created a [design 
> document|https://docs.google.com/document/d/1RV4rOrG1gwlD5IljmrIq_t45rz7H3xs9GbFSEyGzEtM/edit?usp=sharing]
>  showing how we plan to make it happen, including the the user interface.
> As we work on the code migration from Priam to Cassandra, any feedback would 
> be greatly appreciated about the interface or v1 implementation features. I 
> have tried to call out in the document features which we explicitly consider 
> future work (as well as a path forward to implement them in the future) 
> because I would very much like to get this done before the 4.0 merge window 
> closes, and to do that I think aggressively pruning scope is going to be a 
> necessity.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14298) cqlshlib tests broken on b.a.o

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591338#comment-16591338
 ] 

Stefan Podkowinski commented on CASSANDRA-14298:


Thanks for figuring this out, [~ptbannister]! I'll look into rebuilding the 
docker image with the described steps.

> cqlshlib tests broken on b.a.o
> --
>
> Key: CASSANDRA-14298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14298
> Project: Cassandra
>  Issue Type: Bug
>  Components: Build, Testing
>Reporter: Stefan Podkowinski
>Assignee: Patrick Bannister
>Priority: Major
>  Labels: cqlsh, dtest
> Attachments: CASSANDRA-14298-old.txt, CASSANDRA-14298.txt, 
> cqlsh_tests_notes.md
>
>
> It appears that cqlsh-tests on builds.apache.org on all branches stopped 
> working since we removed nosetests from the system environment. See e.g. 
> [here|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-cqlsh-tests/458/cython=no,jdk=JDK%201.8%20(latest),label=cassandra/console].
>  Looks like we either have to make nosetests available again or migrate to 
> pytest as we did with dtests. Giving pytest a quick try resulted in many 
> errors locally, but I haven't inspected them in detail yet. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-14668) Diag events for read repairs

[Stefan Podkowinski (JIRA)]

Stefan Podkowinski created CASSANDRA-14668:
--

 Summary: Diag events for read repairs
 Key: CASSANDRA-14668
 URL: https://issues.apache.org/jira/browse/CASSANDRA-14668
 Project: Cassandra
  Issue Type: Improvement
  Components: Observability
Reporter: Stefan Podkowinski
Assignee: Stefan Podkowinski
 Fix For: 4.x


Read repairs have been a highly discussed topic during the last months and also 
saw some significant code changes. I'd like to be better prepared in case we 
need to investigate any further RR issues in the future, by adding diagnostic 
events that can be enabled for exposing informations such as:
 * contacted endpoints
 * digest responses by endpoint
 * affected partition keys
 * speculated reads / writes

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-8303) Create a capability limitation framework

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590036#comment-16590036
 ] 

Stefan Podkowinski commented on CASSANDRA-8303:
---

{quote}I think it would be much more practical for all capabilities to be 
enabled by default and then restrictions added as required.
{quote}
I think it really depends where you're coming from. If your goal is to keep 
capabilities as minimal as possible (just as you should keep permissions as 
minimal as possible), then the blacklist approach would be tedious and 
confusing for managing a larger number of roles. If you your goal is to simply 
remove a certain capability as needed, then just adding a restriction to a 
user's role would feel intuitive and conveniently simple.

But so far the discussion seems to be centered around a blacklist solution, so 
I really don't want to stale any progress here by opening a whitelist vs 
blacklist discussion again at this point, especially after already have a patch 
for the later.

> Create a capability limitation framework
> 
>
> Key: CASSANDRA-8303
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8303
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Anupam Arora
>Priority: Major
> Fix For: 4.x
>
>
> In addition to our current Auth framework that acts as a white list, and 
> regulates access to data, functions, and roles, it would be beneficial to 
> have a different, capability limitation framework, that would be orthogonal 
> to Auth, and would act as a blacklist.
> Example uses:
> - take away the ability to TRUNCATE from all users but the admin (TRUNCATE 
> itself would still require MODIFY permission)
> - take away the ability to use ALLOW FILTERING from all users but 
> Spark/Hadoop (SELECT would still require SELECT permission)
> - take away the ability to use UNLOGGED BATCH from everyone (the operation 
> itself would still require MODIFY permission)
> - take away the ability to use certain consistency levels (make certain 
> tables LWT-only for all users, for example)
> Original description:
> Please provide a "strict mode" option in cassandra that will kick out any CQL 
> queries that are expensive, e.g. any query with ALLOWS FILTERING, 
> multi-partition queries, secondary index queries, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14545) dtests: fix pytest.raises argument names

[Stefan Podkowinski (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-14545:
---
Resolution: Fixed
  Reviewer: Jason Brown
Status: Resolved  (was: Ready to Commit)

Committed as f210e532e

> dtests: fix pytest.raises argument names
> 
>
> Key: CASSANDRA-14545
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14545
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>  Labels: dtest
> Attachments: CASSANDRA-14545.patch
>
>
> I've been through a couple of [dtest 
> results|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/580/#showFailuresLink]
>  lately and notices some interpreter errors regarding how we call 
> pytest.raises. The 
> [reference|https://docs.pytest.org/en/latest/assert.html#assertions-about-expected-exceptions]
>  is pretty clear on what would be the correct arguments, but still want to 
> make sure we're not working on different pytest versions. 
> [~mkjellman] can you quickly check the following inconsistencies and look at 
> my patch (msg->message, matches->match)?
> {noformat}
> git show 49b2dda4 |egrep 'raises.*, m' {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14689) Add developer docs for creating releases

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606093#comment-16606093
 ] 

Stefan Podkowinski commented on CASSANDRA-14689:


+1, thanks!

> Add developer docs for creating releases
> 
>
> Key: CASSANDRA-14689
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14689
> Project: Cassandra
>  Issue Type: Task
>  Components: Documentation and Website
>Reporter: mck
>Assignee: mck
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 4.x
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Provide an initial outline on the steps Release Managers follow for creating, 
> voting and publishing a release for Apache Cassandra.
> ASF has the following guidelines:
>  * `ASF Release Policy `_.
>  * `ASF Release Distribution Policy 
> `_.
>  * `ASF Release Best Practices 
> `_.
> The project is still doing some things in an outdated manner, eg using 
> people.apache.org URLs for staging artefacts. There is no urgent need to fix 
> these things but by having the docs published it can improved incrementally 
> over time.
> fyi [~mshuler]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org