[jira] [Created] (CASSANDRA-14712) Cassandra 4.0 packaging support
Stefan Podkowinski created CASSANDRA-14712: -- Summary: Cassandra 4.0 packaging support Key: CASSANDRA-14712 URL: https://issues.apache.org/jira/browse/CASSANDRA-14712 Project: Cassandra Issue Type: Bug Components: Packaging Reporter: Stefan Podkowinski Fix For: 4.x Currently it's not possible to build any native packages (.deb/.rpm) for trunk. cassandra-builds - docker/*-image.docker * Add Java11 to debian+centos build image * (packaged ant scripts won't work with Java 11 on centos, so we may have to install ant from tarballs) cassandra-builds - docker/build-*.sh * set JAVA8_HOME to Java8 * set JAVA_HOME to Java11 (4.0) or Java8 (<4.0) cassandra - redhat/cassandra.spec * Check if patches still apply after CASSANDRA-14707 * Add fqltool as %files We may also have to change the version handling in build.xml or build-*.sh, depending how we plan to release packages during beta, or if we plan to do so at all before GA. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14713) Add docker testing image to cassandra-builds
Stefan Podkowinski created CASSANDRA-14713:
--
Summary: Add docker testing image to cassandra-builds
Key: CASSANDRA-14713
URL: https://issues.apache.org/jira/browse/CASSANDRA-14713
Project: Cassandra
Issue Type: New Feature
Components: Testing
Reporter: Stefan Podkowinski
Tests executed on builds.apache.org ({{docker/jenkins/jenkinscommand.sh}}) and
circleCI ({{.circleci/config.yml}}) will currently use the same
[cassandra-test|https://hub.docker.com/r/kjellman/cassandra-test/] docker image
([github|https://github.com/mkjellman/cassandra-test-docker]) by [~mkjellman].
We should manage this image on our own as part of cassandra-builds, to keep it
updated. There's also a [Apache user|https://hub.docker.com/u/apache/?page=1]
on docker hub for publishing images.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14298) cqlshlib tests broken on b.a.o
[ https://issues.apache.org/jira/browse/CASSANDRA-14298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608965#comment-16608965 ] Stefan Podkowinski edited comment on CASSANDRA-14298 at 9/10/18 10:15 AM: -- Thanks [~mkjellman]! Can you please attach your Dockerfile to CASSANDRA-14713? I'll then try to get rid of the ADDed resource dependencies. was (Author: spo...@gmail.com): Thanks [~mkjellman]! Can you please attach your Dockerimage file to CASSANDRA-14713? I'll then try to get rid of the ADDed resource dependencies. > cqlshlib tests broken on b.a.o > -- > > Key: CASSANDRA-14298 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14298 > Project: Cassandra > Issue Type: Bug > Components: Build, Testing >Reporter: Stefan Podkowinski >Assignee: Patrick Bannister >Priority: Major > Labels: cqlsh, dtest > Attachments: CASSANDRA-14298-old.txt, CASSANDRA-14298.txt, > cqlsh_tests_notes.md > > > It appears that cqlsh-tests on builds.apache.org on all branches stopped > working since we removed nosetests from the system environment. See e.g. > [here|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-cqlsh-tests/458/cython=no,jdk=JDK%201.8%20(latest),label=cassandra/console]. > Looks like we either have to make nosetests available again or migrate to > pytest as we did with dtests. Giving pytest a quick try resulted in many > errors locally, but I haven't inspected them in detail yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14298) cqlshlib tests broken on b.a.o
[ https://issues.apache.org/jira/browse/CASSANDRA-14298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608965#comment-16608965 ] Stefan Podkowinski commented on CASSANDRA-14298: Thanks [~mkjellman]! Can you please attach your Dockerimage file to CASSANDRA-14713? I'll then try to get rid of the ADDed resource dependencies. > cqlshlib tests broken on b.a.o > -- > > Key: CASSANDRA-14298 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14298 > Project: Cassandra > Issue Type: Bug > Components: Build, Testing >Reporter: Stefan Podkowinski >Assignee: Patrick Bannister >Priority: Major > Labels: cqlsh, dtest > Attachments: CASSANDRA-14298-old.txt, CASSANDRA-14298.txt, > cqlsh_tests_notes.md > > > It appears that cqlsh-tests on builds.apache.org on all branches stopped > working since we removed nosetests from the system environment. See e.g. > [here|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-cqlsh-tests/458/cython=no,jdk=JDK%201.8%20(latest),label=cassandra/console]. > Looks like we either have to make nosetests available again or migrate to > pytest as we did with dtests. Giving pytest a quick try resulted in many > errors locally, but I haven't inspected them in detail yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Resolved] (CASSANDRA-13348) Duplicate tokens after bootstrap
[
https://issues.apache.org/jira/browse/CASSANDRA-13348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski resolved CASSANDRA-13348.
Resolution: Cannot Reproduce
> Duplicate tokens after bootstrap
>
>
> Key: CASSANDRA-13348
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13348
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tom van der Woerdt
>Assignee: Dikang Gu
>Priority: Blocker
> Fix For: 3.0.x
>
>
> This one is a bit scary, and probably results in data loss. After a bootstrap
> of a few new nodes into an existing cluster, two new nodes have chosen some
> overlapping tokens.
> In fact, of the 256 tokens chosen, 51 tokens were already in use on the other
> node.
> Node 1 log :
> {noformat}
> INFO [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:42:43,461
> StorageService.java:1160 - JOINING: waiting for ring information
> INFO [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:42:43,461
> StorageService.java:1160 - JOINING: waiting for schema information to complete
> INFO [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:42:43,461
> StorageService.java:1160 - JOINING: schema complete, ready to bootstrap
> INFO [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:42:43,462
> StorageService.java:1160 - JOINING: waiting for pending range calculation
> INFO [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:42:43,462
> StorageService.java:1160 - JOINING: calculation complete, ready to bootstrap
> INFO [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:42:43,462
> StorageService.java:1160 - JOINING: getting bootstrap token
> WARN [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:42:43,564
> TokenAllocation.java:61 - Selected tokens [, 2959334889475814712,
> 3727103702384420083, 7183119311535804926, 6013900799616279548,
> -1222135324851761575, 1645259890258332163, -1213352346686661387,
> 7604192574911909354]
> WARN [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:42:43,729
> TokenAllocation.java:65 - Replicated node load in datacentre before
> allocation max 1.00 min 1.00 stddev 0.
> WARN [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:42:43,729
> TokenAllocation.java:66 - Replicated node load in datacentre after allocation
> max 1.00 min 1.00 stddev 0.
> WARN [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:42:43,729
> TokenAllocation.java:70 - Unexpected growth in standard deviation after
> allocation.
> INFO [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:42:44,150
> StorageService.java:1160 - JOINING: sleeping 3 ms for pending range setup
> INFO [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:43:14,151
> StorageService.java:1160 - JOINING: Starting to bootstrap...
> {noformat}
> Node 2 log:
> {noformat}
> INFO [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:55:51,937
> StorageService.java:971 - Joining ring by operator request
> INFO [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:55:52,513
> StorageService.java:1160 - JOINING: waiting for ring information
> INFO [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:55:52,513
> StorageService.java:1160 - JOINING: waiting for schema information to complete
> INFO [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:55:52,513
> StorageService.java:1160 - JOINING: schema complete, ready to bootstrap
> INFO [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:55:52,513
> StorageService.java:1160 - JOINING: waiting for pending range calculation
> INFO [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:55:52,514
> StorageService.java:1160 - JOINING: calculation complete, ready to bootstrap
> INFO [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:55:52,514
> StorageService.java:1160 - JOINING: getting bootstrap token
> WARN [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:55:52,630
> TokenAllocation.java:61 - Selected tokens [.., 2890709530010722764,
> -2416006722819773829, -5820248611267569511, -5990139574852472056,
> 1645259890258332163, 9135021011763659240, -5451286144622276797,
> 7604192574911909354]
> WARN [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:55:52,794
> TokenAllocation.java:65 - Replicated node load in datacentre before
> allocation max 1.02 min 0.98 stddev 0.
> WARN [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:55:52,795
> TokenAllocation.java:66 - Replicated node load in datacentre after allocation
> max 1.00 min 1.00 stddev 0.
> INFO [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:55:53,149
> StorageService.java:1160 - JOINING: sleeping 3 ms for pending range setup
> INFO [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:56:23,149
> StorageService.java:1160 - JOINING: Starting to bootstrap...
> {noformat}
> eg. 7604192574911909354 has been chosen by both.
> The joins were eight days apart, so I don't
[jira] [Commented] (CASSANDRA-14714) `ant artifacts` broken on trunk (4.0); creates no tar artifacts
[
https://issues.apache.org/jira/browse/CASSANDRA-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609722#comment-16609722
]
Stefan Podkowinski commented on CASSANDRA-14714:
I've tried to wrap up some of the 4.0 related build/packaging issues in
CASSANDRA-14712
> `ant artifacts` broken on trunk (4.0); creates no tar artifacts
> ---
>
> Key: CASSANDRA-14714
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14714
> Project: Cassandra
> Issue Type: Bug
>Reporter: Michael Shuler
>Priority: Blocker
> Fix For: 4.0
>
>
> `ant artifacts` on the trunk (4.0) branch currently creates no tar artifacts.
> Additionally, the target does not exit non-zero, so the result is:
> {noformat}
> <...>
> artifacts:
> BUILD SUCCESSFUL
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14668) Diag events for read repairs
[
https://issues.apache.org/jira/browse/CASSANDRA-14668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598377#comment-16598377
]
Stefan Podkowinski commented on CASSANDRA-14668:
Example events from a single read repair as retrieved via JMX.
ReadRepairEvent:
{noformat}
class=org.apache.cassandra.service.reads.repair.ReadRepairEvent,
type=START_REPAIR,
command=SELECT * FROM ks.table1 WHERE key = a LIMIT 100,
consistency=ALL,
speculativeRetry=PERCENTILE,
keyspace=ks,
table=table1,
allEndpoints=[127.0.0.1:7000, 127.0.0.2:7000],
endpointDestinations=[127.0.0.1:7000, 127.0.0.2:7000],
thread=Native-Transport-Requests-1,
digestsByEndpoint={
127.0.0.2:7000={digestHex=d41d8cd98f00b204e9800998ecf8427e,
isDigestResponse=true},
127.0.0.1:7000={digestHex=82577faad6f8c6450786e68df35db686,
isDigestResponse=false}
},
ts=1535700434509
{noformat}
PartitionRepairEvent:
{noformat}
class=org.apache.cassandra.service.reads.repair.PartitionRepairEvent,
type=SEND_INITIAL_REPAIRS,
consistency=ALL,
keyspace=ks,
mutation=Mutation(keyspace='ks', key='61', modifications=[
[ks.table1] key=a partition_deletion=deletedAt=-9223372036854775808,
localDeletion=2147483647 columns=[[] | [c1 v1]]
Row[info=[ts=1535700208937182] ]: EMPTY | [c1=1 ts=1535700208937182], [v1=1
ts=1535700208937182]
]),
destination=127.0.0.2:7000,
thread=Native-Transport-Requests-1,
key=61,
token=-8839064797231613815,
ts=1535700434522
{noformat}
> Diag events for read repairs
>
>
> Key: CASSANDRA-14668
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14668
> Project: Cassandra
> Issue Type: Improvement
> Components: Observability
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
> Fix For: 4.x
>
>
> Read repairs have been a highly discussed topic during the last months and
> also saw some significant code changes. I'd like to be better prepared in
> case we need to investigate any further RR issues in the future, by adding
> diagnostic events that can be enabled for exposing informations such as:
> * contacted endpoints
> * digest responses by endpoint
> * affected partition keys
> * speculated reads / writes
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14594) No validation for repeated fields in cqlsh and misbehaviour in data display
[ https://issues.apache.org/jira/browse/CASSANDRA-14594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597186#comment-16597186 ] Stefan Podkowinski commented on CASSANDRA-14594: Should be fixed already in CASSANDRA-13262 > No validation for repeated fields in cqlsh and misbehaviour in data display > --- > > Key: CASSANDRA-14594 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14594 > Project: Cassandra > Issue Type: Bug > Components: Tools > Environment: Operating System: Ubuntu 14.04(64 bit) (Image : > cqlshbug.png) > Operating System: ubuntu 16.04 (64 bit) (Image: > cqlsh_select_repeated_fields.png) > Apache cassandra version : 3.11.1 >Reporter: Chakravarthi Manepalli >Assignee: Benjamin Lerer >Priority: Major > Attachments: cqlsh bug.png, cqlsh_select_repeated_fields.png > > > In a table, if the fields(columns) are repeated in select call, the > displaying information is not correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13262) Incorrect cqlsh results when selecting same columns multiple times
[
https://issues.apache.org/jira/browse/CASSANDRA-13262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597212#comment-16597212
]
Stefan Podkowinski commented on CASSANDRA-13262:
Fix version is correct. See my [previous
comment|https://issues.apache.org/jira/browse/CASSANDRA-13262?focusedCommentId=15998605=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15998605].
> Incorrect cqlsh results when selecting same columns multiple times
> --
>
> Key: CASSANDRA-13262
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13262
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefan Podkowinski
>Assignee: Murukesh Mohanan
>Priority: Minor
> Labels: lhf
> Fix For: 4.0
>
> Attachments:
> 0001-Fix-incorrect-cqlsh-results-when-selecting-same-colu.patch,
> CASSANDRA-13262-v2.2.txt, CASSANDRA-13262-v3.0.txt, CASSANDRA-13262-v3.11.txt
>
>
> Just stumbled over this on trunk:
> {quote}
> cqlsh:test1> select a, b, c from table1;
> a | b| c
> ---+--+-
> 1 |b | 2
> 2 | null | 2.2
> (2 rows)
> cqlsh:test1> select a, a, b, c from table1;
> a | a| b | c
> ---+--+-+--
> 1 |b | 2 | null
> 2 | null | 2.2 | null
> (2 rows)
> cqlsh:test1> select a, a, a, b, c from table1;
> a | a| a | b| c
> ---+--+---+--+--
> 1 |b | 2.0 | null | null
> 2 | null | 2.2004768 | null | null
> {quote}
> My guess is that his is on the Python side, but haven't really looked into it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14703) Document Apache Cassandra Backup Operations
[ https://issues.apache.org/jira/browse/CASSANDRA-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610359#comment-16610359 ] Stefan Podkowinski commented on CASSANDRA-14703: Already looks promising; please don't forget to confirm "Submit Patch", once you're done! > Document Apache Cassandra Backup Operations > --- > > Key: CASSANDRA-14703 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14703 > Project: Cassandra > Issue Type: Improvement > Components: Documentation and Website >Reporter: pedro vidigal >Priority: Major > Attachments: backups.rst > > > The documentation for the Backup Operations is missing in on the > documentation site > ([https://github.com/apache/cassandra/blob/trunk/doc/source/operating/backups.rst)] > > Branch with the changes: > https://github.com/vidigalp/cassandra/tree/documentation/backup-strategies -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14702) Cassandra Write failed even when the required nodes to Ack(consistency) are up.
[ https://issues.apache.org/jira/browse/CASSANDRA-14702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610335#comment-16610335 ] Stefan Podkowinski commented on CASSANDRA-14702: What are the steps to reproduce the described issue? > Cassandra Write failed even when the required nodes to Ack(consistency) are > up. > --- > > Key: CASSANDRA-14702 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14702 > Project: Cassandra > Issue Type: Bug >Reporter: Rohit Singh >Priority: Blocker > > Hi, > We have following configuration in our project for cassandra. > Total nodes in Cluster-5 > Replication Factor- 3 > Consistency- LOCAL_QUORUM > We get the writetimeout exception from cassandra even when 2 nodes are up and > why does stack trace says that 3 replica were required when consistency is 2? > Below is the exception we got:- > com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout > during write query at consistency LOCAL_QUORUM (3 replica were required but > only 2 acknowledged the write) > at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:59) > at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:37) > at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:289) > at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:269) > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Resolved] (CASSANDRA-14691) Cassandra 2.1 backport - The JVM should exit if jmx fails to bind
[
https://issues.apache.org/jira/browse/CASSANDRA-14691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski resolved CASSANDRA-14691.
Resolution: Won't Fix
Not a critical bug, which is the acceptance criteria for 2.1, sorry.
> Cassandra 2.1 backport - The JVM should exit if jmx fails to bind
> -
>
> Key: CASSANDRA-14691
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14691
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Thomas Steinmaurer
>Priority: Major
> Labels: lhf
> Fix For: 2.1.x
>
>
> If you are already running a cassandra instance, but for some reason try to
> start another one, this happens:
> {noformat}
> INFO 20:57:09 JNA mlockall successful
> WARN 20:57:09 JMX is not enabled to receive remote connections. Please see
> cassandra-env.sh for more info.
> ERROR 20:57:10 Error starting local jmx server:
> java.rmi.server.ExportException: Port already in use: 7199; nested exception
> is:
> java.net.BindException: Address already in use
> at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:340)
> ~[na:1.7.0_76]
> at
> sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:248)
> ~[na:1.7.0_76]
> at
> sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:411)
> ~[na:1.7.0_76]
> at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147)
> ~[na:1.7.0_76]
> at
> sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:207)
> ~[na:1.7.0_76]
> at sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:122)
> ~[na:1.7.0_76]
> at sun.rmi.registry.RegistryImpl.(RegistryImpl.java:98)
> ~[na:1.7.0_76]
> at
> java.rmi.registry.LocateRegistry.createRegistry(LocateRegistry.java:239)
> ~[na:1.7.0_76]
> at
> org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:100)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:222)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:564)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:653)
> [main/:na]
> Caused by: java.net.BindException: Address already in use
> at java.net.PlainSocketImpl.socketBind(Native Method) ~[na:1.7.0_76]
> at
> java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
> ~[na:1.7.0_76]
> at java.net.ServerSocket.bind(ServerSocket.java:376) ~[na:1.7.0_76]
> at java.net.ServerSocket.(ServerSocket.java:237) ~[na:1.7.0_76]
> at
> javax.net.DefaultServerSocketFactory.createServerSocket(ServerSocketFactory.java:231)
> ~[na:1.7.0_76]
> at
> org.apache.cassandra.utils.RMIServerSocketFactoryImpl.createServerSocket(RMIServerSocketFactoryImpl.java:13)
> ~[main/:na]
> at
> sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:666)
> ~[na:1.7.0_76]
> at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:329)
> ~[na:1.7.0_76]
> ... 11 common frames omitted
> {noformat}
> However the startup continues, and ends up replaying commitlogs, which is
> probably not a good thing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14701) Cleanup (and other) compaction type(s) not counted in compaction remaining time
[
https://issues.apache.org/jira/browse/CASSANDRA-14701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610411#comment-16610411
]
Stefan Podkowinski commented on CASSANDRA-14701:
Link to [cassandra-user
thread|https://lists.apache.org/thread.html/6f355d1c2258478dbd7ea3e7960a03ed480daa4cab91d31e71de1000@%3Cuser.cassandra.apache.org%3E].
> Cleanup (and other) compaction type(s) not counted in compaction remaining
> time
> ---
>
> Key: CASSANDRA-14701
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14701
> Project: Cassandra
> Issue Type: Bug
> Components: Observability
>Reporter: Thomas Steinmaurer
>Priority: Critical
>
> Opened a ticket, as discussed in user list.
> Looks like compaction remaining time only includes compactions of type
> COMPACTION and other compaction types like cleanup etc. aren't part of the
> estimation calculation.
> E.g. from one of our environments:
> {noformat}
> nodetool compactionstats -H
> pending tasks: 1
>compaction type keyspace table completed totalunit
> progress
>CleanupXXX YYY 908.16 GB 1.13 TB bytes
> 78.63%
> Active compaction remaining time : 0h00m00s
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14692) join_ring=false populates wrong value into StorageServiceMB and prevents join by nodetool
[
https://issues.apache.org/jira/browse/CASSANDRA-14692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610506#comment-16610506
]
Stefan Podkowinski commented on CASSANDRA-14692:
Can you please double check that the {{-Dcassandra.join_ring=false}} option is
really set correctly and will show up as part of the java process parameters?
> join_ring=false populates wrong value into StorageServiceMB and prevents join
> by nodetool
> -
>
> Key: CASSANDRA-14692
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14692
> Project: Cassandra
> Issue Type: Bug
> Components: Lifecycle
>Reporter: Roland Johann
>Priority: Major
> Fix For: 3.11.3
>
> Attachments: Bildschirmfoto 2018-09-05 um 17.29.54.png,
> cassandra1_log, cassandra1_nodetool_gossipinfo, cassandra1_nodetool_status,
> cassandra2_nodetool_gossipinfo, cassandra2_nodetool_status
>
>
> Restarting a cassandra cluster member with option
> {{-Dcassandra.join_ring=false}} populates wrong value to its
> {{StorageServiceMB}} field {{Joined}} which causes the actual trigger to join
> via {{nodetool join}} to abort due to check if {{Join}} in
> {{StorageServiceMB}} is true. Via jconsole it's possible as there is no check.
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/tools/nodetool/Join.java
> {{nodetool status}} also shows that the node is up and in normal node, on the
> rest of the cluster node status is {{DN}}.
> {{nodetool gossipinfo}} states that the non joined node is in gossip state
> {{hibernate}}.
> Came across this issue while evaluated the problem of zombies to integrate
> into automation processes and the documentation states
> {quote}To avoid this problem, run {{nodetool repair}} on any restored node
> before rejoining it to its cluster.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14709) Global configuration parameter to reject increment repair and allow full repair only
[ https://issues.apache.org/jira/browse/CASSANDRA-14709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-14709: --- Issue Type: Wish (was: Bug) > Global configuration parameter to reject increment repair and allow full > repair only > > > Key: CASSANDRA-14709 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14709 > Project: Cassandra > Issue Type: Wish >Reporter: Thomas Steinmaurer >Priority: Major > Fix For: 2.2.x, 3.0.x, 3.11.x, 4.0.x > > > We are running Cassandra in AWS and On-Premise at customer sites, currently > 2.1 in production with 3.0/3.11 in pre-production stages including loadtest. > In a migration path from 2.1 to 3.11.x, I’m afraid that at some point in time > we end up in incremental repairs being enabled / ran a first time > unintentionally, cause: > a) A lot of online resources / examples do not use the _-full_ command-line > option available since 2.2 (?) > b) Our internal (support) tickets of course also state nodetool repair > command without the -full option, as these examples are for 2.1 > Especially for On-Premise customers (with less control than with our AWS > deployments), this asks a bit for getting out-of-control once we have 3.11 > out and nodetool repair being run without the -full command-line option. > With troubles incremental repair are introducing and incremental being the > default since 2.2 (?), what do you think about a JVM system property, > cassandra.yaml setting or whatever … to basically let the cluster > administrator chose if incremental repairs are allowed or not? I know, such a > flag still can be flipped then (by the customer), but as a first safety stage > possibly sufficient enough. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14692) join_ring=false populates wrong value into StorageServiceMB and prevents join by nodetool
[
https://issues.apache.org/jira/browse/CASSANDRA-14692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610548#comment-16610548
]
Stefan Podkowinski commented on CASSANDRA-14692:
Ok, I didn't realize you're trying this with a node that has been successfully
joined the ring before. This is in fact easy to reproduce. No idea why
{{isJoined}} has been implemented like it is and {{nodetool join}} will not
work for the described situation. But this doesn't seem to be a regression, and
the log output clearly describes the correct command to use instead. Have there
been any other issues related to that, except from the failing {{nodetool
join}} command?
> join_ring=false populates wrong value into StorageServiceMB and prevents join
> by nodetool
> -
>
> Key: CASSANDRA-14692
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14692
> Project: Cassandra
> Issue Type: Bug
> Components: Lifecycle
>Reporter: Roland Johann
>Priority: Major
> Fix For: 3.11.3
>
> Attachments: Bildschirmfoto 2018-09-05 um 17.29.54.png,
> cassandra1_log, cassandra1_nodetool_gossipinfo, cassandra1_nodetool_status,
> cassandra2_nodetool_gossipinfo, cassandra2_nodetool_status
>
>
> Restarting a cassandra cluster member with option
> {{-Dcassandra.join_ring=false}} populates wrong value to its
> {{StorageServiceMB}} field {{Joined}} which causes the actual trigger to join
> via {{nodetool join}} to abort due to check if {{Join}} in
> {{StorageServiceMB}} is true. Via jconsole it's possible as there is no check.
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/tools/nodetool/Join.java
> {{nodetool status}} also shows that the node is up and in normal node, on the
> rest of the cluster node status is {{DN}}.
> {{nodetool gossipinfo}} states that the non joined node is in gossip state
> {{hibernate}}.
> Came across this issue while evaluated the problem of zombies to integrate
> into automation processes and the documentation states
> {quote}To avoid this problem, run {{nodetool repair}} on any restored node
> before rejoining it to its cluster.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14701) Cleanup (and other) compaction type(s) not counted in compaction remaining time
[
https://issues.apache.org/jira/browse/CASSANDRA-14701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-14701:
---
Priority: Major (was: Critical)
> Cleanup (and other) compaction type(s) not counted in compaction remaining
> time
> ---
>
> Key: CASSANDRA-14701
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14701
> Project: Cassandra
> Issue Type: Bug
> Components: Observability
>Reporter: Thomas Steinmaurer
>Priority: Major
>
> Opened a ticket, as discussed in user list.
> Looks like compaction remaining time only includes compactions of type
> COMPACTION and other compaction types like cleanup etc. aren't part of the
> estimation calculation.
> E.g. from one of our environments:
> {noformat}
> nodetool compactionstats -H
> pending tasks: 1
>compaction type keyspace table completed totalunit
> progress
>CleanupXXX YYY 908.16 GB 1.13 TB bytes
> 78.63%
> Active compaction remaining time : 0h00m00s
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14691) Cassandra 2.1 backport - The JVM should exit if jmx fails to bind
[
https://issues.apache.org/jira/browse/CASSANDRA-14691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610452#comment-16610452
]
Stefan Podkowinski commented on CASSANDRA-14691:
For me, critical also implies that we'd have to cut a new 2.1 release for such
a patch and urge users to update to the new version, and I don't see how that
would be justified in this case.
> Cassandra 2.1 backport - The JVM should exit if jmx fails to bind
> -
>
> Key: CASSANDRA-14691
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14691
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Thomas Steinmaurer
>Priority: Major
> Labels: lhf
> Fix For: 2.1.x
>
>
> If you are already running a cassandra instance, but for some reason try to
> start another one, this happens:
> {noformat}
> INFO 20:57:09 JNA mlockall successful
> WARN 20:57:09 JMX is not enabled to receive remote connections. Please see
> cassandra-env.sh for more info.
> ERROR 20:57:10 Error starting local jmx server:
> java.rmi.server.ExportException: Port already in use: 7199; nested exception
> is:
> java.net.BindException: Address already in use
> at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:340)
> ~[na:1.7.0_76]
> at
> sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:248)
> ~[na:1.7.0_76]
> at
> sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:411)
> ~[na:1.7.0_76]
> at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147)
> ~[na:1.7.0_76]
> at
> sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:207)
> ~[na:1.7.0_76]
> at sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:122)
> ~[na:1.7.0_76]
> at sun.rmi.registry.RegistryImpl.(RegistryImpl.java:98)
> ~[na:1.7.0_76]
> at
> java.rmi.registry.LocateRegistry.createRegistry(LocateRegistry.java:239)
> ~[na:1.7.0_76]
> at
> org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:100)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:222)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:564)
> [main/:na]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:653)
> [main/:na]
> Caused by: java.net.BindException: Address already in use
> at java.net.PlainSocketImpl.socketBind(Native Method) ~[na:1.7.0_76]
> at
> java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
> ~[na:1.7.0_76]
> at java.net.ServerSocket.bind(ServerSocket.java:376) ~[na:1.7.0_76]
> at java.net.ServerSocket.(ServerSocket.java:237) ~[na:1.7.0_76]
> at
> javax.net.DefaultServerSocketFactory.createServerSocket(ServerSocketFactory.java:231)
> ~[na:1.7.0_76]
> at
> org.apache.cassandra.utils.RMIServerSocketFactoryImpl.createServerSocket(RMIServerSocketFactoryImpl.java:13)
> ~[main/:na]
> at
> sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:666)
> ~[na:1.7.0_76]
> at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:329)
> ~[na:1.7.0_76]
> ... 11 common frames omitted
> {noformat}
> However the startup continues, and ends up replaying commitlogs, which is
> probably not a good thing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14558) dtest: log-watching thread leak and thread starvation
Stefan Podkowinski created CASSANDRA-14558:
--
Summary: dtest: log-watching thread leak and thread starvation
Key: CASSANDRA-14558
URL: https://issues.apache.org/jira/browse/CASSANDRA-14558
Project: Cassandra
Issue Type: Bug
Components: Testing
Reporter: Stefan Podkowinski
Assignee: Stefan Podkowinski
We get occasional build timeouts on b.a.o after pytest becomes unresponsive for
over 20 minutes. This will result in a thread dump like this one:
[https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-2.2-dtest/117/consoleFull]
If you look for "Log-watching thread starting" messages and the dump, it
becomes quite obvious whats the issue here.
I had a quick look at the python3 / pytest related changes in CASSANDRA-14134
and it seems that some of the handling around dtest_setup's
{{log_watch_thread}} var has been changed in a way that would prevent
eventually yielding the allocated thread.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13971) Automatic certificate management using Vault
[ https://issues.apache.org/jira/browse/CASSANDRA-13971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-13971: --- Reviewer: (was: Jason Brown) > Automatic certificate management using Vault > > > Key: CASSANDRA-13971 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13971 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Major > Labels: security > Fix For: 4.x > > Attachments: start_vault_ssl.sh > > > We've been adding security features during the last years to enable users to > secure their clusters, if they are willing to use them and do so correctly. > Some features are powerful and easy to work with, such as role based > authorization. Other features that require to manage a local keystore are > rather painful to deal with. Think about setting up SSL.. > To be fair, keystore related issues and certificate handling hasn't been > invented by us. We're just following Java standards there. But that doesn't > mean that we absolutely have to, if there are better options. I'd like to > give it a shoot and find out if we can automate certificate/key handling > (PKI) by using external APIs. In this case, the implementation will be based > on [Vault|https://vaultproject.io]. But certificate management services > offered by cloud providers may also be able to handle the use-case and I > intend to create a generic, pluggable API for that. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14558) dtest: log-watching thread leak and thread starvation
[
https://issues.apache.org/jira/browse/CASSANDRA-14558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16534660#comment-16534660
]
Stefan Podkowinski commented on CASSANDRA-14558:
[~KurtG], can you elaborate a bit on why you think this isn't an actual issue,
as you mentioned in #dev? Shouldn't we have "Log-watching thread exiting"
messages in the log, if the behaviour isn't broken in a way as described in
this ticket?
> dtest: log-watching thread leak and thread starvation
> -
>
> Key: CASSANDRA-14558
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14558
> Project: Cassandra
> Issue Type: Bug
> Components: Testing
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>
> We get occasional build timeouts on b.a.o after pytest becomes unresponsive
> for over 20 minutes. This will result in a thread dump like this one:
> [https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-2.2-dtest/117/consoleFull]
> If you look for "Log-watching thread starting" messages and the dump, it
> becomes quite obvious whats the issue here.
> I had a quick look at the python3 / pytest related changes in CASSANDRA-14134
> and it seems that some of the handling around dtest_setup's
> {{log_watch_thread}} var has been changed in a way that would prevent
> eventually yielding the allocated thread.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14560) rebuild_test.py: Fix broken pytest.raises using incorrect kwarg
[ https://issues.apache.org/jira/browse/CASSANDRA-14560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535701#comment-16535701 ] Stefan Podkowinski commented on CASSANDRA-14560: This has been also addressed in CASSANDRA-14545 along with another related migration kwarg oversight, which you may want to have a look at as well. > rebuild_test.py: Fix broken pytest.raises using incorrect kwarg > --- > > Key: CASSANDRA-14560 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14560 > Project: Cassandra > Issue Type: Bug > Components: Testing >Reporter: Ariel Weisberg >Assignee: Ariel Weisberg >Priority: Major > Fix For: 4.0 > > > This was missed in the Python 2/3 conversion I guess. It uses the wrong > signature for the keyword argument and for some reason it has started showing > up as an error now. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14566) Cache CSM.onlyPurgeRepairedTombstones()
Stefan Podkowinski created CASSANDRA-14566:
--
Summary: Cache CSM.onlyPurgeRepairedTombstones()
Key: CASSANDRA-14566
URL: https://issues.apache.org/jira/browse/CASSANDRA-14566
Project: Cassandra
Issue Type: Improvement
Components: Compaction
Reporter: Stefan Podkowinski
We currently call {{CompactionStrategyManager.onlyPurgeRepairedTombstones()}}
*a lot* during compaction, I think at least for every key. So we should
probably cache the value, instead of constantly reading from a volatile and
calling parseBoolean.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14566) Cache CSM.onlyPurgeRepairedTombstones()
[
https://issues.apache.org/jira/browse/CASSANDRA-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-14566:
---
Labels: lhf performance (was: performance)
> Cache CSM.onlyPurgeRepairedTombstones()
> ---
>
> Key: CASSANDRA-14566
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14566
> Project: Cassandra
> Issue Type: Improvement
> Components: Compaction
>Reporter: Stefan Podkowinski
>Priority: Minor
> Labels: lhf, performance
>
> We currently call {{CompactionStrategyManager.onlyPurgeRepairedTombstones()}}
> *a lot* during compaction, I think at least for every key. So we should
> probably cache the value, instead of constantly reading from a volatile and
> calling parseBoolean.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14204) Nodetool garbagecollect AssertionError
[
https://issues.apache.org/jira/browse/CASSANDRA-14204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542907#comment-16542907
]
Stefan Podkowinski commented on CASSANDRA-14204:
So the patched implementation pretty much follows {{performSSTableRewrite()}},
which looks like the correct way to handle this to me. We could modify
{{GcCompactionTest}} a bit to make some of the effects more testable, see
[ebd7de7|https://github.com/spodkowinski/cassandra/commit/ebd7de758b48a6f924d60eeecbc615c355c87257].
> Nodetool garbagecollect AssertionError
> --
>
> Key: CASSANDRA-14204
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14204
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Vincent White
>Assignee: Vincent White
>Priority: Minor
> Fix For: 3.11.x, 4.x
>
>
> When manually running a garbage collection compaction across a table with
> unrepaired sstables and only_purge_repaired_tombstones set to true an
> assertion error is thrown. This is because the unrepaired sstables aren't
> being removed from the transaction as they are filtered out in
> filterSSTables().
> ||3.11||trunk||
> |[branch|https://github.com/vincewhite/cassandra/commit/e13c822736edd3df3403c02e8ef90816f158cde2]|[branch|https://github.com/vincewhite/cassandra/commit/cc8828576404e72504d9b334be85f84c90e77aa7]|
> The stacktrace:
> {noformat}
> -- StackTrace --
> java.lang.AssertionError
> at
> org.apache.cassandra.db.compaction.CompactionManager.parallelAllSSTableOperation(CompactionManager.java:339)
> at
> org.apache.cassandra.db.compaction.CompactionManager.performGarbageCollection(CompactionManager.java:476)
> at
> org.apache.cassandra.db.ColumnFamilyStore.garbageCollect(ColumnFamilyStore.java:1579)
> at
> org.apache.cassandra.service.StorageService.garbageCollect(StorageService.java:3069)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
> at
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
> at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
> at sun.rmi.transport.Transport$1.run(Transport.java:200)
> at sun.rmi.transport.Transport$1.run(Transport.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
>
[jira] [Resolved] (CASSANDRA-14558) dtest: log-watching thread leak and thread starvation
[
https://issues.apache.org/jira/browse/CASSANDRA-14558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski resolved CASSANDRA-14558.
Resolution: Fixed
Reviewer: Kurt Greaves
Committed as c98469d86 , thanks for bringing it up and reviewing my patch, Kurt!
> dtest: log-watching thread leak and thread starvation
> -
>
> Key: CASSANDRA-14558
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14558
> Project: Cassandra
> Issue Type: Bug
> Components: Testing
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>
> We get occasional build timeouts on b.a.o after pytest becomes unresponsive
> for over 20 minutes. This will result in a thread dump like this one:
> [https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-2.2-dtest/117/consoleFull]
> If you look for "Log-watching thread starting" messages and the dump, it
> becomes quite obvious whats the issue here.
> I had a quick look at the python3 / pytest related changes in CASSANDRA-14134
> and it seems that some of the handling around dtest_setup's
> {{log_watch_thread}} var has been changed in a way that would prevent
> eventually yielding the allocated thread.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14561) Make token-generator Python3 compatible
Stefan Podkowinski created CASSANDRA-14561:
--
Summary: Make token-generator Python3 compatible
Key: CASSANDRA-14561
URL: https://issues.apache.org/jira/browse/CASSANDRA-14561
Project: Cassandra
Issue Type: Improvement
Components: Testing, Tools
Reporter: Stefan Podkowinski
Among all scripts in {{tools/bin/}}, {{token-generator}} is bit of an oddball
there. While all other scripts are simple shell script wrappers around Java
tools, the generator is a standalone Python script. Although it seems to be
still working just fine, it's not Python3 compatible yet, which causes
permanent test failures for {{token_generator_test.py}}, after we migrated to
Python3.
It would be great if we could make the script work with both Python2+3.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Resolved] (CASSANDRA-14181) RPM package has too many executable files
[ https://issues.apache.org/jira/browse/CASSANDRA-14181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski resolved CASSANDRA-14181. Resolution: Fixed Assignee: Troels Arvin Reviewer: Stefan Podkowinski Fix Version/s: 4.0 3.11.2 3.0.16 2.2.12 2.1.20 Again, thanks for the patch! I've tested it locally on Fedora 26 and the affected files are now installed and owned by root using 644 permissions, just as expected. Committed as 5ba9e6da94ed74c11f6ea37199dbe8a501859e7c to 2.1 upwards. > RPM package has too many executable files > - > > Key: CASSANDRA-14181 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14181 > Project: Cassandra > Issue Type: Bug > Components: Packaging >Reporter: Troels Arvin >Assignee: Troels Arvin >Priority: Minor > Fix For: 2.1.20, 2.2.12, 3.0.16, 3.11.2, 4.0 > > Attachments: cassandra-permissions-fix.patch > > > When installing using the RPM files: > In /etc/cassandra/conf, the files should not be execuable, as they are either > * properties-like files > * readme-like files, or > * files to be sourced by shell scripts > I'm adding a patch (cassandra-permissions-fix.patch) to the cassandra.spec > file which fixes this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14128) Prune dead links from cassandra.apache.org
[ https://issues.apache.org/jira/browse/CASSANDRA-14128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390102#comment-16390102 ] Stefan Podkowinski edited comment on CASSANDRA-14128 at 3/7/18 8:12 PM: I don't feel fully comfortable to promote companies the way it is proposed here, just a single click from the homepage. We already throw around names of lot's of companies on the homepage directly and we (the Apache org) need to make sure to not present us as a collaboration between those, but as a commercially independent organization. My suggestion for a compromise would be to add a "Third Party Support" section at the bottom of the "Community" page and list companies in order of number of contributions (patches). We can phrase it like e.g. "the following companies are actively contributing to Apache Cassandra as part of the community and may also be able to offer professional support", or something like that. But just list and link them, without logos. Short line of text might be acceptable. But we should present them as part of the community, that's my main point here, whether on the community page or not. was (Author: spo...@gmail.com): I don't feel fully comfortable to promote companies the way it is proposed here, just a single click from the homepage. We already throw around names of lot's of companies on the homepage directly and we (the Apache org) need to make sure to not present us as a collaboration between those, but as a commercially independent organization. My suggestion for a compromise would be to add a "Professional Services" section at the bottom of the "Community" page and list companies in order of number of contributions (patches). We can phrase it like e.g. "the following companies are actively contributing to Apache Cassandra as part of the community and may also be able to offer professional support", or something like that. But just list and link them, without logos. Short line of text might be acceptable. But we should present them as part of the community, that's my main point here. > Prune dead links from cassandra.apache.org > -- > > Key: CASSANDRA-14128 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14128 > Project: Cassandra > Issue Type: Task > Components: Documentation and Website >Reporter: Jeff Jirsa >Assignee: Kurt Greaves >Priority: Trivial > Labels: lhf > Attachments: 14128-site.patch > > > Lots of dead links on the site, including the homepage. Should do some > pruning cleanup. > Site repo is [here|https://svn.apache.org/repos/asf/cassandra/site/] (in svn) > for anyone who wishes to give this a shot. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14128) Prune dead links from cassandra.apache.org
[ https://issues.apache.org/jira/browse/CASSANDRA-14128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390238#comment-16390238 ] Stefan Podkowinski commented on CASSANDRA-14128: You're right Kenneth, changing the listing the way I suggested is probably not completely fair to everyone. Especially since it's hard to measure contributions. I'll have to give this some more thoughts. > Prune dead links from cassandra.apache.org > -- > > Key: CASSANDRA-14128 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14128 > Project: Cassandra > Issue Type: Task > Components: Documentation and Website >Reporter: Jeff Jirsa >Assignee: Kurt Greaves >Priority: Trivial > Labels: lhf > Attachments: 14128-site.patch > > > Lots of dead links on the site, including the homepage. Should do some > pruning cleanup. > Site repo is [here|https://svn.apache.org/repos/asf/cassandra/site/] (in svn) > for anyone who wishes to give this a shot. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14128) Prune dead links from cassandra.apache.org
[ https://issues.apache.org/jira/browse/CASSANDRA-14128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390102#comment-16390102 ] Stefan Podkowinski commented on CASSANDRA-14128: I don't feel fully comfortable to promote companies the way it is proposed here, just a single click from the homepage. We already throw around names of lot's of companies on the homepage directly and we (the Apache org) need to make sure to not present us as a collaboration between those, but as a commercially independent organization. My suggestion for a compromise would be to add a "Professional Services" section at the bottom of the "Community" page and list companies in order of number of contributions (patches). We can phrase it like e.g. "the following companies are actively contributing to Apache Cassandra as part of the community and may also be able to offer professional support", or something like that. But just list and link them, without logos. Short line of text might be acceptable. But we should present them as part of the community, that's my main point here. > Prune dead links from cassandra.apache.org > -- > > Key: CASSANDRA-14128 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14128 > Project: Cassandra > Issue Type: Task > Components: Documentation and Website >Reporter: Jeff Jirsa >Assignee: Kurt Greaves >Priority: Trivial > Labels: lhf > Attachments: 14128-site.patch > > > Lots of dead links on the site, including the homepage. Should do some > pruning cleanup. > Site repo is [here|https://svn.apache.org/repos/asf/cassandra/site/] (in svn) > for anyone who wishes to give this a shot. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14061) trunk eclipse-warnings
[
https://issues.apache.org/jira/browse/CASSANDRA-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391173#comment-16391173
]
Stefan Podkowinski commented on CASSANDRA-14061:
+1
> trunk eclipse-warnings
> --
>
> Key: CASSANDRA-14061
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14061
> Project: Cassandra
> Issue Type: Bug
> Components: Testing
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
>
> {noformat}
> eclipse-warnings:
> [mkdir] Created dir: /home/ubuntu/cassandra/build/ecj
> [echo] Running Eclipse Code Analysis. Output logged to
> /home/ubuntu/cassandra/build/ecj/eclipse_compiler_checks.txt
> [java] --
> [java] 1. ERROR in
> /home/ubuntu/cassandra/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
> (at line 59)
> [java] return new SSTableIdentityIterator(sstable, key,
> partitionLevelDeletion, file.getPath(), iterator);
> [java]
> ^^^
> [java] Potential resource leak: 'iterator' may not be closed at this
> location
> [java] --
> [java] 2. ERROR in
> /home/ubuntu/cassandra/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
> (at line 79)
> [java] return new SSTableIdentityIterator(sstable, key,
> partitionLevelDeletion, dfile.getPath(), iterator);
> [java]
>
> [java] Potential resource leak: 'iterator' may not be closed at this
> location
> [java] --
> [java] 2 problems (2 errors)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14061) trunk eclipse-warnings
[
https://issues.apache.org/jira/browse/CASSANDRA-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-14061:
---
Status: Ready to Commit (was: Patch Available)
> trunk eclipse-warnings
> --
>
> Key: CASSANDRA-14061
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14061
> Project: Cassandra
> Issue Type: Bug
> Components: Testing
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
>
> {noformat}
> eclipse-warnings:
> [mkdir] Created dir: /home/ubuntu/cassandra/build/ecj
> [echo] Running Eclipse Code Analysis. Output logged to
> /home/ubuntu/cassandra/build/ecj/eclipse_compiler_checks.txt
> [java] --
> [java] 1. ERROR in
> /home/ubuntu/cassandra/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
> (at line 59)
> [java] return new SSTableIdentityIterator(sstable, key,
> partitionLevelDeletion, file.getPath(), iterator);
> [java]
> ^^^
> [java] Potential resource leak: 'iterator' may not be closed at this
> location
> [java] --
> [java] 2. ERROR in
> /home/ubuntu/cassandra/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
> (at line 79)
> [java] return new SSTableIdentityIterator(sstable, key,
> partitionLevelDeletion, dfile.getPath(), iterator);
> [java]
>
> [java] Potential resource leak: 'iterator' may not be closed at this
> location
> [java] --
> [java] 2 problems (2 errors)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13457) Diag. Events: Add base classes
[
https://issues.apache.org/jira/browse/CASSANDRA-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391026#comment-16391026
]
Stefan Podkowinski commented on CASSANDRA-13457:
{quote}diagnostic_events_enabled: true
After recent discussions on the dev ML around the use of experimental flags, eg
on MV, would it make more sense that this was false by default?
{quote}
Publishing events without any subscribers should be noop anyways. I intended
the flag rather as a kill switch, in case the feature is causing any issues in
production. But you're right, it's a new feature and it should probably not be
enabled by default. It's not a big deal to change it to true if you're going to
use it.
([13b1e8adbd|https://github.com/apache/cassandra/commit/13b1e8adbd3f597311b7f67d71318dc939dd54fb])
{quote}DiagnosticEvent.addIfNotNull(..)
this seems it could be a bit trivial… Can we just enforce toString
serialisation in the subclasses instead? (like Hint does it)
{quote}
I'm not really a fan of such helper methods either. I guess my intention was to
keep null values absent from the return mapped and also prevent NPEs. But we
should catch any exception from {{toMap}} anyways, so I'm fine removing it in
favor of {{String.valueOf}} on the caller side.
([5770638d6|https://github.com/apache/cassandra/commit/5770638d689333d59f57a9ef977dbb1fa6964e30])
{quote}Can we avoid the static fields? So to be avoiding adding to the
CASSANDRA-7837 problems… I don't think C* has a better habit in place for this?
But a singleton would be one better than all static fields…
{quote}
I don't mind as long as we try to keep things consistent. What's the latest
trend handling this? {{instance()}}?
But I don't think swapping out the implementation like in {{Trace}} is a use
cases that we have to support here. Basically the "service" is just a
broadcaster that implements publish/subscribe semantics and I can't really
think of any reason someone wants to change that and use their own
implementation.
([ba09523d|https://github.com/apache/cassandra/commit/ba09523de7cad3b3732c0e7e60b072f84e809e21])
{quote}Not too sure why a number of fields in existing classes were changed
from private to package-protected, for example in Gossiper. If it's for tests
in latter branches should they deserve the @VisibleForTesting annotation? And
should that change also happen in the latter branches
{quote}
The {{GossiperEvent}} will access these members from the constructor to collect
relevant details for the event. Passing these values directly from {{Gossiper}}
to {{GossiperEvent.*}} would mean that {{Gossiper}} needs to generate some
garbage by creating immutable copies of some of the values, while at the same
time events might not even be enabled and would be discarded anyways.
{quote}Should the event classes be included in the client jarfile (or some
'type and deserialisation' part of the event classes). This would then
introduce issues of compatibility, (eg enums). If event classes are not exposed
client-side, could they then be package private? (they're not used outside
their package)
{quote}
The implementation in CASSANDRA-13459 would only expose events serialized as
strings. I'd not serialize any internal classes, as we don't have any
versioning in place and classes could change quickly enough to cause
deserialization issues, even on bugfix releases. The only reason for keeping
complex objects as event members would be to make them accessible in unit
tests, see CASSANDRA-13458. That's also the reason I'd prefer to keep classes
public, so they can be used for unit testing.
{quote}What about conglomeration between metrics, diag events, and tracing
events? For example i wonder if the latter two will often pair?, good example
in HintsDispatcher
{quote}
There's probably some overlap, but I'd expect to rather see metrics on hot
paths where you want histograms and need to do local buffering and sampling to
cope with the load. Diag. events should be emitted key events only. As you
mentioned the HintsDispatcher, one of the first versions had events for
dispatched hints. After some basic testing I quickly realized that way too much
events would be produced, if you happen to subscribe to them. You could easily
maintain a histogram for that, but not emit diag events. So it makes sense to
limit emitted events for the hints service to dispatcher and maybe pages.
Although there's some overlap when it comes to looking at the outcome, yes. But
other diag events such as dispatcherCreated, abortRequested, would not make
much sense to have as a metric, as you need a bit more context to make them
useful for further analyzis and the HintEvent has that (hostId, address,
dispatcher,..).
As for tracing events, I'd expect only some pairing between them and diag
events for repairs at this point. Avoiding redundancy should be possible, e.g.
by replacing
[jira] [Created] (CASSANDRA-14298) cqlsh tests broken on b.a.o
Stefan Podkowinski created CASSANDRA-14298: -- Summary: cqlsh tests broken on b.a.o Key: CASSANDRA-14298 URL: https://issues.apache.org/jira/browse/CASSANDRA-14298 Project: Cassandra Issue Type: Bug Components: Build, Testing Reporter: Stefan Podkowinski It appears that cqlsh-tests on builds.apache.org on all branches stopped working since we removed nosetests from the system environment. See e.g. [here|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-cqlsh-tests/458/cython=no,jdk=JDK%201.8%20(latest),label=cassandra/console]. Looks like we either have to make nosetests available again or migrate to pytest as we did with dtests. Giving pytest a quick try resulted in many errors locally, but I haven't inspected them in detail yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14298) cqlshlib tests broken on b.a.o
[ https://issues.apache.org/jira/browse/CASSANDRA-14298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-14298: --- Summary: cqlshlib tests broken on b.a.o (was: cqlsh tests broken on b.a.o) > cqlshlib tests broken on b.a.o > -- > > Key: CASSANDRA-14298 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14298 > Project: Cassandra > Issue Type: Bug > Components: Build, Testing >Reporter: Stefan Podkowinski >Priority: Major > > It appears that cqlsh-tests on builds.apache.org on all branches stopped > working since we removed nosetests from the system environment. See e.g. > [here|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-cqlsh-tests/458/cython=no,jdk=JDK%201.8%20(latest),label=cassandra/console]. > Looks like we either have to make nosetests available again or migrate to > pytest as we did with dtests. Giving pytest a quick try resulted in many > errors locally, but I haven't inspected them in detail yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-7622) Implement virtual tables
[ https://issues.apache.org/jira/browse/CASSANDRA-7622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398915#comment-16398915 ] Stefan Podkowinski commented on CASSANDRA-7622: --- I was rather skeptical at the beginning of the discussion on this issue. But I like the way we moved forward here and the proposed implementation looks promising. Definitely worth the effort. Please keep in mind that it should also be possible to grant permissions based on our RBAC model to virtual tables, too (could be done in another ticket though). I've not looked into it in detail and it might not be a big deal to implement, but it should be possible by working with GRANTs on table level. This requires that any vtables will be known from the start and not will not be registered dynamically. E.g. I'd like to be able to grant select permissions for a certain role on the table 'tablemetrics', instead of having dynamic table names, if that's planed at all. > Implement virtual tables > > > Key: CASSANDRA-7622 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7622 > Project: Cassandra > Issue Type: Improvement >Reporter: Tupshin Harper >Assignee: Chris Lohfink >Priority: Major > Fix For: 4.x > > > There are a variety of reasons to want virtual tables, which would be any > table that would be backed by an API, rather than data explicitly managed and > stored as sstables. > One possible use case would be to expose JMX data through CQL as a > resurrection of CASSANDRA-3527. > Another is a more general framework to implement the ability to expose yaml > configuration information. So it would be an alternate approach to > CASSANDRA-7370. > A possible implementation would be in terms of CASSANDRA-7443, but I am not > presupposing. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14301) Error running cqlsh: unexpected keyword argument 'no_compact'
[
https://issues.apache.org/jira/browse/CASSANDRA-14301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392767#comment-16392767
]
Stefan Podkowinski commented on CASSANDRA-14301:
Looks like you have to bundle a newer version of the python driver in the brew
formula. But we only test and support the python driver shipped as part of the
official distribution, so you might run into unexpected issues using a
different driver artifact.
There seems to be already a brew github issue
[24977|https://github.com/Homebrew/homebrew-core/issues/24977] for that as well.
> Error running cqlsh: unexpected keyword argument 'no_compact'
> -
>
> Key: CASSANDRA-14301
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14301
> Project: Cassandra
> Issue Type: Bug
>Reporter: M. Justin
>Priority: Major
> Labels: cqlsh
>
> I recently installed Cassandra 3.11.2 on my Mac using Homebrew. When I run
> the "cqlsh" command, I get the following error:
> {code:none}
> $ cqlsh
> Traceback (most recent call last):
> File "/usr/local/Cellar/cassandra/3.11.2/libexec/bin/cqlsh.py", line 2443,
> in
> main(*read_options(sys.argv[1:], os.environ))
> File "/usr/local/Cellar/cassandra/3.11.2/libexec/bin/cqlsh.py", line 2421,
> in main
> encoding=options.encoding)
> File "/usr/local/Cellar/cassandra/3.11.2/libexec/bin/cqlsh.py", line 488,
> in __init__
> **kwargs)
> File "cassandra/cluster.py", line 735, in
> cassandra.cluster.Cluster.__init__ (cassandra/cluster.c:10935)
> TypeError: __init__() got an unexpected keyword argument 'no_compact'
> {code}
> Commenting out [line 483 of
> cqlsh.py|https://github.com/apache/cassandra/blob/cassandra-3.11.2/bin/cqlsh.py#L483]
> works around the issue:
> {code}
> # no_compact=no_compact
> {code}
> I am not the only person impacted, as evidenced by [this existing Stack
> Overflow
> post|https://stackoverflow.com/questions/48885984/was-cqlsh-5-0-1-broken-in-cassandra-3-11-2-release]
> from 2/20/2018.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14301) Error running cqlsh: unexpected keyword argument 'no_compact'
[
https://issues.apache.org/jira/browse/CASSANDRA-14301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-14301:
---
Labels: cqlsh proposed-wontfix (was: cqlsh)
> Error running cqlsh: unexpected keyword argument 'no_compact'
> -
>
> Key: CASSANDRA-14301
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14301
> Project: Cassandra
> Issue Type: Bug
>Reporter: M. Justin
>Priority: Major
> Labels: cqlsh, proposed-wontfix
>
> I recently installed Cassandra 3.11.2 on my Mac using Homebrew. When I run
> the "cqlsh" command, I get the following error:
> {code:none}
> $ cqlsh
> Traceback (most recent call last):
> File "/usr/local/Cellar/cassandra/3.11.2/libexec/bin/cqlsh.py", line 2443,
> in
> main(*read_options(sys.argv[1:], os.environ))
> File "/usr/local/Cellar/cassandra/3.11.2/libexec/bin/cqlsh.py", line 2421,
> in main
> encoding=options.encoding)
> File "/usr/local/Cellar/cassandra/3.11.2/libexec/bin/cqlsh.py", line 488,
> in __init__
> **kwargs)
> File "cassandra/cluster.py", line 735, in
> cassandra.cluster.Cluster.__init__ (cassandra/cluster.c:10935)
> TypeError: __init__() got an unexpected keyword argument 'no_compact'
> {code}
> Commenting out [line 483 of
> cqlsh.py|https://github.com/apache/cassandra/blob/cassandra-3.11.2/bin/cqlsh.py#L483]
> works around the issue:
> {code}
> # no_compact=no_compact
> {code}
> I am not the only person impacted, as evidenced by [this existing Stack
> Overflow
> post|https://stackoverflow.com/questions/48885984/was-cqlsh-5-0-1-broken-in-cassandra-3-11-2-release]
> from 2/20/2018.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12041) Add CDC to describe table
[
https://issues.apache.org/jira/browse/CASSANDRA-12041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392750#comment-16392750
]
Stefan Podkowinski commented on CASSANDRA-12041:
I was going through some of the tests failures produced by running
{{pylib/cqlshlib/test}} and it seems we have a little regression going on:
{{nosetests
test_cqlsh_output.py:TestCqlshOutput.test_describe_columnfamily_output}}
{noformat}
FAIL: test_describe_columnfamily_output
(cqlshlib.test.test_cqlsh_output.TestCqlshOutput)
--
Traceback (most recent call last):
File
"/home/spod/git/cassandra-trunk/pylib/cqlshlib/test/test_cqlsh_output.py", line
638, in test_describe_columnfamily_output
self.assertSequenceEqual(output.split('\n'), table_desc3.split('\n'))
AssertionError: Sequences differ: ['', 'CREATE TABLE "CqlshTests... != ['',
'CREATE TABLE "CqlshTests...
First differing element 20:
"AND comment = ''"
'AND cdc = false'
{noformat}
If I understand correctly, the cdc flag is supposed to show up in the output,
which it is not.
> Add CDC to describe table
> -
>
> Key: CASSANDRA-12041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12041
> Project: Cassandra
> Issue Type: Sub-task
> Components: Tools
>Reporter: Carl Yeksigian
>Assignee: Stefania
>Priority: Major
> Labels: client-impacting
> Fix For: 3.8
>
>
> Currently we do not output CDC with {{DESCRIBE TABLE}}, but should include
> that for 3.8+ tables.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14299) cqlsh: ssl setting not read from cqlshrc in 3.11
[
https://issues.apache.org/jira/browse/CASSANDRA-14299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-14299:
---
Assignee: Stefan Podkowinski
Fix Version/s: 4.x
3.11.x
Status: Patch Available (was: Open)
Looks like a merge oversight in
[6d429cd|https://github.com/apache/cassandra/commit/6d429cd]. Trivial fix is
linked, do you mind take a look [~ifesdjeen]?
> cqlsh: ssl setting not read from cqlshrc in 3.11
> -
>
> Key: CASSANDRA-14299
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14299
> Project: Cassandra
> Issue Type: Bug
>Reporter: Christian Becker
>Assignee: Stefan Podkowinski
>Priority: Major
> Fix For: 3.11.x, 4.x
>
>
> With CASSANDRA-10458 an option was added to read the {{--ssl}} flag from
> cqlshrc, however the commit seems to have been incorrectly merged or the
> changes were dropped somehow.
> Currently adding the following has no effect:
> {code:java}
> [connection]
> ssl = true{code}
> When looking at the current tree it's obvious that the flag is not read:
> [https://github.com/apache/cassandra/blame/cassandra-3.11/bin/cqlsh.py#L2247]
> However it should have been added with
> [https://github.com/apache/cassandra/commit/70649a8d65825144fcdbde136d9b6354ef1fb911]
> The values like {{DEFAULT_SSL = False}} are present, but the
> {{option_with_default()}} call is missing.
> Git blame also shows no change to that line which would have reverted the
> change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14061) trunk eclipse-warnings
[
https://issues.apache.org/jira/browse/CASSANDRA-14061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389523#comment-16389523
]
Stefan Podkowinski commented on CASSANDRA-14061:
Looks like {{eclipse-warnings}} is breaking
[trunk-test-all|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-test-all/]
on b.a.o, so we should get this fixed at some point. There's also a new type
of error. Would you like to rebase and look at this again, [~jay.zhuang]? The
suggested SuppressWarnings annotation should be safe, so let's not
unnecessarily complicate matters and go with the suggested changes.
/cc [~aweisberg]
> trunk eclipse-warnings
> --
>
> Key: CASSANDRA-14061
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14061
> Project: Cassandra
> Issue Type: Bug
> Components: Testing
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
>
> {noformat}
> eclipse-warnings:
> [mkdir] Created dir: /home/ubuntu/cassandra/build/ecj
> [echo] Running Eclipse Code Analysis. Output logged to
> /home/ubuntu/cassandra/build/ecj/eclipse_compiler_checks.txt
> [java] --
> [java] 1. ERROR in
> /home/ubuntu/cassandra/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
> (at line 59)
> [java] return new SSTableIdentityIterator(sstable, key,
> partitionLevelDeletion, file.getPath(), iterator);
> [java]
> ^^^
> [java] Potential resource leak: 'iterator' may not be closed at this
> location
> [java] --
> [java] 2. ERROR in
> /home/ubuntu/cassandra/src/java/org/apache/cassandra/io/sstable/SSTableIdentityIterator.java
> (at line 79)
> [java] return new SSTableIdentityIterator(sstable, key,
> partitionLevelDeletion, dfile.getPath(), iterator);
> [java]
>
> [java] Potential resource leak: 'iterator' may not be closed at this
> location
> [java] --
> [java] 2 problems (2 errors)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389656#comment-16389656
]
Stefan Podkowinski commented on CASSANDRA-12151:
To further summarize, let me quote [~jolynch]'s list of collected use cases
once more and add another use case highlighted with bold letters.
{quote} # Logging for security
# Logging for business accounting compliance (e.g. SOX).
# Logging for monetary transaction compliance (e.g. PCI).
# *Logging for data protection compliance (GDPR)*.
# Logging for replay later (e.g. for correctness testing)
# Logging for debugging{quote}
Why is user auditing relevant to comply with GDPR regulations? [Art.
30|https://gdpr-info.eu/art-30-gdpr/] mandates that organizations must maintain
records of processing activities. As already mentioned, this doesn't have to
take place in Cassandra directly and can often be done more effectively at
application level. But if data is manipulated manually, or say you have some
small adhoc tools that change data but won't produce valid logs, you need to be
able to enable auditing logging for these cases, to be able to document how
data was changed.
It's also required by [art. 32|https://gdpr-info.eu/art-32-gdpr/] that any
natural person must only process personal data as instructed, ie. you need to
have an activity log to be able to prove that this is actually the case.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
Stefan Podkowinski created CASSANDRA-14373:
--
Summary: Allow using custom script for chronicle queue BinLog
archival
Key: CASSANDRA-14373
URL: https://issues.apache.org/jira/browse/CASSANDRA-14373
Project: Cassandra
Issue Type: Improvement
Reporter: Stefan Podkowinski
Fix For: 4.x
It would be nice to allow the user to configure an archival script that will be
executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log, just
as we do in {{CommitLogArchiver}}. The script should be able to copy the
released file to an external location or do whatever the author hand in mind.
Deleting the log file should be delegated to the script as well.
See CASSANDRA-13983, CASSANDRA-12151 for use cases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431899#comment-16431899
]
Stefan Podkowinski commented on CASSANDRA-12151:
The native transport integration for diagnostic events is basically just
pushing events one by one over the control connection to the subscribed client.
It's not really designed as a generic, fully scalable server-side data push
solution. There are also no delivery guarantees, as it's really just supposed
for debugging and analysis and not for implementing any control instances on
top if it. The use case I have in mind is to have 1-2 clients subscribing to
some kind of event, either ad-hoc or constantly running in the background. But
I don't really see any use case for having a large fanout of e.g. compaction
events. For that, the solution proposed in CASSANDRA-13459 should be
sufficient. But we should probably discuss further details there, as it's
slightly off-topic for this ticket.
{quote}I think specifying a shell script is probably OK although if someone
specifies the script we should run it immediately once Chronicle rolls the
file. Also if the script is specified we probably shouldn't delete artifacts.
{quote}
I've created a new ticket (CASSANDRA-14373) for this, as it's not strictly an
auditing feature.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13460) Diag. Events: Add local persistency
[
https://issues.apache.org/jira/browse/CASSANDRA-13460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431911#comment-16431911
]
Stefan Podkowinski commented on CASSANDRA-13460:
The proposed solution should be reconsidered using the chronicle queue based
BinLog, instead of writing to a local keyspace. This should be a better
solution for storing temporary, time based and sequentially retrieved events.
We also get better portability by being able to simply copy already rolled over
log files and read them on external systems. E.g. you could ask a user to
enable diag event logging for compactions and have him send you an archive with
all bin logs the next day, just by working with files.
> Diag. Events: Add local persistency
> ---
>
> Key: CASSANDRA-13460
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13460
> Project: Cassandra
> Issue Type: Sub-task
> Components: Observability
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>
> Some generated events will be rather less frequent but very useful for
> retroactive troubleshooting. E.g. all events related to bootstraping and
> gossip would probably be worth saving, as they might provide valuable
> insights and will consume very little resources in low quantities. Imaging if
> we could e.g. in case of CASSANDRA-13348 just ask the user to -run a tool
> like {{./bin/diagdump BootstrapEvent}} on each host, to get us a detailed log
> of all relevant events- provide a dump of all events as described in the
> [documentation|https://github.com/spodkowinski/cassandra/blob/WIP-13460/doc/source/operating/diag_events.rst].
>
> This could be done by saving events white-listed in cassandra.yaml to a local
> table. Maybe using a TTL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13460) Diag. Events: Add local persistency
[
https://issues.apache.org/jira/browse/CASSANDRA-13460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-13460:
---
Status: In Progress (was: Patch Available)
> Diag. Events: Add local persistency
> ---
>
> Key: CASSANDRA-13460
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13460
> Project: Cassandra
> Issue Type: Sub-task
> Components: Observability
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>
> Some generated events will be rather less frequent but very useful for
> retroactive troubleshooting. E.g. all events related to bootstraping and
> gossip would probably be worth saving, as they might provide valuable
> insights and will consume very little resources in low quantities. Imaging if
> we could e.g. in case of CASSANDRA-13348 just ask the user to -run a tool
> like {{./bin/diagdump BootstrapEvent}} on each host, to get us a detailed log
> of all relevant events- provide a dump of all events as described in the
> [documentation|https://github.com/spodkowinski/cassandra/blob/WIP-13460/doc/source/operating/diag_events.rst].
>
> This could be done by saving events white-listed in cassandra.yaml to a local
> table. Maybe using a TTL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14346) Scheduled Repair in Cassandra
[ https://issues.apache.org/jira/browse/CASSANDRA-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426773#comment-16426773 ] Stefan Podkowinski commented on CASSANDRA-14346: If we keep the scope of this ticket to schedule repairs in Cassandra, we should really talk a bit more about the different requirements users have and how using the described solution in practice would look like. There are several aspect to consider for coming up with a working repair schedule: * number of tables (from a single table per cluster to hundreds of tables) * priority in repairing tables (some tables should be repaired more often, others never at all) * data size per table (large table should not block repairs for smaller more important ones) * predictable cluster load (try to schedule repairs off hours) * sustainable repair intensity (repair sessions should not leak into peak hours) * different gc_grace periods (plan intervals for each table so we can tolerate missing a repair run) Repair schedules, which will take these aspects into account, require a certain flexibility and some more careful configuration. Tools, such as reaper, allow you to put together such plans already. Looking at the configuration options described in the design document, I'd probably still want to use such an external tool. That would be mostly due to the use of delays instead of recurring repair times and the way you'd have to configure repairs on table level, which probably gets a bit "messy" fast when you have a lot of tables. The lack of any reporting doesn't help either to further tune these config options afterwards. I think the intention is to keep the scope of this ticket to "integrated repair scheduling and execution", so I'll spare you any of my thoughts about how we should coordinate and execute repairs differently in a post CASSANDRA-9143 world. But if we want to solve scheduling on top of our existing repair implementation, we have to make sure that we can compete with existing 3rd party solutions. So far it was already suggested to move on incrementally. But then we also have to think about how improvements could be implemented on top of the proposed solution. I'd assume that optimizations would be easier to implement in external tools or sidecars that communicates via an IPC interface, compared to a baked in solution, which is using the yaml config, table properties, or has to deal with upgrade paths. From my impression, 3rd party projects are probably also a better place to quickly iterate on these kind of problems. > Scheduled Repair in Cassandra > - > > Key: CASSANDRA-14346 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14346 > Project: Cassandra > Issue Type: Improvement > Components: Repair >Reporter: Joseph Lynch >Priority: Major > Labels: CommunityFeedbackRequested > Fix For: 4.0 > > Attachments: ScheduledRepairV1_20180327.pdf > > > There have been many attempts to automate repair in Cassandra, which makes > sense given that it is necessary to give our users eventual consistency. Most > recently CASSANDRA-10070, CASSANDRA-8911 and CASSANDRA-13924 have all looked > for ways to solve this problem. > At Netflix we've built a scheduled repair service within Priam (our sidecar), > which we spoke about last year at NGCC. Given the positive feedback at NGCC > we focussed on getting it production ready and have now been using it in > production to repair hundreds of clusters, tens of thousands of nodes, and > petabytes of data for the past six months. Also based on feedback at NGCC we > have invested effort in figuring out how to integrate this natively into > Cassandra rather than open sourcing it as an external service (e.g. in Priam). > As such, [~vinaykumarcse] and I would like to re-work and merge our > implementation into Cassandra, and have created a [design > document|https://docs.google.com/document/d/1RV4rOrG1gwlD5IljmrIq_t45rz7H3xs9GbFSEyGzEtM/edit?usp=sharing] > showing how we plan to make it happen, including the the user interface. > As we work on the code migration from Priam to Cassandra, any feedback would > be greatly appreciated about the interface or v1 implementation features. I > have tried to call out in the document features which we explicitly consider > future work (as well as a path forward to implement them in the future) > because I would very much like to get this done before the 4.0 merge window > closes, and to do that I think aggressively pruning scope is going to be a > necessity. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail:
[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426915#comment-16426915
]
Stefan Podkowinski edited comment on CASSANDRA-12151 at 4/5/18 1:25 PM:
I was just giving the BinLogger a try and it looks like the way the bin log
files are created, should make it possible to archive them before they get
deleted. The most simple way would probably to run a rsync job for pulling the
latest chronicle bin logs from the nodes. The timestamp based naming of the
files should make this trivial.
Another option would be to allow the user to configure an archival script that
will be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin
log, just as we do in {{CommitLogArchiver}}. WDYT [~aweisberg]?
I'm going to write a custom {{IAuditLogger}} implementation tomorrow and will
get back with some more feedback.
was (Author: spo...@gmail.com):
I was just giving the BinLogger a try and it looks like the way the bin log
files are created, should make it possible to archive them before they get
deleted. The most simple way would probably to run a rsync job for pulling the
latest chronicle bin logs from the nodes. The timestamp based naming of the
files should make this trivial.
Another option would be to allow the user to configure an archival script that
will be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin
log, just as we do in {{CommitLogArchiver}}. WDYT [~aweisberg]?
I'm going to write a custom implementation tomorrow and will get back with some
more feedback.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426915#comment-16426915
]
Stefan Podkowinski commented on CASSANDRA-12151:
I was just giving the BinLogger a try and it looks like the way the bin log
files are created, should make it possible to archive them before they get
deleted. The most simple way would probably to run a rsync job for pulling the
latest chronicle bin logs from the nodes. The timestamp based naming of the
files should make this trivial.
Another option would be to allow the user to configure an archival script that
will be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin
log, just as we do in {{CommitLogArchiver}}. WDYT [~aweisberg]?
I'm going to write a custom implementation tomorrow and will get back with some
more feedback.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13971) Automatic certificate management using Vault
[ https://issues.apache.org/jira/browse/CASSANDRA-13971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433902#comment-16433902 ] Stefan Podkowinski commented on CASSANDRA-13971: Any update on the review status [~jasobrown], now that the 4.0 window is about to close? > Automatic certificate management using Vault > > > Key: CASSANDRA-13971 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13971 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Major > Labels: security > Fix For: 4.x > > > We've been adding security features during the last years to enable users to > secure their clusters, if they are willing to use them and do so correctly. > Some features are powerful and easy to work with, such as role based > authorization. Other features that require to manage a local keystore are > rather painful to deal with. Think about setting up SSL.. > To be fair, keystore related issues and certificate handling hasn't been > invented by us. We're just following Java standards there. But that doesn't > mean that we absolutely have to, if there are better options. I'd like to > give it a shoot and find out if we can automate certificate/key handling > (PKI) by using external APIs. In this case, the implementation will be based > on [Vault|https://vaultproject.io]. But certificate management services > offered by cloud providers may also be able to handle the use-case and I > intend to create a generic, pluggable API for that. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13459) Diag. Events: Native transport integration
[
https://issues.apache.org/jira/browse/CASSANDRA-13459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433832#comment-16433832
]
Stefan Podkowinski commented on CASSANDRA-13459:
{quote}So I was just thinking that forward looking restricting this mechanism
to diagnostic events might not make sense. I was thinking a more generic
subscription mechanism where diagnostic are events is a subset of what clients
can conditionally subscribe to means we don't end up with naming issues in the
future.
{quote}
We could specify a subscription mechanism for native transport that is not
specific to diag events. But what would the subject look like to subscribe to?
If we want to support more powerful publish/subscribe semantics, we'd have to
allow clients to specify event matchers in a more generic way, e.g. by using
some kind of query language.
Examples
All auditing events for "ks" keyspace updates:
{{SUBSCRIBE diag_events "event=AuditEvent#UPDATE(keyspace=ks)"}}
Full query replication for ks.table1:
{{SUBSCRIBE full_query_log "keyspace=ks,table=table1"}}
Subscribe to any row updates matching a query:
{{SUBSCRIBE cdc "select * from order_status where order_id = 1"}}
Not a big fan of language-in-language, but I'm open to discuss any options, if
that's something we should add.
{quote}For V1 of this functionality my only sticking point is that even with
1-2 clients consuming diagnostic events we have to handle backpressure somehow.
AFAIK we hold onto messages pending to a client for a while (indefinitely?). I
am not actually sure what kind fo timeouts or health checks we do for clients.
{quote}
The current implementation will simply write and flush any event message to the
netty stack. Netty should use it's own event loop, but I'm not sure how
buffering is handled in detail. Maybe we could also use the
{{Message.Dispatcher}} instead and add even messages to the (unbounded) queue
of items to flush. But we don't do that either for "classic" schema/topo/status
change messages.
> Diag. Events: Native transport integration
> --
>
> Key: CASSANDRA-13459
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13459
> Project: Cassandra
> Issue Type: Sub-task
> Components: CQL
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
> Labels: client-impacting
>
> Events should be consumable by clients that would received subscribed events
> from the connected node. This functionality is designed to work on top of
> native transport with minor modifications to the protocol standard (see
> [original
> proposal|https://docs.google.com/document/d/1uEk7KYgxjNA0ybC9fOuegHTcK3Yi0hCQN5nTp5cNFyQ/edit?usp=sharing]
> for further considered options). First we have to add another value for
> existing event types. Also, we have to extend the protocol a bit to be able
> to specify a sub-class and sub-type value. E.g.
> {{DIAGNOSTIC_EVENT(GossiperEvent, MAJOR_STATE_CHANGE_HANDLED)}}. This still
> has to be worked out and I'd appreciate any feedback.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435587#comment-16435587 ] Stefan Podkowinski commented on CASSANDRA-12151: I must have missed that log statement when filtering by user. Its a bit strange that failed login attempts don't get logged, if you filter by a particular user. But that's probably more a minor issue compared to the the question regarding how we should go on with prepared statements. Cause I can't see how we can get away with not logging these, if we want to address any compliance or security use cases. Any ideas? > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, CASSANDRA_12151-benchmark.html, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13668) Diag. events for user audit logging
[ https://issues.apache.org/jira/browse/CASSANDRA-13668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-13668: --- Status: In Progress (was: Patch Available) > Diag. events for user audit logging > --- > > Key: CASSANDRA-13668 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13668 > Project: Cassandra > Issue Type: Improvement > Components: Observability >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Major > Fix For: 4.x > > > With the availability of CASSANDRA-13459, any native transport enabled client > will be able to subscribe to internal Cassandra events. External tools can > take advantage by monitoring these events in various ways. Use-cases for this > can be e.g. auditing tools for compliance and security purposes. > The scope of this ticket is to add diagnostic events that are raised around > authentication and CQL operations. These events can then be consumed and used > by external tools to implement a Cassandra user auditing solution. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13668) Diag. events for user audit logging
[ https://issues.apache.org/jira/browse/CASSANDRA-13668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-13668: --- Summary: Diag. events for user audit logging (was: Database user auditing events) > Diag. events for user audit logging > --- > > Key: CASSANDRA-13668 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13668 > Project: Cassandra > Issue Type: Improvement > Components: Observability >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Major > Fix For: 4.x > > > With the availability of CASSANDRA-13459, any native transport enabled client > will be able to subscribe to internal Cassandra events. External tools can > take advantage by monitoring these events in various ways. Use-cases for this > can be e.g. auditing tools for compliance and security purposes. > The scope of this ticket is to add diagnostic events that are raised around > authentication and CQL operations. These events can then be consumed and used > by external tools to implement a Cassandra user auditing solution. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430448#comment-16430448
]
Stefan Podkowinski edited comment on CASSANDRA-12151 at 4/9/18 1:08 PM:
I've now managed to update CASSANDRA-13668 by implementing IAuditLogger and
expose audit events as diagnostic events via native transport to subscribed
clients. Went pretty much as expected and seems to work fine.
Smaller issues I've came across:
* AuditLogUtil doesn't look very useful. DEFAULT_SOURCE is only used once,
SYSTEM_USER not at all.
* Do we really want to keep IAuditLogger.error()? Please provide a description
of the log/error semantics in context with audit logging for possible
subclasses, or get rid of error().
* AuditLogFilter.isFiltered(): proposed logic will ignore includeSet if
excludeSet is provided (may not make sense to do so, but not strictly forbidden
by cassandra.yaml either), e.g. exclude(A), include(A,B,C) should only have B,C
pass
There are also a couple of limitations:
* Username will not be provided for failed authentications
* Bound values will not get logged for prepared statements (as already pointed
out in this ticket's discussion)
I haven't found a quick way to work around these, but being able to avoid the
audit log by using prepared statements, is something we have to address. It's
probably not going to be that much of an issue for my use case logging ad-hoc
commands for regular users, once we have CASSANDRA-8303 and can disable
prepared statement for them. But for logging all activity for application
users, I don't know.
[~laxmikant99]
{quote}Can we have a configurable property like exitOnAuditFailure ? This
fulfills requirement of strict auditing .. I mean in case auditing fails for a
db operation, then the operation should not get executed.
{quote}
Any logging to the BinLogger should block by default. But it doesn't exit the
JVM.
was (Author: spo...@gmail.com):
I've now managed to update CASSANDRA-13668 by implementing IAuditLogger and
expose audit events as diagnostic events via native transport to subscribed
clients. Went pretty much as expected and seems to work fine.
Smaller issues I've came across:
* AuditLogUtil doesn't look very useful. DEFAULT_SOURCE is only used once,
SYSTEM_USER not at all.
* Do we really want to keep IAuditLogger.error()? Please provide a description
of the log/error semantics in context with audit logging for possible
subclasses, or get rid of error().
* AuditLogFilter.isFiltered(): proposed logic will ignore includeSet if
excludeSet is provided (may not make sense to do so, but not strictly forbidden
by cassandra.yaml either), e.g. exclude(A), include(A,B,C) should only have B,C
pass
There are also a couple of limitations:
* Username will not be provided for failed authentications
* Bound values will not get logged for prepared statements
I haven't found a quick way to work around these, but being able to avoid the
audit log by using prepared statements, is something we have to address. It's
probably not going to be that much of an issue for my use case logging ad-hoc
commands for regular users, once we have CASSANDRA-8303 and can disable
prepared statement for them. But for logging all activity for application
users, I don't know.
[~laxmikant99]
{quote}Can we have a configurable property like exitOnAuditFailure ? This
fulfills requirement of strict auditing .. I mean in case auditing fails for a
db operation, then the operation should not get executed.
{quote}
Any logging to the BinLogger should block by default. But it doesn't exit the
JVM.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430448#comment-16430448
]
Stefan Podkowinski commented on CASSANDRA-12151:
I've now managed to update CASSANDRA-13668 by implementing IAuditLogger and
expose audit events as diagnostic events via native transport to subscribed
clients. Went pretty much as expected and seems to work fine.
Smaller issues I've came across:
* AuditLogUtil doesn't look very useful. DEFAULT_SOURCE is only used once,
SYSTEM_USER not at all.
* Do we really want to keep IAuditLogger.error()? Please provide a description
of the log/error semantics in context with audit logging for possible
subclasses, or get rid of error().
* AuditLogFilter.isFiltered(): proposed logic will ignore includeSet if
excludeSet is provided (may not make sense to do so, but not strictly forbidden
by cassandra.yaml either), e.g. exclude(A), include(A,B,C) should only have B,C
pass
There are also a couple of limitations:
* Username will not be provided for failed authentications
* Bound values will not get logged for prepared statements
I haven't found a quick way to work around these, but being able to avoid the
audit log by using prepared statements, is something we have to address. It's
probably not going to be that much of an issue for my use case logging ad-hoc
commands for regular users, once we have CASSANDRA-8303 and can disable
prepared statement for them. But for logging all activity for application
users, I don't know.
[~laxmikant99]
{quote}Can we have a configurable property like exitOnAuditFailure ? This
fulfills requirement of strict auditing .. I mean in case auditing fails for a
db operation, then the operation should not get executed.
{quote}
Any logging to the BinLogger should block by default. But it doesn't exit the
JVM.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14298) cqlshlib tests broken on b.a.o
[ https://issues.apache.org/jira/browse/CASSANDRA-14298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436944#comment-16436944 ] Stefan Podkowinski commented on CASSANDRA-14298: Maybe you would also be interested to review CASSANDRA-14299, Patrick? > cqlshlib tests broken on b.a.o > -- > > Key: CASSANDRA-14298 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14298 > Project: Cassandra > Issue Type: Bug > Components: Build, Testing >Reporter: Stefan Podkowinski >Assignee: Patrick Bannister >Priority: Major > > It appears that cqlsh-tests on builds.apache.org on all branches stopped > working since we removed nosetests from the system environment. See e.g. > [here|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-cqlsh-tests/458/cython=no,jdk=JDK%201.8%20(latest),label=cassandra/console]. > Looks like we either have to make nosetests available again or migrate to > pytest as we did with dtests. Giving pytest a quick try resulted in many > errors locally, but I haven't inspected them in detail yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13348) Duplicate tokens after bootstrap
[
https://issues.apache.org/jira/browse/CASSANDRA-13348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437217#comment-16437217
]
Stefan Podkowinski commented on CASSANDRA-13348:
[~dikanggu], are you still working on this or should we close this ticket as
"Unable to reproduce"?
[~tvdw], did the issue happen again in the mean time?
> Duplicate tokens after bootstrap
>
>
> Key: CASSANDRA-13348
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13348
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tom van der Woerdt
>Assignee: Dikang Gu
>Priority: Blocker
> Fix For: 3.0.x
>
>
> This one is a bit scary, and probably results in data loss. After a bootstrap
> of a few new nodes into an existing cluster, two new nodes have chosen some
> overlapping tokens.
> In fact, of the 256 tokens chosen, 51 tokens were already in use on the other
> node.
> Node 1 log :
> {noformat}
> INFO [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:42:43,461
> StorageService.java:1160 - JOINING: waiting for ring information
> INFO [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:42:43,461
> StorageService.java:1160 - JOINING: waiting for schema information to complete
> INFO [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:42:43,461
> StorageService.java:1160 - JOINING: schema complete, ready to bootstrap
> INFO [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:42:43,462
> StorageService.java:1160 - JOINING: waiting for pending range calculation
> INFO [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:42:43,462
> StorageService.java:1160 - JOINING: calculation complete, ready to bootstrap
> INFO [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:42:43,462
> StorageService.java:1160 - JOINING: getting bootstrap token
> WARN [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:42:43,564
> TokenAllocation.java:61 - Selected tokens [, 2959334889475814712,
> 3727103702384420083, 7183119311535804926, 6013900799616279548,
> -1222135324851761575, 1645259890258332163, -1213352346686661387,
> 7604192574911909354]
> WARN [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:42:43,729
> TokenAllocation.java:65 - Replicated node load in datacentre before
> allocation max 1.00 min 1.00 stddev 0.
> WARN [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:42:43,729
> TokenAllocation.java:66 - Replicated node load in datacentre after allocation
> max 1.00 min 1.00 stddev 0.
> WARN [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:42:43,729
> TokenAllocation.java:70 - Unexpected growth in standard deviation after
> allocation.
> INFO [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:42:44,150
> StorageService.java:1160 - JOINING: sleeping 3 ms for pending range setup
> INFO [RMI TCP Connection(107)-127.0.0.1] 2017-03-09 07:43:14,151
> StorageService.java:1160 - JOINING: Starting to bootstrap...
> {noformat}
> Node 2 log:
> {noformat}
> INFO [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:55:51,937
> StorageService.java:971 - Joining ring by operator request
> INFO [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:55:52,513
> StorageService.java:1160 - JOINING: waiting for ring information
> INFO [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:55:52,513
> StorageService.java:1160 - JOINING: waiting for schema information to complete
> INFO [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:55:52,513
> StorageService.java:1160 - JOINING: schema complete, ready to bootstrap
> INFO [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:55:52,513
> StorageService.java:1160 - JOINING: waiting for pending range calculation
> INFO [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:55:52,514
> StorageService.java:1160 - JOINING: calculation complete, ready to bootstrap
> INFO [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:55:52,514
> StorageService.java:1160 - JOINING: getting bootstrap token
> WARN [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:55:52,630
> TokenAllocation.java:61 - Selected tokens [.., 2890709530010722764,
> -2416006722819773829, -5820248611267569511, -5990139574852472056,
> 1645259890258332163, 9135021011763659240, -5451286144622276797,
> 7604192574911909354]
> WARN [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:55:52,794
> TokenAllocation.java:65 - Replicated node load in datacentre before
> allocation max 1.02 min 0.98 stddev 0.
> WARN [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:55:52,795
> TokenAllocation.java:66 - Replicated node load in datacentre after allocation
> max 1.00 min 1.00 stddev 0.
> INFO [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:55:53,149
> StorageService.java:1160 - JOINING: sleeping 3 ms for pending range setup
> INFO [RMI TCP Connection(380)-127.0.0.1] 2017-03-17 15:56:23,149
>
[jira] [Updated] (CASSANDRA-14299) cqlsh: ssl setting not read from cqlshrc in 3.11
[
https://issues.apache.org/jira/browse/CASSANDRA-14299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-14299:
---
Resolution: Fixed
Fix Version/s: (was: 3.11.x)
(was: 4.x)
3.11.3
Status: Resolved (was: Patch Available)
Merged as 845243db93a5add273f6e on 3.11
Thx Jay!
> cqlsh: ssl setting not read from cqlshrc in 3.11
> -
>
> Key: CASSANDRA-14299
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14299
> Project: Cassandra
> Issue Type: Bug
>Reporter: Christian Becker
>Assignee: Stefan Podkowinski
>Priority: Major
> Fix For: 3.11.3
>
>
> With CASSANDRA-10458 an option was added to read the {{--ssl}} flag from
> cqlshrc, however the commit seems to have been incorrectly merged or the
> changes were dropped somehow.
> Currently adding the following has no effect:
> {code:java}
> [connection]
> ssl = true{code}
> When looking at the current tree it's obvious that the flag is not read:
> [https://github.com/apache/cassandra/blame/cassandra-3.11/bin/cqlsh.py#L2247]
> However it should have been added with
> [https://github.com/apache/cassandra/commit/70649a8d65825144fcdbde136d9b6354ef1fb911]
> The values like {{DEFAULT_SSL = False}} are present, but the
> {{option_with_default()}} call is missing.
> Git blame also shows no change to that line which would have reverted the
> change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-7544) Allow storage port to be configurable per node
[ https://issues.apache.org/jira/browse/CASSANDRA-7544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440600#comment-16440600 ] Stefan Podkowinski commented on CASSANDRA-7544: --- Having both --port and --with-port options for nodetool is slightly confusing. Can we rename --with-port to something like --print-port? > Allow storage port to be configurable per node > -- > > Key: CASSANDRA-7544 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7544 > Project: Cassandra > Issue Type: Improvement >Reporter: Sam Overton >Assignee: Ariel Weisberg >Priority: Major > Fix For: 4.0 > > > Currently storage_port must be configured identically on all nodes in a > cluster and it is assumed that this is the case when connecting to a remote > node. > This prevents running in any environment that requires multiple nodes to be > able to bind to the same network interface, such as with many automatic > provisioning/deployment frameworks. > The current solutions seems to be > * use a separate network interface for each node deployed to the same box. > This puts a big requirement on IP allocation at large scale. > * allow multiple clusters to be provisioned from the same resource pool, but > restrict allocation to a maximum of one node per host from each cluster, > assuming each cluster is running on a different storage port. > It would make operations much simpler in these kind of environments if the > environment provisioning the resources could assign the ports to be used when > bringing up a new node on shared hardware. > The changes required would be at least the following: > 1. configure seeds as IP:port instead of just IP > 2. gossip the storage port as part of a node's ApplicationState > 3. refer internally to nodes by hostID instead of IP, since there will be > multiple nodes with the same IP > (1) & (2) are mostly trivial and I already have a patch for these. The bulk > of the work to enable this is (3), and I would structure this as a separate > pre-requisite patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13047) Point cqlsh help to the new doc
[ https://issues.apache.org/jira/browse/CASSANDRA-13047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440571#comment-16440571 ] Stefan Podkowinski commented on CASSANDRA-13047: We should probably wait until CASSANDRA-13907 and point to the correct document version, instead of always showing the most recent CQL spec. > Point cqlsh help to the new doc > --- > > Key: CASSANDRA-13047 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13047 > Project: Cassandra > Issue Type: Bug > Components: CQL >Reporter: Sylvain Lebresne >Assignee: Patrick Bannister >Priority: Major > Fix For: 4.0 > > Attachments: 13047-3.11.txt > > > Cqlsh still points to the "old" textile CQL doc, but that's not really > maintain anymore now that we have the new doc (which include everything the > old doc had and more). We should update cqlsh to point to the new doc so we > can remove the old one completely. > Any takers? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14298) cqlshlib tests broken on b.a.o
[ https://issues.apache.org/jira/browse/CASSANDRA-14298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443213#comment-16443213 ] Stefan Podkowinski commented on CASSANDRA-14298: You do use [nose2pytest|https://github.com/pytest-dev/nose2pytest] for converting the tests to pytest, are you? It should be able to convert most of the code automatically. > cqlshlib tests broken on b.a.o > -- > > Key: CASSANDRA-14298 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14298 > Project: Cassandra > Issue Type: Bug > Components: Build, Testing >Reporter: Stefan Podkowinski >Assignee: Patrick Bannister >Priority: Major > Attachments: cqlsh_tests_notes.md > > > It appears that cqlsh-tests on builds.apache.org on all branches stopped > working since we removed nosetests from the system environment. See e.g. > [here|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-cqlsh-tests/458/cython=no,jdk=JDK%201.8%20(latest),label=cassandra/console]. > Looks like we either have to make nosetests available again or migrate to > pytest as we did with dtests. Giving pytest a quick try resulted in many > errors locally, but I haven't inspected them in detail yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14392) Rename nodetool --with-port to --print-port to disambiguate from --port
[ https://issues.apache.org/jira/browse/CASSANDRA-14392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-14392: --- Reviewer: Stefan Podkowinski > Rename nodetool --with-port to --print-port to disambiguate from --port > --- > > Key: CASSANDRA-14392 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14392 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Ariel Weisberg >Assignee: Ariel Weisberg >Priority: Major > Fix For: 4.0 > > > Right now it looks kind of like a third way to set the port number. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-9430) Add startup options to cqlshrc
[
https://issues.apache.org/jira/browse/CASSANDRA-9430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-9430:
--
Reviewer: Stefan Podkowinski
> Add startup options to cqlshrc
> --
>
> Key: CASSANDRA-9430
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9430
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
>Reporter: Jeremy Hanna
>Priority: Minor
> Labels: cqlsh, lhf
>
> There are certain settings that would be nice to set defaults for in the
> cqlshrc file. For example, a user may want to set the paging to off by
> default for their environment. You can't simply do
> {code}
> echo "paging off;" | cqlsh
> {code}
> because this would disable paging and immediately exit cqlsh.
> So it would be nice to have a section of the cqlshrc to include default
> settings on startup.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14392) Rename nodetool --with-port to --print-port to disambiguate from --port
[ https://issues.apache.org/jira/browse/CASSANDRA-14392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442071#comment-16442071 ] Stefan Podkowinski commented on CASSANDRA-14392: +1 > Rename nodetool --with-port to --print-port to disambiguate from --port > --- > > Key: CASSANDRA-14392 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14392 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Ariel Weisberg >Assignee: Ariel Weisberg >Priority: Major > Fix For: 4.0 > > > Right now it looks kind of like a third way to set the port number. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14298) cqlshlib tests broken on b.a.o
[ https://issues.apache.org/jira/browse/CASSANDRA-14298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-14298: --- Reviewer: Stefan Podkowinski > cqlshlib tests broken on b.a.o > -- > > Key: CASSANDRA-14298 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14298 > Project: Cassandra > Issue Type: Bug > Components: Build, Testing >Reporter: Stefan Podkowinski >Assignee: Patrick Bannister >Priority: Major > Attachments: cqlsh_tests_notes.md > > > It appears that cqlsh-tests on builds.apache.org on all branches stopped > working since we removed nosetests from the system environment. See e.g. > [here|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-cqlsh-tests/458/cython=no,jdk=JDK%201.8%20(latest),label=cassandra/console]. > Looks like we either have to make nosetests available again or migrate to > pytest as we did with dtests. Giving pytest a quick try resulted in many > errors locally, but I haven't inspected them in detail yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14400) Subrange repair doesn't always mark as repaired
[
https://issues.apache.org/jira/browse/CASSANDRA-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-14400:
---
Since Version: 4.0
Fix Version/s: (was: 4.0)
> Subrange repair doesn't always mark as repaired
> ---
>
> Key: CASSANDRA-14400
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14400
> Project: Cassandra
> Issue Type: Bug
>Reporter: Kurt Greaves
>Priority: Major
>
> So was just messing around with subrange repair on trunk and found that if I
> generated an SSTable with a single token and then tried to repair that
> SSTable using subrange repairs it wouldn't get marked as repaired.
>
> Before repair:
> {code:java}
> First token: -9223362383595311662 (derphead4471291)
> Last token: -9223362383595311662 (derphead4471291)
> Repaired at: 0
> Pending repair: 862395e0-4394-11e8-8f20-3b8ee110d005
> {code}
> Repair command:
> {code}
> ccm node1 nodetool "repair -st -9223362383595311663 -et -9223362383595311661
> aoeu"
> [2018-04-19 05:44:42,806] Starting repair command #7
> (c23f76c0-4394-11e8-8f20-3b8ee110d005), repairing keyspace aoeu with repair
> options (parallelism: parallel, primary range: false, incremental: true, job
> threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], previewKind:
> NONE, # of ranges: 1, pull repair: false, force repair: false, optimise
> streams: false)
> [2018-04-19 05:44:42,843] Repair session c242d220-4394-11e8-8f20-3b8ee110d005
> for range [(-9223362383595311663,-9223362383595311661]] finished (progress:
> 20%)
> [2018-04-19 05:44:43,139] Repair completed successfully
> [2018-04-19 05:44:43,140] Repair command #7 finished in 0 seconds
> {code}
> After repair SSTable hasn't changed and sstablemetadata outputs:
> {code}
> First token: -9223362383595311662 (derphead4471291)
> Last token: -9223362383595311662 (derphead4471291)
> Repaired at: 0
> Pending repair: 862395e0-4394-11e8-8f20-3b8ee110d005
> {code}
> And parent_repair_history states that the repair is complete/range was
> successful:
> {code}
> select * from system_distributed.parent_repair_history where
> parent_id=862395e0-4394-11e8-8f20-3b8ee110d005 ;
> parent_id| columnfamily_names |
> exception_message | exception_stacktrace | finished_at |
> keyspace_name | options
>
>
> | requested_ranges
> | started_at | successful_ranges
> --++---+--+-+---++-+-+-
> 862395e0-4394-11e8-8f20-3b8ee110d005 | {'aoeu'} |
> null | null | 2018-04-19 05:43:14.578000+ | aoeu
> | {'dataCenters': '', 'forceRepair': 'false', 'hosts': '', 'incremental':
> 'true', 'jobThreads': '1', 'optimiseStreams': 'false', 'parallelism':
> 'parallel', 'previewKind': 'NONE', 'primaryRange': 'false', 'pullRepair':
> 'false', 'sub_range_repair': 'true', 'trace': 'false'} |
> {'(-9223362383595311663,-9223362383595311661]'} | 2018-04-19
> 05:43:01.952000+ | {'(-9223362383595311663,-9223362383595311661]'}
> {code}
> Subrange repairs seem to work fine over large ranges and set {{Repaired at}}
> as expected, but I haven't figured out why it works for a large range versus
> a small range so far.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14400) Subrange repair doesn't always mark as repaired
[
https://issues.apache.org/jira/browse/CASSANDRA-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443660#comment-16443660
]
Stefan Podkowinski commented on CASSANDRA-14400:
The main reason for not marking sstables as repaired on sub-range repairs, was
to avoid anti-compaction. Creating lots of small tables for small repair ranges
will be inefficient and should also not be necessary, as incremental repairs
should be run often enough to keep the unrepaired set reasonably small.
> Subrange repair doesn't always mark as repaired
> ---
>
> Key: CASSANDRA-14400
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14400
> Project: Cassandra
> Issue Type: Bug
>Reporter: Kurt Greaves
>Priority: Major
>
> So was just messing around with subrange repair on trunk and found that if I
> generated an SSTable with a single token and then tried to repair that
> SSTable using subrange repairs it wouldn't get marked as repaired.
>
> Before repair:
> {code:java}
> First token: -9223362383595311662 (derphead4471291)
> Last token: -9223362383595311662 (derphead4471291)
> Repaired at: 0
> Pending repair: 862395e0-4394-11e8-8f20-3b8ee110d005
> {code}
> Repair command:
> {code}
> ccm node1 nodetool "repair -st -9223362383595311663 -et -9223362383595311661
> aoeu"
> [2018-04-19 05:44:42,806] Starting repair command #7
> (c23f76c0-4394-11e8-8f20-3b8ee110d005), repairing keyspace aoeu with repair
> options (parallelism: parallel, primary range: false, incremental: true, job
> threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], previewKind:
> NONE, # of ranges: 1, pull repair: false, force repair: false, optimise
> streams: false)
> [2018-04-19 05:44:42,843] Repair session c242d220-4394-11e8-8f20-3b8ee110d005
> for range [(-9223362383595311663,-9223362383595311661]] finished (progress:
> 20%)
> [2018-04-19 05:44:43,139] Repair completed successfully
> [2018-04-19 05:44:43,140] Repair command #7 finished in 0 seconds
> {code}
> After repair SSTable hasn't changed and sstablemetadata outputs:
> {code}
> First token: -9223362383595311662 (derphead4471291)
> Last token: -9223362383595311662 (derphead4471291)
> Repaired at: 0
> Pending repair: 862395e0-4394-11e8-8f20-3b8ee110d005
> {code}
> And parent_repair_history states that the repair is complete/range was
> successful:
> {code}
> select * from system_distributed.parent_repair_history where
> parent_id=862395e0-4394-11e8-8f20-3b8ee110d005 ;
> parent_id| columnfamily_names |
> exception_message | exception_stacktrace | finished_at |
> keyspace_name | options
>
>
> | requested_ranges
> | started_at | successful_ranges
> --++---+--+-+---++-+-+-
> 862395e0-4394-11e8-8f20-3b8ee110d005 | {'aoeu'} |
> null | null | 2018-04-19 05:43:14.578000+ | aoeu
> | {'dataCenters': '', 'forceRepair': 'false', 'hosts': '', 'incremental':
> 'true', 'jobThreads': '1', 'optimiseStreams': 'false', 'parallelism':
> 'parallel', 'previewKind': 'NONE', 'primaryRange': 'false', 'pullRepair':
> 'false', 'sub_range_repair': 'true', 'trace': 'false'} |
> {'(-9223362383595311663,-9223362383595311661]'} | 2018-04-19
> 05:43:01.952000+ | {'(-9223362383595311663,-9223362383595311661]'}
> {code}
> Subrange repairs seem to work fine over large ranges and set {{Repaired at}}
> as expected, but I haven't figured out why it works for a large range versus
> a small range so far.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14400) Subrange repair doesn't always mark as repaired
[
https://issues.apache.org/jira/browse/CASSANDRA-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443660#comment-16443660
]
Stefan Podkowinski edited comment on CASSANDRA-14400 at 4/19/18 7:49 AM:
-
The main reason for not marking sstables as repaired on sub-range repairs, was
to avoid anti-compaction. Creating lots of small tables for small repair ranges
will be inefficient and should also not be necessary, as incremental repairs
should be run often enough to keep the unrepaired set reasonably small. See
also CASSANDRA-13818.
was (Author: spo...@gmail.com):
The main reason for not marking sstables as repaired on sub-range repairs, was
to avoid anti-compaction. Creating lots of small tables for small repair ranges
will be inefficient and should also not be necessary, as incremental repairs
should be run often enough to keep the unrepaired set reasonably small.
> Subrange repair doesn't always mark as repaired
> ---
>
> Key: CASSANDRA-14400
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14400
> Project: Cassandra
> Issue Type: Bug
>Reporter: Kurt Greaves
>Priority: Major
>
> So was just messing around with subrange repair on trunk and found that if I
> generated an SSTable with a single token and then tried to repair that
> SSTable using subrange repairs it wouldn't get marked as repaired.
>
> Before repair:
> {code:java}
> First token: -9223362383595311662 (derphead4471291)
> Last token: -9223362383595311662 (derphead4471291)
> Repaired at: 0
> Pending repair: 862395e0-4394-11e8-8f20-3b8ee110d005
> {code}
> Repair command:
> {code}
> ccm node1 nodetool "repair -st -9223362383595311663 -et -9223362383595311661
> aoeu"
> [2018-04-19 05:44:42,806] Starting repair command #7
> (c23f76c0-4394-11e8-8f20-3b8ee110d005), repairing keyspace aoeu with repair
> options (parallelism: parallel, primary range: false, incremental: true, job
> threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], previewKind:
> NONE, # of ranges: 1, pull repair: false, force repair: false, optimise
> streams: false)
> [2018-04-19 05:44:42,843] Repair session c242d220-4394-11e8-8f20-3b8ee110d005
> for range [(-9223362383595311663,-9223362383595311661]] finished (progress:
> 20%)
> [2018-04-19 05:44:43,139] Repair completed successfully
> [2018-04-19 05:44:43,140] Repair command #7 finished in 0 seconds
> {code}
> After repair SSTable hasn't changed and sstablemetadata outputs:
> {code}
> First token: -9223362383595311662 (derphead4471291)
> Last token: -9223362383595311662 (derphead4471291)
> Repaired at: 0
> Pending repair: 862395e0-4394-11e8-8f20-3b8ee110d005
> {code}
> And parent_repair_history states that the repair is complete/range was
> successful:
> {code}
> select * from system_distributed.parent_repair_history where
> parent_id=862395e0-4394-11e8-8f20-3b8ee110d005 ;
> parent_id| columnfamily_names |
> exception_message | exception_stacktrace | finished_at |
> keyspace_name | options
>
>
> | requested_ranges
> | started_at | successful_ranges
> --++---+--+-+---++-+-+-
> 862395e0-4394-11e8-8f20-3b8ee110d005 | {'aoeu'} |
> null | null | 2018-04-19 05:43:14.578000+ | aoeu
> | {'dataCenters': '', 'forceRepair': 'false', 'hosts': '', 'incremental':
> 'true', 'jobThreads': '1', 'optimiseStreams': 'false', 'parallelism':
> 'parallel', 'previewKind': 'NONE', 'primaryRange': 'false', 'pullRepair':
> 'false', 'sub_range_repair': 'true', 'trace': 'false'} |
> {'(-9223362383595311663,-9223362383595311661]'} | 2018-04-19
> 05:43:01.952000+ | {'(-9223362383595311663,-9223362383595311661]'}
> {code}
> Subrange repairs seem to work fine over large ranges and set {{Repaired at}}
> as expected, but I haven't figured out why it works for a large range versus
> a small range so far.
--
This message was sent by Atlassian JIRA
[jira] [Commented] (CASSANDRA-13971) Automatic certificate management using Vault
[
https://issues.apache.org/jira/browse/CASSANDRA-13971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445687#comment-16445687
]
Stefan Podkowinski commented on CASSANDRA-13971:
I've now rebased my patch on trunk, which required to resolve some conflicts
related to CASSANDRA-14222, CASSANDRA-14314. The dtest is now migrated to
Python3 and pytest as well. Vault was also upgrade to 0.10.0, but that was
trivial.
I've now attached a script {{start_vault_ssl.sh}} to the ticket. It will
bootstrap a local PKI enabled vault instance and print the matching
cassandra.yaml config. My suggestion would be to run it on the conf folder,
append the output to the yaml and manually enable encryption afterwards. Hope
this makes testing even easier.
As already mentioned, the trunk patch doesn't contain any new libraries or
third party code. It's all HTTP REST communication with any Vault server over
the network. Nothing added in lib.
> Automatic certificate management using Vault
>
>
> Key: CASSANDRA-13971
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13971
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
> Labels: security
> Fix For: 4.x
>
> Attachments: start_vault_ssl.sh
>
>
> We've been adding security features during the last years to enable users to
> secure their clusters, if they are willing to use them and do so correctly.
> Some features are powerful and easy to work with, such as role based
> authorization. Other features that require to manage a local keystore are
> rather painful to deal with. Think about setting up SSL..
> To be fair, keystore related issues and certificate handling hasn't been
> invented by us. We're just following Java standards there. But that doesn't
> mean that we absolutely have to, if there are better options. I'd like to
> give it a shoot and find out if we can automate certificate/key handling
> (PKI) by using external APIs. In this case, the implementation will be based
> on [Vault|https://vaultproject.io]. But certificate management services
> offered by cloud providers may also be able to handle the use-case and I
> intend to create a generic, pluggable API for that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13971) Automatic certificate management using Vault
[ https://issues.apache.org/jira/browse/CASSANDRA-13971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-13971: --- Attachment: start_vault_ssl.sh > Automatic certificate management using Vault > > > Key: CASSANDRA-13971 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13971 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Major > Labels: security > Fix For: 4.x > > Attachments: start_vault_ssl.sh > > > We've been adding security features during the last years to enable users to > secure their clusters, if they are willing to use them and do so correctly. > Some features are powerful and easy to work with, such as role based > authorization. Other features that require to manage a local keystore are > rather painful to deal with. Think about setting up SSL.. > To be fair, keystore related issues and certificate handling hasn't been > invented by us. We're just following Java standards there. But that doesn't > mean that we absolutely have to, if there are better options. I'd like to > give it a shoot and find out if we can automate certificate/key handling > (PKI) by using external APIs. In this case, the implementation will be based > on [Vault|https://vaultproject.io]. But certificate management services > offered by cloud providers may also be able to handle the use-case and I > intend to create a generic, pluggable API for that. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14298) cqlshlib tests broken on b.a.o
[ https://issues.apache.org/jira/browse/CASSANDRA-14298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452331#comment-16452331 ] Stefan Podkowinski commented on CASSANDRA-14298: If you have a patch to get some of the cqlsh dtests work again with Python 3 and pytest, that would be great. We can get back to the then temporarily disabled dtest leftovers after updating cqlshlib. As for doing a full Python 3 transition or keeping the code cross compatible, I don't know. Guess it depends on how much work it is and how big the change set is going to be. If we can get away with a few future imports and a couple of changed print statements, we can even consider patching 3.11. But if it's too much work, or to hard to be compatible, we might as well fully migrate trunk to Python 3. But's thats something we should discuss on dev- based on your findings. > cqlshlib tests broken on b.a.o > -- > > Key: CASSANDRA-14298 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14298 > Project: Cassandra > Issue Type: Bug > Components: Build, Testing >Reporter: Stefan Podkowinski >Assignee: Patrick Bannister >Priority: Major > Attachments: cqlsh_tests_notes.md > > > It appears that cqlsh-tests on builds.apache.org on all branches stopped > working since we removed nosetests from the system environment. See e.g. > [here|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-cqlsh-tests/458/cython=no,jdk=JDK%201.8%20(latest),label=cassandra/console]. > Looks like we either have to make nosetests available again or migrate to > pytest as we did with dtests. Giving pytest a quick try resulted in many > errors locally, but I haven't inspected them in detail yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14298) cqlshlib tests broken on b.a.o
[ https://issues.apache.org/jira/browse/CASSANDRA-14298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16448100#comment-16448100 ] Stefan Podkowinski commented on CASSANDRA-14298: "It's broken right now because cqlshlib isn't compatible with Python 3." We'll have to address this at some point anyways. Time's probably better spend migrating code to Python 3, instead of creating workarounds to make the tests run with Python 2 again. > cqlshlib tests broken on b.a.o > -- > > Key: CASSANDRA-14298 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14298 > Project: Cassandra > Issue Type: Bug > Components: Build, Testing >Reporter: Stefan Podkowinski >Assignee: Patrick Bannister >Priority: Major > Attachments: cqlsh_tests_notes.md > > > It appears that cqlsh-tests on builds.apache.org on all branches stopped > working since we removed nosetests from the system environment. See e.g. > [here|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-cqlsh-tests/458/cython=no,jdk=JDK%201.8%20(latest),label=cassandra/console]. > Looks like we either have to make nosetests available again or migrate to > pytest as we did with dtests. Giving pytest a quick try resulted in many > errors locally, but I haven't inspected them in detail yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14298) cqlshlib tests broken on b.a.o
[
https://issues.apache.org/jira/browse/CASSANDRA-14298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450051#comment-16450051
]
Stefan Podkowinski commented on CASSANDRA-14298:
{quote}For example, do we want to go all the way to Python 3, or would we
prefer it to be 2/3 cross-compatible?
{quote}
Being cross-compatible would be the most convenient option for users I guess.
But I haven't really any experience converting Python 2 code in such a way.
Would you up to give this a try, Patrick?
{quote}I agree with this, only casually observing this ticket. Is it worth
bringing up on the dev@ ML?
{quote}
Sure. Maybe we can bring it up with an actual proposal after checking our
options.
{quote}However, for the impacted copy tests, we can only completely avoid these
ugly workarounds if we port cqlshlib to Python 3 not only in trunk, but also in
all other supported versions.
{quote}
Depending on how invasive those changes will turn out to be, we can discuss
patching 3.11, but I don't think that's going to happen for older branches.
Anyone running Cassandra 2.x/3.0 should be able to keep using Python 2 until
EOL 2020, which should be well past the Cassandra EOL date.
{quote}Any branch of C* left behind on Python 2.7 will either have to be
skipped for the copy tests, or else tested through some kind of alternate
approach such as the awful hack I'm working on right now.
{quote}
We currently run 0% tests on b.a.o. If we can get all but the copy tests
working in a straight forward way, that would be a good start.
> cqlshlib tests broken on b.a.o
> --
>
> Key: CASSANDRA-14298
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14298
> Project: Cassandra
> Issue Type: Bug
> Components: Build, Testing
>Reporter: Stefan Podkowinski
>Assignee: Patrick Bannister
>Priority: Major
> Attachments: cqlsh_tests_notes.md
>
>
> It appears that cqlsh-tests on builds.apache.org on all branches stopped
> working since we removed nosetests from the system environment. See e.g.
> [here|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-cqlsh-tests/458/cython=no,jdk=JDK%201.8%20(latest),label=cassandra/console].
> Looks like we either have to make nosetests available again or migrate to
> pytest as we did with dtests. Giving pytest a quick try resulted in many
> errors locally, but I haven't inspected them in detail yet.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420389#comment-16420389
]
Stefan Podkowinski commented on CASSANDRA-12151:
We should not provide solutions that we know will have significant performance
issues on busy production systems. I don’t mind keeping the FileAuditLogger
class, as it’s really trivial. But please remove references to it from
cassandra.yaml and the corresponding entries in logback.xml (don’t like to have
an empty audit.log file on all nodes either). Let’s nudge users to use to
BinAuditLogger right away as the recommended solution.
If you don’t think adding an option like include_auditlog_types should be
necessary, that’s fine. But then let me use my own implementation for filtering
logs, if my requirements are different. This means that I’d have to be able to
specify additional parameters (e.g. custom filtering options) along with the
class name for my implementation. So I'd suggest to use ParameterizedClass for
the logger in cassandra.yaml to make that easily possible.
Looking at IAuditLogger and thinking about how to filter log events makes me a
bit worried about the design in general there. We keep generating AuditLogEntry
instances and create unnecessary garbage, even if we’re only interested in some
specific entry types. Maybe we should move filtering either into the
IAuditLogger implementation or make it possible to use a custom AuditLogFilter
as well (IAuditLogger.getLogFilter() ?).
Just looking at AuditLogFilter makes me also think that the isFiltered logic
should be reconsidered, as null values will cause entries to always pass, even
if the include set is not empty.
{quote}How do you suggest we approach this problem? We cannot keep the audit
log files around indefinitely. Perhaps we can specify a disk quota for audit
log files in the configuration? We can expose this setting in cassandra.yaml?
{quote}
How does this work with BinLogger? I haven't looked at that part in detail yet.
Does it overwrite old entries automatically? How would you suggest users should
archive logs from there?
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418918#comment-16418918
]
Stefan Podkowinski commented on CASSANDRA-12151:
I'd not use the logback logger for query logging in production. As shown
recently in CASSANDRA-14318, the performance impact is significant and your
benchmark results show that as well. Rotating out log files by simply deleting
them is probably also not what you'd expect from a auditing solution. But maybe
we can keep the logback logger for some simple auth related logging and make it
also useful for users who would not enable auditing in first place. How about
enabling the {{FileAuditLogger}} by default and let it create an {{auth.log}},
which would log all failed login attempts? Maybe add a comment on how to enable
successful attempts as well.
Looks like this won't be possible with the current {{included_categories}}
filtering, which is based on the DDL, DML, .. categories. I'd suggest to make
the filter work for both the category and actual AuditLogEntryType (SELECT,
UPDATE, DELETE,..).
Full query logging should be done using the BinLogger or a custom
implementation. It would be nice to be able to use mutliple implementations in
parallel, in case we want to enable FileAuditLogger by default.
NIT: check logger.isEnabled before toString in FileAuditLogger
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt, CASSANDRA_12151-benchmark.html,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log connection attempt and changes to the
> user/roles.
> I was thinking of making a new keyspace and insert an entry for every
> activity that occurs.
> Then It would be possible to query for specific activity or a query targeting
> a specific keyspace and column family.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14326) Handle verbose logging at a different level than DEBUG
[ https://issues.apache.org/jira/browse/CASSANDRA-14326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-14326: --- Fix Version/s: (was: 4.0) 4.x > Handle verbose logging at a different level than DEBUG > -- > > Key: CASSANDRA-14326 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14326 > Project: Cassandra > Issue Type: Improvement >Reporter: Alexander Dejanovski >Priority: Major > Fix For: 4.x > > > CASSANDRA-10241 introduced debug logging turned on by default to act as a > verbose system.log and help troubleshoot production issues. > One of the consequence was to severely affect read performance in 2.2 as > contributors weren't all up to speed on how to use logging levels > (CASSANDRA-14318). > As DEBUG level has a very specific meaning in dev, it is confusing to use it > for always on verbose logging and should probably not be used this way in > Cassandra. > Options so far are : > # Bring back common loggings to INFO level (compactions, flushes, etc...) > and disable debug logging by default > # Use files named as verbose-system.log instead of debug.log and use a > custom logging level instead of DEBUG for verbose tracing, that would be > enabled by default. Debug logging would still exist and be disabled by > default and the root logger level (not just filtered at the appender level). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-14329) Update logging guidelines and move in-tree
[ https://issues.apache.org/jira/browse/CASSANDRA-14329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski reassigned CASSANDRA-14329: -- Assignee: Stefan Podkowinski > Update logging guidelines and move in-tree > -- > > Key: CASSANDRA-14329 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14329 > Project: Cassandra > Issue Type: Improvement > Components: Documentation and Website >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Major > > We should update the existing [logging > guidelines|https://wiki.apache.org/cassandra/LoggingGuidelines] while moving > them in-tree. Maybe also split up between "Configuring Cassandra" and > "Contributing to Cassandra". -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14326) Handle verbose logging at a different level than DEBUG
[ https://issues.apache.org/jira/browse/CASSANDRA-14326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406374#comment-16406374 ] Stefan Podkowinski commented on CASSANDRA-14326: I'd prefer option #2 over what we have now, if we can use a SLF4J marker for filtering messages in a practical way, as suggested by [~jjordan]. The only thing I can think of that will be a bit odd, is to have two different logs, both having INFO messages with one being a superset of the other. Although I'm aware of the reasons behind this, it's probably not that obvious why you need to keep system.log around with verbose-system.log enabled, too. But this is really more a minor issue, compared to the benefit of preventing performance regressions by just adding a debug statement in hot code paths. > Handle verbose logging at a different level than DEBUG > -- > > Key: CASSANDRA-14326 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14326 > Project: Cassandra > Issue Type: Improvement >Reporter: Alexander Dejanovski >Priority: Major > Fix For: 4.x > > > CASSANDRA-10241 introduced debug logging turned on by default to act as a > verbose system.log and help troubleshoot production issues. > One of the consequence was to severely affect read performance in 2.2 as > contributors weren't all up to speed on how to use logging levels > (CASSANDRA-14318). > As DEBUG level has a very specific meaning in dev, it is confusing to use it > for always on verbose logging and should probably not be used this way in > Cassandra. > Options so far are : > # Bring back common loggings to INFO level (compactions, flushes, etc...) > and disable debug logging by default > # Use files named as verbose-system.log instead of debug.log and use a > custom logging level instead of DEBUG for verbose tracing, that would be > enabled by default. Debug logging would still exist and be disabled by > default and the root logger level (not just filtered at the appender level). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14329) Update logging guidelines and move in-tree
Stefan Podkowinski created CASSANDRA-14329: -- Summary: Update logging guidelines and move in-tree Key: CASSANDRA-14329 URL: https://issues.apache.org/jira/browse/CASSANDRA-14329 Project: Cassandra Issue Type: Improvement Components: Documentation and Website Reporter: Stefan Podkowinski We should update the existing [logging guidelines|https://wiki.apache.org/cassandra/LoggingGuidelines] while moving them in-tree. Maybe also split up between "Configuring Cassandra" and "Contributing to Cassandra". -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[
https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386137#comment-16386137
]
Stefan Podkowinski commented on CASSANDRA-12151:
[~eanujwa] wrote:
{quote}If auditing for a database application user is only at application
level, what would happen in case direct queries are executed on the database
using the the application user credentials (using tools such as cqlsh)?
{quote}
If the operator decided not to enable audit logging for the application users,
nothing would get logged. If someone steals the credentials, none of the
activities would show up in the logs. If you think that this is unacceptable,
then you can enable auditing for the app user as well, which I would not, but
it’s really up to you.
[~jolynch] wrote:
{quote}Doesn't the category filter adequately achieve this (you could exclude
DML or QUERY)? Do we need per user query logging when there is already per user
permissions limiting their access to the database in the first place?
{quote}
Although I understand the motivation behind the query categories from a
developer perspective, it’s probably a feature that is not as easy to
understand and handled by operators. Let alone managers and compliance
officers, who lack the technical understanding for discussing which categories
needs to be enabled for which tables. Once you’ll notice that you have to limit
the audit logging due to the performance impact, it’s going to be a rather
painful discussion how categories and tables should be configured in this
regard. I’d rather give ops people more simpler options that are easy to
understand and reason about, when discussed between ops and non-technical
stakeholders. Offering “all or nothing” audit logging on user basis would fit
this requirements for me (at least for CQL, login events could be a different
story).
I'd also really like to avoid luring people into the idea that Cassandra would
be a one-stop solution for audit logging and you don't have to implement your
own logging on application level anymore, since Cassandra does handle this for
you. It probably won't, as logging all statements for applications will be too
much of a performance impact. Now the question for me is to offer operators
options to optimize their way out of this situation, or simply tell people that
it's not possible to handle this use-case (audit logging for application users
on high-load systems) in Cassandra. I'd clearly go with the later, as many
users think that Cassandra already puts too much of a burden on operators.
{quote}While I agree use case #1 (security) does not require this, use cases #2
and #3 very much do. For #2 or similar you typically have to prove that only
authorized applications manipulated the database and a typical way to do that
is to produce query logs showing that only trusted application IP addresses and
specific credentials made DML statements (but QUERY is less important). For #3
the requirements are even greater, e.g. you may have to be able to prove that
user data was not exfiltrated at all, requiring auditing of QUERY statements.
Yes it's higher overhead but if you can turn it off with the category filters I
think it's fine don't you?
{quote}
Can you really “prove” unauthorised data manipulation did *not* happen through
the audit logs? Say I have an application that is updating a credit score and
now I use that user to do an update manually, how would you ever notice that
from the audit logs? My manual statement looks just like the million others
executed by the app. IP based access restrictions should be in place anyways,
that’s also not something that would necessarily give you additional clues.
As was already suggested, let’s take an incremental approach that will allow us
to introduce a simple audit logging that can be understood by both technical
and non-technical people and configured in little time. If any external
compliance authority raises some concerns, e.g. that DCL statements need to be
handled differently from queries, we can take care of that. But any 90%
solutions would still be a big win over what we currently have.
> Audit logging for database activity
> ---
>
> Key: CASSANDRA-12151
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12151
> Project: Cassandra
> Issue Type: New Feature
>Reporter: stefan setyadi
>Assignee: Vinay Chella
>Priority: Major
> Fix For: 4.x
>
> Attachments: 12151.txt,
> DesignProposal_AuditingFeature_ApacheCassandra_v1.docx
>
>
> we would like a way to enable cassandra to log database activity being done
> on our server.
> It should show username, remote address, timestamp, action type, keyspace,
> column family, and the query statement.
> it should also be able to log
[jira] [Updated] (CASSANDRA-13668) Database user auditing events
[ https://issues.apache.org/jira/browse/CASSANDRA-13668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski updated CASSANDRA-13668: --- Status: Patch Available (was: In Progress) > Database user auditing events > - > > Key: CASSANDRA-13668 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13668 > Project: Cassandra > Issue Type: Improvement > Components: Observability >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Major > Fix For: 4.x > > > With the availability of CASSANDRA-13459, any native transport enabled client > will be able to subscribe to internal Cassandra events. External tools can > take advantage by monitoring these events in various ways. Use-cases for this > can be e.g. auditing tools for compliance and security purposes. > The scope of this ticket is to add diagnostic events that are raised around > authentication and CQL operations. These events can then be consumed and used > by external tools to implement a Cassandra user auditing solution. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12151) Audit logging for database activity
[ https://issues.apache.org/jira/browse/CASSANDRA-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383488#comment-16383488 ] Stefan Podkowinski commented on CASSANDRA-12151: [~vinaykumarcse] wrote: bq. Seems like a complicated configuration, would like to understand the use cases more here and see if anyone else needs this functionality. All the usecases that I see here are Auditing at the cluster or no auditing, but not specific to user. I would love to hear if there are any other users with this usecase. Usually you'll see two kind of users on production systems: privileged users and application users. Auditing privileged users (admins or developers) will almost always make sense, in order to be able to detect unauthorized access and data manipulation. There's only a limited amount of statements to log, as these will be executed manually. It also shouldn't matter which keyspaces or tables are access by the users; he is either monitored or not. However, auditing queries of application users has a very limited security and data privacy benefit, but adds a great deal of load to the database. Those queries will be automatically generated by the application and there will be no way to tell if the query or statement was authorized, as you don't know on behalf of whom it was executed. Any auditing functionality for these operations must therefor take place at application level. Eg. a help desk tool, which is used by a support employee to access personal data of a customer in Cassandra, must keep an activity log for that directly. It doesn't make sense to log queries for the generic help desk tool Cassandra user on the database side. Therefor we need a way to enable CQL query auditing on user level. > Audit logging for database activity > --- > > Key: CASSANDRA-12151 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12151 > Project: Cassandra > Issue Type: New Feature >Reporter: stefan setyadi >Assignee: Vinay Chella >Priority: Major > Fix For: 4.x > > Attachments: 12151.txt, > DesignProposal_AuditingFeature_ApacheCassandra_v1.docx > > > we would like a way to enable cassandra to log database activity being done > on our server. > It should show username, remote address, timestamp, action type, keyspace, > column family, and the query statement. > it should also be able to log connection attempt and changes to the > user/roles. > I was thinking of making a new keyspace and insert an entry for every > activity that occurs. > Then It would be possible to query for specific activity or a query targeting > a specific keyspace and column family. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13459) Diag. Events: Native transport integration
[
https://issues.apache.org/jira/browse/CASSANDRA-13459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454195#comment-16454195
]
Stefan Podkowinski commented on CASSANDRA-13459:
We do implement some basic backpressure for inter-node communication
(CASSANDRA-8457) based on Netty's {{Channel.isWritable()}} indicator (see
{{MessageOutHandler.channelWritabilityChanged()}},
{{OutboundConnectionParams}}, {{QueuedMessage}}). And there's also
{{CoalescingStrategy}}. But I don't think we're doing any of this for native
transport. My guess would be that this kind of optimizations won't be very
effective for single connection based request/response semantics, in contrast
to the bulky async messaging for inter-node communication.
But we could still optimize native transport a bit for diag events by
implementing load shedding, as we only provide events on best effort basis, see
[5121a848|https://github.com/spodkowinski/cassandra/commit/5121a848855f35894f2fb5176d7dc46f7b2a8571].
Ideally we would then also indicate that messages have been dropped/omitted,
but we'll also have to introduce a corresponding message to the protocol spec.
{quote}Looking at what you have now there is no query language correct? You
subscribe to these events via the wire protocol not the query language?
{quote}
Yes, exactly. I'd assume most drivers would have a hard time to support
publish/subscribe semantics in existing request/response query execution models.
> Diag. Events: Native transport integration
> --
>
> Key: CASSANDRA-13459
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13459
> Project: Cassandra
> Issue Type: Sub-task
> Components: CQL
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
> Labels: client-impacting
>
> Events should be consumable by clients that would received subscribed events
> from the connected node. This functionality is designed to work on top of
> native transport with minor modifications to the protocol standard (see
> [original
> proposal|https://docs.google.com/document/d/1uEk7KYgxjNA0ybC9fOuegHTcK3Yi0hCQN5nTp5cNFyQ/edit?usp=sharing]
> for further considered options). First we have to add another value for
> existing event types. Also, we have to extend the protocol a bit to be able
> to specify a sub-class and sub-type value. E.g.
> {{DIAGNOSTIC_EVENT(GossiperEvent, MAJOR_STATE_CHANGE_HANDLED)}}. This still
> has to be worked out and I'd appreciate any feedback.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14423) SSTables stop being compacted
[
https://issues.apache.org/jira/browse/CASSANDRA-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-14423:
---
Status: Ready to Commit (was: Patch Available)
> SSTables stop being compacted
> -
>
> Key: CASSANDRA-14423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14423
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Blocker
> Fix For: 2.2.13, 3.0.17, 3.11.3
>
>
> So seeing a problem in 3.11.0 where SSTables are being lost from the view and
> not being included in compactions/as candidates for compaction. It seems to
> get progressively worse until there's only 1-2 SSTables in the view which
> happen to be the most recent SSTables and thus compactions completely stop
> for that table.
> The SSTables seem to still be included in reads, just not compactions.
> The issue can be fixed by restarting C*, as it will reload all SSTables into
> the view, but this is only a temporary fix. User defined/major compactions
> still work - not clear if they include the result back in the view but is not
> a good work around.
> This also results in a discrepancy between SSTable count and SSTables in
> levels for any table using LCS.
> {code:java}
> Keyspace : xxx
> Read Count: 57761088
> Read Latency: 0.10527088681224288 ms.
> Write Count: 2513164
> Write Latency: 0.018211106398149903 ms.
> Pending Flushes: 0
> Table: xxx
> SSTable count: 10
> SSTables in each level: [2, 0, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 894498746
> Space used (total): 894498746
> Space used by snapshots (total): 0
> Off heap memory used (total): 11576197
> SSTable Compression Ratio: 0.6956629530569777
> Number of keys (estimate): 3562207
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 87
> Local read count: 57761088
> Local read latency: 0.108 ms
> Local write count: 2513164
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 86.33
> Bloom filter false positives: 43
> Bloom filter false ratio: 0.0
> Bloom filter space used: 8046104
> Bloom filter off heap memory used: 8046024
> Index summary off heap memory used: 3449005
> Compression metadata off heap memory used: 81168
> Compacted partition minimum bytes: 104
> Compacted partition maximum bytes: 5722
> Compacted partition mean bytes: 175
> Average live cells per slice (last five minutes): 1.0
> Maximum live cells per slice (last five minutes): 1
> Average tombstones per slice (last five minutes): 1.0
> Maximum tombstones per slice (last five minutes): 1
> Dropped Mutations: 0
> {code}
> Also for STCS we've confirmed that SSTable count will be different to the
> number of SSTables reported in the Compaction Bucket's. In the below example
> there's only 3 SSTables in a single bucket - no more are listed for this
> table. Compaction thresholds haven't been modified for this table and it's a
> very basic KV schema.
> {code:java}
> Keyspace : yyy
> Read Count: 30485
> Read Latency: 0.06708991307200263 ms.
> Write Count: 57044
> Write Latency: 0.02204061776873992 ms.
> Pending Flushes: 0
> Table: yyy
> SSTable count: 19
> Space used (live): 18195482
> Space used (total): 18195482
> Space used by snapshots (total): 0
> Off heap memory used (total): 747376
> SSTable Compression Ratio: 0.7607394576769735
> Number of keys (estimate): 116074
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 39
> Local read count: 30485
> Local read latency: NaN ms
> Local write count: 57044
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 79.76
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 690912
> Bloom filter off heap memory used: 690760
> Index summary off heap memory used: 54736
> Compression metadata off heap memory used: 1880
> Compacted partition minimum bytes: 73
> Compacted partition maximum bytes: 124
> Compacted partition mean bytes: 96
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
> Dropped Mutations: 0
> {code}
> {code:java}
> Apr 27 03:10:39 cassandra[9263]: TRACE o.a.c.d.c.SizeTieredCompactionStrategy
> Compaction buckets are
>
[jira] [Commented] (CASSANDRA-14528) Provide stacktraces for various error logs
[
https://issues.apache.org/jira/browse/CASSANDRA-14528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16518041#comment-16518041
]
Stefan Podkowinski commented on CASSANDRA-14528:
Maybe telling which one of the cql statements caused the error would also be
possible by looking only at the message. But since this code is triggered by a
user, it shouldn't spam the logs too much in any case..
> Provide stacktraces for various error logs
> --
>
> Key: CASSANDRA-14528
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14528
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
> Fix For: 4.x
>
>
> We should reintroduce some stack traces that have gone missing since
> CASSANDRA-13723 (ba87ab4e954ad2). The cleanest way would probably to use
> {{String.format}} for any custom messages, e.g.
> {{logger.error(String.format("Error using param {}", param), e)}}, so we make
> this more implicit and robust for coming api changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14423) SSTables stop being compacted
[
https://issues.apache.org/jira/browse/CASSANDRA-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526070#comment-16526070
]
Stefan Podkowinski commented on CASSANDRA-14423:
[~KurtG], do you mind to go on by committing 2.2/3.0/3.11 patches now and
address trunk in a separate ticket?
> SSTables stop being compacted
> -
>
> Key: CASSANDRA-14423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14423
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Kurt Greaves
>Assignee: Kurt Greaves
>Priority: Blocker
> Fix For: 2.2.13, 3.0.17, 3.11.3
>
>
> So seeing a problem in 3.11.0 where SSTables are being lost from the view and
> not being included in compactions/as candidates for compaction. It seems to
> get progressively worse until there's only 1-2 SSTables in the view which
> happen to be the most recent SSTables and thus compactions completely stop
> for that table.
> The SSTables seem to still be included in reads, just not compactions.
> The issue can be fixed by restarting C*, as it will reload all SSTables into
> the view, but this is only a temporary fix. User defined/major compactions
> still work - not clear if they include the result back in the view but is not
> a good work around.
> This also results in a discrepancy between SSTable count and SSTables in
> levels for any table using LCS.
> {code:java}
> Keyspace : xxx
> Read Count: 57761088
> Read Latency: 0.10527088681224288 ms.
> Write Count: 2513164
> Write Latency: 0.018211106398149903 ms.
> Pending Flushes: 0
> Table: xxx
> SSTable count: 10
> SSTables in each level: [2, 0, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 894498746
> Space used (total): 894498746
> Space used by snapshots (total): 0
> Off heap memory used (total): 11576197
> SSTable Compression Ratio: 0.6956629530569777
> Number of keys (estimate): 3562207
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 87
> Local read count: 57761088
> Local read latency: 0.108 ms
> Local write count: 2513164
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 86.33
> Bloom filter false positives: 43
> Bloom filter false ratio: 0.0
> Bloom filter space used: 8046104
> Bloom filter off heap memory used: 8046024
> Index summary off heap memory used: 3449005
> Compression metadata off heap memory used: 81168
> Compacted partition minimum bytes: 104
> Compacted partition maximum bytes: 5722
> Compacted partition mean bytes: 175
> Average live cells per slice (last five minutes): 1.0
> Maximum live cells per slice (last five minutes): 1
> Average tombstones per slice (last five minutes): 1.0
> Maximum tombstones per slice (last five minutes): 1
> Dropped Mutations: 0
> {code}
> Also for STCS we've confirmed that SSTable count will be different to the
> number of SSTables reported in the Compaction Bucket's. In the below example
> there's only 3 SSTables in a single bucket - no more are listed for this
> table. Compaction thresholds haven't been modified for this table and it's a
> very basic KV schema.
> {code:java}
> Keyspace : yyy
> Read Count: 30485
> Read Latency: 0.06708991307200263 ms.
> Write Count: 57044
> Write Latency: 0.02204061776873992 ms.
> Pending Flushes: 0
> Table: yyy
> SSTable count: 19
> Space used (live): 18195482
> Space used (total): 18195482
> Space used by snapshots (total): 0
> Off heap memory used (total): 747376
> SSTable Compression Ratio: 0.7607394576769735
> Number of keys (estimate): 116074
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 39
> Local read count: 30485
> Local read latency: NaN ms
> Local write count: 57044
> Local write latency: NaN ms
> Pending flushes: 0
> Percent repaired: 79.76
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 690912
> Bloom filter off heap memory used: 690760
> Index summary off heap memory used: 54736
> Compression metadata off heap memory used: 1880
> Compacted partition minimum bytes: 73
> Compacted partition maximum bytes: 124
> Compacted partition mean bytes: 96
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
> Dropped Mutations: 0
> {code}
> {code:java}
> Apr 27 03:10:39 cassandra[9263]: TRACE o.a.c.d.c.SizeTieredCompactionStrategy
>
[jira] [Commented] (CASSANDRA-14821) Make it possible to run multi-node coordinator/replica tests in a single JVM
[ https://issues.apache.org/jira/browse/CASSANDRA-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648278#comment-16648278 ] Stefan Podkowinski commented on CASSANDRA-14821: What kind of bugs are you aiming to test for using this approach? Can you please add more complex, real-world test cases first, so it's possible to tell the advantages over existing testing approaches? Doesn't even have to compile, just to get an idea on where this is going... > Make it possible to run multi-node coordinator/replica tests in a single JVM > > > Key: CASSANDRA-14821 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14821 > Project: Cassandra > Issue Type: Test >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: Major > > Currently, dtests are complex to write, hard to modify and slow to run. The > only option to manipulate a cluster state is either to shut down nodes or run > unreliable Byteman queries. > In order to improve the situation, a new Distributed Tester is proposed. It > fires up multiple Cassandra Instances in a single JVM. It is done through > having distinct class loaders in order to work around the singleton problem > in Cassandra. In order to be able to pass some information between the nodes, > a common class loader is used that loads up java standard library and several > helper classes. Tests look a lot like CQLTester tests would usually look like. > Each Cassandra Instance, with its distinct class loader is using > serialisation and class loading mechanisms in order to run instance-local > queries and execute node state manipulation code, hooks, callbacks etc. > First version mocks out Messaging Service and simplifies schema management by > simply running schema change commands on each of the instances separately. > Internode communication is mocked by passing ByteBuffers through shared class > loader. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14821) Make it possible to run multi-node coordinator/replica tests in a single JVM
[ https://issues.apache.org/jira/browse/CASSANDRA-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648891#comment-16648891 ] Stefan Podkowinski commented on CASSANDRA-14821: I understand the intention and motivation behind this effort. No justification needed. Alex asked for early feedback and I responded that more complex, real-world test cases would help evaluating the usefulness and advantages of the provided code over existing testing approaches. His Jira comment and the corresponding PR now seem to be deleted/closed, so it looks like this is still largely work in progress at this point. > Make it possible to run multi-node coordinator/replica tests in a single JVM > > > Key: CASSANDRA-14821 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14821 > Project: Cassandra > Issue Type: Test >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: Major > > Currently, dtests are complex to write, hard to modify and slow to run. The > only option to manipulate a cluster state is either to shut down nodes or run > unreliable Byteman queries. > In order to improve the situation, a new Distributed Tester is proposed. It > fires up multiple Cassandra Instances in a single JVM. It is done through > having distinct class loaders in order to work around the singleton problem > in Cassandra. In order to be able to pass some information between the nodes, > a common class loader is used that loads up java standard library and several > helper classes. Tests look a lot like CQLTester tests would usually look like. > Each Cassandra Instance, with its distinct class loader is using > serialisation and class loading mechanisms in order to run instance-local > queries and execute node state manipulation code, hooks, callbacks etc. > First version mocks out Messaging Service and simplifies schema management by > simply running schema change commands on each of the instances separately. > Internode communication is mocked by passing ByteBuffers through shared class > loader. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14813) Crash frequently due to fatal error caused by "StubRoutines::updateBytesCRC32"
[
https://issues.apache.org/jira/browse/CASSANDRA-14813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647572#comment-16647572
]
Stefan Podkowinski commented on CASSANDRA-14813:
Is this also happening for writes without your custom indexer at work?
> Crash frequently due to fatal error caused by "StubRoutines::updateBytesCRC32"
> --
>
> Key: CASSANDRA-14813
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14813
> Project: Cassandra
> Issue Type: Bug
> Environment: *OS:*
> {code:java}
> CentOS release 6.9 (Final){code}
> *JAVA:*
> {noformat}
> java version "1.8.0_101"
> Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode){noformat}
>
> *Memory:*
> {noformat}
> 256GB{noformat}
> *CPU:*
> {noformat}
> 4* Intel® Xeon® Processor E5-2620 v4{noformat}
> *DISK:*
> {noformat}
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda3 423G 28G 374G 7% /
> tmpfs 126G 68K 126G 1% /dev/shm
> /dev/sda1 240M 40M 188M 18% /boot
> /dev/sdb 3.7T 33M 3.7T 1% /mpp-data/c/cache
> /dev/sdc 3.7T 2.7T 984G 74% /mpp-data/c/data00
> /dev/sdd 3.7T 2.5T 1.2T 68% /mpp-data/c/data01
> /dev/sde 3.7T 2.7T 1.1T 72% /mpp-data/c/data02
> /dev/sdf 3.7T 2.5T 1.2T 68% /mpp-data/c/data03
> /dev/sdg 3.7T 2.4T 1.3T 66% /mpp-data/c/data04
> /dev/sdh 3.7T 2.6T 1.2T 69% /mpp-data/c/data05
> /dev/sdi 3.7T 2.6T 1.2T 70% /mpp-data/c/data06{noformat}
>Reporter: Jinchao Zhang
>Priority: Major
> Attachments: f3.png, f4.png, hs1.png, hs2.png, hs_err_pid26350.log
>
>
> Recently, we encountered the same problem described by CASSANDRA-14283 in our
> production system, which runs on Cassandra 3.11.2. We noticed that this issue
> has been resolved in Cassandra 3.11.3 (CASSANDRA-14284), thus we upgrade our
> system to 3.11.3. However, this induce more frequent crash，as shown in the
> following screenshots, and the reduced hs file is posted here as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-14810) Upgrade dtests to pytest-3.8
[
https://issues.apache.org/jira/browse/CASSANDRA-14810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski reassigned CASSANDRA-14810:
--
Assignee: Thomas Pouget-Abadie
> Upgrade dtests to pytest-3.8
>
>
> Key: CASSANDRA-14810
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14810
> Project: Cassandra
> Issue Type: Improvement
> Components: Testing
>Reporter: Stefan Podkowinski
>Assignee: Thomas Pouget-Abadie
>Priority: Minor
> Labels: lhf
>
> The [dtest project|https://github.com/apache/cassandra-dtest] uses pytest as
> test runner of choice for executing tests on builds.apache.org or CircleCI.
> The pytest dependency has recently been upgrade to 3.6, but couldn't be
> upgrade to the most recent 3.8 version, due to issues described below.
> Before test execution, the {{run_dtests.py}} script will gather a list of all
> tests:
> {{./run_dtests.py --dtest-print-tests-only}}
> Afterwards pytest can be started with any of the output lines as argument.
> With pytest-3.8 however, the output format changed and preventing pytest to
> find the test:
> {{pytest
> upgrade_tests/upgrade_supercolumns_test.py::TestSCUpgrade::test_upgrade_super_columns_through_limited_versions}}
> vs
> {{pytest
> upgrade_supercolumns_test.py::TestSCUpgrade::test_upgrade_super_columns_through_limited_versions}}
> The underlying issue appears to be the changes in the {{pytest
> --collect-only}} output, consumed in {{run_dtests.py}}, which now includes a
> element that needs to be parsed as well to derive at the path as we
> did before. We'd have to parse the new output and assemble the correct paths
> again, so we can use run_dtests.py as we did before with pytest 3.6.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14713) Update docker image used for testing
[
https://issues.apache.org/jira/browse/CASSANDRA-14713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-14713:
---
Resolution: Fixed
Fix Version/s: 4.0
3.11.4
3.0.18
2.2.14
Status: Resolved (was: Ready to Commit)
I haven't noticed any regressions from the dtest updates so far. Looks like we
can take the final step switching to the new docker image as well.
CircleCI config update committed as dcd92a9 and merged upwards.
> Update docker image used for testing
>
>
> Key: CASSANDRA-14713
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14713
> Project: Cassandra
> Issue Type: New Feature
> Components: Testing
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
> Fix For: 2.2.14, 3.0.18, 3.11.4, 4.0
>
> Attachments: Dockerfile
>
>
> Tests executed on builds.apache.org ({{docker/jenkins/jenkinscommand.sh}})
> and circleCI ({{.circleci/config.yml}}) will currently use the same
> [cassandra-test|https://hub.docker.com/r/kjellman/cassandra-test/] docker
> image ([github|https://github.com/mkjellman/cassandra-test-docker]) by
> [~mkjellman].
> We should manage this image on our own as part of cassandra-builds, to keep
> it updated. There's also a [Apache
> user|https://hub.docker.com/u/apache/?page=1] on docker hub for publishing
> images.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14810) Upgrade dtests to pytest-3.8
[
https://issues.apache.org/jira/browse/CASSANDRA-14810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-14810:
---
Reviewer: Stefan Podkowinski
Status: Patch Available (was: Open)
> Upgrade dtests to pytest-3.8
>
>
> Key: CASSANDRA-14810
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14810
> Project: Cassandra
> Issue Type: Improvement
> Components: Testing
>Reporter: Stefan Podkowinski
>Assignee: Thomas Pouget-Abadie
>Priority: Minor
> Labels: lhf
> Attachments: 14810-master.txt
>
>
> The [dtest project|https://github.com/apache/cassandra-dtest] uses pytest as
> test runner of choice for executing tests on builds.apache.org or CircleCI.
> The pytest dependency has recently been upgrade to 3.6, but couldn't be
> upgrade to the most recent 3.8 version, due to issues described below.
> Before test execution, the {{run_dtests.py}} script will gather a list of all
> tests:
> {{./run_dtests.py --dtest-print-tests-only}}
> Afterwards pytest can be started with any of the output lines as argument.
> With pytest-3.8 however, the output format changed and preventing pytest to
> find the test:
> {{pytest
> upgrade_tests/upgrade_supercolumns_test.py::TestSCUpgrade::test_upgrade_super_columns_through_limited_versions}}
> vs
> {{pytest
> upgrade_supercolumns_test.py::TestSCUpgrade::test_upgrade_super_columns_through_limited_versions}}
> The underlying issue appears to be the changes in the {{pytest
> --collect-only}} output, consumed in {{run_dtests.py}}, which now includes a
> element that needs to be parsed as well to derive at the path as we
> did before. We'd have to parse the new output and assemble the correct paths
> again, so we can use run_dtests.py as we did before with pytest 3.6.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14690) Add dtests for fqltool replay/compare
[
https://issues.apache.org/jira/browse/CASSANDRA-14690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-14690:
---
Reviewer: Stefan Podkowinski
> Add dtests for fqltool replay/compare
> -
>
> Key: CASSANDRA-14690
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14690
> Project: Cassandra
> Issue Type: Test
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Major
> Labels: fqltool
>
> We should add some basic round-trip dtests for {{fqltool replay}} and
> {{compare}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14806) CircleCI workflow improvements and Java 11 support
[
https://issues.apache.org/jira/browse/CASSANDRA-14806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-14806:
---
Description:
The current CircleCI config could use some cleanup and improvements. First of
all, the config has been made more modular by using the new CircleCI 2.1
executors and command elements. Based on CASSANDRA-14713, there's now also a
Java 11 executor that will allow running tests under Java 11. The {{build}}
step will be done using Java 11 in all cases, so we can catch any regressions
for that and also test the Java 11 multi-jar artifact during dtests, that we'd
also create during the release process.
The job workflow has now also been changed to make use of the [manual job
approval|https://circleci.com/docs/2.0/workflows/#holding-a-workflow-for-a-manual-approval]
feature, which now allows running dtest jobs only on request and not
automatically with every commit. The Java8 unit tests still do, but that could
also be easily changed if needed. See [example
workflow|https://circleci.com/workflow-run/be25579d-3cbb-4258-9e19-b1f571873850]
with start_ jobs being triggers needed manual approval for starting the actual
jobs.
was:
The current CircleCI config could use some cleanup and improvements. First of
all, the config has been made more modular by using the new CircleCI 2.1
executors and command elements. Based on CASSANDRA-14713, there's now also a
Java 11 executor that will allow running tests under Java 11. The {{build}}
step will be done using Java 11 in all cases, so we can catch any regressions
for that and also test the Java 11 multi-jar artifact during dtests, that we'd
also create during the release process.
The job workflow has now also been changed to make use of the [manual job
approval|https://circleci.com/docs/2.0/workflows/#holding-a-workflow-for-a-manual-approval]
feature, which now allows running dtest jobs only on request and not
automatically with every commit. The Java8 unit tests still do, but that could
also be easily changed if needed. See [example
workflow|https://circleci.com/workflow-run/08ecb879-9aaa-4d75-84d6-b00dc9628425]
with start_ jobs being triggers needed manual approval for starting the actual
jobs.
There was some churn in manually editing the config for paid and non-paid
resource tiers before. This has been mostly mitigated now by using project
settings instead, for overriding lower defaults (see below) and scheduling
dtests on request, which will only run on paid accounts nonetheless, so we use
high settings for these right away. The only issue left is the question how we
may be able to dynamically adjust the {{resource_class}} and {{parallelism}}
settings for unit tests. So at this point, the CircleCI config will work for
both paid and non-paid by default, but paid accounts will see slower unit test
results, as only medium instances are used (ie. 15min instead of 4min).
Attention CircleCI paid account users: you'll have to add "{{CCM_MAX_HEAP_SIZE:
2048M}}" and "{{CCM_HEAP_NEWSIZE: 512M}}" to your project's environment
settings or create a context, to override the lower defaults for free instances!
> CircleCI workflow improvements and Java 11 support
> --
>
> Key: CASSANDRA-14806
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14806
> Project: Cassandra
> Issue Type: Improvement
> Components: Build, Testing
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>
> The current CircleCI config could use some cleanup and improvements. First of
> all, the config has been made more modular by using the new CircleCI 2.1
> executors and command elements. Based on CASSANDRA-14713, there's now also a
> Java 11 executor that will allow running tests under Java 11. The {{build}}
> step will be done using Java 11 in all cases, so we can catch any regressions
> for that and also test the Java 11 multi-jar artifact during dtests, that
> we'd also create during the release process.
> The job workflow has now also been changed to make use of the [manual job
> approval|https://circleci.com/docs/2.0/workflows/#holding-a-workflow-for-a-manual-approval]
> feature, which now allows running dtest jobs only on request and not
> automatically with every commit. The Java8 unit tests still do, but that
> could also be easily changed if needed. See [example
> workflow|https://circleci.com/workflow-run/be25579d-3cbb-4258-9e19-b1f571873850]
> with start_ jobs being triggers needed manual approval for starting the
> actual jobs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional
[jira] [Commented] (CASSANDRA-14806) CircleCI workflow improvements and Java 11 support
[
https://issues.apache.org/jira/browse/CASSANDRA-14806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649871#comment-16649871
]
Stefan Podkowinski commented on CASSANDRA-14806:
I've now basically rewritten the CircleCI config to make heavily use of
parameters to avoid redundant copy and pasting of script snippets. Combined
with job approvals, we can now have a broader selection of tests. So far I've
added: ant long-test, test-burn, cql-test, test-compression, stress-test,
fqltool-test. Cqlsh dtests will be added in CASSANDRA-14298.
Unfortunately I wasn't able to solve the requirement for not having to manually
edit the config for switching between high/free resource settings. CircleCI
support was so kind to look at the issue and latest config, but wasn't able to
advise a solution either. So I guess we still have to live with that situation
for a while.
> CircleCI workflow improvements and Java 11 support
> --
>
> Key: CASSANDRA-14806
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14806
> Project: Cassandra
> Issue Type: Improvement
> Components: Build, Testing
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Major
>
> The current CircleCI config could use some cleanup and improvements. First of
> all, the config has been made more modular by using the new CircleCI 2.1
> executors and command elements. Based on CASSANDRA-14713, there's now also a
> Java 11 executor that will allow running tests under Java 11. The {{build}}
> step will be done using Java 11 in all cases, so we can catch any regressions
> for that and also test the Java 11 multi-jar artifact during dtests, that
> we'd also create during the release process.
> The job workflow has now also been changed to make use of the [manual job
> approval|https://circleci.com/docs/2.0/workflows/#holding-a-workflow-for-a-manual-approval]
> feature, which now allows running dtest jobs only on request and not
> automatically with every commit. The Java8 unit tests still do, but that
> could also be easily changed if needed. See [example
> workflow|https://circleci.com/workflow-run/08ecb879-9aaa-4d75-84d6-b00dc9628425]
> with start_ jobs being triggers needed manual approval for starting the
> actual jobs.
> There was some churn in manually editing the config for paid and non-paid
> resource tiers before. This has been mostly mitigated now by using project
> settings instead, for overriding lower defaults (see below) and scheduling
> dtests on request, which will only run on paid accounts nonetheless, so we
> use high settings for these right away. The only issue left is the question
> how we may be able to dynamically adjust the {{resource_class}} and
> {{parallelism}} settings for unit tests. So at this point, the CircleCI
> config will work for both paid and non-paid by default, but paid accounts
> will see slower unit test results, as only medium instances are used (ie.
> 15min instead of 4min).
> Attention CircleCI paid account users: you'll have to add
> "{{CCM_MAX_HEAP_SIZE: 2048M}}" and "{{CCM_HEAP_NEWSIZE: 512M}}" to your
> project's environment settings or create a context, to override the lower
> defaults for free instances!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-14228) Add expiration date overflow notice and recovery instructions to doc
[ https://issues.apache.org/jira/browse/CASSANDRA-14228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefan Podkowinski reassigned CASSANDRA-14228: -- Assignee: Andrew Baker > Add expiration date overflow notice and recovery instructions to doc > > > Key: CASSANDRA-14228 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14228 > Project: Cassandra > Issue Type: Task > Components: Documentation and Website >Reporter: Paulo Motta >Assignee: Andrew Baker >Priority: Minor > Labels: lhf > > On CASSANDRA-14092 we added a new > [CASSANDRA-14092.txt|https://github.com/apache/cassandra/blob/trunk/CASSANDRA-14092.txt] > file with the maximum ttl expiration notice and recovery instructions for > affected users. > We should probably also add the contents of this file to the documentation > with some basic formatting. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13487) Generate snapshot packages through Jenkins
[
https://issues.apache.org/jira/browse/CASSANDRA-13487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-13487:
---
Attachment: 13487.patch
> Generate snapshot packages through Jenkins
> --
>
> Key: CASSANDRA-13487
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13487
> Project: Cassandra
> Issue Type: New Feature
> Components: Build
>Reporter: Stefan Podkowinski
>Priority: Major
> Attachments: 13487.patch
>
>
> Creating packages through the new docker based build scripts now work pretty
> much independent from any local environment, as long as docker is available,
> e.g. also on Jenkins. Having daily snapshots available for deb/rpm artifacts
> would enable us to provide users dev-releases for testing and validating
> fixes.
> I've created a branch for the Jenkins integration, which can be found here:
> https://github.com/spodkowinski/cassandra-builds/tree/jenkins_debrpm
> The major issue I'm currently struggling with is the handling of the actual
> version value. We need to find a way to have Jenkins recognize the correct
> version for the branch being build. Also we must create $version-SNAPSHOT
> packages, as builds are not official releases and we should not have any
> packages for versions that aren't published yet.
> The Debian build process will use the version defined in
> {{debian/changelog}}. Adding a -SNAPSHOT suffix for the version should work,
> but this has to be handled manually and care must be taken to change the
> value back again for a regular release.
> With RPMs, the version must be set for {{cassandra.spec}}, which is currently
> done by running {noformat}rpmbuild --define="version ${CASSANDRA_VERSION}"
> -ba ./redhat/cassandra.spec{noformat}, where the version is passed as a
> parameter by {{build-scripts/cassandra-rpm-packaging.sh}}. Maybe we could
> grep the version from build.xml here?
> So I wonder if there any way we can keep track of the version in a single
> place, such as build.xml or CHANGES. Afterwards we also need to enable a
> SNAPSHOT mode, maybe by setting a Jenkins environment value.
> /cc [~mshuler]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13487) Generate snapshot packages through Jenkins
[
https://issues.apache.org/jira/browse/CASSANDRA-13487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-13487:
---
Attachment: 13487-v2.patch
> Generate snapshot packages through Jenkins
> --
>
> Key: CASSANDRA-13487
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13487
> Project: Cassandra
> Issue Type: New Feature
> Components: Build
>Reporter: Stefan Podkowinski
>Priority: Major
> Attachments: 13487-v2.patch, 13487.patch
>
>
> Creating packages through the new docker based build scripts now work pretty
> much independent from any local environment, as long as docker is available,
> e.g. also on Jenkins. Having daily snapshots available for deb/rpm artifacts
> would enable us to provide users dev-releases for testing and validating
> fixes.
> I've created a branch for the Jenkins integration, which can be found here:
> https://github.com/spodkowinski/cassandra-builds/tree/jenkins_debrpm
> The major issue I'm currently struggling with is the handling of the actual
> version value. We need to find a way to have Jenkins recognize the correct
> version for the branch being build. Also we must create $version-SNAPSHOT
> packages, as builds are not official releases and we should not have any
> packages for versions that aren't published yet.
> The Debian build process will use the version defined in
> {{debian/changelog}}. Adding a -SNAPSHOT suffix for the version should work,
> but this has to be handled manually and care must be taken to change the
> value back again for a regular release.
> With RPMs, the version must be set for {{cassandra.spec}}, which is currently
> done by running {noformat}rpmbuild --define="version ${CASSANDRA_VERSION}"
> -ba ./redhat/cassandra.spec{noformat}, where the version is passed as a
> parameter by {{build-scripts/cassandra-rpm-packaging.sh}}. Maybe we could
> grep the version from build.xml here?
> So I wonder if there any way we can keep track of the version in a single
> place, such as build.xml or CHANGES. Afterwards we also need to enable a
> SNAPSHOT mode, maybe by setting a Jenkins environment value.
> /cc [~mshuler]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
