[jira] [Created] (CASSANDRA-13812) Missing system keyspace tables are not created

[ZhaoYang (JIRA)]

ZhaoYang created CASSANDRA-13812:


 Summary: Missing system keyspace tables are not created
 Key: CASSANDRA-13812
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13812
 Project: Cassandra
  Issue Type: Bug
  Components: Distributed Metadata
Reporter: ZhaoYang


Auth/Trace/Distributed Keyspaces or Tables dropped are not created on startup 
although a log message {{MigrationManager.java:220 - Create new table: 
TableMetadata...}} appears.

Steps to reproduce:
# Start node
# {{DROP TABLE system_distributed.view_build_status;}}
# {{DROP TABLE system_distributed.repair_history;}}
# Stop node
# Start node
# Tables are *not* created, but log messages appear

Cause:
System's keyspaces or tables are created with timestamp 0 in CASSANDRA-13441




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-8727) nodetool command to get the status of things that can be enable/disable'd

[Kenya Ukai (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143382#comment-16143382
 ] 

Kenya Ukai commented on CASSANDRA-8727:
---

I submitted PR https://github.com/apache/cassandra/pull/142
Please review.

> nodetool command to get the status of things that can be enable/disable'd
> -
>
> Key: CASSANDRA-8727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8727
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter:  Brian Hess
>  Labels: lhf
>
> It is possible to say enableautocompaction or disableautocompaction via 
> nodetool.  But it doesn't appear that there is a nodetool command to display 
> the current state of autocompaction - enabled or disabled.  
> It would be good to be able to get the status of these binary commands so 
> that you can determine the status, possibly make a change to 
> troubleshoot/experiment/etc, and return the status to the way it was when you 
> arrived.
> This applies for autocompaction, backup, binary, gossip, handoff, and thrift.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-8727) nodetool command to get the status of things that can be enable/disable'd

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143379#comment-16143379
 ] 

ASF GitHub Bot commented on CASSANDRA-8727:
---

GitHub user cormoran opened a pull request:

https://github.com/apache/cassandra/pull/142

CASSANDRA-8727 add command 'statusautocompaction'

Added nodetool command 'statusautocompaction'
By default, the status is summarized: running, not running or partially 
running (some of the specified tables have it enabled, some don't).
```
$ nodetool statusautocompaction
partially running
```
The `--all` option can be used to see a detailed status output.
```
$ nodetool statusautocompaction -a
Keyspace   Table  Status
system_schema  dropped_columnsnot running
system_schema  tables not running
...
system built_viewsrunning
system sstable_activity   running
...
test   test2  not running
test   test_cfnot running
system_authrole_permissions   not running
system_authresource_role_permissons_index not running
```

You can specify keyspace (and table).
```
$ nodetool statusautocompaction test test2 -a
Keyspace Table Status
test test2 not running
```


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cormoran/cassandra CASSANDRA-8727

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/cassandra/pull/142.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #142


commit 9bd85f7402f566fb7bd6bcbd969486446f08a21b
Author: cormoran 
Date:   2017-08-28T04:18:57Z

CASSANDRA-8727 add command 'statusautocompaction'




> nodetool command to get the status of things that can be enable/disable'd
> -
>
> Key: CASSANDRA-8727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8727
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter:  Brian Hess
>  Labels: lhf
>
> It is possible to say enableautocompaction or disableautocompaction via 
> nodetool.  But it doesn't appear that there is a nodetool command to display 
> the current state of autocompaction - enabled or disabled.  
> It would be good to be able to get the status of these binary commands so 
> that you can determine the status, possibly make a change to 
> troubleshoot/experiment/etc, and return the status to the way it was when you 
> arrived.
> This applies for autocompaction, backup, binary, gossip, handoff, and thrift.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Resolved] (CASSANDRA-13783) Updating the plugin url link

[Amitkumar Ghatwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amitkumar Ghatwal resolved CASSANDRA-13783.
---
Resolution: Fixed
  Reviewer: Amitkumar Ghatwal

Not an Issue , hence closing.

> Updating the plugin url link
> 
>
> Key: CASSANDRA-13783
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13783
> Project: Cassandra
>  Issue Type: Task
>  Components: Documentation and Website
>Reporter: Amitkumar Ghatwal
>Priority: Critical
> Fix For: 4.x
>
>
> Hi All,
>  
> Updating index.rst with url of plugin
> Moved the capi-row cache plugin github repository from an individual github 
> account to a more generic one.
> Older plugin repo location : https://github.com/hhorii/capi-rowcache
> New plugin repo location : https://github.com/ppc64le/capi-rowcache
> PR link : 
> https://github.com/apache/cassandra/pull/139/commits/05851cdabc953ec17480b0bd325b1b5adb7b53bb
> Reference : https://issues.apache.org/jira/browse/CASSANDRA-13581



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13783) Updating the plugin url link

[Amitkumar Ghatwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143377#comment-16143377
 ] 

Amitkumar Ghatwal commented on CASSANDRA-13783:
---

The updated url is already reflecting here : 
https://github.com/apache/cassandra/blob/trunk/doc/source/plugins/index.rst .

Closing this ticket.

> Updating the plugin url link
> 
>
> Key: CASSANDRA-13783
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13783
> Project: Cassandra
>  Issue Type: Task
>  Components: Documentation and Website
>Reporter: Amitkumar Ghatwal
>Priority: Critical
> Fix For: 4.x
>
>
> Hi All,
>  
> Updating index.rst with url of plugin
> Moved the capi-row cache plugin github repository from an individual github 
> account to a more generic one.
> Older plugin repo location : https://github.com/hhorii/capi-rowcache
> New plugin repo location : https://github.com/ppc64le/capi-rowcache
> PR link : 
> https://github.com/apache/cassandra/pull/139/commits/05851cdabc953ec17480b0bd325b1b5adb7b53bb
> Reference : https://issues.apache.org/jira/browse/CASSANDRA-13581



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-9430) Add startup options to cqlshrc

[Murukesh Mohanan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-9430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143349#comment-16143349
 ] 

Murukesh Mohanan commented on CASSANDRA-9430:
-

(Just a note to anybody interested in implementing this.) If this were to be 
implemented with an option like Python's {{-i}}, I still wouldn't expect {{echo 
 "paging off;" | cqlsh -i}} to give me an interactive shell. That would require 
{{cqlsh}} to detect whether it is connected to a TTY (since stdin naturally 
would be from the pipe), then provide a prompt on that TTY, and read from that 
TTY. That could be done ({{sudo}} is a command which does that), but it would 
be quite hacky. I think it would be best if cqlsh didn't try to play guessing 
games with stdin, and simply take it as given. Reading from a file given as an 
option argument is a different case, of course.

> Add startup options to cqlshrc
> --
>
> Key: CASSANDRA-9430
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9430
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Jeremy Hanna
>Priority: Minor
>  Labels: cqlsh, lhf
>
> There are certain settings that would be nice to set defaults for in the 
> cqlshrc file.  For example, a user may want to set the paging to off by 
> default for their environment.  You can't simply do
> {code}
> echo "paging off;" | cqlsh
> {code}
> because this would disable paging and immediately exit cqlsh.
> So it would be nice to have a section of the cqlshrc to include default 
> settings on startup.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13780) ADD Node streaming throughput performance

[Kurt Greaves (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143315#comment-16143315
 ] 

Kurt Greaves commented on CASSANDRA-13780:
--

To clarify, you are using vnodes? When you bootstrap the new node, how many 
nodes stream to it?

> ADD Node streaming throughput performance
> -
>
> Key: CASSANDRA-13780
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13780
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
> Environment: Linux 2.6.32-696.3.2.el6.x86_64 #1 SMP Mon Jun 19 
> 11:55:55 PDT 2017 x86_64 x86_64 x86_64 GNU/Linux
> Architecture:  x86_64
> CPU op-mode(s):32-bit, 64-bit
> Byte Order:Little Endian
> CPU(s):40
> On-line CPU(s) list:   0-39
> Thread(s) per core:2
> Core(s) per socket:10
> Socket(s): 2
> NUMA node(s):  2
> Vendor ID: GenuineIntel
> CPU family:6
> Model: 79
> Model name:Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
> Stepping:  1
> CPU MHz:   2199.869
> BogoMIPS:  4399.36
> Virtualization:VT-x
> L1d cache: 32K
> L1i cache: 32K
> L2 cache:  256K
> L3 cache:  25600K
> NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
> NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
>  total   used   free sharedbuffers cached
> Mem:  252G   217G34G   708K   308M   149G
> -/+ buffers/cache:67G   185G
> Swap:  16G 0B16G
>Reporter: Kevin Rivait
> Fix For: 3.0.9
>
>
> Problem: Adding a new node to a large cluster runs at least 1000x slower than 
> what the network and node hardware capacity can support, taking several days 
> per new node.  Adjusting stream throughput and other YAML parameters seems to 
> have no effect on performance.  Essentially, it appears that Cassandra has an 
> architecture scalability growth problem when adding new nodes to a moderate 
> to high data ingestion cluster because Cassandra cannot add new node capacity 
> fast enough to keep up with increasing data ingestion volumes and growth.
> Initial Configuration: 
> Running 3.0.9 and have implemented TWCS on one of our largest table.
> Largest table partitioned on (ID, MM)  using 1 day buckets with a TTL of 
> 60 days.
> Next release will change partitioning to (ID, MMDD) so that partitions 
> are aligned with daily TWCS buckets.
> Each node is currently creating roughly a 30GB SSTable per day.
> TWCS working as expected,  daily SSTables are dropping off daily after 70 
> days ( 60 + 10 day grace)
> Current deployment is a 28 node 2 datacenter cluster, 14 nodes in each DC , 
> replication factor 3
> Data directories are backed with 4 - 2TB SSDs on each node  and a 1 800GB SSD 
> for commit logs.
> Requirement is to double cluster size, capacity, and ingestion volume within 
> a few weeks.
> Observed Behavior:
> 1. streaming throughput during add node – we observed maximum 6 Mb/s 
> streaming from each of the 14 nodes on a 20Gb/s switched network, taking at 
> least 106 hours for each node to join cluster and each node is only about 2.2 
> TB is size.
> 2. compaction on the newly added node - compaction has fallen behind, with 
> anywhere from 4,000 to 10,000 SSTables at any given time.  It took 3 weeks 
> for compaction to finish on each newly added node.   Increasing number of 
> compaction threads to match number of CPU (40)  and increasing compaction 
> throughput to 32MB/s seemed to be the sweet spot. 
> 3. TWCS buckets on new node, data streamed to this node over 4 1/2 days.  
> Compaction correctly placed the data in daily files, but the problem is the 
> file dates reflect when compaction created the file and not the date of the 
> last record written in the TWCS bucket, which will cause the files to remain 
> around much longer than necessary.  
> Two Questions:
> 1. What can be done to substantially improve the performance of adding a new 
> node?
> 2. Can compaction on TWCS partitions for newly added nodes change the file 
> create date to match the highest date record in the file -or- add another 
> piece of meta-data to the TWCS files that reflect the file drop date so that 
> TWCS partitions can be dropped consistently?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13787) RangeTombstoneMarker and ParitionDeletion is not properly included in MV

[ZhaoYang (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138159#comment-16138159
 ] 

ZhaoYang edited comment on CASSANDRA-13787 at 8/28/17 2:50 AM:
---

The first problem is not exactly a MV bug, it's SSTableReader issue of handling 
multiple slices which is probably only used by MV. The existing code consumes 
the {{original RangeTombstoneMarker(eg. Deletion@c1=1@10)}} for the first slice 
to generate a RangeTombstoneMarker(eg. Deletion@c1=1=0@10) with first 
slice's clustering. So there is no tombstones generated for other slices. The 
fix is to make sure tombstones are generated for each slices within the range 
of original RangeTombstoneMarker and do not close the openMarker til reading 
its paired closeMarker.

The second problem is that if there is no existing live data, empty existing 
row is given to the ViewUpdateGenerator which may resurrect deleted cells. The 
fix is to include current-open-deletion into the empty existing row. So view 
could shadow the deleted cells.


[Draft|https://github.com/jasonstack/cassandra/commits/CASSANDRA-13787-3.0] 
patch base on cassandra-3.0 . This needs to be fixed in 3.0/3.11/trunk.




was (Author: jasonstack):
The first problem is not exactly a MV bug, it's SSTableReader issue of handling 
multiple slices which is probably only used by MV. The existing code consumes 
the {{original RangeTombstoneMarker(eg. Deletion@c1=1@10)}} for the first slice 
to generate a RangeTombstoneMarker(eg. Deletion@c1=1=0@10) with first 
slice's clustering. So there is no tombstones generated for other slices. The 
fix is change the ClusteringPrefix comparison, to make sure tombstones are 
generated for each slices within the range of original RangeTombstoneMarker.

The second problem is that if there is no existing live data, empty existing 
row is given to the ViewUpdateGenerator which may resurrect deleted cells. The 
fix is to include current-open-deletion into the empty existing row. So view 
could shadow the deleted cells.


[Draft|https://github.com/jasonstack/cassandra/commits/CASSANDRA-13787-3.0] 
patch base on cassandra-3.0 . This needs to be fixed in 3.0/3.11/trunk.



> RangeTombstoneMarker and ParitionDeletion is not properly included in MV
> 
>
> Key: CASSANDRA-13787
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13787
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths, Materialized Views
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>
> Found two problems related to MV tombstone. 
> 1. Range-tombstone-Marker being ignored after shadowing first row, subsequent 
> base rows are not shadowed in TableViews.
> If the range tombstone was not flushed, it was used as deleted row to 
> shadow new updates. It works correctly.
> After range tombstone was flushed, it was used as RangeTombstoneMarker 
> and being skipped after shadowing first update. The bound of 
> RangeTombstoneMarker seems wrong, it contained full clustering, but it should 
> contain range or it should be multiple RangeTombstoneMarkers for multiple 
> slices(aka. new updates)
> 2. Partition tombstone is not used when no existing live data, it will 
> resurrect deleted cells. It was found in 11500 and included in that patch.
> In order not to make 11500 patch more complicated, I will try fix 
> range/partition tombstone issue here.
> {code:title=Tests to reproduce}
> @Test
> public void testExistingRangeTombstoneWithFlush() throws Throwable
> {
> testExistingRangeTombstone(true);
> }
> @Test
> public void testExistingRangeTombstoneWithoutFlush() throws Throwable
> {
> testExistingRangeTombstone(false);
> }
> public void testExistingRangeTombstone(boolean flush) throws Throwable
> {
> createTable("CREATE TABLE %s (k1 int, c1 int, c2 int, v1 int, v2 int, 
> PRIMARY KEY (k1, c1, c2))");
> execute("USE " + keyspace());
> executeNet(protocolVersion, "USE " + keyspace());
> createView("view1",
>"CREATE MATERIALIZED VIEW view1 AS SELECT * FROM %%s WHERE 
> k1 IS NOT NULL AND c1 IS NOT NULL AND c2 IS NOT NULL PRIMARY KEY (k1, c2, 
> c1)");
> updateView("DELETE FROM %s USING TIMESTAMP 10 WHERE k1 = 1 and c1=1");
> if (flush)
> 
> Keyspace.open(keyspace()).getColumnFamilyStore(currentTable()).forceBlockingFlush();
> String table = KEYSPACE + "." + currentTable();
> updateView("BEGIN BATCH " +
> "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 0, 
> 0, 0, 0) USING TIMESTAMP 5; " +
> "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 0, 
> 1, 0, 1) USING TIMESTAMP 5; " +
> "INSERT 

[jira] [Commented] (CASSANDRA-13798) Disallow filtering on non-primary-key base column for MV

[Kurt Greaves (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143300#comment-16143300
 ] 

Kurt Greaves commented on CASSANDRA-13798:
--

I think something should be done on 3.11 and blocking the behaviour with a 
workaround flag as [~pauloricardomg] suggested is probably the best way to go 
about it. MV's are already causing a lot of issues for people doing dev on 
3.11, and I know we have users with this case, that will undoubtedly run into 
issues sooner or later.

However I'm not convinced about also committing this to 4.0. If we're not 
committing to fixing it in 4.0 then MV's will be pretty useless until 5.0 if it 
does take massive storage changes, which could be a long way away. Which will 
be problematic for people who have already gone down this path, and we'll be 
missing a much wanted feature for quite a long time. IMO we made the mistake of 
pushing it out too fast, we should fix it asap.

> Disallow filtering on non-primary-key base column for MV
> 
>
> Key: CASSANDRA-13798
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13798
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>
> We should probably consider disallow filtering conditions on non-primary-key 
> base column for Materialized View which is introduced in CASSANDRA-10368.
> The main problem is that the liveness of view row is now depending on 
> multiple base columns (multiple filtered non-pk base column + base column 
> used in view pk) and this semantic could not be properly support without 
> drastic storage format changes. (SEE CASSANDRA-11500, 
> [background|https://issues.apache.org/jira/browse/CASSANDRA-11500?focusedCommentId=16119823=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16119823])
> We should step back and re-consider the non-primary-key filtering feature 
> together with supporting multiple non-PK cols in MV clustering key in 
> CASSANDRA-10226.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13811) Unable to find table . at maybeLoadschemainfo (StressProfile.java)

[Akshay Jindal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akshay Jindal updated CASSANDRA-13811:
--
Description: 
* Please find attached my .yaml and .sh file.
* Now the problem is if I run stress-script.sh the first time, just after 
firing up cassandra, it is working fine on the cluster, but when I again run 
stress-script.sh, it is giving the following error:-

*Unable to find prutorStress3node.code*
at 
org.apache.cassandra.stress.StressProfile.maybeLoadSchemaInfo(StressProfile.java:306)
at 
org.apache.cassandra.stress.StressProfile.maybeCreateSchema(StressProfile.java:273)
at 
org.apache.cassandra.stress.StressProfile.newGenerator(StressProfile.java:676)
at 
org.apache.cassandra.stress.StressProfile.printSettings(StressProfile.java:129)
at 
org.apache.cassandra.stress.settings.StressSettings.printSettings(StressSettings.java:383)
at org.apache.cassandra.stress.Stress.run(Stress.java:95)
at org.apache.cassandra.stress.Stress.main(Stress.java:62)

In the file 
[https://insight.io/github.com/apache/cassandra/blob/trunk/tools/stress/src/org/apache/cassandra/stress/StressProfile.java?line=289]
 ,I saw that table metadata is being populated to NULL. I tried to make sense 
of the stack trace, but was not able to make anything of it. Please give me 
some directions as to what might have gone wrong?

  was:
* Please find attached my .yaml and .sh file.
* Now the problem is if I run this the first time, just after firing up 
cassandra, it is working fine, but when I again run this query, it is giving 
the following error:-

*Unable to find prutorStress3node.code*
at 
org.apache.cassandra.stress.StressProfile.maybeLoadSchemaInfo(StressProfile.java:306)
at 
org.apache.cassandra.stress.StressProfile.maybeCreateSchema(StressProfile.java:273)
at 
org.apache.cassandra.stress.StressProfile.newGenerator(StressProfile.java:676)
at 
org.apache.cassandra.stress.StressProfile.printSettings(StressProfile.java:129)
at 
org.apache.cassandra.stress.settings.StressSettings.printSettings(StressSettings.java:383)
at org.apache.cassandra.stress.Stress.run(Stress.java:95)
at org.apache.cassandra.stress.Stress.main(Stress.java:62)

In the file 
[https://insight.io/github.com/apache/cassandra/blob/trunk/tools/stress/src/org/apache/cassandra/stress/StressProfile.java?line=289]
 ,I saw that table metadata is being populated to NULL. I tried to make sense 
of the stack trace, but was not able to make anything of it. Please give me 
some directions as to what might have gone wrong?


> Unable to find table . at maybeLoadschemainfo 
> (StressProfile.java)
> 
>
> Key: CASSANDRA-13811
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13811
> Project: Cassandra
>  Issue Type: Bug
>  Components: Stress
> Environment: 3 node cluster
> Node 1 --> 172.27.21.16(Seed Node)
> Node 2 --> 172.27.21.18
> Node 3 --> 172.27.21.19
> *cassandra.yaml paramters for all the nodes:-*
> 1) seeds: "172.27.21.16"
> 2) write_request_timeout_in_ms: 5000
> 3) listen_address: 172.27.21.1(6,8,9
> 4) rpc_address: 172.27.21.1(6,8,9)
>Reporter: Akshay Jindal
> Fix For: 3.10
>
> Attachments: code.yaml, stress-script.sh
>
>
> * Please find attached my .yaml and .sh file.
> * Now the problem is if I run stress-script.sh the first time, just after 
> firing up cassandra, it is working fine on the cluster, but when I again run 
> stress-script.sh, it is giving the following error:-
> *Unable to find prutorStress3node.code*
> at 
> org.apache.cassandra.stress.StressProfile.maybeLoadSchemaInfo(StressProfile.java:306)
>   at 
> org.apache.cassandra.stress.StressProfile.maybeCreateSchema(StressProfile.java:273)
>   at 
> org.apache.cassandra.stress.StressProfile.newGenerator(StressProfile.java:676)
>   at 
> org.apache.cassandra.stress.StressProfile.printSettings(StressProfile.java:129)
>   at 
> org.apache.cassandra.stress.settings.StressSettings.printSettings(StressSettings.java:383)
>   at org.apache.cassandra.stress.Stress.run(Stress.java:95)
>   at org.apache.cassandra.stress.Stress.main(Stress.java:62)
> In the file 
> [https://insight.io/github.com/apache/cassandra/blob/trunk/tools/stress/src/org/apache/cassandra/stress/StressProfile.java?line=289]
>  ,I saw that table metadata is being populated to NULL. I tried to make sense 
> of the stack trace, but was not able to make anything of it. Please give me 
> some directions as to what might have gone wrong?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: 

[jira] [Updated] (CASSANDRA-13811) Unable to find table . at maybeLoadschemainfo (StressProfile.java)

[Akshay Jindal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akshay Jindal updated CASSANDRA-13811:
--
Description: 
* Please find attached my .yaml and .sh file.
* Now the problem is if I run this the first time, just after firing up 
cassandra, it is working fine, but when I again run this query, it is giving 
the following error:-

*Unable to find prutorStress3node.code*
at 
org.apache.cassandra.stress.StressProfile.maybeLoadSchemaInfo(StressProfile.java:306)
at 
org.apache.cassandra.stress.StressProfile.maybeCreateSchema(StressProfile.java:273)
at 
org.apache.cassandra.stress.StressProfile.newGenerator(StressProfile.java:676)
at 
org.apache.cassandra.stress.StressProfile.printSettings(StressProfile.java:129)
at 
org.apache.cassandra.stress.settings.StressSettings.printSettings(StressSettings.java:383)
at org.apache.cassandra.stress.Stress.run(Stress.java:95)
at org.apache.cassandra.stress.Stress.main(Stress.java:62)

In the file 
[https://insight.io/github.com/apache/cassandra/blob/trunk/tools/stress/src/org/apache/cassandra/stress/StressProfile.java?line=289]
 ,I saw that table metadata is being populated to NULL. I tried to make sense 
of the stack trace, but was not able to make anything of it. Please give me 
some directions as to what might have gone wrong?

  was:
* Please find attached my .yaml and .sh file.
* Now the problem is if I run this the first time, just after firing up 
cassandra, it is working fine, but when I again run this query, it is giving 
the following error:-

*Unable to find prutorStress3node.code*
at 
org.apache.cassandra.stress.StressProfile.maybeLoadSchemaInfo(StressProfile.java:306)
at 
org.apache.cassandra.stress.StressProfile.maybeCreateSchema(StressProfile.java:273)
at 
org.apache.cassandra.stress.StressProfile.newGenerator(StressProfile.java:676)
at 
org.apache.cassandra.stress.StressProfile.printSettings(StressProfile.java:129)
at 
org.apache.cassandra.stress.settings.StressSettings.printSettings(StressSettings.java:383)
at org.apache.cassandra.stress.Stress.run(Stress.java:95)
at org.apache.cassandra.stress.Stress.main(Stress.java:62)

In the file 
[https://insight.io/github.com/apache/cassandra/blob/trunk/tools/stress/src/org/apache/cassandra/stress/settings/StressSettings.java?line=121]
 ,I saw that table metadata is being populated to NULL. I tried to make sense 
of the stack trace, but was not able to make anything of it. Please give me 
some directions as to what might have gone wrong?


> Unable to find table . at maybeLoadschemainfo 
> (StressProfile.java)
> 
>
> Key: CASSANDRA-13811
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13811
> Project: Cassandra
>  Issue Type: Bug
>  Components: Stress
> Environment: 3 node cluster
> Node 1 --> 172.27.21.16(Seed Node)
> Node 2 --> 172.27.21.18
> Node 3 --> 172.27.21.19
> *cassandra.yaml paramters for all the nodes:-*
> 1) seeds: "172.27.21.16"
> 2) write_request_timeout_in_ms: 5000
> 3) listen_address: 172.27.21.1(6,8,9
> 4) rpc_address: 172.27.21.1(6,8,9)
>Reporter: Akshay Jindal
> Fix For: 3.10
>
> Attachments: code.yaml, stress-script.sh
>
>
> * Please find attached my .yaml and .sh file.
> * Now the problem is if I run this the first time, just after firing up 
> cassandra, it is working fine, but when I again run this query, it is giving 
> the following error:-
> *Unable to find prutorStress3node.code*
> at 
> org.apache.cassandra.stress.StressProfile.maybeLoadSchemaInfo(StressProfile.java:306)
>   at 
> org.apache.cassandra.stress.StressProfile.maybeCreateSchema(StressProfile.java:273)
>   at 
> org.apache.cassandra.stress.StressProfile.newGenerator(StressProfile.java:676)
>   at 
> org.apache.cassandra.stress.StressProfile.printSettings(StressProfile.java:129)
>   at 
> org.apache.cassandra.stress.settings.StressSettings.printSettings(StressSettings.java:383)
>   at org.apache.cassandra.stress.Stress.run(Stress.java:95)
>   at org.apache.cassandra.stress.Stress.main(Stress.java:62)
> In the file 
> [https://insight.io/github.com/apache/cassandra/blob/trunk/tools/stress/src/org/apache/cassandra/stress/StressProfile.java?line=289]
>  ,I saw that table metadata is being populated to NULL. I tried to make sense 
> of the stack trace, but was not able to make anything of it. Please give me 
> some directions as to what might have gone wrong?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: 

[jira] [Created] (CASSANDRA-13811) Unable to find table . at maybeLoadschemainfo (StressProfile.java)

[Akshay Jindal (JIRA)]

Akshay Jindal created CASSANDRA-13811:
-

 Summary: Unable to find table . at 
maybeLoadschemainfo (StressProfile.java)
 Key: CASSANDRA-13811
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13811
 Project: Cassandra
  Issue Type: Bug
  Components: Stress
 Environment: 3 node cluster
Node 1 --> 172.27.21.16(Seed Node)
Node 2 --> 172.27.21.18
Node 3 --> 172.27.21.19

*cassandra.yaml paramters for all the nodes:-*
1) seeds: "172.27.21.16"
2) write_request_timeout_in_ms: 5000
3) listen_address: 172.27.21.1(6,8,9
4) rpc_address: 172.27.21.1(6,8,9)

Reporter: Akshay Jindal
 Fix For: 3.10
 Attachments: code.yaml, stress-script.sh

* Please find attached my .yaml and .sh file.
* Now the problem is if I run this the first time, just after firing up 
cassandra, it is working fine, but when I again run this query, it is giving 
the following error:-

*Unable to find prutorStress3node.code*
at 
org.apache.cassandra.stress.StressProfile.maybeLoadSchemaInfo(StressProfile.java:306)
at 
org.apache.cassandra.stress.StressProfile.maybeCreateSchema(StressProfile.java:273)
at 
org.apache.cassandra.stress.StressProfile.newGenerator(StressProfile.java:676)
at 
org.apache.cassandra.stress.StressProfile.printSettings(StressProfile.java:129)
at 
org.apache.cassandra.stress.settings.StressSettings.printSettings(StressSettings.java:383)
at org.apache.cassandra.stress.Stress.run(Stress.java:95)
at org.apache.cassandra.stress.Stress.main(Stress.java:62)

In the file 
[https://insight.io/github.com/apache/cassandra/blob/trunk/tools/stress/src/org/apache/cassandra/stress/settings/StressSettings.java?line=121]
 ,I saw that table metadata is being populated to NULL. I tried to make sense 
of the stack trace, but was not able to make anything of it. Please give me 
some directions as to what might have gone wrong?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13461) Update circle.yml to run dtests and utests in parallel across containers

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143226#comment-16143226
 ] 

Jeff Jirsa commented on CASSANDRA-13461:


Removing myself as reviewer, let me know if something changes and I should 
revisit.

> Update circle.yml to run dtests and utests in parallel across containers
> 
>
> Key: CASSANDRA-13461
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13461
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
>
> I have a circle.yml that parallelizes the dtests and utests over the 4 free 
> available containers. It can be tweaked to support however many containers 
> are available.
> The unit tests pass normally. The dtests run mostly normally. There are 10 or 
> so tests that fail on trunk, but 30 that fail when run in CircleCI. It's 
> still better than not running the dtests IMO. I am currently working on 
> figuring out why the test failures don't match.
> {noformat}
> version: 2
> jobs:
>   build:
> resource_class: xlarge
> working_directory: ~/
> parallelism: 4
> docker:
>   - image: ubuntu:xenial-20170410
> steps:
>   - run:
>   name: apt-get install packages
>   command: |
> echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu xenial 
> main" | tee /etc/apt/sources.list.d/webupd8team-java.list
> echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu 
> xenial main" | tee -a /etc/apt/sources.list.d/webupd8team-java.list
> apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 
> EEA14886
> echo oracle-java8-installer shared/accepted-oracle-license-v1-1 
> select true | /usr/bin/debconf-set-selections
> apt-get update
> apt-get install -y git-core npm python python-pip python-dev ant 
> ant-optional oracle-java8-installer net-tools
> ln -s /usr/bin/nodejs /usr/bin/node || true
>   - run:
>   name: Log environment information
>   command: |
>   echo '*** id ***'
>   id
>   echo '*** cat /proc/cpuinfo ***'
>   cat /proc/cpuinfo
>   echo '*** free -m ***'
>   free -m
>   echo '*** df -m ***'
>   df -m
>   echo '*** ifconfig -a ***'
>   ifconfig -a
>   echo '*** uname -a ***'
>   uname -a
>   echo '*** mount ***'
>   mount
>   echo '*** env ***'
>   env
>   - run:
>   name: Clone git repos
>   command: |
> git clone --single-branch --depth 1 
> https://github.com/riptano/cassandra-dtest ~/cassandra-dtest
> git clone --single-branch --depth 1 --branch $CIRCLE_BRANCH 
> git://github.com/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME.git 
> ~/cassandra
>   - run:
>   name: Install junit-merge
>   command: npm install -g junit-merge
>   - run:
>   name: Install virtualenv
>   command: pip install virtualenv 
>   - run:
>   name: Configure virtualenv and python dependencies
>   command: |
> virtualenv --python=python2 --no-site-packages venv
> source venv/bin/activate
> export CASS_DRIVER_NO_EXTENSIONS=true
> export CASS_DRIVER_NO_CYTHON=true
> pip install -r ~/cassandra-dtest/requirements.txt
> pip freeze
>   - run:
>   name: Build Cassandra
>   command: |
> cd ~/cassandra
> # Loop to prevent failure due to maven-ant-tasks not downloading 
> a jar..
> for x in $(seq 1 3); do
> ant clean jar
> RETURN="$?"
> if [ "${RETURN}" -eq "0" ]; then
> break
> fi
> done
> # Exit, if we didn't build successfully
> if [ "${RETURN}" -ne "0" ]; then
> echo "Build failed with exit code: ${RETURN}"
> exit ${RETURN}
> fi
>   no_output_timeout: 20m
>   - run:
>   name: Determine tests to run
>   no_output_timeout: 10m
>   command: |
> #"$HOME/cassandra-dtest/**/*.py"
> echo `circleci tests glob "$HOME/cassandra/test/unit/**" 
> "$HOME/cassandra-dtest/**/*.py" | grep -v upgrade_tests | grep -v 
> "cassandra-thrift" | grep -v "thrift_bindings" | grep -v "tools" | grep -v 
> "dtest.py" | grep -v "$HOME/cassandra-dtest/bin" | grep -v "plugins" | 
> circleci tests split --split-by=timings --timings-type=filename` > 
> /tmp/tests.txt
> echo "***processed tests***"
> cat 

[jira] [Comment Edited] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables

[mck (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142240#comment-16142240
 ] 

mck edited comment on CASSANDRA-13418 at 8/27/17 9:22 PM:
--

{quote}N.B: I tried to apply the syle guide found in 
.idea/codeStyleSettings.xml but it is changing me a lot of things. Do you know 
if it is up to date ?{quote}
I don't use IntelliJ so I can't answer that for you, sry. [~krummas]?
Otherwise you can ask on irc #cassandra or on the user mailing list.

{quote}I enable uncheckedTombstoneCompaction when ignoreOverlaps is 
activated{quote}
I'm -1 on this for the moment. While it holds a logic argument, as you explain, 
it's not intuitive for the user. The user has to know that this happens (via 
docs or via code). I'd be more comfortable expecting the users using an 
advanced toggle like this (requires system properties and table option) to 
appreciate the difference between {{uncheckedTombstoneCompaction}} and 
{{unsafe_aggressive_sstable_expiration}} and to enable both. Any smarts can be 
added latter on with further user feedback and experience.

Could we, instead of setting {{uncheckedTombstoneCompaction}}, log a warning 
telling the user that they probably want to {{uncheckedTombstoneCompaction}} 
set as well?


was (Author: michaelsembwever):
{quote}N.B: I tried to apply the syle guide found in 
.idea/codeStyleSettings.xml but it is changing me a lot of things. Do you know 
if it is up to date ?{quote}
I don't use IntelliJ so I can't answer that for you, sry. [~krummas]?
Otherwise you can ask on irc #cassandra or on the user mailing list.

{quote}I enable uncheckedTombstoneCompaction when ignoreOverlaps is 
activated{quote}
I'm -1 on this for the moment. While it holds a logic argument, as you explain, 
it's not intuitive for the user. The user has to know that this happens (via 
docs or via code). I'd be more comfortable expecting the users using an 
advanced toggle like this (requires system properties and table option) to 
appreciate the difference between {{uncheckedTombstoneCompaction}} and 
{{unsafe_aggressive_sstable_expiration}} and to enable both. Any smarts can be 
added latter on with further user feedback and experience.

> Allow TWCS to ignore overlaps when dropping fully expired sstables
> --
>
> Key: CASSANDRA-13418
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13418
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Corentin Chary
>  Labels: twcs
> Attachments: twcs-cleanup.png
>
>
> http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If 
> you really want read-repairs you're going to have sstables blocking the 
> expiration of other fully expired SSTables because they overlap.
> You can set unchecked_tombstone_compaction = true or tombstone_threshold to a 
> very low value and that will purge the blockers of old data that should 
> already have expired, thus removing the overlaps and allowing the other 
> SSTables to expire.
> The thing is that this is rather CPU intensive and not optimal. If you have 
> time series, you might not care if all your data doesn't exactly expire at 
> the right time, or if data re-appears for some time, as long as it gets 
> deleted as soon as it can. And in this situation I believe it would be really 
> beneficial to allow users to simply ignore overlapping SSTables when looking 
> for fully expired ones.
> To the question: why would you need read-repairs ?
> - Full repairs basically take longer than the TTL of the data on my dataset, 
> so this isn't really effective.
> - Even with a 10% chances of doing a repair, we found out that this would be 
> enough to greatly reduce entropy of the most used data (and if you have 
> timeseries, you're likely to have a dashboard doing the same important 
> queries over and over again).
> - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow.
> I'll try to come up with a patch demonstrating how this would work, try it on 
> our system and report the effects.
> cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13780) ADD Node streaming throughput performance

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143223#comment-16143223
 ] 

Jeff Jirsa commented on CASSANDRA-13780:


De-prioritizing - Blocker is reserved for bugs that MUST be fixed before a 
release (ie: corruption bugs).


> ADD Node streaming throughput performance
> -
>
> Key: CASSANDRA-13780
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13780
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
> Environment: Linux 2.6.32-696.3.2.el6.x86_64 #1 SMP Mon Jun 19 
> 11:55:55 PDT 2017 x86_64 x86_64 x86_64 GNU/Linux
> Architecture:  x86_64
> CPU op-mode(s):32-bit, 64-bit
> Byte Order:Little Endian
> CPU(s):40
> On-line CPU(s) list:   0-39
> Thread(s) per core:2
> Core(s) per socket:10
> Socket(s): 2
> NUMA node(s):  2
> Vendor ID: GenuineIntel
> CPU family:6
> Model: 79
> Model name:Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
> Stepping:  1
> CPU MHz:   2199.869
> BogoMIPS:  4399.36
> Virtualization:VT-x
> L1d cache: 32K
> L1i cache: 32K
> L2 cache:  256K
> L3 cache:  25600K
> NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
> NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
>  total   used   free sharedbuffers cached
> Mem:  252G   217G34G   708K   308M   149G
> -/+ buffers/cache:67G   185G
> Swap:  16G 0B16G
>Reporter: Kevin Rivait
> Fix For: 3.0.9
>
>
> Problem: Adding a new node to a large cluster runs at least 1000x slower than 
> what the network and node hardware capacity can support, taking several days 
> per new node.  Adjusting stream throughput and other YAML parameters seems to 
> have no effect on performance.  Essentially, it appears that Cassandra has an 
> architecture scalability growth problem when adding new nodes to a moderate 
> to high data ingestion cluster because Cassandra cannot add new node capacity 
> fast enough to keep up with increasing data ingestion volumes and growth.
> Initial Configuration: 
> Running 3.0.9 and have implemented TWCS on one of our largest table.
> Largest table partitioned on (ID, MM)  using 1 day buckets with a TTL of 
> 60 days.
> Next release will change partitioning to (ID, MMDD) so that partitions 
> are aligned with daily TWCS buckets.
> Each node is currently creating roughly a 30GB SSTable per day.
> TWCS working as expected,  daily SSTables are dropping off daily after 70 
> days ( 60 + 10 day grace)
> Current deployment is a 28 node 2 datacenter cluster, 14 nodes in each DC , 
> replication factor 3
> Data directories are backed with 4 - 2TB SSDs on each node  and a 1 800GB SSD 
> for commit logs.
> Requirement is to double cluster size, capacity, and ingestion volume within 
> a few weeks.
> Observed Behavior:
> 1. streaming throughput during add node – we observed maximum 6 Mb/s 
> streaming from each of the 14 nodes on a 20Gb/s switched network, taking at 
> least 106 hours for each node to join cluster and each node is only about 2.2 
> TB is size.
> 2. compaction on the newly added node - compaction has fallen behind, with 
> anywhere from 4,000 to 10,000 SSTables at any given time.  It took 3 weeks 
> for compaction to finish on each newly added node.   Increasing number of 
> compaction threads to match number of CPU (40)  and increasing compaction 
> throughput to 32MB/s seemed to be the sweet spot. 
> 3. TWCS buckets on new node, data streamed to this node over 4 1/2 days.  
> Compaction correctly placed the data in daily files, but the problem is the 
> file dates reflect when compaction created the file and not the date of the 
> last record written in the TWCS bucket, which will cause the files to remain 
> around much longer than necessary.  
> Two Questions:
> 1. What can be done to substantially improve the performance of adding a new 
> node?
> 2. Can compaction on TWCS partitions for newly added nodes change the file 
> create date to match the highest date record in the file -or- add another 
> piece of meta-data to the TWCS files that reflect the file drop date so that 
> TWCS partitions can be dropped consistently?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13780) ADD Node streaming throughput performance

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13780:
---
Priority: Major  (was: Blocker)

> ADD Node streaming throughput performance
> -
>
> Key: CASSANDRA-13780
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13780
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
> Environment: Linux 2.6.32-696.3.2.el6.x86_64 #1 SMP Mon Jun 19 
> 11:55:55 PDT 2017 x86_64 x86_64 x86_64 GNU/Linux
> Architecture:  x86_64
> CPU op-mode(s):32-bit, 64-bit
> Byte Order:Little Endian
> CPU(s):40
> On-line CPU(s) list:   0-39
> Thread(s) per core:2
> Core(s) per socket:10
> Socket(s): 2
> NUMA node(s):  2
> Vendor ID: GenuineIntel
> CPU family:6
> Model: 79
> Model name:Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
> Stepping:  1
> CPU MHz:   2199.869
> BogoMIPS:  4399.36
> Virtualization:VT-x
> L1d cache: 32K
> L1i cache: 32K
> L2 cache:  256K
> L3 cache:  25600K
> NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
> NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
>  total   used   free sharedbuffers cached
> Mem:  252G   217G34G   708K   308M   149G
> -/+ buffers/cache:67G   185G
> Swap:  16G 0B16G
>Reporter: Kevin Rivait
> Fix For: 3.0.9
>
>
> Problem: Adding a new node to a large cluster runs at least 1000x slower than 
> what the network and node hardware capacity can support, taking several days 
> per new node.  Adjusting stream throughput and other YAML parameters seems to 
> have no effect on performance.  Essentially, it appears that Cassandra has an 
> architecture scalability growth problem when adding new nodes to a moderate 
> to high data ingestion cluster because Cassandra cannot add new node capacity 
> fast enough to keep up with increasing data ingestion volumes and growth.
> Initial Configuration: 
> Running 3.0.9 and have implemented TWCS on one of our largest table.
> Largest table partitioned on (ID, MM)  using 1 day buckets with a TTL of 
> 60 days.
> Next release will change partitioning to (ID, MMDD) so that partitions 
> are aligned with daily TWCS buckets.
> Each node is currently creating roughly a 30GB SSTable per day.
> TWCS working as expected,  daily SSTables are dropping off daily after 70 
> days ( 60 + 10 day grace)
> Current deployment is a 28 node 2 datacenter cluster, 14 nodes in each DC , 
> replication factor 3
> Data directories are backed with 4 - 2TB SSDs on each node  and a 1 800GB SSD 
> for commit logs.
> Requirement is to double cluster size, capacity, and ingestion volume within 
> a few weeks.
> Observed Behavior:
> 1. streaming throughput during add node – we observed maximum 6 Mb/s 
> streaming from each of the 14 nodes on a 20Gb/s switched network, taking at 
> least 106 hours for each node to join cluster and each node is only about 2.2 
> TB is size.
> 2. compaction on the newly added node - compaction has fallen behind, with 
> anywhere from 4,000 to 10,000 SSTables at any given time.  It took 3 weeks 
> for compaction to finish on each newly added node.   Increasing number of 
> compaction threads to match number of CPU (40)  and increasing compaction 
> throughput to 32MB/s seemed to be the sweet spot. 
> 3. TWCS buckets on new node, data streamed to this node over 4 1/2 days.  
> Compaction correctly placed the data in daily files, but the problem is the 
> file dates reflect when compaction created the file and not the date of the 
> last record written in the TWCS bucket, which will cause the files to remain 
> around much longer than necessary.  
> Two Questions:
> 1. What can be done to substantially improve the performance of adding a new 
> node?
> 2. Can compaction on TWCS partitions for newly added nodes change the file 
> create date to match the highest date record in the file -or- add another 
> piece of meta-data to the TWCS files that reflect the file drop date so that 
> TWCS partitions can be dropped consistently?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-12948) Upgrade 2.2.4 to 3.0.10 failed

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-12948:
---
Priority: Major  (was: Blocker)

> Upgrade 2.2.4 to 3.0.10 failed
> --
>
> Key: CASSANDRA-12948
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12948
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: CentOS7, dsc-cassandra 2.2.4 / 3.0.10
>Reporter: Romain
>
> Hi guys,
> I have a problem when I try to migrate my cluster 2.2.4 to 3.0.10.
> In production environnement, I don't want to stop my cluster, so I'll have 
> mixing cluster during the migration.
> Here is the cluster description in development and my tests:
> Today, I have three nodes in 2.2.4, each one with 256 vnodes.
> When I spawn a blank node 2.2.4, my keyspaces are well propagate to the new 
> node.
> When I spawn a blank node 3.0.10, my keyspaces are not propagate to the new 
> node.
> After the new node was spawned my application generate this error on the new 
> node:
> {code}
> WARN  17:08:14 UnknownColumnFamilyException reading from socket; closing
> cassandra3_1| 
> org.apache.cassandra.db.UnknownColumnFamilyException: Got legacy paged range 
> command for nonexistent table docker.user.
> {code}
> It seems normal because keyspaces are not propagate.
> Here is the output of the describecluster command:
> {code}
> Cluster Information:
>   Name: Test Cluster
>   Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
>   Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>   Schema versions:
>   a34bf9e6-9139-3696-b85e-3fdd47aca08f: [172.17.0.2, 172.17.0.3, 
> 172.17.0.5]
>   59adb24e-f3cd-3e02-97f0-5b395827453f: [172.17.0.4]
> {code}
> Now, a new test. 3 nodes in 2.2.4, 0 in 3.0.10
> I upgrade an existing 2.2.4 with this instruction 
> https://docs.datastax.com/en/upgrade/doc/upgrade/cassandra/upgrdCassandraDetails.html.
> So I have this describecluster:
> {code}
> Cluster Information:
>   Name: Test Cluster
>   Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
>   Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>   Schema versions:
>   a34bf9e6-9139-3696-b85e-3fdd47aca08f: [172.17.0.2, 172.17.0.3]
>   59adb24e-f3cd-3e02-97f0-5b395827453f: [172.17.0.5]
> {code}
> The node 172.17.0.5 is the one in 3.0.10. It has same keyspaces as other 
> nodes.
> When I use my application (protocol version to 3), it try to contact the 
> 3.0.10 node and I get this error:
> {code}
> Cassandra timeout during write query at consistency LOCAL_ONE (1 replica were 
> required but only 0 acknowledged the write)
> {code}
> For all my test, the seeder is 172.17.0.2 and replication factor is 1.
> What is wrong in my migration ? Or is it a bug in the migration procedure ?
> Why the blank node in 3.0.10 doesn't receive keyspaces description ?
> Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13546) Getting error while adding a node in existing cluster

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143222#comment-16143222
 ] 

Jeff Jirsa commented on CASSANDRA-13546:


Sonu, looks like you were using Cassandra 2.1.2, which is now very old. Have 
you reproduced this with a modern version of Cassandra?


> Getting error while adding a node in existing cluster
> -
>
> Key: CASSANDRA-13546
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13546
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
> Environment: Unix
>Reporter: Sonu Gupta
>Priority: Blocker
> Fix For: 2.1.2
>
>
> Getting error after adding a node in existing cluster, error are coming after 
> 1 hour after started service on seed node and due to this load got increased 
> vastly on seed node not on new node.
> Below errors from system.log on seed node:
> {code}
> ERROR [NonPeriodicTasks:1] 2017-05-18 10:03:01,819 CassandraDaemon.java:153 - 
> Exception in thread Thread[NonPeriodicTasks:1,5,main]
> java.lang.AssertionError: null
> at org.apache.cassandra.io.util.Memory.free(Memory.java:300) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.utils.obs.OffHeapBitSet.close(OffHeapBitSet.java:143) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.utils.BloomFilter.close(BloomFilter.java:116) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.io.sstable.SSTableReader$6.run(SSTableReader.java:645) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown 
> Source) ~[na:1.7.0_71]
> at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_71]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown
>  Source) ~[na:1.7.0_71]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>  Source) ~[na:1.7.0_71]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 
> [na:1.7.0_71]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
> [na:1.7.0_71]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
> INFO  [CompactionExecutor:162] 2017-05-18 10:03:01,820 
> ColumnFamilyStore.java:840 - Enqueuing flush of compactions_in_progress: 164 
> (0%) on-heap, 0 (0%) off-heap
> INFO  [MemtableFlushWriter:152] 2017-05-18 10:03:01,821 Memtable.java:325 - 
> Writing Memtable-compactions_in_progress@731506998(0 serialized bytes, 1 ops, 
> 0%/0% of on/off-heap limit)
> INFO  [MemtableFlushWriter:152] 2017-05-18 10:03:01,825 Memtable.java:364 - 
> Completed flushing 
> /var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-52434-Data.db
>  (42 bytes) for commitlog position ReplayPosition(segmentId=1495106933696, 
> position=9156004)
> ERROR [CompactionExecutor:162] 2017-05-18 10:03:01,829 
> CassandraDaemon.java:153 -
>  Exception in thread Thread[CompactionExecutor:162,1,main]
> java.lang.AssertionError: null
> at org.apache.cassandra.io.util.Memory.free(Memory.java:300) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.utils.obs.OffHeapBitSet.close(OffHeapBitSet.java:143) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.utils.BloomFilter.close(BloomFilter.java:116) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:345) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.io.sstable.SSTableRewriter.abort(SSTableRewriter.java:198)
>  ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:204)
>  ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>  ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75)
>  ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
>  ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:232)
>  ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 

[jira] [Updated] (CASSANDRA-13546) Getting error while adding a node in existing cluster

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13546:
---
Priority: Major  (was: Blocker)

> Getting error while adding a node in existing cluster
> -
>
> Key: CASSANDRA-13546
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13546
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
> Environment: Unix
>Reporter: Sonu Gupta
> Fix For: 2.1.2
>
>
> Getting error after adding a node in existing cluster, error are coming after 
> 1 hour after started service on seed node and due to this load got increased 
> vastly on seed node not on new node.
> Below errors from system.log on seed node:
> {code}
> ERROR [NonPeriodicTasks:1] 2017-05-18 10:03:01,819 CassandraDaemon.java:153 - 
> Exception in thread Thread[NonPeriodicTasks:1,5,main]
> java.lang.AssertionError: null
> at org.apache.cassandra.io.util.Memory.free(Memory.java:300) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.utils.obs.OffHeapBitSet.close(OffHeapBitSet.java:143) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.utils.BloomFilter.close(BloomFilter.java:116) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.io.sstable.SSTableReader$6.run(SSTableReader.java:645) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown 
> Source) ~[na:1.7.0_71]
> at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_71]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown
>  Source) ~[na:1.7.0_71]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>  Source) ~[na:1.7.0_71]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 
> [na:1.7.0_71]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
> [na:1.7.0_71]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
> INFO  [CompactionExecutor:162] 2017-05-18 10:03:01,820 
> ColumnFamilyStore.java:840 - Enqueuing flush of compactions_in_progress: 164 
> (0%) on-heap, 0 (0%) off-heap
> INFO  [MemtableFlushWriter:152] 2017-05-18 10:03:01,821 Memtable.java:325 - 
> Writing Memtable-compactions_in_progress@731506998(0 serialized bytes, 1 ops, 
> 0%/0% of on/off-heap limit)
> INFO  [MemtableFlushWriter:152] 2017-05-18 10:03:01,825 Memtable.java:364 - 
> Completed flushing 
> /var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-52434-Data.db
>  (42 bytes) for commitlog position ReplayPosition(segmentId=1495106933696, 
> position=9156004)
> ERROR [CompactionExecutor:162] 2017-05-18 10:03:01,829 
> CassandraDaemon.java:153 -
>  Exception in thread Thread[CompactionExecutor:162,1,main]
> java.lang.AssertionError: null
> at org.apache.cassandra.io.util.Memory.free(Memory.java:300) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.utils.obs.OffHeapBitSet.close(OffHeapBitSet.java:143) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.utils.BloomFilter.close(BloomFilter.java:116) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:345) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.io.sstable.SSTableRewriter.abort(SSTableRewriter.java:198)
>  ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:204)
>  ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
>  ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) 
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75)
>  ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
>  ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at 
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:232)
>  ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown 
> Source) ~[na:1.7.0_71]
> at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_71]
> at 

[jira] [Commented] (CASSANDRA-12683) Batch statement fails with consistency ONE when only 1 node up in DC

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143219#comment-16143219
 ] 

Jeff Jirsa commented on CASSANDRA-12683:


Hi Andy,

What's the replication strategy configured on your keyspace? 


> Batch statement fails with consistency ONE when only 1 node up in DC
> 
>
> Key: CASSANDRA-12683
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12683
> Project: Cassandra
>  Issue Type: Bug
>  Components: Coordination
> Environment: 3 Cassandra nodes (N1, N2, N3)
> 2 Data Centers (DC1, DC2)
> N1 and N2 members of DC1
> N3 a member of DC2
> 1 keyspace using SimpleReplicationStrategy with RF=3 (can also be 2):
> CREATE KEYSPACE ks WITH 
> REPLICATION={'class':'SimpleStrategy','replication_factor':3};
> 1 column family in the keyspace. For simplicity, a partition key of bigint 
> and one other field of any type. No cluster key needed:
> CREATE TABLE ks.test (
> id  bigint,
> flagboolean,
> PRIMARY KEY(id)
> );
>Reporter: Andy Klages
>Priority: Minor
>
> If Cassandra node N2 is stopped, that only leaves one node (N1) in DC1 
> running. Output from "nodetool status" is:
> Datacenter: DC1
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  AddressLoad   Tokens  OwnsHost ID 
>   Rack
> DN  N2...
> UN  N1...
> Datacenter: DC2
> ==
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  AddressLoad   Tokens  OwnsHost ID 
>   Rack
> UN  N3...
> The following batch statement will fail when executed on N1 with consistency 
> level of ONE using cqlsh (also fails with Java using Datastax driver):
> CONSISTENCY ONE
> BEGIN BATCH
> UPDATE ks.test SET flag=true where id=1;
> UPDATE ks.test SET flag=true where id=2;
> APPLY BATCH;
> The failure is:
>   File 
> "apache-cassandra-2.1.15/bin/../lib/cassandra-driver-internal-only-2.7.2.zip/cassandra-driver-2.7.2/cassandra/cluster.py",
>  line 3347, in result
> raise self._final_exception
> Unavailable: code=1000 [Unavailable exception] message="Cannot achieve 
> consistency level ONE" info={'required_replicas': 1, 'alive_replicas': 0, 
> 'consistency': 'ONE'}
> There are 2 replicas alive but for some reason, Cassandra thinks there are 
> none.
> If each statement is executed individually (i.e. no batch), each one succeeds.
> This same batch query succeeds on N3, which is the other node in the cluster 
> that is running. My analysis shows this happens when the query is executed on 
> a node in a DC where all other nodes in that DC is down. The batch statement 
> has 2 or more queries with different partition keys. If all of the partition 
> keys are the same value, it succeeds. As the replication factor is set to the 
> number of nodes in the cluster (full replication) and the consistency level 
> is ONE, the batch statement should succeed.
> 2 workarounds for this are:
> 1. Set the consistency level to ANY.
> 2. Use an unlogged batch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12696) Allow to change logging levels based on components

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143218#comment-16143218
 ] 

Jeff Jirsa commented on CASSANDRA-12696:


[~cnlwsu] is this something you may be interested in reviewing?


> Allow to change logging levels based on components
> --
>
> Key: CASSANDRA-12696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12696
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: lhf
> Attachments: 12696-trunk.patch
>
>
> Currently users are able to dynamically change logging configuration by using 
> {{nodetool setlogginglevel  }}. Unfortunately this requires to 
> know a bit about the Cassandra package hierarchy and gathering all the 
> involved packages/classes can be tedious, especially in troubleshooting 
> situations. What I'd like to have is a way to tell a user to "_when X 
> happens, enable debug logs for bootstrapping/repair/compactions/.._" by 
> simply running e.g. {{nodetool setlogginglevel bootstrap DEBUG}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12870) Calculate pending ranges for identical KS settings just once

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143216#comment-16143216
 ] 

Jeff Jirsa commented on CASSANDRA-12870:


[~jkni] - are you still open to reviewing this?


> Calculate pending ranges for identical KS settings just once
> 
>
> Key: CASSANDRA-12870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12870
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
> Fix For: 4.x
>
> Attachments: 12870-trunk.patch
>
>
> The {{TokenMetadata.calculatePendingRanges()}} operation can be very 
> expensive and already has been subject to micro optimization in 
> CASSANDRA-9258. Instead of further optimizing the calculation itself, I'd 
> like to prevent it from being executed as often by only calling it just once 
> for all keyspaces sharing identical replication settings. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Resolved] (CASSANDRA-12964) Reported load higher than available disk space

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa resolved CASSANDRA-12964.

Resolution: Duplicate

Almost certainly a dupe of CASSANDRA-13738


> Reported load higher than available disk space
> --
>
> Key: CASSANDRA-12964
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12964
> Project: Cassandra
>  Issue Type: Bug
> Environment: cassandra 3.9
>Reporter: Feras
>
> Nodetool status is reporting a node to have almost 2x the amount of space 
> that it actually physically have. We are running 6x825 GB HDDs in JBOD per 
> node. Currently the last node alcas11 is in bootstrap state. Nodetool is 
> consistently reporting the following:
> {quote}
> [alcas11-p2e] ~ # /opt/cassandra/bin/nodetool -h alcas11-p2e  status -r
> Datacenter: datacenter1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address  Load   Tokens   Owns (effective)  Host ID
>Rack
> UN  node8   3.27 TiB   256  29.6% 
> 44cdab3e-ebb2-4f9a-9c7a-0b720ab81b7e  rack1
> UN  node5  8.29 TiB   256  31.1% 
> 46d2dc3e-e2c8-487a-ab9f-cc499f45d68f  rack1
> UN  node9  3.32 TiB   256  30.2% 
> 08e0bd5e-4247-42f1-976b-c4478f52277b  rack1
> UN  node10 3.36 TiB   256  29.7% 
> 0d62d6df-e49c-44bd-a749-834ea8e612c0  rack1
> UN  node1  3.42 TiB   256  30.5% 
> e6763183-0a1c-4078-a8a9-8d28ab8b42ab  rack1
> UN  node4  3.34 TiB   256  30.4% 
> d40d4db3-36d3-4311-adac-55754d1a13c3  rack1
> UN  node3  3.28 TiB   256  29.5% 
> 5a689467-e835-472d-807f-be38c03a4231  rack1
> UN  node2  3.21 TiB   256  29.1% 
> 0b66f5e9-a05c-45ec-9723-3b8fb700f510  rack1
> UN  node7  3.23 TiB   256  29.4% 
> d43c8527-edd8-4683-ba3e-54fc10a414c7  rack1
> UN  node6  3.49 TiB   256  30.7% 
> a64cb95d-ba63-4340-9e81-63d5e189217f  rack1
> UJ  node11-p2e  1.49 TiB   256  ? 
> 6df0c42a-f3aa-4408-8876-51e0772ca2f2  rack1
> {quote}
> while the physical disk utilization on the machine itself is:
> {quote}
> [node5] ~ # df -h
> Filesystem  Size  Used Avail Use% Mounted on
> /dev/sdb1   825G  638G  145G  82% /data1
> /dev/sdc1   825G  532G  251G  68% /data2
> /dev/sdd1   825G  601G  182G  77% /data3
> /dev/sde1   825G  572G  212G  74% /data5
> /dev/sdf1   825G  552G  231G  71% /data6
> /dev/sdg1   825G  724G   59G  93% /data4
> {quote}
> Any clue what might cause this or any other information you need me to 
> provide?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13738) Load is over calculated after each IndexSummaryRedistribution

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13738:
---
Reviewer:   (was: Jeff Jirsa)

> Load is over calculated after each IndexSummaryRedistribution
> -
>
> Key: CASSANDRA-13738
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13738
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
> Attachments: sizeIssue.png
>
>
> For example, here is one of our cluster with about 500GB per node, but 
> {{nodetool status}} shows far more load than it actually is and keeps 
> increasing, restarting the process will reset the load, but keeps increasing 
> afterwards:
> {noformat}
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  AddressLoad   Tokens   Owns (effective)  Host ID  
>  Rack
> UN  IP1*   13.52 TB   256  100.0%
> c4c31e0a-3f01-49f7-8a22-33043737975d  rac1
> UN  IP2*   14.25 TB   256  100.0%
> efec4980-ec9e-4424-8a21-ce7ddaf80aa0  rac1
> UN  IP3*   13.52 TB   256  100.0%
> 7dbcfdfc-9c07-4b1a-a4b9-970b715ebed8  rac1
> UN  IP4*   22.13 TB   256  100.0%
> 8879e6c4-93e3-4cc5-b957-f999c6b9b563  rac1
> UN  IP5*   18.02 TB   256  100.0%
> 4a1eaf22-4a83-4736-9e1c-12f898d685fa  rac1
> UN  IP6*   11.68 TB   256  100.0%
> d633c591-28af-42cc-bc5e-47d1c8bcf50f  rac1
> {noformat}
> !sizeIssue.png|test!
> The root cause is if the SSTable index summary is redistributed (typically 
> executes hourly), the updated SSTable size is added again.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13738) Load is over calculated after each IndexSummaryRedistribution

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143212#comment-16143212
 ] 

Jeff Jirsa commented on CASSANDRA-13738:


[~iamaleksey] - are you willing to take review on this?



> Load is over calculated after each IndexSummaryRedistribution
> -
>
> Key: CASSANDRA-13738
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13738
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
> Attachments: sizeIssue.png
>
>
> For example, here is one of our cluster with about 500GB per node, but 
> {{nodetool status}} shows far more load than it actually is and keeps 
> increasing, restarting the process will reset the load, but keeps increasing 
> afterwards:
> {noformat}
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  AddressLoad   Tokens   Owns (effective)  Host ID  
>  Rack
> UN  IP1*   13.52 TB   256  100.0%
> c4c31e0a-3f01-49f7-8a22-33043737975d  rac1
> UN  IP2*   14.25 TB   256  100.0%
> efec4980-ec9e-4424-8a21-ce7ddaf80aa0  rac1
> UN  IP3*   13.52 TB   256  100.0%
> 7dbcfdfc-9c07-4b1a-a4b9-970b715ebed8  rac1
> UN  IP4*   22.13 TB   256  100.0%
> 8879e6c4-93e3-4cc5-b957-f999c6b9b563  rac1
> UN  IP5*   18.02 TB   256  100.0%
> 4a1eaf22-4a83-4736-9e1c-12f898d685fa  rac1
> UN  IP6*   11.68 TB   256  100.0%
> d633c591-28af-42cc-bc5e-47d1c8bcf50f  rac1
> {noformat}
> !sizeIssue.png|test!
> The root cause is if the SSTable index summary is redistributed (typically 
> executes hourly), the updated SSTable size is added again.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13082) Suspect OnDiskIndex.IteratorOrder.startAt code for ASC

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13082:
---
Description: 
startAt for ASC does
{code}
case ASC:
if (found.cmp < 0) // search term was bigger then whole 
data set
return found.index;
return inclusive && (found.cmp == 0 || found.cmp < 0) ? 
found.index : found.index - 1;
{code}
which is equivalent to
{code}
case ASC:
if (found.cmp < 0) // search term was bigger then whole 
data set
return found.index;
return inclusive ? found.index : found.index - 1;
{code}
which seems wrong. Is the parenthesis wrong here?

  was:
startAt for ASC does

case ASC:
if (found.cmp < 0) // search term was bigger then whole 
data set
return found.index;
return inclusive && (found.cmp == 0 || found.cmp < 0) ? 
found.index : found.index - 1;

which is equivalent to

case ASC:
if (found.cmp < 0) // search term was bigger then whole 
data set
return found.index;
return inclusive ? found.index : found.index - 1;

which seems wrong. Is the parenthesis wrong here?


> Suspect OnDiskIndex.IteratorOrder.startAt code for ASC
> --
>
> Key: CASSANDRA-13082
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13082
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Dave Brosius
>Priority: Trivial
> Fix For: 4.x
>
>
> startAt for ASC does
> {code}
> case ASC:
> if (found.cmp < 0) // search term was bigger then whole 
> data set
> return found.index;
> return inclusive && (found.cmp == 0 || found.cmp < 0) ? 
> found.index : found.index - 1;
> {code}
> which is equivalent to
> {code}
> case ASC:
> if (found.cmp < 0) // search term was bigger then whole 
> data set
> return found.index;
> return inclusive ? found.index : found.index - 1;
> {code}
> which seems wrong. Is the parenthesis wrong here?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13107) Remove -XX:ThreadPriorityPolicy=42 jvm flag

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143209#comment-16143209
 ] 

Jeff Jirsa commented on CASSANDRA-13107:


Compaction has changed so much since CASSANDRA-1181 that I really dont believe 
we still need to deprioritize it in 2017. [~krummas] / [~ato...@datastax.com] / 
[~tjake] , how do you three feel about removing that flag entirely?


> Remove -XX:ThreadPriorityPolicy=42 jvm flag
> ---
>
> Key: CASSANDRA-13107
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13107
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Paulo Motta
>Priority: Minor
>
> CASSANDRA-1181 added {{-XX:ThreadPriorityPolicy=42}} jvm flag to support 
> setting native thread priority without root (based on the workaround 
> described on 
> http://tech.stolsvik.com/2010/01/linux-java-thread-priorities-workaround.html).
> On Java9 this wokaround will not longer be possible due to [JEP 
> 245|https://bugs.openjdk.java.net/browse/JDK-8059557] so this flag should be 
> removed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13165) Read repair process.

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143206#comment-16143206
 ] 

Jeff Jirsa commented on CASSANDRA-13165:


I suspect the short version is something like this:

Replica A,B,C gets CQL row 1 @ TS=1
Replica A goes offline.
Replica B, C gets CQL row 2 @ TS=2
Replica A comes up, replica B goes down
Replica A, C get CQL row 3 @ TS=3
Replica B comes up, replica C goes down
Replica A, B get CQL row 4 @ TS=4

You do a SELECT, and you get timestamps 1,3, 4 from A, 1,2, 4 FROM B, 1,2,3  
from C. If A is the coordinator, it has all of the winning timestamps for its 
three rows, but it doesn't know that it's missing a row. When the digest 
mismatch happens, it still needs to do a full data read to realize it's missing 
row 2.



> Read repair process.
> 
>
> Key: CASSANDRA-13165
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13165
> Project: Cassandra
>  Issue Type: Wish
>  Components: Coordination
>Reporter: Andrey
>Priority: Minor
>
> Why can't we send timestamps together with digests? It will reduce 
> unnecessary communications between coordinator and replicas. I think the size 
> of messages is not that matter here, because a latency( due to network) is 
> way more harmful. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13175) Integrate "Error Prone" Code Analyzer

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143205#comment-16143205
 ] 

Jeff Jirsa commented on CASSANDRA-13175:


[~aweisberg] and [~krummas] have been looking at CI for the project recently, 
perhaps this is interesting enough to one of them to review.



> Integrate "Error Prone" Code Analyzer
> -
>
> Key: CASSANDRA-13175
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13175
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
> Attachments: 0001-Add-Error-Prone-code-analyzer.patch, 
> checks-2_2.out, checks-3_0.out, checks-trunk.out
>
>
> I've been playing with [Error Prone|http://errorprone.info/] by integrating 
> it into the build process and to see what kind of warnings it would produce. 
> So far I'm positively impressed by the coverage and usefulness of some of the 
> implemented checks. See attachments for results.
> Unfortunately there are still some issues on how the analyzer is effecting 
> generated code and used guava versions, see 
> [#492|https://github.com/google/error-prone/issues/492]. In case those issues 
> have been solved and the resulting code isn't affected by the analyzer, I'd 
> suggest to add it to trunk with warn only behaviour and some less useful 
> checks disabled. Alternatively a new ant target could be added, maybe with 
> build breaking checks and CI integration.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13175) Integrate "Error Prone" Code Analyzer

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13175:
---
Status: Patch Available  (was: Open)

> Integrate "Error Prone" Code Analyzer
> -
>
> Key: CASSANDRA-13175
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13175
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
> Attachments: 0001-Add-Error-Prone-code-analyzer.patch, 
> checks-2_2.out, checks-3_0.out, checks-trunk.out
>
>
> I've been playing with [Error Prone|http://errorprone.info/] by integrating 
> it into the build process and to see what kind of warnings it would produce. 
> So far I'm positively impressed by the coverage and usefulness of some of the 
> implemented checks. See attachments for results.
> Unfortunately there are still some issues on how the analyzer is effecting 
> generated code and used guava versions, see 
> [#492|https://github.com/google/error-prone/issues/492]. In case those issues 
> have been solved and the resulting code isn't affected by the analyzer, I'd 
> suggest to add it to trunk with warn only behaviour and some less useful 
> checks disabled. Alternatively a new ant target could be added, maybe with 
> build breaking checks and CI integration.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13184) Document un-revokable SELECT on certain system key-spaces

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13184:
---
Resolution: Fixed
  Reviewer: Jeff Jirsa
Status: Resolved  (was: Ready to Commit)

Thanks [~rspitzer], committed as {{68c2d2e6afc1c0ace103cd1e02e7df7b430f9f4d}}

> Document un-revokable SELECT on certain system key-spaces
> -
>
> Key: CASSANDRA-13184
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13184
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Russell Spitzer
>Assignee: Russell Spitzer
> Attachments: 13184-trunk.txt
>
>
> The documentation currently does not have a section describing certain tables 
> which are accessible to all authorized users. 
> https://github.com/apache/cassandra/blob/c1fa21458777b51a9b21795330ed6f298103b436/src/java/org/apache/cassandra/service/ClientState.java#L63-L85
> I believe this should be described here
> https://cassandra.apache.org/doc/latest/cql/security.html#revoke-permission



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13184) Document un-revokable SELECT on certain system key-spaces

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13184:
---
Status: Ready to Commit  (was: Patch Available)

> Document un-revokable SELECT on certain system key-spaces
> -
>
> Key: CASSANDRA-13184
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13184
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Russell Spitzer
>Assignee: Russell Spitzer
> Attachments: 13184-trunk.txt
>
>
> The documentation currently does not have a section describing certain tables 
> which are accessible to all authorized users. 
> https://github.com/apache/cassandra/blob/c1fa21458777b51a9b21795330ed6f298103b436/src/java/org/apache/cassandra/service/ClientState.java#L63-L85
> I believe this should be described here
> https://cassandra.apache.org/doc/latest/cql/security.html#revoke-permission



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

cassandra git commit: Document Un-revokable Select on System Keyspaces

[jjirsa]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 7bea74b76 -> 68c2d2e6a


Document Un-revokable Select on System Keyspaces

Patch by Russell Spitzer; Reviewed by Jeff Jirsa for CASSANDRA-13184


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/68c2d2e6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/68c2d2e6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/68c2d2e6

Branch: refs/heads/trunk
Commit: 68c2d2e6afc1c0ace103cd1e02e7df7b430f9f4d
Parents: 7bea74b
Author: Russell Spitzer 
Authored: Fri Feb 3 10:35:37 2017 -0800
Committer: Jeff Jirsa 
Committed: Sun Aug 27 12:56:51 2017 -0700

--
 doc/source/cql/security.rst | 9 +
 1 file changed, 9 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/68c2d2e6/doc/source/cql/security.rst
--
diff --git a/doc/source/cql/security.rst b/doc/source/cql/security.rst
index 9efe27f..ada6517 100644
--- a/doc/source/cql/security.rst
+++ b/doc/source/cql/security.rst
@@ -472,6 +472,15 @@ For instance::
 REVOKE EXECUTE ON FUNCTION keyspace1.user_function( int ) FROM 
report_writer;
 REVOKE DESCRIBE ON ALL ROLES FROM role_admin;
 
+Because of their function in normal driver operations, certain tables cannot 
have their `SELECT` permissions
+revoked. The following tables will be available to all authorized users 
regardless of their assigned role::
+
+* `system_schema.keyspaces`
+* `system_schema.columns`
+* `system_schema.tables`
+* `system.local`
+* `system.peers`
+
 .. _list-permissions-statement:
 
 LIST PERMISSIONS


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13235) All thread blocked and writes pending.

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143203#comment-16143203
 ] 

Jeff Jirsa commented on CASSANDRA-13235:


[~zhaoyan] do you still see this after CASSANDRA-12689 ? 


> All thread blocked and writes pending.
> --
>
> Key: CASSANDRA-13235
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13235
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: jdk8
> cassandra 2.1.15
>Reporter: zhaoyan
>
> I found cassandra many pending  MutationStage task
> {code}
> NFO  [Service Thread] 2017-02-17 16:00:14,440 StatusLogger.java:51 - Pool 
> NameActive   Pending  Completed   Blocked  All Time 
> Blocked
> INFO  [Service Thread] 2017-02-17 16:00:14,440 StatusLogger.java:66 - 
> MutationStage   384  4553 4294213082 0
>  0
> INFO  [Service Thread] 2017-02-17 16:00:14,441 StatusLogger.java:66 - 
> RequestResponseStage  0 0 2172612382 0
>  0
> INFO  [Service Thread] 2017-02-17 16:00:14,441 StatusLogger.java:66 - 
> ReadRepairStage   0 05378852 0
>  0
> INFO  [Service Thread] 2017-02-17 16:00:14,441 StatusLogger.java:66 - 
> CounterMutationStage  0 0  0 0
>  0
> INFO  [Service Thread] 2017-02-17 16:00:14,441 StatusLogger.java:66 - 
> ReadStage 5 0  577242284 0
>  0
> INFO  [Service Thread] 2017-02-17 16:00:14,441 StatusLogger.java:66 - 
> MiscStage 0 0  0 0
>  0
> INFO  [Service Thread] 2017-02-17 16:00:14,441 StatusLogger.java:66 - 
> HintedHandoff 0 0   1480 0
>  0
> INFO  [Service Thread] 2017-02-17 16:00:14,441 StatusLogger.java:66 - 
> GossipStage   0 09342250 0
>  0
> {code}
> And I found there are many blocked thread with jstack
> {code}
> "SharedPool-Worker-28" #416 daemon prio=5 os_prio=0 tid=0x01fb8000 
> nid=0x7459 waiting for monitor entry [0x7fdd83ca]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at sun.misc.Unsafe.monitorEnter(Native Method)
> at 
> org.apache.cassandra.utils.concurrent.Locks.monitorEnterUnsafe(Locks.java:46)
> at 
> org.apache.cassandra.db.AtomicBTreeColumns.addAllWithSizeDelta(AtomicBTreeColumns.java:202)
> at org.apache.cassandra.db.Memtable.put(Memtable.java:210)
> at 
> org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1244)
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396)
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:359)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
> at 
> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
> at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at 
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> To use "grep BLOCKED |wc -l",  get Number is 384 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13285) Lower bound [INCL_START_BOUND(event_source) ]is bigger than first returned value

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13285:
---
Description: 
We do the test of upgrade 1.2.4->3.9 , everything seems ok. but when we query 
some data (using thrift) server throw this exception:

..  1:07:40 PM
{code}
java.lang.AssertionError: Lower bound [INCL_START_BOUND(event_source) ]is 
bigger than first returned value [Row: column1=action | value=] for sstable 
/opt/cassandra/upgrade/data/OpsCenter/events/mc-10-big-Data.db
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:122)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:46)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:500)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:360)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.rows.UnfilteredRowIterator.isEmpty(UnfilteredRowIterator.java:70)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.SinglePartitionReadCommand.withSSTablesIterated(SinglePartitionReadCommand.java:666)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:615)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:492)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:358)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:397) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1801)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2486)
 ~[apache-cassandra-3.9.jar:3.9]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_121]
at 
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
 ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) 
[apache-cassandra-3.9.jar:3.9]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
{code}




There was no problems when I use cassandra-cli under 1.2.4 cluster and cqlsh 
under 3.9 clusters to do the query, it occurred only if when I use thrift api 
to search it. 

  was:
We do the test of upgrade 1.2.4->3.9 , everything seems ok. but when we query 
some data (using thrift) server throw this exception:

..  1:07:40 PM
java.lang.AssertionError: Lower bound [INCL_START_BOUND(event_source) ]is 
bigger than first returned value [Row: column1=action | value=] for sstable 
/opt/cassandra/upgrade/data/OpsCenter/events/mc-10-big-Data.db
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:122)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.computeNext(UnfilteredRowIteratorWithLowerBound.java:46)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:374)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:186)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:155)
 

[jira] [Updated] (CASSANDRA-13339) java.nio.BufferOverflowException: null

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13339:
---
Description: 
I'm seeing the following exception running Cassandra 3.9 (with Netty updated to 
4.1.8.Final) running on a 2 node cluster.  It would have been processing around 
50 queries/second at the time (mixture of inserts/updates/selects/deletes) : 
there's a collection of tables (some with counters some without) and a single 
materialized view.

{code}
ERROR [MutationStage-4] 2017-03-15 22:50:33,052 StorageProxy.java:1353 - Failed 
to apply mutation locally : {}
java.nio.BufferOverflowException: null
at 
org.apache.cassandra.io.util.DataOutputBufferFixed.doFlush(DataOutputBufferFixed.java:52)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.writeUnsignedVInt(BufferedDataOutputStreamPlus.java:262)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.rows.EncodingStats$Serializer.serialize(EncodingStats.java:233)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.SerializationHeader$Serializer.serializeForMessaging(SerializationHeader.java:380)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:122)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:89)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.serialize(PartitionUpdate.java:790)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.Mutation$MutationSerializer.serialize(Mutation.java:393)
 ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:279) 
~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:493) 
~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) 
~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:215) 
~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:227) 
~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:241) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.StorageProxy$8.runMayThrow(StorageProxy.java:1347) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2539)
 [apache-cassandra-3.9.jar:3.9]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[na:1.8.0_121]
at 
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
 [apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
 [apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) 
[apache-cassandra-3.9.jar:3.9]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
{code}

and then again shortly afterwards

{code}
ERROR [MutationStage-3] 2017-03-15 23:27:36,198 StorageProxy.java:1353 - Failed 
to apply mutation locally : {}
java.nio.BufferOverflowException: null
at 
org.apache.cassandra.io.util.DataOutputBufferFixed.doFlush(DataOutputBufferFixed.java:52)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.writeUnsignedVInt(BufferedDataOutputStreamPlus.java:262)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.rows.EncodingStats$Serializer.serialize(EncodingStats.java:233)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.SerializationHeader$Serializer.serializeForMessaging(SerializationHeader.java:380)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:122)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:89)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.serialize(PartitionUpdate.java:790)
 ~[apache-cassandra-3.9.jar:3.9]
at 

[jira] [Commented] (CASSANDRA-13378) DS Cassandra3.0 Adding a datacenter to a cluster procedure not working for us

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143199#comment-16143199
 ] 

Jeff Jirsa commented on CASSANDRA-13378:


Resolving as 'not a problem', since the problem was an error in your dc name.


> DS Cassandra3.0  Adding a datacenter to a cluster procedure not working for us
> --
>
> Key: CASSANDRA-13378
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13378
> Project: Cassandra
>  Issue Type: Bug
>  Components: Configuration
> Environment: 5 node cluster , 
> • Server Model: Dell R730xd
> • Processor: 2 * Intel Xeon E5-2630 – Total 16 cores 
> • Memory: 192 GB  
> • OS Drive: 2 * 120 GB 
> • Direct attached storage – 8 TB (4 * 2TB SSD) + 1 * 800GB SSD
> • 10Gb Dual Port + 1Gb Dual Port Network Daughter Card 
> OS - OEL 2.6.32-642.15.1.el6.x86_64
> java version "1.8.0_91"
> Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
> Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
> 6 nodes in other DC are identical
>Reporter: Kevin Rivait
>Priority: Blocker
> Fix For: 3.0.9
>
> Attachments: cassandra-rackdc.properties_from_dc1_node, 
> cassandra-rackdc.properties_from_dc2_node, cassandra.yaml, 
> DS-add-a-dc-C3.0.rtf
>
>
> i have replicated the issue on my personal cluster using VMs.
> we have many keyspaces and users developing on the Dev Cluster we are trying 
> to stretch.
> With my current VMs (10 total) i can create a 2 DC cluster  dc1 and dc2.
> i rebuild all nodes - clear data dirs and restart dc1.
> i also clear data dirs on dc2 nodes,  i donot restart yet,
> now i have single dc1 5 nodes cluster.
> snitch - GossipingPropertyFileSnitch
> i create keyspaces on DC1
> after i alter keyspaces with replication dc1 3   dc2 0i can no longer 
> query tables - not enough replicas available for query at consistency ONE.
> same error with CQL using consistency local_one
> continue with procedure, startup dc2 nodes,  
> alter replication on keyspaces to  dc1 3  dc2 2
> from dc2 nodes , nodetool rebuild -- dc1  fails
> i am attaching detailed steps with errors  and cassandra.yaml



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Resolved] (CASSANDRA-13378) DS Cassandra3.0 Adding a datacenter to a cluster procedure not working for us

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa resolved CASSANDRA-13378.

Resolution: Not A Problem

> DS Cassandra3.0  Adding a datacenter to a cluster procedure not working for us
> --
>
> Key: CASSANDRA-13378
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13378
> Project: Cassandra
>  Issue Type: Bug
>  Components: Configuration
> Environment: 5 node cluster , 
> • Server Model: Dell R730xd
> • Processor: 2 * Intel Xeon E5-2630 – Total 16 cores 
> • Memory: 192 GB  
> • OS Drive: 2 * 120 GB 
> • Direct attached storage – 8 TB (4 * 2TB SSD) + 1 * 800GB SSD
> • 10Gb Dual Port + 1Gb Dual Port Network Daughter Card 
> OS - OEL 2.6.32-642.15.1.el6.x86_64
> java version "1.8.0_91"
> Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
> Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
> 6 nodes in other DC are identical
>Reporter: Kevin Rivait
>Priority: Blocker
> Fix For: 3.0.9
>
> Attachments: cassandra-rackdc.properties_from_dc1_node, 
> cassandra-rackdc.properties_from_dc2_node, cassandra.yaml, 
> DS-add-a-dc-C3.0.rtf
>
>
> i have replicated the issue on my personal cluster using VMs.
> we have many keyspaces and users developing on the Dev Cluster we are trying 
> to stretch.
> With my current VMs (10 total) i can create a 2 DC cluster  dc1 and dc2.
> i rebuild all nodes - clear data dirs and restart dc1.
> i also clear data dirs on dc2 nodes,  i donot restart yet,
> now i have single dc1 5 nodes cluster.
> snitch - GossipingPropertyFileSnitch
> i create keyspaces on DC1
> after i alter keyspaces with replication dc1 3   dc2 0i can no longer 
> query tables - not enough replicas available for query at consistency ONE.
> same error with CQL using consistency local_one
> continue with procedure, startup dc2 nodes,  
> alter replication on keyspaces to  dc1 3  dc2 2
> from dc2 nodes , nodetool rebuild -- dc1  fails
> i am attaching detailed steps with errors  and cassandra.yaml



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Resolved] (CASSANDRA-13496) table with lots of column ( tens of thousands columns), system is very slow

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa resolved CASSANDRA-13496.

Resolution: Won't Fix

As mentioned, this is unlikely to be fixed anytime soon, as is really a case 
where a new data model should be considered. Without any further 
quantification, closing as wont-fix.


> table with lots of column ( tens of thousands columns), system is very slow 
> 
>
> Key: CASSANDRA-13496
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13496
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
> Environment: linux redhat 6.5 cassandra 3.10 using 2i
> cpu : 4 core , memory: 16G  . 5 servers
> using gocql client
>Reporter: lizg
>Priority: Minor
>  Labels: usability
> Attachments: debug.log, sql.txt
>
>
> In debug.log there are lots of 
> DEBUG [COMMIT-LOG-ALLOCATOR] 2017-05-04 08:54:35,769 
> AbstractCommitLogSegmentManager.java:107 - No segments in reserve; creating a 
> fresh one
> DEBUG [MemtablePostFlush:54] 2017-05-04 08:49:19,699 
> AbstractCommitLogSegmentManager.java:329 - Segment 
> CommitLogSegment(/home/matrix/cassandra/commitlog/CommitLog-6-1493809341337.log)
>  is no longer active and will be deleted now



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13544) Exceptions encountered for concurrent range deletes with mixed cluster keys

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143193#comment-16143193
 ] 

Jeff Jirsa commented on CASSANDRA-13544:


[~slebresne] - any intuition about what this may be a dupe of? Can we close it 
as 'cannot reproduce' anyway [~urandom] ? 


> Exceptions encountered for concurrent range deletes with mixed cluster keys
> ---
>
> Key: CASSANDRA-13544
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13544
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core, Local Write-Read Paths
> Environment: Cassandra 3.7, Debian Linux
>Reporter: Eric Evans
>
> Using a schema that looks something like...
> {code:sql}
> CREATE TABLE data (
> key text,
> rev int,
> tid timeuuid,
> value blob,
> PRIMARY KEY (key, rev, tid)
> ) WITH CLUSTERING ORDER BY (rev DESC, tid DESC)
> {code}
> ...we are performing range deletes using inequality operators on both {{rev}} 
> and {{tid}} ({{WHERE key = ? AND rev < ?}} and {{WHERE key = ? AND rev = ? 
> AND  tid < ?}}).  These range deletes are interleaved with normal writes 
> probabilistically, and (apparently) when two such range deletes occur 
> concurrently, the following exceptions result.
> {noformat}
> ERROR [SharedPool-Worker-18] 2017-05-19 17:30:22,426 Message.java:611 - 
> Unexpected exception during request; channel = [id: 0x793a853b, 
> L:/10.64.0.36:9042 - R:/10.64.32.112:550
> 48]
> java.lang.AssertionError: null
> at 
> org.apache.cassandra.db.ClusteringBoundOrBoundary.(ClusteringBoundOrBoundary.java:31)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.ClusteringBoundary.(ClusteringBoundary.java:15) 
> ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.ClusteringBoundOrBoundary.inclusiveCloseExclusiveOpen(ClusteringBoundOrBoundary.java:78)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.rows.RangeTombstoneBoundaryMarker.inclusiveCloseExclusiveOpen(RangeTombstoneBoundaryMarker.java:54)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.rows.RangeTombstoneMarker$Merger.merge(RangeTombstoneMarker.java:139)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:521)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:478)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:217)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:156)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:460)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:320)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) 
> ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:113) 
> ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.transform.FilteredRows.isEmpty(FilteredRows.java:30) 
> ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.transform.Filter.closeIfEmpty(Filter.java:53) 
> ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.transform.Filter.applyToPartition(Filter.java:23) 
> ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.transform.Filter.applyToPartition(Filter.java:6) 
> ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:76)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:735)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:410)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:363)
>  ~[apache-cassandra-3.7.3.jar:3.7.3]
> at 
> 

[jira] [Commented] (CASSANDRA-13556) Corrupted SSTables

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143192#comment-16143192
 ] 

Jeff Jirsa commented on CASSANDRA-13556:


That sounds very much like corruption caused by CASSANDRA-13004 - did you 
recently add or remove any columns from that table?


> Corrupted SSTables
> --
>
> Key: CASSANDRA-13556
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13556
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
> Environment: CentOS Linux release 7.3.1611 (Core)
> openjdk version "1.8.0_121"
> OpenJDK Runtime Environment (build 1.8.0_121-b13)
> OpenJDK 64-Bit Server VM (build 25.121-b13, mixed mode)
> Python cassandra (DataStax) driver v3.6.0
>Reporter: Ihor Prokopov
>Priority: Critical
> Fix For: 3.9
>
>
> After 3 month of working, we noticed that number of compaction tasks were 
> growing (~600 pending tasks). SStables verification shows that some of them 
> were corrupted. Repairing didn't help (it was crashing with error). 
> Also some of requests (f.e. select * from fetcher where 
> domain=8289511971670945261 and uri=-5417197141545933706; ) fails with next 
> error:
> {color:red}
> Traceback (most recent call last):
>   File "/var/cassandra/apache-cassandra-3.9/bin/cqlsh.py", line 1264, in 
> perform_simple_statement
> result = future.result()
>   File 
> "/var/cassandra/apache-cassandra-3.9/bin/../lib/cassandra-driver-internal-only-3.5.0.post0-d8d0456.zip/cassandra-driver-3.5.0.post0-d8d0456/cassandra/cluster.py",
>  line 3650, in result
> raise self._final_exception
> error: unpack requires a string argument of length 4
> {color}
> Table chema:
> {quote}
> CREATE TABLE fetcher (
> domain bigint,
> uri bigint,
> date date,
> content_length int,
> elapsed float,
> encoding text,
> fetched_time bigint,
> flinks frozen,
> flinks_count int,
> html_fingerprint bigint,
> indexed boolean,
> adult boolean,
> kws_count int,
> lang_id int,
> last_updated bigint,
> redirect_url tuple,
> revisit_date date,
> revisit_interval int,
> status_code int,
> tokens_fingerprint bigint,
> uris frozen url text,
> PRIMARY KEY (domain, uri, date)
> ) WITH CLUSTERING ORDER BY (uri ASC, date DESC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = \{'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = 'fetcher history'
> AND compaction = \{'class': 
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy',
>   'sstable_size_in_mb': '256',
>   'tombstone_threshold': '.2'}
> AND compression = \{'chunk_length_in_kb': '64',
>'class': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.5
> AND speculative_retry = '99PERCENTILE';
> {quote}
> Corrupted 
> [SSTable|https://drive.google.com/file/d/0B4ZaUOv0G9oMcHpERTdlb3ozSVk/view?usp=sharing].



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13561) Purge TTL on expiration

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143191#comment-16143191
 ] 

Jeff Jirsa commented on CASSANDRA-13561:


Sorry, I actually think I confused issues previously:

CASSANDRA-13643 - converting expired ttl cells to tombstones causing 
unnecessary digest mismatches


> Purge TTL on expiration
> ---
>
> Key: CASSANDRA-13561
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13561
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Andrew Whang
>Assignee: Andrew Whang
>Priority: Minor
> Fix For: 4.0
>
>
> Tables with mostly TTL columns tend to suffer from high droppable tombstone 
> ratio, which results in higher read latency, cpu utilization, and disk usage. 
> Expired TTL data become tombstones, and the nature of purging tombstones 
> during compaction (due to checking for overlapping SSTables) make them 
> susceptible to surviving much longer than expected. A table option to purge 
> TTL on expiration would address this issue, by preventing them from becoming 
> tombstones. A boolean purge_ttl_on_expiration table setting would allow users 
> to easily turn the feature on or off. 
> Being more aggressive with gc_grace could also address the problem of long 
> lasting tombstones, but that would affect tombstones from deletes as well. 
> Even if a purged [expired] cell is revived via repair from a node that hasn't 
> yet compacted away the cell, it would be revived as an expiring cell with the 
> same localDeletionTime, so reads should properly handle them. As well, it 
> would be purged in the next compaction. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Resolved] (CASSANDRA-13613) Import cassandra-dtest project into it's own repository

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa resolved CASSANDRA-13613.

Resolution: Fixed

> Import cassandra-dtest project into it's own repository 
> 
>
> Key: CASSANDRA-13613
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13613
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Nate McCall
>
> Given the completion of CASSANDRA-13584, we can now import 
> {{cassandra-dtest}} into ASF infrastructure. This is a top level issue for 
> tracking the bits and pieces of this task. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13637) data updated to old value after flushing largest CFS

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143189#comment-16143189
 ] 

Jeff Jirsa commented on CASSANDRA-13637:


Do you know what consistency level you're using on the inserts and selects?


> data updated to old value after flushing largest CFS
> 
>
> Key: CASSANDRA-13637
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13637
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: RHEL 6.6 64 bits
> Cassandra 2.2.6 with replication factor of 2
> 2 servers in the Cassandra ring. 
> One server is down for maintenance for more than 24 hours
>Reporter: Hervé Toulan
> Fix For: 2.2.6
>
> Attachments: local.zip
>
>
> VitualMachine 1 : application server  + Cassandra 1
> VirtualMachine 2: application server  + Cassandra 2
> Cassandra1 and Cassandra2 are the only members of the Cassandra ring.
> I expect Cassandra to always be synchronized as soon as VM1 and Vm2 can 
> communicate.
> Applicaiton server (AS1) of VM1  writes a value to Cassandra (a date).
> VM1 is shot down.
> VM2 reads the value written by AS1  as expected.
> Cassandra of VM2 perform a flush operation.
> VM2 reads the value older than the exp
> ected one, the last value written by AS1.
> I've lost the last inserted data, and I don't understand why.
> In attached log bcs.log the read in DB:
> 2.4.0.1-SNAPSHOT - LCL - 26 Jun 2017 07:15:47,959 INFO  [RMI TCP 
> Connection(111)-172.25.153.163] - Cassandra supervisor : check OK : last 
> generation of CDR is : {color:red}Sun Jun 25 00:10:53 CEST 2017{color}
> 2.4.0.1-SNAPSHOT - LCL - 26 Jun 2017 07:16:47,963 INFO  [RMI TCP 
> Connection(113)-172.25.153.163] - Cassandra supervisor : check OK : last 
> generation of CDR is : {color:red}Thu Jun 22 07:16:10 CEST 2017{color}
> Nothing has been inserted in database between the 2 logs but in debug.log 
> from Cassandra I can see a flush operation the 2017-06-24 at  07:16:37,056.
> What did we miss ?
> How data  can be overwritten  ?
> Regards,
> Hervé



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13799) Fix some alerts raised by lgtm.com

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143188#comment-16143188
 ] 

Jeff Jirsa commented on CASSANDRA-13799:


If they're nontrivial, I encourage you to do a JIRA per issue.


> Fix some alerts raised by lgtm.com
> --
>
> Key: CASSANDRA-13799
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13799
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Malcolm Taylor
>Assignee: Malcolm Taylor
>
> lgtm.com has identified a number of issues  where there may be scope for 
> improving the code 
> ([https://lgtm.com/projects/g/apache/cassandra/alerts/?mode=tree=error]).
>  This issue is to address some of the more straightforward cases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13600) sstabledump possible problem

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143165#comment-16143165
 ] 

Jeff Jirsa commented on CASSANDRA-13600:


We have some docs on contributing 
[here|http://cassandra.apache.org/doc/latest/development/patches.html] - you 
can upload the patches here, or do GH PRs, whatever's easiest. Please do 
identify all of the relevant branches and do a patch for each if a single patch 
doesn't apply cleanly to all of them.


> sstabledump possible problem
> 
>
> Key: CASSANDRA-13600
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13600
> Project: Cassandra
>  Issue Type: Bug
> Environment: Official cassandra docker image (last) under Win10
>Reporter: a8775
>Assignee: Varun Barala
>  Labels: patch
> Fix For: 3.10
>
> Attachments: CASSANDRA-13600.patch
>
>
> h2. Possible bug in sstabledump
> {noformat}
> cqlsh> show version
> [cqlsh 5.0.1 | Cassandra 3.10 | CQL spec 3.4.4 | Native protocol v4]
> {noformat}
> h2. Execute script in cqlsh in new keyspace
> {noformat}
> CREATE TABLE IF NOT EXISTS test_data (   
> // partitioning key
> PK TEXT, 
> // data
> Data TEXT,
> 
> PRIMARY KEY (PK)
> );
> insert into test_data(PK,Data) values('0','');
> insert into test_data(PK,Data) values('1','');
> insert into test_data(PK,Data) values('2','');
> delete from test_data where PK='1';
> insert into test_data(PK,Data) values('1','');
> {noformat}
> h2. Execute the following commands
> {noformat}
> nodetool flush
> nodetool compact
> sstabledump mc-2-big-Data.db
> sstabledump -d mc-2-big-Data.db
> {noformat}
> h3. default dump - missing data for partiotion key = "1"
> {noformat}
> [
>   {
> "partition" : {
>   "key" : [ "0" ],
>   "position" : 0
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 15,
> "liveness_info" : { "tstamp" : "2017-06-14T12:23:13.529389Z" },
> "cells" : [
>   { "name" : "data", "value" : "" }
> ]
>   }
> ]
>   },
>   {
> "partition" : {
>   "key" : [ "2" ],
>   "position" : 26
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 41,
> "liveness_info" : { "tstamp" : "2017-06-14T12:23:13.544132Z" },
> "cells" : [
>   { "name" : "data", "value" : "" }
> ]
>   }
> ]
>   },
>   {
> "partition" : {
>   "key" : [ "1" ],
>   "position" : 53,
>   "deletion_info" : { "marked_deleted" : "2017-06-14T12:23:13.545988Z", 
> "local_delete_time" : "2017-06-14T12:23:13Z" }
> }
>   }
> ]
> {noformat}
> h3. dump with -d option - correct data for partiotion key = "1"
> {noformat}
> [0]@0 Row[info=[ts=1497442993529389] ]:  | [data= ts=1497442993529389]
> [2]@26 Row[info=[ts=1497442993544132] ]:  | [data= ts=1497442993544132]
> [1]@53 deletedAt=1497442993545988, localDeletion=1497442993
> [1]@53 Row[info=[ts=1497442993550159] ]:  | [data= ts=1497442993550159]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13600) sstabledump possible problem

[Varun Barala (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143156#comment-16143156
 ] 

Varun Barala commented on CASSANDRA-13600:
--

[~jjirsa] okay, I'll write unit test case for this. Shall I raise GitHub pull 
request or patch, which one will be more convenient? Thanks!

> sstabledump possible problem
> 
>
> Key: CASSANDRA-13600
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13600
> Project: Cassandra
>  Issue Type: Bug
> Environment: Official cassandra docker image (last) under Win10
>Reporter: a8775
>Assignee: Varun Barala
>  Labels: patch
> Fix For: 3.10
>
> Attachments: CASSANDRA-13600.patch
>
>
> h2. Possible bug in sstabledump
> {noformat}
> cqlsh> show version
> [cqlsh 5.0.1 | Cassandra 3.10 | CQL spec 3.4.4 | Native protocol v4]
> {noformat}
> h2. Execute script in cqlsh in new keyspace
> {noformat}
> CREATE TABLE IF NOT EXISTS test_data (   
> // partitioning key
> PK TEXT, 
> // data
> Data TEXT,
> 
> PRIMARY KEY (PK)
> );
> insert into test_data(PK,Data) values('0','');
> insert into test_data(PK,Data) values('1','');
> insert into test_data(PK,Data) values('2','');
> delete from test_data where PK='1';
> insert into test_data(PK,Data) values('1','');
> {noformat}
> h2. Execute the following commands
> {noformat}
> nodetool flush
> nodetool compact
> sstabledump mc-2-big-Data.db
> sstabledump -d mc-2-big-Data.db
> {noformat}
> h3. default dump - missing data for partiotion key = "1"
> {noformat}
> [
>   {
> "partition" : {
>   "key" : [ "0" ],
>   "position" : 0
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 15,
> "liveness_info" : { "tstamp" : "2017-06-14T12:23:13.529389Z" },
> "cells" : [
>   { "name" : "data", "value" : "" }
> ]
>   }
> ]
>   },
>   {
> "partition" : {
>   "key" : [ "2" ],
>   "position" : 26
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 41,
> "liveness_info" : { "tstamp" : "2017-06-14T12:23:13.544132Z" },
> "cells" : [
>   { "name" : "data", "value" : "" }
> ]
>   }
> ]
>   },
>   {
> "partition" : {
>   "key" : [ "1" ],
>   "position" : 53,
>   "deletion_info" : { "marked_deleted" : "2017-06-14T12:23:13.545988Z", 
> "local_delete_time" : "2017-06-14T12:23:13Z" }
> }
>   }
> ]
> {noformat}
> h3. dump with -d option - correct data for partiotion key = "1"
> {noformat}
> [0]@0 Row[info=[ts=1497442993529389] ]:  | [data= ts=1497442993529389]
> [2]@26 Row[info=[ts=1497442993544132] ]:  | [data= ts=1497442993544132]
> [1]@53 deletedAt=1497442993545988, localDeletion=1497442993
> [1]@53 Row[info=[ts=1497442993550159] ]:  | [data= ts=1497442993550159]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13600) sstabledump possible problem

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143154#comment-16143154
 ] 

Jeff Jirsa commented on CASSANDRA-13600:


[~varuna] - thanks for the patch. Would you be able to write a unit test that 
fails before your fix, and succeeds with your fix, to help prevent this from 
regressing in the future?


> sstabledump possible problem
> 
>
> Key: CASSANDRA-13600
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13600
> Project: Cassandra
>  Issue Type: Bug
> Environment: Official cassandra docker image (last) under Win10
>Reporter: a8775
>Assignee: Varun Barala
>  Labels: patch
> Fix For: 3.10
>
> Attachments: CASSANDRA-13600.patch
>
>
> h2. Possible bug in sstabledump
> {noformat}
> cqlsh> show version
> [cqlsh 5.0.1 | Cassandra 3.10 | CQL spec 3.4.4 | Native protocol v4]
> {noformat}
> h2. Execute script in cqlsh in new keyspace
> {noformat}
> CREATE TABLE IF NOT EXISTS test_data (   
> // partitioning key
> PK TEXT, 
> // data
> Data TEXT,
> 
> PRIMARY KEY (PK)
> );
> insert into test_data(PK,Data) values('0','');
> insert into test_data(PK,Data) values('1','');
> insert into test_data(PK,Data) values('2','');
> delete from test_data where PK='1';
> insert into test_data(PK,Data) values('1','');
> {noformat}
> h2. Execute the following commands
> {noformat}
> nodetool flush
> nodetool compact
> sstabledump mc-2-big-Data.db
> sstabledump -d mc-2-big-Data.db
> {noformat}
> h3. default dump - missing data for partiotion key = "1"
> {noformat}
> [
>   {
> "partition" : {
>   "key" : [ "0" ],
>   "position" : 0
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 15,
> "liveness_info" : { "tstamp" : "2017-06-14T12:23:13.529389Z" },
> "cells" : [
>   { "name" : "data", "value" : "" }
> ]
>   }
> ]
>   },
>   {
> "partition" : {
>   "key" : [ "2" ],
>   "position" : 26
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 41,
> "liveness_info" : { "tstamp" : "2017-06-14T12:23:13.544132Z" },
> "cells" : [
>   { "name" : "data", "value" : "" }
> ]
>   }
> ]
>   },
>   {
> "partition" : {
>   "key" : [ "1" ],
>   "position" : 53,
>   "deletion_info" : { "marked_deleted" : "2017-06-14T12:23:13.545988Z", 
> "local_delete_time" : "2017-06-14T12:23:13Z" }
> }
>   }
> ]
> {noformat}
> h3. dump with -d option - correct data for partiotion key = "1"
> {noformat}
> [0]@0 Row[info=[ts=1497442993529389] ]:  | [data= ts=1497442993529389]
> [2]@26 Row[info=[ts=1497442993544132] ]:  | [data= ts=1497442993544132]
> [1]@53 deletedAt=1497442993545988, localDeletion=1497442993
> [1]@53 Row[info=[ts=1497442993550159] ]:  | [data= ts=1497442993550159]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-13600) sstabledump possible problem

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa reassigned CASSANDRA-13600:
--

Assignee: Varun Barala

> sstabledump possible problem
> 
>
> Key: CASSANDRA-13600
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13600
> Project: Cassandra
>  Issue Type: Bug
> Environment: Official cassandra docker image (last) under Win10
>Reporter: a8775
>Assignee: Varun Barala
>  Labels: patch
> Fix For: 3.10
>
> Attachments: CASSANDRA-13600.patch
>
>
> h2. Possible bug in sstabledump
> {noformat}
> cqlsh> show version
> [cqlsh 5.0.1 | Cassandra 3.10 | CQL spec 3.4.4 | Native protocol v4]
> {noformat}
> h2. Execute script in cqlsh in new keyspace
> {noformat}
> CREATE TABLE IF NOT EXISTS test_data (   
> // partitioning key
> PK TEXT, 
> // data
> Data TEXT,
> 
> PRIMARY KEY (PK)
> );
> insert into test_data(PK,Data) values('0','');
> insert into test_data(PK,Data) values('1','');
> insert into test_data(PK,Data) values('2','');
> delete from test_data where PK='1';
> insert into test_data(PK,Data) values('1','');
> {noformat}
> h2. Execute the following commands
> {noformat}
> nodetool flush
> nodetool compact
> sstabledump mc-2-big-Data.db
> sstabledump -d mc-2-big-Data.db
> {noformat}
> h3. default dump - missing data for partiotion key = "1"
> {noformat}
> [
>   {
> "partition" : {
>   "key" : [ "0" ],
>   "position" : 0
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 15,
> "liveness_info" : { "tstamp" : "2017-06-14T12:23:13.529389Z" },
> "cells" : [
>   { "name" : "data", "value" : "" }
> ]
>   }
> ]
>   },
>   {
> "partition" : {
>   "key" : [ "2" ],
>   "position" : 26
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 41,
> "liveness_info" : { "tstamp" : "2017-06-14T12:23:13.544132Z" },
> "cells" : [
>   { "name" : "data", "value" : "" }
> ]
>   }
> ]
>   },
>   {
> "partition" : {
>   "key" : [ "1" ],
>   "position" : 53,
>   "deletion_info" : { "marked_deleted" : "2017-06-14T12:23:13.545988Z", 
> "local_delete_time" : "2017-06-14T12:23:13Z" }
> }
>   }
> ]
> {noformat}
> h3. dump with -d option - correct data for partiotion key = "1"
> {noformat}
> [0]@0 Row[info=[ts=1497442993529389] ]:  | [data= ts=1497442993529389]
> [2]@26 Row[info=[ts=1497442993544132] ]:  | [data= ts=1497442993544132]
> [1]@53 deletedAt=1497442993545988, localDeletion=1497442993
> [1]@53 Row[info=[ts=1497442993550159] ]:  | [data= ts=1497442993550159]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13810) Overload because of hint pressure + MVs

[Tom van der Woerdt (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143117#comment-16143117
 ] 

Tom van der Woerdt commented on CASSANDRA-13810:


btw, I've also seen this happen when our weekly repair runs and large 
partitions get repaired (and go through the MV path). It triggers the same 1M 
mutations/s behavior that either fixes itself after a few minutes or needs 
manual attention to clean up. I'm assuming it's the same bug, so I won't create 
another ticket for it.

> Overload because of hint pressure + MVs
> ---
>
> Key: CASSANDRA-13810
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13810
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Tom van der Woerdt
>
> Cluster setup: 3 DCs, 20 Cassandra nodes each, all 3.0.14, with approx. 200GB 
> data per machine. Many tables have MVs associated.
> During some maintenance we did a rolling restart of all nodes in the cluster. 
> This caused a buildup of hints/batches, as expected. Most nodes came back 
> just fine, except for two nodes.
> These two nodes came back with a loadavg of >100, and 'nodetool tpstats' 
> showed a million (not exaggerating) MutationStage tasks per second(!). It was 
> clear that these were mostly (all?) mutations coming from hints, as indicated 
> by thousands of log entries per second in debug.log :
> {noformat}
> DEBUG [SharedPool-Worker-107] 2017-08-27 13:16:51,098 HintVerbHandler.java:95 
> - Failed to apply hint
> java.util.concurrent.CompletionException: 
> org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out - 
> received only 0 responses.
> at 
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
>  ~[na:1.8.0_144]
> at 
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
>  ~[na:1.8.0_144]
> at 
> java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:647) 
> ~[na:1.8.0_144]
> at 
> java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
>  ~[na:1.8.0_144]
> at 
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
>  ~[na:1.8.0_144]
> at 
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
>  ~[na:1.8.0_144]
> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:481) 
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at 
> org.apache.cassandra.db.Keyspace.lambda$applyInternal$0(Keyspace.java:495) 
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_144]
> at 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>  ~[apache-cassandra-3.0.14.jar:3.0.14]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) 
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144]
> Caused by: org.apache.cassandra.exceptions.WriteTimeoutException: Operation 
> timed out - received only 0 responses.
> ... 6 common frames omitted
> {noformat}
> After reading the relevant code, it seems that a hint is considered 
> droppable, and in the mutation path when the table contains a MV and the lock 
> fails to acquire and the mutation is droppable, it throws a WTE without 
> waiting until the timeout expires. This explains why Cassandra is able to 
> process a million mutations per second without actually considering them 
> 'dropped' in the 'nodetool tpstats' output.
> I managed to recover the two nodes by stopping handoffs on all nodes in the 
> cluster and reenabling them one at a time. It's likely that the hint/batchlog 
> settings were sub-optimal on this cluster, but I think that the retry 
> behavior(?) of hints should be improved as it's hard to express hint 
> throughput in kb/s when the mutations can involve MVs.
> More data available upon request -- I'm not sure which bits are relevant and 
> which aren't.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-13810) Overload because of hint pressure + MVs

[Tom van der Woerdt (JIRA)]

Tom van der Woerdt created CASSANDRA-13810:
--

 Summary: Overload because of hint pressure + MVs
 Key: CASSANDRA-13810
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13810
 Project: Cassandra
  Issue Type: Bug
Reporter: Tom van der Woerdt


Cluster setup: 3 DCs, 20 Cassandra nodes each, all 3.0.14, with approx. 200GB 
data per machine. Many tables have MVs associated.

During some maintenance we did a rolling restart of all nodes in the cluster. 
This caused a buildup of hints/batches, as expected. Most nodes came back just 
fine, except for two nodes.

These two nodes came back with a loadavg of >100, and 'nodetool tpstats' showed 
a million (not exaggerating) MutationStage tasks per second(!). It was clear 
that these were mostly (all?) mutations coming from hints, as indicated by 
thousands of log entries per second in debug.log :

{noformat}
DEBUG [SharedPool-Worker-107] 2017-08-27 13:16:51,098 HintVerbHandler.java:95 - 
Failed to apply hint
java.util.concurrent.CompletionException: 
org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out - 
received only 0 responses.
at 
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
 ~[na:1.8.0_144]
at 
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
 ~[na:1.8.0_144]
at 
java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:647) 
~[na:1.8.0_144]
at 
java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632)
 ~[na:1.8.0_144]
at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) 
~[na:1.8.0_144]
at 
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
 ~[na:1.8.0_144]
at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:481) 
~[apache-cassandra-3.0.14.jar:3.0.14]
at 
org.apache.cassandra.db.Keyspace.lambda$applyInternal$0(Keyspace.java:495) 
~[apache-cassandra-3.0.14.jar:3.0.14]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_144]
at 
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
 ~[apache-cassandra-3.0.14.jar:3.0.14]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) 
~[apache-cassandra-3.0.14.jar:3.0.14]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144]
Caused by: org.apache.cassandra.exceptions.WriteTimeoutException: Operation 
timed out - received only 0 responses.
... 6 common frames omitted
{noformat}

After reading the relevant code, it seems that a hint is considered droppable, 
and in the mutation path when the table contains a MV and the lock fails to 
acquire and the mutation is droppable, it throws a WTE without waiting until 
the timeout expires. This explains why Cassandra is able to process a million 
mutations per second without actually considering them 'dropped' in the 
'nodetool tpstats' output.

I managed to recover the two nodes by stopping handoffs on all nodes in the 
cluster and reenabling them one at a time. It's likely that the hint/batchlog 
settings were sub-optimal on this cluster, but I think that the retry 
behavior(?) of hints should be improved as it's hard to express hint throughput 
in kb/s when the mutations can involve MVs.

More data available upon request -- I'm not sure which bits are relevant and 
which aren't.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13600) sstabledump possible problem

[Varun Barala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Barala updated CASSANDRA-13600:
-
   Labels: patch  (was: )
Reproduced In: 3.10
   Status: Patch Available  (was: Open)

sstabledump tool does not handle the cases where a partition goes through 
{{insert, delete, and re-insert}} cycle.

This is fixed in 
https://github.com/apache/cassandra/commit/883c9f0f743139d78996f5faf191508a9be338b5
 for {{trunk}} and {{3.0.11}}

{{partition.staticRow() != null}} instead of null checking, should be checking 
for {{!partition.staticRow().isEmpty()}}


{code:java}
/**
 * The static part corresponding to this partition (this can be an empty
 * row but cannot be {@code null}).
 */
public Row staticRow();
{code}


> sstabledump possible problem
> 
>
> Key: CASSANDRA-13600
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13600
> Project: Cassandra
>  Issue Type: Bug
> Environment: Official cassandra docker image (last) under Win10
>Reporter: a8775
>  Labels: patch
> Fix For: 3.10
>
> Attachments: CASSANDRA-13600.patch
>
>
> h2. Possible bug in sstabledump
> {noformat}
> cqlsh> show version
> [cqlsh 5.0.1 | Cassandra 3.10 | CQL spec 3.4.4 | Native protocol v4]
> {noformat}
> h2. Execute script in cqlsh in new keyspace
> {noformat}
> CREATE TABLE IF NOT EXISTS test_data (   
> // partitioning key
> PK TEXT, 
> // data
> Data TEXT,
> 
> PRIMARY KEY (PK)
> );
> insert into test_data(PK,Data) values('0','');
> insert into test_data(PK,Data) values('1','');
> insert into test_data(PK,Data) values('2','');
> delete from test_data where PK='1';
> insert into test_data(PK,Data) values('1','');
> {noformat}
> h2. Execute the following commands
> {noformat}
> nodetool flush
> nodetool compact
> sstabledump mc-2-big-Data.db
> sstabledump -d mc-2-big-Data.db
> {noformat}
> h3. default dump - missing data for partiotion key = "1"
> {noformat}
> [
>   {
> "partition" : {
>   "key" : [ "0" ],
>   "position" : 0
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 15,
> "liveness_info" : { "tstamp" : "2017-06-14T12:23:13.529389Z" },
> "cells" : [
>   { "name" : "data", "value" : "" }
> ]
>   }
> ]
>   },
>   {
> "partition" : {
>   "key" : [ "2" ],
>   "position" : 26
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 41,
> "liveness_info" : { "tstamp" : "2017-06-14T12:23:13.544132Z" },
> "cells" : [
>   { "name" : "data", "value" : "" }
> ]
>   }
> ]
>   },
>   {
> "partition" : {
>   "key" : [ "1" ],
>   "position" : 53,
>   "deletion_info" : { "marked_deleted" : "2017-06-14T12:23:13.545988Z", 
> "local_delete_time" : "2017-06-14T12:23:13Z" }
> }
>   }
> ]
> {noformat}
> h3. dump with -d option - correct data for partiotion key = "1"
> {noformat}
> [0]@0 Row[info=[ts=1497442993529389] ]:  | [data= ts=1497442993529389]
> [2]@26 Row[info=[ts=1497442993544132] ]:  | [data= ts=1497442993544132]
> [1]@53 deletedAt=1497442993545988, localDeletion=1497442993
> [1]@53 Row[info=[ts=1497442993550159] ]:  | [data= ts=1497442993550159]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13600) sstabledump possible problem

[Varun Barala (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Barala updated CASSANDRA-13600:
-
Attachment: CASSANDRA-13600.patch

> sstabledump possible problem
> 
>
> Key: CASSANDRA-13600
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13600
> Project: Cassandra
>  Issue Type: Bug
> Environment: Official cassandra docker image (last) under Win10
>Reporter: a8775
>  Labels: patch
> Fix For: 3.10
>
> Attachments: CASSANDRA-13600.patch
>
>
> h2. Possible bug in sstabledump
> {noformat}
> cqlsh> show version
> [cqlsh 5.0.1 | Cassandra 3.10 | CQL spec 3.4.4 | Native protocol v4]
> {noformat}
> h2. Execute script in cqlsh in new keyspace
> {noformat}
> CREATE TABLE IF NOT EXISTS test_data (   
> // partitioning key
> PK TEXT, 
> // data
> Data TEXT,
> 
> PRIMARY KEY (PK)
> );
> insert into test_data(PK,Data) values('0','');
> insert into test_data(PK,Data) values('1','');
> insert into test_data(PK,Data) values('2','');
> delete from test_data where PK='1';
> insert into test_data(PK,Data) values('1','');
> {noformat}
> h2. Execute the following commands
> {noformat}
> nodetool flush
> nodetool compact
> sstabledump mc-2-big-Data.db
> sstabledump -d mc-2-big-Data.db
> {noformat}
> h3. default dump - missing data for partiotion key = "1"
> {noformat}
> [
>   {
> "partition" : {
>   "key" : [ "0" ],
>   "position" : 0
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 15,
> "liveness_info" : { "tstamp" : "2017-06-14T12:23:13.529389Z" },
> "cells" : [
>   { "name" : "data", "value" : "" }
> ]
>   }
> ]
>   },
>   {
> "partition" : {
>   "key" : [ "2" ],
>   "position" : 26
> },
> "rows" : [
>   {
> "type" : "row",
> "position" : 41,
> "liveness_info" : { "tstamp" : "2017-06-14T12:23:13.544132Z" },
> "cells" : [
>   { "name" : "data", "value" : "" }
> ]
>   }
> ]
>   },
>   {
> "partition" : {
>   "key" : [ "1" ],
>   "position" : 53,
>   "deletion_info" : { "marked_deleted" : "2017-06-14T12:23:13.545988Z", 
> "local_delete_time" : "2017-06-14T12:23:13Z" }
> }
>   }
> ]
> {noformat}
> h3. dump with -d option - correct data for partiotion key = "1"
> {noformat}
> [0]@0 Row[info=[ts=1497442993529389] ]:  | [data= ts=1497442993529389]
> [2]@26 Row[info=[ts=1497442993544132] ]:  | [data= ts=1497442993544132]
> [1]@53 deletedAt=1497442993545988, localDeletion=1497442993
> [1]@53 Row[info=[ts=1497442993550159] ]:  | [data= ts=1497442993550159]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org