date:20170905


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154822#comment-16154822
 ] 

Jeff Jirsa commented on CASSANDRA-13760:


Just realized that those are both fixed by 
{{e1aa7d32c95ff3f06de97803a186ff432237ecab}}, which isn't in your branch, so 
I've rebased and force-pushed. The new dtest will be @ 
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/283/
 



> presize collections
> ---
>
> Key: CASSANDRA-13760
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13760
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Dave Brosius
>Assignee: Dave Brosius
>Priority: Trivial
> Fix For: 4.x
>
> Attachments: 0001-Presize-collections.patch, 13760.txt, 
> test-output.png
>
>
> presize collections where sane, to avoid reallocs, or excess garbage



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13760) presize collections


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154822#comment-16154822
 ] 

Jeff Jirsa edited comment on CASSANDRA-13760 at 9/6/17 5:18 AM:


Just realized that those are both fixed by 
{{e1aa7d32c95ff3f06de97803a186ff432237ecab}}, which isn't in (y)our branch, so 
I've rebased and 
[force-pushed|https://github.com/jeffjirsa/cassandra/commits/cassandra-13760]. 
The new dtest will be @ 
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/283/
 




was (Author: jjirsa):
Just realized that those are both fixed by 
{{e1aa7d32c95ff3f06de97803a186ff432237ecab}}, which isn't in your branch, so 
I've rebased and force-pushed. The new dtest will be @ 
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/283/
 



> presize collections
> ---
>
> Key: CASSANDRA-13760
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13760
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Dave Brosius
>Assignee: Dave Brosius
>Priority: Trivial
> Fix For: 4.x
>
> Attachments: 0001-Presize-collections.patch, 13760.txt, 
> test-output.png
>
>
> presize collections where sane, to avoid reallocs, or excess garbage



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13760) presize collections


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154818#comment-16154818
 ] 

Jeff Jirsa commented on CASSANDRA-13760:


Attached screenshot of dtest results - 2 failures aren't typical failures:

!test-output.png|thumbnail!

Checked console output, 
{{repair_tests.incremental_repair_test.TestIncRepair.sstable_marking_test}} 
looks like a netty error: 

{code}
Unexpected error in node2 log, error: 
WARN  [epollEventLoopGroup-2-8] 2017-08-31 09:12:42,153 Slf4JLogger.java:151 - 
An exceptionCaught() event was fired, and it reached at the tail of the 
pipeline. It usually means the last handler in the pipeline did not handle the 
exception.
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown 
Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
{code}


Meanwhile {{native_transport_ssl_test.NativeTransportSSL.connect_to_ssl_test}} 
throws an actual ssl error:

{code}
WARN  [epollEventLoopGroup-2-7] 2017-08-31 07:46:04,104 Slf4JLogger.java:151 - 
An exceptionCaught() event was fired, and it reached at the tail of the 
pipeline. It usually means the last handler in the pipeline did not handle the 
exception.
io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 
04100100160001000b43514c5f56455253494f4e0005332e302e30
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1057) 
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at 
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
 ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at 
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
 ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
 ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
 ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
 ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at 
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
 ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
 ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
 ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at 
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
 ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at 
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:934)
 ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at 
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:397) 
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:302) 
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
 [netty-all-4.0.44.Final.jar:4.0.44.Final]
at 
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
 [netty-all-4.0.44.Final.jar:4.0.44.Final]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
{code}


> presize collections
> ---
>
> Key: CASSANDRA-13760
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13760
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Dave Brosius
>Assignee: Dave Brosius
>Priority: Trivial
> Fix For: 4.x
>
> Attachments: 0001-Presize-collections.patch, 13760.txt, 
> test-output.png
>
>
> presize collections where sane, to avoid reallocs, or excess garbage



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13760) presize collections


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13760:
---
Attachment: test-output.png

> presize collections
> ---
>
> Key: CASSANDRA-13760
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13760
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Dave Brosius
>Assignee: Dave Brosius
>Priority: Trivial
> Fix For: 4.x
>
> Attachments: 0001-Presize-collections.patch, 13760.txt, 
> test-output.png
>
>
> presize collections where sane, to avoid reallocs, or excess garbage



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-09-05 Thread Kurt Greaves (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154716#comment-16154716
 ] 

Kurt Greaves commented on CASSANDRA-11500:
--

Looks good. Thanks for all the hard work [~jasonstack] and [~pauloricardomg]

> Obsolete MV entry may not be properly deleted
> -
>
> Key: CASSANDRA-11500
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11500
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: Sylvain Lebresne
>Assignee: ZhaoYang
> Fix For: 3.0.15, 3.11.1, 4.0
>
>
> When a Materialized View uses a non-PK base table column in its PK, if an 
> update changes that column value, we add the new view entry and remove the 
> old one. When doing that removal, the current code uses the same timestamp 
> than for the liveness info of the new entry, which is the max timestamp for 
> any columns participating to the view PK. This is not correct for the 
> deletion as the old view entry could have other columns with higher timestamp 
> which won't be deleted as can easily shown by the failing of the following 
> test:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS 
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 4 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1;
> SELECT * FROM mv WHERE k = 1; // This currently return 2 entries, the old 
> (invalid) and the new one
> {noformat}
> So the correct timestamp to use for the deletion is the biggest timestamp in 
> the old view entry (which we know since we read the pre-existing base row), 
> and that is what CASSANDRA-11475 does (the test above thus doesn't fail on 
> that branch).
> Unfortunately, even then we can still have problems if further updates 
> requires us to overide the old entry. Consider the following case:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS 
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; // This will delete the 
> entry for a=1 with timestamp 10
> UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; // This needs to re-insert 
> an entry for a=1 but shouldn't be deleted by the prior deletion
> UPDATE t USING TIMESTAMP 4 SET a = 2 WHERE k = 1; // ... and we can play this 
> game more than once
> UPDATE t USING TIMESTAMP 5 SET a = 1 WHERE k = 1;
> ...
> {noformat}
> In a way, this is saying that the "shadowable" deletion mechanism is not 
> general enough: we need to be able to re-insert an entry when a prior one had 
> been deleted before, but we can't rely on timestamps being strictly bigger on 
> the re-insert. In that sense, this can be though as a similar problem than 
> CASSANDRA-10965, though the solution there of a single flag is not enough 
> since we can have to replace more than once.
> I think the proper solution would be to ship enough information to always be 
> able to decide when a view deletion is shadowed. Which means that both 
> liveness info (for updates) and shadowable deletion would need to ship the 
> timestamp of any base table column that is part the view PK (so {{a}} in the 
> example below).  It's doable (and not that hard really), but it does require 
> a change to the sstable and intra-node protocol, which makes this a bit 
> painful right now.
> But I'll also note that as CASSANDRA-1096 shows, the timestamp is not even 
> enough since on equal timestamp the value can be the deciding factor. So in 
> theory we'd have to ship the value of those columns (in the case of a 
> deletion at least since we have it in the view PK for updates). That said, on 
> that last problem, my preference would be that we start prioritizing 
> CASSANDRA-6123 seriously so we don't have to care about conflicting timestamp 
> anymore, which would make this problem go away.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13692) CompactionAwareWriter_getWriteDirectory throws incompatible exceptions


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154343#comment-16154343
 ] 

Dimitar Dimitrov edited comment on CASSANDRA-13692 at 9/5/17 9:50 PM:
--

I ended up rebasing the 3.0, 3.11, and trunk versions of the change, and the 
results are as follows:
* No more consistency_test.TestConsistency.test_13747 failures for the 3.0 and 
3.11 changes. Small differences between actual and expected dtest failures, 
which seem to be flaking out.
* The trunk changes are still hitting what seems to be an existing problem 
plaguing cassci.datastax.com trunk dtest jobs for the last 10 days or so (see 
https://cassci.datastax.com/job/trunk_dtest/).
** The problem is currently being investigated.

Now it looks like the 2.2, 3.0, and 3.11 changes can be accepted as passing the 
testall and dtest criteria, only the trunk changes need to be verified, after 
the problem affecting trunk gets resolved. I'll post an update once this 
happens, but in the meantime, it's possible to mark this as ready for review.


was (Author: dimitarndimitrov):
I ended up rebasing the 3.0, 3.11, and trunk versions of the change, and the 
results are as follows:
* No more consistency_test.TestConsistency.test_13747 failures for the 3.0 and 
3.11 changes. Small differences between actual and expected dtest failures, 
which seem to be flaking out.
* The trunk changes are still hitting what seems to be an existing problem 
plaguing cassci.datastax.com trunk dtest jobs for the last 10 days or so (see 
https://cassci.datastax.com/job/trunk_dtest/).
** The problem is currently being investigated.

Now it looks like the 2.2, 3.0, and 3.11 changes can be accepted as passing the 
testall and dtest criteria, only the trunk changes need to be verified, after 
the problem affecting trunk gets resolved. I'll post an update once this 
happens, but in the meantime, I'd assume it's safe to mark this as "Awaiting 
Feedback".

> CompactionAwareWriter_getWriteDirectory throws incompatible exceptions
> --
>
> Key: CASSANDRA-13692
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13692
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Hao Zhong
>Assignee: Dimitar Dimitrov
>  Labels: lhf
> Attachments: c13692-2.2-dtest-results.PNG, 
> c13692-2.2-testall-results.PNG, c13692-3.0-dtest-results.PNG, 
> c13692-3.0-testall-results.PNG, c13692-3.11-dtest-results.PNG, 
> c13692-3.11-testall-results.PNG, c13692-dtest-results.PNG, 
> c13692-testall-results.PNG
>
>
> The CompactionAwareWriter_getWriteDirectory throws RuntimeException:
> {code}
> public Directories.DataDirectory getWriteDirectory(Iterable 
> sstables, long estimatedWriteSize)
> {
> File directory = null;
> for (SSTableReader sstable : sstables)
> {
> if (directory == null)
> directory = sstable.descriptor.directory;
> if (!directory.equals(sstable.descriptor.directory))
> {
> logger.trace("All sstables not from the same disk - putting 
> results in {}", directory);
> break;
> }
> }
> Directories.DataDirectory d = 
> getDirectories().getDataDirectoryForFile(directory);
> if (d != null)
> {
> long availableSpace = d.getAvailableSpace();
> if (availableSpace < estimatedWriteSize)
> throw new RuntimeException(String.format("Not enough space to 
> write %s to %s (%s available)",
>  
> FBUtilities.prettyPrintMemory(estimatedWriteSize),
>  d.location,
>  
> FBUtilities.prettyPrintMemory(availableSpace)));
> logger.trace("putting compaction results in {}", directory);
> return d;
> }
> d = getDirectories().getWriteableLocation(estimatedWriteSize);
> if (d == null)
> throw new RuntimeException(String.format("Not enough disk space 
> to store %s",
>  
> FBUtilities.prettyPrintMemory(estimatedWriteSize)));
> return d;
> }
> {code}
> However, the thrown exception does not  trigger the failure policy. 
> CASSANDRA-11448 fixed a similar problem. The buggy code is:
> {code}
> protected Directories.DataDirectory getWriteDirectory(long writeSize)
> {
> Directories.DataDirectory directory = 
> getDirectories().getWriteableLocation(writeSize);
> if (directory == null)
> throw new RuntimeException("Insufficient disk space to write " + 
> writeSize + " bytes");
>

[jira] [Commented] (CASSANDRA-13692) CompactionAwareWriter_getWriteDirectory throws incompatible exceptions


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154343#comment-16154343
 ] 

Dimitar Dimitrov commented on CASSANDRA-13692:
--

I ended up rebasing the 3.0, 3.11, and trunk versions of the change, and the 
results are as follows:
* No more consistency_test.TestConsistency.test_13747 failures for the 3.0 and 
3.11 changes. Small differences between actual and expected dtest failures, 
which seem to be flaking out.
* The trunk changes are still hitting what seems to be an existing problem 
plaguing cassci.datastax.com trunk dtest jobs for the last 10 days or so (see 
https://cassci.datastax.com/job/trunk_dtest/).
** The problem is currently being investigated.

Now it looks like the 2.2, 3.0, and 3.11 changes can be accepted as passing the 
testall and dtest criteria, only the trunk changes need to be verified, after 
the problem affecting trunk gets resolved. I'll post an update once this 
happens, but in the meantime, I'd assume it's safe to mark this as "Awaiting 
Feedback".

> CompactionAwareWriter_getWriteDirectory throws incompatible exceptions
> --
>
> Key: CASSANDRA-13692
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13692
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Hao Zhong
>Assignee: Dimitar Dimitrov
>  Labels: lhf
> Attachments: c13692-2.2-dtest-results.PNG, 
> c13692-2.2-testall-results.PNG, c13692-3.0-dtest-results.PNG, 
> c13692-3.0-testall-results.PNG, c13692-3.11-dtest-results.PNG, 
> c13692-3.11-testall-results.PNG, c13692-dtest-results.PNG, 
> c13692-testall-results.PNG
>
>
> The CompactionAwareWriter_getWriteDirectory throws RuntimeException:
> {code}
> public Directories.DataDirectory getWriteDirectory(Iterable 
> sstables, long estimatedWriteSize)
> {
> File directory = null;
> for (SSTableReader sstable : sstables)
> {
> if (directory == null)
> directory = sstable.descriptor.directory;
> if (!directory.equals(sstable.descriptor.directory))
> {
> logger.trace("All sstables not from the same disk - putting 
> results in {}", directory);
> break;
> }
> }
> Directories.DataDirectory d = 
> getDirectories().getDataDirectoryForFile(directory);
> if (d != null)
> {
> long availableSpace = d.getAvailableSpace();
> if (availableSpace < estimatedWriteSize)
> throw new RuntimeException(String.format("Not enough space to 
> write %s to %s (%s available)",
>  
> FBUtilities.prettyPrintMemory(estimatedWriteSize),
>  d.location,
>  
> FBUtilities.prettyPrintMemory(availableSpace)));
> logger.trace("putting compaction results in {}", directory);
> return d;
> }
> d = getDirectories().getWriteableLocation(estimatedWriteSize);
> if (d == null)
> throw new RuntimeException(String.format("Not enough disk space 
> to store %s",
>  
> FBUtilities.prettyPrintMemory(estimatedWriteSize)));
> return d;
> }
> {code}
> However, the thrown exception does not  trigger the failure policy. 
> CASSANDRA-11448 fixed a similar problem. The buggy code is:
> {code}
> protected Directories.DataDirectory getWriteDirectory(long writeSize)
> {
> Directories.DataDirectory directory = 
> getDirectories().getWriteableLocation(writeSize);
> if (directory == null)
> throw new RuntimeException("Insufficient disk space to write " + 
> writeSize + " bytes");
> return directory;
> }
> {code}
> The fixed code is:
> {code}
> protected Directories.DataDirectory getWriteDirectory(long writeSize)
> {
> Directories.DataDirectory directory = 
> getDirectories().getWriteableLocation(writeSize);
> if (directory == null)
> throw new FSWriteError(new IOException("Insufficient disk space 
> to write " + writeSize + " bytes"), "");
> return directory;
> }
> {code}
> The fixed code throws FSWE and triggers the failure policy.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Issue Comment Deleted] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables

2017-09-05 Thread mck (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mck updated CASSANDRA-13418:

Comment: was deleted

(was: thanks for the patch. is going to make a big difference to a number of
users that have append-only datamodels!





-- 
Mick Semb Wever
Australia

The Last Pickle
Apache Cassandra Consulting
http://www.thelastpickle.com
)

> Allow TWCS to ignore overlaps when dropping fully expired sstables
> --
>
> Key: CASSANDRA-13418
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13418
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Corentin Chary
>Assignee: Romain GERARD
>  Labels: twcs
> Fix For: 3.11.1, 4.0
>
> Attachments: twcs-cleanup.png
>
>
> http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If 
> you really want read-repairs you're going to have sstables blocking the 
> expiration of other fully expired SSTables because they overlap.
> You can set unchecked_tombstone_compaction = true or tombstone_threshold to a 
> very low value and that will purge the blockers of old data that should 
> already have expired, thus removing the overlaps and allowing the other 
> SSTables to expire.
> The thing is that this is rather CPU intensive and not optimal. If you have 
> time series, you might not care if all your data doesn't exactly expire at 
> the right time, or if data re-appears for some time, as long as it gets 
> deleted as soon as it can. And in this situation I believe it would be really 
> beneficial to allow users to simply ignore overlapping SSTables when looking 
> for fully expired ones.
> To the question: why would you need read-repairs ?
> - Full repairs basically take longer than the TTL of the data on my dataset, 
> so this isn't really effective.
> - Even with a 10% chances of doing a repair, we found out that this would be 
> enough to greatly reduce entropy of the most used data (and if you have 
> timeseries, you're likely to have a dashboard doing the same important 
> queries over and over again).
> - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow.
> I'll try to come up with a patch demonstrating how this would work, try it on 
> our system and report the effects.
> cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables

2017-09-05 Thread Mick Semb Wever (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154249#comment-16154249
 ] 

Mick Semb Wever commented on CASSANDRA-13418:
-

thanks for the patch. is going to make a big difference to a number of
users that have append-only datamodels!





-- 
Mick Semb Wever
Australia

The Last Pickle
Apache Cassandra Consulting
http://www.thelastpickle.com


> Allow TWCS to ignore overlaps when dropping fully expired sstables
> --
>
> Key: CASSANDRA-13418
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13418
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Corentin Chary
>Assignee: Romain GERARD
>  Labels: twcs
> Fix For: 3.11.1, 4.0
>
> Attachments: twcs-cleanup.png
>
>
> http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If 
> you really want read-repairs you're going to have sstables blocking the 
> expiration of other fully expired SSTables because they overlap.
> You can set unchecked_tombstone_compaction = true or tombstone_threshold to a 
> very low value and that will purge the blockers of old data that should 
> already have expired, thus removing the overlaps and allowing the other 
> SSTables to expire.
> The thing is that this is rather CPU intensive and not optimal. If you have 
> time series, you might not care if all your data doesn't exactly expire at 
> the right time, or if data re-appears for some time, as long as it gets 
> deleted as soon as it can. And in this situation I believe it would be really 
> beneficial to allow users to simply ignore overlapping SSTables when looking 
> for fully expired ones.
> To the question: why would you need read-repairs ?
> - Full repairs basically take longer than the TTL of the data on my dataset, 
> so this isn't really effective.
> - Even with a 10% chances of doing a repair, we found out that this would be 
> enough to greatly reduce entropy of the most used data (and if you have 
> timeseries, you're likely to have a dashboard doing the same important 
> queries over and over again).
> - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow.
> I'll try to come up with a patch demonstrating how this would work, try it on 
> our system and report the effects.
> cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13844) sstableloader doesn't support non default storage_port and ssl_storage_port


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154191#comment-16154191
 ] 

Jason Brown commented on CASSANDRA-13844:
-

Sure, but we're only taking critical bugs for 2.1/2.2 right now, and I'm trying 
to judge if this meets that criteria.

> sstableloader doesn't support non default storage_port and ssl_storage_port
> ---
>
> Key: CASSANDRA-13844
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13844
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Eduard Tudenhoefner
>Assignee: Eduard Tudenhoefner
> Fix For: 2.2.11, 2.1.19
>
>
> Currently *storage_port* and *ssl_storage_port* are hardcoded to the 
> defaults. The problem was already fixed in CASSANDRA-13518 for C* 3.0+, so 
> this here is just backporting it to C* 2.1/2.2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13844) sstableloader doesn't support non default storage_port and ssl_storage_port


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154185#comment-16154185
 ] 

Eduard Tudenhoefner commented on CASSANDRA-13844:
-

[~jasobrown] honestly I haven't tried using the C* {{BulkLoader}} to connect to 
a 2.1/2.2 cluster. I just saw that it wasn't fixed in 2.1/2.2 and it seemed 
easy enough to do it

> sstableloader doesn't support non default storage_port and ssl_storage_port
> ---
>
> Key: CASSANDRA-13844
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13844
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Eduard Tudenhoefner
>Assignee: Eduard Tudenhoefner
> Fix For: 2.2.11, 2.1.19
>
>
> Currently *storage_port* and *ssl_storage_port* are hardcoded to the 
> defaults. The problem was already fixed in CASSANDRA-13518 for C* 3.0+, so 
> this here is just backporting it to C* 2.1/2.2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13844) sstableloader doesn't support non default storage_port and ssl_storage_port


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154179#comment-16154179
 ] 

Jason Brown commented on CASSANDRA-13844:
-

[~eduard.tudenhoefner] you can't use the 3.0 {{BulkLoader}} to connect to a 
2.1/2.2 cluster, correct?

> sstableloader doesn't support non default storage_port and ssl_storage_port
> ---
>
> Key: CASSANDRA-13844
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13844
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Eduard Tudenhoefner
>Assignee: Eduard Tudenhoefner
> Fix For: 2.2.11, 2.1.19
>
>
> Currently *storage_port* and *ssl_storage_port* are hardcoded to the 
> defaults. The problem was already fixed in CASSANDRA-13518 for C* 3.0+, so 
> this here is just backporting it to C* 2.1/2.2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13618) CassandraRoleManager setup task improvement


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154147#comment-16154147
 ] 

Jeff Jirsa commented on CASSANDRA-13618:


Didnt get a chance to triage dtests before they rolled over back in July, so 
kicked off anew:

3.0 - 
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/280/
3.11 - 
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/281/
trunk - 
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/282/


> CassandraRoleManager setup task improvement
> ---
>
> Key: CASSANDRA-13618
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13618
> Project: Cassandra
>  Issue Type: Bug
>  Components: Auth
>Reporter: Jeff Jirsa
>Assignee: Jeff Jirsa
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> {{CassandraRoleManager}} blocks some functionality during setup, using a 
> delay added in CASSANDRA-9761 . Unfortunately, this setup is scheduled for 
> 10s after startup, and may not be necessary, meaning immediately after 
> startup some auth related queries may not behave as intended. We can skip 
> this delay without any additional risk.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13603) Change repair midpoint logging from CASSANDRA-13052


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13603:
---
   Resolution: Fixed
Fix Version/s: (was: 3.11.x)
   (was: 4.x)
   (was: 3.0.x)
   4.0
   3.11.1
   3.0.15
   Status: Resolved  (was: Ready to Commit)

Thanks. Committed as {{bc5c2316c5acfc1ee0ef101a3ebf00d03c89e283}} to 3.0 and 
merged 3.11 and trunk.


> Change repair midpoint logging from  CASSANDRA-13052
> 
>
> Key: CASSANDRA-13603
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13603
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Jeff Jirsa
>Assignee: Jeff Jirsa
>Priority: Trivial
> Fix For: 3.0.15, 3.11.1, 4.0
>
>
> In CASSANDRA-13052 , we changed the way we handle repairs on small ranges to 
> make them more sane in general, but {{MerkleTree.differenceHelper}} now 
> erroneously logs at error.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/90b39dc6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/90b39dc6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/90b39dc6

Branch: refs/heads/trunk
Commit: 90b39dc619d75096e897fb8e90dd38b04e67f848
Parents: 3f5f702 bc5c231
Author: Jeff Jirsa 
Authored: Tue Sep 5 11:58:31 2017 -0700
Committer: Jeff Jirsa 
Committed: Tue Sep 5 11:59:25 2017 -0700

--
 CHANGES.txt |  1 +
 .../org/apache/cassandra/utils/MerkleTree.java  | 23 +
 .../apache/cassandra/utils/MerkleTreeTest.java  | 34 
 3 files changed, 51 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/90b39dc6/CHANGES.txt
--
diff --cc CHANGES.txt
index 5a53e50,4302fdf..75d9bd6
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,14 -1,5 +1,15 @@@
 -3.0.15
 +3.11.1
 + * Add a compaction option to TWCS to ignore sstables overlapping checks 
(CASSANDRA-13418)
 + * BTree.Builder memory leak (CASSANDRA-13754)
 + * Revert CASSANDRA-10368 of supporting non-pk column filtering due to 
correctness (CASSANDRA-13798)
 + * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)
 + * Better bootstrap failure message when blocked by (potential) range 
movement (CASSANDRA-13744)
 + * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
 + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
 + * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
 + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
 +Merged from 3.0:
+  * Change repair midpoint logging for tiny ranges (CASSANDRA-13603)
   * Better handle corrupt final commitlog segment (CASSANDRA-11995)
   * StreamingHistogram is not thread safe (CASSANDRA-13756)
   * Fix MV timestamp issues (CASSANDRA-11500)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/90b39dc6/src/java/org/apache/cassandra/utils/MerkleTree.java
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/dcef7c7e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/dcef7c7e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/dcef7c7e

Branch: refs/heads/trunk
Commit: dcef7c7ed9bac3484322ed5b20eaa20d373833e7
Parents: fa36044 90b39dc
Author: Jeff Jirsa 
Authored: Tue Sep 5 11:59:35 2017 -0700
Committer: Jeff Jirsa 
Committed: Tue Sep 5 12:00:25 2017 -0700

--
 CHANGES.txt |  1 +
 .../org/apache/cassandra/utils/MerkleTree.java  | 23 +
 .../apache/cassandra/utils/MerkleTreeTest.java  | 34 
 3 files changed, 51 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/dcef7c7e/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/dcef7c7e/src/java/org/apache/cassandra/utils/MerkleTree.java
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[2/6] cassandra git commit: Change repair midpoint logging for tiny ranges

Change repair midpoint logging for tiny ranges

Patch by Jeff Jirsa; Reviewed by Blake Eggleston for CASSANDRA-13603


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bc5c2316
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bc5c2316
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bc5c2316

Branch: refs/heads/cassandra-3.11
Commit: bc5c2316c5acfc1ee0ef101a3ebf00d03c89e283
Parents: 0493545
Author: Jeff Jirsa 
Authored: Tue Sep 5 11:57:58 2017 -0700
Committer: Jeff Jirsa 
Committed: Tue Sep 5 11:57:58 2017 -0700

--
 CHANGES.txt |  1 +
 .../org/apache/cassandra/utils/MerkleTree.java  | 23 +
 .../apache/cassandra/utils/MerkleTreeTest.java  | 34 
 3 files changed, 51 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/bc5c2316/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index a3eccf2..4302fdf 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Change repair midpoint logging for tiny ranges (CASSANDRA-13603)
  * Better handle corrupt final commitlog segment (CASSANDRA-11995)
  * StreamingHistogram is not thread safe (CASSANDRA-13756)
  * Fix MV timestamp issues (CASSANDRA-11500)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bc5c2316/src/java/org/apache/cassandra/utils/MerkleTree.java
--
diff --git a/src/java/org/apache/cassandra/utils/MerkleTree.java 
b/src/java/org/apache/cassandra/utils/MerkleTree.java
index 0d5a469..22b61e8 100644
--- a/src/java/org/apache/cassandra/utils/MerkleTree.java
+++ b/src/java/org/apache/cassandra/utils/MerkleTree.java
@@ -22,6 +22,7 @@ import java.io.IOException;
 import java.io.Serializable;
 import java.util.*;
 
+import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Preconditions;
 import com.google.common.collect.PeekingIterator;
 
@@ -246,12 +247,20 @@ public class MerkleTree implements Serializable
 
 if (lhash != null && rhash != null && !Arrays.equals(lhash, rhash))
 {
-logger.debug("Digest mismatch detected, traversing trees [{}, 
{}]", ltree, rtree);
-if (FULLY_INCONSISTENT == differenceHelper(ltree, rtree, diff, 
active))
+if(lnode instanceof  Leaf || rnode instanceof Leaf)
 {
-logger.debug("Range {} fully inconsistent", active);
+logger.debug("Digest mismatch detected among leaf nodes {}, 
{}", lnode, rnode);
 diff.add(active);
 }
+else
+{
+logger.debug("Digest mismatch detected, traversing trees [{}, 
{}]", ltree, rtree);
+if (FULLY_INCONSISTENT == differenceHelper(ltree, rtree, diff, 
active))
+{
+logger.debug("Range {} fully inconsistent", active);
+diff.add(active);
+}
+}
 }
 else if (lhash == null || rhash == null)
 diff.add(active);
@@ -265,6 +274,7 @@ public class MerkleTree implements Serializable
  * Takes two trees and a range for which they have hashes, but are 
inconsistent.
  * @return FULLY_INCONSISTENT if active is inconsistent, 
PARTIALLY_INCONSISTENT if only a subrange is inconsistent.
  */
+@VisibleForTesting
 static int differenceHelper(MerkleTree ltree, MerkleTree rtree, 
List diff, TreeRange active)
 {
 if (active.depth == Byte.MAX_VALUE)
@@ -274,10 +284,9 @@ public class MerkleTree implements Serializable
 // sanity check for midpoint calculation, see CASSANDRA-13052
 if (midpoint.equals(active.left) || midpoint.equals(active.right))
 {
-// Unfortunately we can't throw here to abort the validation 
process, as the code is executed in it's own
-// thread with the caller waiting for a condition to be signaled 
after completion and without an option
-// to indicate an error (2.x only).
-logger.error("Invalid midpoint {} for [{},{}], range will be 
reported inconsistent", midpoint, active.left, active.right);
+// If the midpoint equals either the left or the right, we have a 
range that's too small to split - we'll simply report the
+// whole range as inconsistent
+logger.debug("({}) No sane midpoint ({}) for range {} , marking 
whole range as inconsistent", active.depth, midpoint, active);
 return FULLY_INCONSISTENT;
 }

[1/6] cassandra git commit: Change repair midpoint logging for tiny ranges

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 0493545dd -> bc5c2316c
  refs/heads/cassandra-3.11 3f5f70281 -> 90b39dc61
  refs/heads/trunk fa36044a5 -> dcef7c7ed


Change repair midpoint logging for tiny ranges

Patch by Jeff Jirsa; Reviewed by Blake Eggleston for CASSANDRA-13603


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bc5c2316
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bc5c2316
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bc5c2316

Branch: refs/heads/cassandra-3.0
Commit: bc5c2316c5acfc1ee0ef101a3ebf00d03c89e283
Parents: 0493545
Author: Jeff Jirsa 
Authored: Tue Sep 5 11:57:58 2017 -0700
Committer: Jeff Jirsa 
Committed: Tue Sep 5 11:57:58 2017 -0700

--
 CHANGES.txt |  1 +
 .../org/apache/cassandra/utils/MerkleTree.java  | 23 +
 .../apache/cassandra/utils/MerkleTreeTest.java  | 34 
 3 files changed, 51 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/bc5c2316/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index a3eccf2..4302fdf 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Change repair midpoint logging for tiny ranges (CASSANDRA-13603)
  * Better handle corrupt final commitlog segment (CASSANDRA-11995)
  * StreamingHistogram is not thread safe (CASSANDRA-13756)
  * Fix MV timestamp issues (CASSANDRA-11500)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bc5c2316/src/java/org/apache/cassandra/utils/MerkleTree.java
--
diff --git a/src/java/org/apache/cassandra/utils/MerkleTree.java 
b/src/java/org/apache/cassandra/utils/MerkleTree.java
index 0d5a469..22b61e8 100644
--- a/src/java/org/apache/cassandra/utils/MerkleTree.java
+++ b/src/java/org/apache/cassandra/utils/MerkleTree.java
@@ -22,6 +22,7 @@ import java.io.IOException;
 import java.io.Serializable;
 import java.util.*;
 
+import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Preconditions;
 import com.google.common.collect.PeekingIterator;
 
@@ -246,12 +247,20 @@ public class MerkleTree implements Serializable
 
 if (lhash != null && rhash != null && !Arrays.equals(lhash, rhash))
 {
-logger.debug("Digest mismatch detected, traversing trees [{}, 
{}]", ltree, rtree);
-if (FULLY_INCONSISTENT == differenceHelper(ltree, rtree, diff, 
active))
+if(lnode instanceof  Leaf || rnode instanceof Leaf)
 {
-logger.debug("Range {} fully inconsistent", active);
+logger.debug("Digest mismatch detected among leaf nodes {}, 
{}", lnode, rnode);
 diff.add(active);
 }
+else
+{
+logger.debug("Digest mismatch detected, traversing trees [{}, 
{}]", ltree, rtree);
+if (FULLY_INCONSISTENT == differenceHelper(ltree, rtree, diff, 
active))
+{
+logger.debug("Range {} fully inconsistent", active);
+diff.add(active);
+}
+}
 }
 else if (lhash == null || rhash == null)
 diff.add(active);
@@ -265,6 +274,7 @@ public class MerkleTree implements Serializable
  * Takes two trees and a range for which they have hashes, but are 
inconsistent.
  * @return FULLY_INCONSISTENT if active is inconsistent, 
PARTIALLY_INCONSISTENT if only a subrange is inconsistent.
  */
+@VisibleForTesting
 static int differenceHelper(MerkleTree ltree, MerkleTree rtree, 
List diff, TreeRange active)
 {
 if (active.depth == Byte.MAX_VALUE)
@@ -274,10 +284,9 @@ public class MerkleTree implements Serializable
 // sanity check for midpoint calculation, see CASSANDRA-13052
 if (midpoint.equals(active.left) || midpoint.equals(active.right))
 {
-// Unfortunately we can't throw here to abort the validation 
process, as the code is executed in it's own
-// thread with the caller waiting for a condition to be signaled 
after completion and without an option
-// to indicate an error (2.x only).
-logger.error("Invalid midpoint {} for [{},{}], range will be 
reported inconsistent", midpoint, active.left, active.right);
+// If the midpoint equals either the left or the right, we have a 
range that's too small to split - we'll simply report the
+// whole range as inconsistent
+logger.debug("({}) No sane midpoint ({}) for range {} , marking 
whole range as inconsistent", active.depth,

[3/6] cassandra git commit: Change repair midpoint logging for tiny ranges

Change repair midpoint logging for tiny ranges

Patch by Jeff Jirsa; Reviewed by Blake Eggleston for CASSANDRA-13603


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bc5c2316
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bc5c2316
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bc5c2316

Branch: refs/heads/trunk
Commit: bc5c2316c5acfc1ee0ef101a3ebf00d03c89e283
Parents: 0493545
Author: Jeff Jirsa 
Authored: Tue Sep 5 11:57:58 2017 -0700
Committer: Jeff Jirsa 
Committed: Tue Sep 5 11:57:58 2017 -0700

--
 CHANGES.txt |  1 +
 .../org/apache/cassandra/utils/MerkleTree.java  | 23 +
 .../apache/cassandra/utils/MerkleTreeTest.java  | 34 
 3 files changed, 51 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/bc5c2316/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index a3eccf2..4302fdf 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Change repair midpoint logging for tiny ranges (CASSANDRA-13603)
  * Better handle corrupt final commitlog segment (CASSANDRA-11995)
  * StreamingHistogram is not thread safe (CASSANDRA-13756)
  * Fix MV timestamp issues (CASSANDRA-11500)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bc5c2316/src/java/org/apache/cassandra/utils/MerkleTree.java
--
diff --git a/src/java/org/apache/cassandra/utils/MerkleTree.java 
b/src/java/org/apache/cassandra/utils/MerkleTree.java
index 0d5a469..22b61e8 100644
--- a/src/java/org/apache/cassandra/utils/MerkleTree.java
+++ b/src/java/org/apache/cassandra/utils/MerkleTree.java
@@ -22,6 +22,7 @@ import java.io.IOException;
 import java.io.Serializable;
 import java.util.*;
 
+import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Preconditions;
 import com.google.common.collect.PeekingIterator;
 
@@ -246,12 +247,20 @@ public class MerkleTree implements Serializable
 
 if (lhash != null && rhash != null && !Arrays.equals(lhash, rhash))
 {
-logger.debug("Digest mismatch detected, traversing trees [{}, 
{}]", ltree, rtree);
-if (FULLY_INCONSISTENT == differenceHelper(ltree, rtree, diff, 
active))
+if(lnode instanceof  Leaf || rnode instanceof Leaf)
 {
-logger.debug("Range {} fully inconsistent", active);
+logger.debug("Digest mismatch detected among leaf nodes {}, 
{}", lnode, rnode);
 diff.add(active);
 }
+else
+{
+logger.debug("Digest mismatch detected, traversing trees [{}, 
{}]", ltree, rtree);
+if (FULLY_INCONSISTENT == differenceHelper(ltree, rtree, diff, 
active))
+{
+logger.debug("Range {} fully inconsistent", active);
+diff.add(active);
+}
+}
 }
 else if (lhash == null || rhash == null)
 diff.add(active);
@@ -265,6 +274,7 @@ public class MerkleTree implements Serializable
  * Takes two trees and a range for which they have hashes, but are 
inconsistent.
  * @return FULLY_INCONSISTENT if active is inconsistent, 
PARTIALLY_INCONSISTENT if only a subrange is inconsistent.
  */
+@VisibleForTesting
 static int differenceHelper(MerkleTree ltree, MerkleTree rtree, 
List diff, TreeRange active)
 {
 if (active.depth == Byte.MAX_VALUE)
@@ -274,10 +284,9 @@ public class MerkleTree implements Serializable
 // sanity check for midpoint calculation, see CASSANDRA-13052
 if (midpoint.equals(active.left) || midpoint.equals(active.right))
 {
-// Unfortunately we can't throw here to abort the validation 
process, as the code is executed in it's own
-// thread with the caller waiting for a condition to be signaled 
after completion and without an option
-// to indicate an error (2.x only).
-logger.error("Invalid midpoint {} for [{},{}], range will be 
reported inconsistent", midpoint, active.left, active.right);
+// If the midpoint equals either the left or the right, we have a 
range that's too small to split - we'll simply report the
+// whole range as inconsistent
+logger.debug("({}) No sane midpoint ({}) for range {} , marking 
whole range as inconsistent", active.depth, midpoint, active);
 return FULLY_INCONSISTENT;
 }
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bc5c2316/test/unit/org/apache/cassandra/utils/MerkleTreeTest.java

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/90b39dc6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/90b39dc6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/90b39dc6

Branch: refs/heads/cassandra-3.11
Commit: 90b39dc619d75096e897fb8e90dd38b04e67f848
Parents: 3f5f702 bc5c231
Author: Jeff Jirsa 
Authored: Tue Sep 5 11:58:31 2017 -0700
Committer: Jeff Jirsa 
Committed: Tue Sep 5 11:59:25 2017 -0700

--
 CHANGES.txt |  1 +
 .../org/apache/cassandra/utils/MerkleTree.java  | 23 +
 .../apache/cassandra/utils/MerkleTreeTest.java  | 34 
 3 files changed, 51 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/90b39dc6/CHANGES.txt
--
diff --cc CHANGES.txt
index 5a53e50,4302fdf..75d9bd6
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,14 -1,5 +1,15 @@@
 -3.0.15
 +3.11.1
 + * Add a compaction option to TWCS to ignore sstables overlapping checks 
(CASSANDRA-13418)
 + * BTree.Builder memory leak (CASSANDRA-13754)
 + * Revert CASSANDRA-10368 of supporting non-pk column filtering due to 
correctness (CASSANDRA-13798)
 + * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)
 + * Better bootstrap failure message when blocked by (potential) range 
movement (CASSANDRA-13744)
 + * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
 + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
 + * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
 + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
 +Merged from 3.0:
+  * Change repair midpoint logging for tiny ranges (CASSANDRA-13603)
   * Better handle corrupt final commitlog segment (CASSANDRA-11995)
   * StreamingHistogram is not thread safe (CASSANDRA-13756)
   * Fix MV timestamp issues (CASSANDRA-11500)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/90b39dc6/src/java/org/apache/cassandra/utils/MerkleTree.java
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-11995) Commitlog replaced with all NULs


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-11995:
---
   Resolution: Fixed
 Reviewer: Branimir Lambov
Fix Version/s: 4.0
   3.11.1
   3.0.15
   Status: Resolved  (was: Ready to Commit)

Thanks for the 'ready-to-commit' change. Committed as 
{{0493545dd08d29c34d757bb2f1d90052d03d24c6}} to 3.0 and merged up through 3.11 
and trunk.



> Commitlog replaced with all NULs
> 
>
> Key: CASSANDRA-11995
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11995
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: Windows 10 Enterprise 1511
> DataStax Cassandra Community Server 2.2.3
>Reporter: James Howe
>Assignee: Jeff Jirsa
> Fix For: 3.0.15, 3.11.1, 4.0
>
>
> I noticed this morning that Cassandra was failing to start, after being shut 
> down on Friday.
> {code}
> ERROR 09:13:37 Exiting due to error while processing commit log during 
> initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
> Could not read commit log descriptor in file C:\Program Files\DataStax 
> Community\data\commitlog\CommitLog-5-1465571056722.log
>   at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:622)
>  [apache-cassandra-2.2.3.jar:2.2.3]
>   at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:302)
>  [apache-cassandra-2.2.3.jar:2.2.3]
>   at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147)
>  [apache-cassandra-2.2.3.jar:2.2.3]
>   at 
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189) 
> [apache-cassandra-2.2.3.jar:2.2.3]
>   at 
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169) 
> [apache-cassandra-2.2.3.jar:2.2.3]
>   at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:273) 
> [apache-cassandra-2.2.3.jar:2.2.3]
>   at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:513)
>  [apache-cassandra-2.2.3.jar:2.2.3]
>   at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:622) 
> [apache-cassandra-2.2.3.jar:2.2.3]
> {code}
> Checking the referenced file reveals it comprises 33,554,432 (32 * 1024 * 
> 1024) NUL bytes.
> No logs (stdout, stderr, prunsrv) from the shutdown show any other issues and 
> appear exactly as normal.
> Is installed as a service via DataStax's distribution.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fa36044a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fa36044a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fa36044a

Branch: refs/heads/trunk
Commit: fa36044a585d2268d6dd61603f3fb21e6b8ec286
Parents: 733ea88 3f5f702
Author: Jeff Jirsa 
Authored: Tue Sep 5 11:48:04 2017 -0700
Committer: Jeff Jirsa 
Committed: Tue Sep 5 11:48:40 2017 -0700

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/commitlog/CommitLogReader.java  |  3 ++-
 .../apache/cassandra/db/commitlog/CommitLogTest.java| 12 +++-
 3 files changed, 14 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/fa36044a/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/fa36044a/src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java
--
diff --cc src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java
index 301c832,8c04329..864325b
--- a/src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java
+++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java
@@@ -147,6 -149,7 +147,7 @@@ public class CommitLogReade
  if (desc == null)
  {
  // don't care about whether or not the handler thinks we can 
continue. We can't w/out descriptor.
 -// whether or not we continue with startup will depend on 
whether this is the last segment
++// whether or not we can continue depends on whether this is 
the last segment
  handler.handleUnrecoverableError(new CommitLogReadException(
  String.format("Could not read commit log descriptor in 
file %s", file),
  CommitLogReadErrorReason.UNRECOVERABLE_DESCRIPTOR_ERROR,

http://git-wip-us.apache.org/repos/asf/cassandra/blob/fa36044a/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java
--
diff --cc test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java
index f98dd6b,aab55a5..956a05d
--- a/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java
+++ b/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java
@@@ -153,9 -155,22 +156,16 @@@ public class CommitLogTes
  }
  
  @Test
 -public void testRecoveryWithFinalEmptyLog() throws Exception
++public void testRecoveryWithEmptyFinalLog() throws Exception
+ {
 -// Even though it's empty, it's the last commitlog segment, so 
allowTruncation=true should allow it to pass
 -CommitLog.instance.recoverFiles(new 
File[]{tmpFile(CommitLogDescriptor.current_version)});
++
CommitLog.instance.recoverFiles(tmpFile(CommitLogDescriptor.current_version));
+ }
+ 
 -@Test
 -public void testRecoveryWithEmptyLog20() throws Exception
 -{
 -
CommitLog.instance.recoverFiles(tmpFile(CommitLogDescriptor.VERSION_20));
 -}
+ 
+ @Test
  public void testRecoveryWithZeroLog() throws Exception
  {
 -testRecovery(new byte[10], null);
 +testRecovery(new byte[10], CommitLogReplayException.class);
  }
  
  @Test


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3f5f7028
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3f5f7028
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3f5f7028

Branch: refs/heads/trunk
Commit: 3f5f70281ee87c09d312d2169d6335e5b0abf815
Parents: d44a0d2 0493545
Author: Jeff Jirsa 
Authored: Tue Sep 5 11:47:18 2017 -0700
Committer: Jeff Jirsa 
Committed: Tue Sep 5 11:47:55 2017 -0700

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/commitlog/CommitLogReader.java  |  3 ++-
 .../apache/cassandra/db/commitlog/CommitLogTest.java| 12 +++-
 3 files changed, 14 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3f5f7028/CHANGES.txt
--
diff --cc CHANGES.txt
index 0faed3d,a3eccf2..5a53e50
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,18 -1,10 +1,19 @@@
 -3.0.15
 +3.11.1
 + * Add a compaction option to TWCS to ignore sstables overlapping checks 
(CASSANDRA-13418)
 + * BTree.Builder memory leak (CASSANDRA-13754)
 + * Revert CASSANDRA-10368 of supporting non-pk column filtering due to 
correctness (CASSANDRA-13798)
 + * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)
 + * Better bootstrap failure message when blocked by (potential) range 
movement (CASSANDRA-13744)
 + * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
 + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
 + * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
 + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
 +Merged from 3.0:
+  * Better handle corrupt final commitlog segment (CASSANDRA-11995)
   * StreamingHistogram is not thread safe (CASSANDRA-13756)
   * Fix MV timestamp issues (CASSANDRA-11500)
 - * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) 
 + * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626)
   * Fix race condition in read command serialization (CASSANDRA-13363)
 - * Enable segement creation before recovering commitlogs (CASSANDRA-13587)
   * Fix AssertionError in short read protection (CASSANDRA-13747)
   * Don't skip corrupted sstables on startup (CASSANDRA-13620)
   * Fix the merging of cells with different user type versions 
(CASSANDRA-13776)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3f5f7028/src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java
--
diff --cc src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java
index d1cb8d6,000..8c04329
mode 100644,00..100644
--- a/src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java
+++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java
@@@ -1,514 -1,0 +1,515 @@@
 +/*
 + * Licensed to the Apache Software Foundation (ASF) under one
 + * or more contributor license agreements.  See the NOTICE file
 + * distributed with this work for additional information
 + * regarding copyright ownership.  The ASF licenses this file
 + * to you under the Apache License, Version 2.0 (the
 + * "License"); you may not use this file except in compliance
 + * with the License.  You may obtain a copy of the License at
 + *
 + * http://www.apache.org/licenses/LICENSE-2.0
 + *
 + * Unless required by applicable law or agreed to in writing, software
 + * distributed under the License is distributed on an "AS IS" BASIS,
 + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 + * See the License for the specific language governing permissions and
 + * limitations under the License.
 + */
 +package org.apache.cassandra.db.commitlog;
 +
 +import java.io.*;
 +import java.util.*;
 +import java.util.concurrent.atomic.AtomicInteger;
 +import java.util.zip.CRC32;
 +
 +import com.google.common.annotations.VisibleForTesting;
 +import org.apache.commons.lang3.StringUtils;
 +import org.slf4j.Logger;
 +import org.slf4j.LoggerFactory;
 +
 +import org.apache.cassandra.config.DatabaseDescriptor;
 +import org.apache.cassandra.db.Mutation;
 +import org.apache.cassandra.db.UnknownColumnFamilyException;
 +import 
org.apache.cassandra.db.commitlog.CommitLogReadHandler.CommitLogReadErrorReason;
 +import 
org.apache.cassandra.db.commitlog.CommitLogReadHandler.CommitLogReadException;
 +import org.apache.cassandra.db.partitions.PartitionUpdate;
 +import org.apache.cassandra.db.rows.SerializationHelper;
 +import org.apache.cassandra.io.util.DataInputBuffer;
 +import org.apache.cassandra.io.util.FileDataInput;
 +import

[2/6] cassandra git commit: On log replay, the last segment header may be truncated (or null). We should tolerate failure to read the header the same way we tolerate truncation of any of the rest of t

On log replay, the last segment header may be truncated (or null). We should 
tolerate failure to read the header the same way we tolerate truncation of any 
of the rest of the final segment

Patch by Jeff Jirsa; Reviewed by Branimir Lambov for CASSANDRA-11995


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0493545d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0493545d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0493545d

Branch: refs/heads/cassandra-3.11
Commit: 0493545dd08d29c34d757bb2f1d90052d03d24c6
Parents: ef5ac1a
Author: Jeff Jirsa 
Authored: Fri Mar 17 13:54:03 2017 -0700
Committer: Jeff Jirsa 
Committed: Tue Sep 5 11:46:03 2017 -0700

--
 CHANGES.txt|  1 +
 .../cassandra/db/commitlog/CommitLogReplayer.java  |  4 +++-
 .../apache/cassandra/db/commitlog/CommitLogTest.java   | 13 -
 3 files changed, 16 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/0493545d/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 853bf61..a3eccf2 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Better handle corrupt final commitlog segment (CASSANDRA-11995)
  * StreamingHistogram is not thread safe (CASSANDRA-13756)
  * Fix MV timestamp issues (CASSANDRA-11500)
  * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0493545d/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java
--
diff --git a/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java 
b/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java
index 205c36a..b3b26dd 100644
--- a/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java
+++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java
@@ -355,7 +355,9 @@ public class CommitLogReplayer
 desc = null;
 }
 if (desc == null) {
-handleReplayError(false, "Could not read commit log descriptor 
in file %s", file);
+// Presumably a failed CRC or other IO error occurred, which 
may be ok if it's the last segment
+// where we tolerate (and expect) truncation
+handleReplayError(tolerateTruncation, "Could not read commit 
log descriptor in file %s", file);
 return;
 }
 if (segmentId != desc.id)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0493545d/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java 
b/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java
index 90dc258..1543415 100644
--- a/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java
+++ b/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java
@@ -117,13 +117,24 @@ public class CommitLogTest
 @Test
 public void testRecoveryWithEmptyLog() throws Exception
 {
+// The first empty file we expect to throw as it's invalid
+// We need to pass the second as well, because allowTruncation will be 
set to true for the final segment
 runExpecting(() -> {
-CommitLog.instance.recover(new File[]{ 
tmpFile(CommitLogDescriptor.current_version) });
+CommitLog.instance.recover(new File[]{
+tmpFile(CommitLogDescriptor.current_version),
+tmpFile(CommitLogDescriptor.current_version)  });
 return null;
 }, CommitLogReplayException.class);
 }
 
 @Test
+public void testRecoveryWithEmptyFinalLog() throws Exception
+{
+// Even though it's empty, it's the last commitlog segment, so 
allowTruncation=true should allow it to pass
+CommitLog.instance.recover(new File[]{ 
tmpFile(CommitLogDescriptor.current_version)  });
+}
+
+@Test
 public void testRecoveryWithEmptyLog20() throws Exception
 {
 CommitLog.instance.recover(new File[]{ 
tmpFile(CommitLogDescriptor.VERSION_20) });


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[1/6] cassandra git commit: On log replay, the last segment header may be truncated (or null). We should tolerate failure to read the header the same way we tolerate truncation of any of the rest of t

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 ef5ac1a4a -> 0493545dd
  refs/heads/cassandra-3.11 d44a0d25b -> 3f5f70281
  refs/heads/trunk 733ea88fa -> fa36044a5


On log replay, the last segment header may be truncated (or null). We should 
tolerate failure to read the header the same way we tolerate truncation of any 
of the rest of the final segment

Patch by Jeff Jirsa; Reviewed by Branimir Lambov for CASSANDRA-11995


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0493545d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0493545d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0493545d

Branch: refs/heads/cassandra-3.0
Commit: 0493545dd08d29c34d757bb2f1d90052d03d24c6
Parents: ef5ac1a
Author: Jeff Jirsa 
Authored: Fri Mar 17 13:54:03 2017 -0700
Committer: Jeff Jirsa 
Committed: Tue Sep 5 11:46:03 2017 -0700

--
 CHANGES.txt|  1 +
 .../cassandra/db/commitlog/CommitLogReplayer.java  |  4 +++-
 .../apache/cassandra/db/commitlog/CommitLogTest.java   | 13 -
 3 files changed, 16 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/0493545d/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 853bf61..a3eccf2 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Better handle corrupt final commitlog segment (CASSANDRA-11995)
  * StreamingHistogram is not thread safe (CASSANDRA-13756)
  * Fix MV timestamp issues (CASSANDRA-11500)
  * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0493545d/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java
--
diff --git a/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java 
b/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java
index 205c36a..b3b26dd 100644
--- a/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java
+++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java
@@ -355,7 +355,9 @@ public class CommitLogReplayer
 desc = null;
 }
 if (desc == null) {
-handleReplayError(false, "Could not read commit log descriptor 
in file %s", file);
+// Presumably a failed CRC or other IO error occurred, which 
may be ok if it's the last segment
+// where we tolerate (and expect) truncation
+handleReplayError(tolerateTruncation, "Could not read commit 
log descriptor in file %s", file);
 return;
 }
 if (segmentId != desc.id)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0493545d/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java 
b/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java
index 90dc258..1543415 100644
--- a/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java
+++ b/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java
@@ -117,13 +117,24 @@ public class CommitLogTest
 @Test
 public void testRecoveryWithEmptyLog() throws Exception
 {
+// The first empty file we expect to throw as it's invalid
+// We need to pass the second as well, because allowTruncation will be 
set to true for the final segment
 runExpecting(() -> {
-CommitLog.instance.recover(new File[]{ 
tmpFile(CommitLogDescriptor.current_version) });
+CommitLog.instance.recover(new File[]{
+tmpFile(CommitLogDescriptor.current_version),
+tmpFile(CommitLogDescriptor.current_version)  });
 return null;
 }, CommitLogReplayException.class);
 }
 
 @Test
+public void testRecoveryWithEmptyFinalLog() throws Exception
+{
+// Even though it's empty, it's the last commitlog segment, so 
allowTruncation=true should allow it to pass
+CommitLog.instance.recover(new File[]{ 
tmpFile(CommitLogDescriptor.current_version)  });
+}
+
+@Test
 public void testRecoveryWithEmptyLog20() throws Exception
 {
 CommitLog.instance.recover(new File[]{ 
tmpFile(CommitLogDescriptor.VERSION_20) });


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3f5f7028
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3f5f7028
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3f5f7028

Branch: refs/heads/cassandra-3.11
Commit: 3f5f70281ee87c09d312d2169d6335e5b0abf815
Parents: d44a0d2 0493545
Author: Jeff Jirsa 
Authored: Tue Sep 5 11:47:18 2017 -0700
Committer: Jeff Jirsa 
Committed: Tue Sep 5 11:47:55 2017 -0700

--
 CHANGES.txt |  1 +
 .../apache/cassandra/db/commitlog/CommitLogReader.java  |  3 ++-
 .../apache/cassandra/db/commitlog/CommitLogTest.java| 12 +++-
 3 files changed, 14 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/3f5f7028/CHANGES.txt
--
diff --cc CHANGES.txt
index 0faed3d,a3eccf2..5a53e50
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,18 -1,10 +1,19 @@@
 -3.0.15
 +3.11.1
 + * Add a compaction option to TWCS to ignore sstables overlapping checks 
(CASSANDRA-13418)
 + * BTree.Builder memory leak (CASSANDRA-13754)
 + * Revert CASSANDRA-10368 of supporting non-pk column filtering due to 
correctness (CASSANDRA-13798)
 + * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)
 + * Better bootstrap failure message when blocked by (potential) range 
movement (CASSANDRA-13744)
 + * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
 + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
 + * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
 + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
 +Merged from 3.0:
+  * Better handle corrupt final commitlog segment (CASSANDRA-11995)
   * StreamingHistogram is not thread safe (CASSANDRA-13756)
   * Fix MV timestamp issues (CASSANDRA-11500)
 - * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) 
 + * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626)
   * Fix race condition in read command serialization (CASSANDRA-13363)
 - * Enable segement creation before recovering commitlogs (CASSANDRA-13587)
   * Fix AssertionError in short read protection (CASSANDRA-13747)
   * Don't skip corrupted sstables on startup (CASSANDRA-13620)
   * Fix the merging of cells with different user type versions 
(CASSANDRA-13776)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/3f5f7028/src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java
--
diff --cc src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java
index d1cb8d6,000..8c04329
mode 100644,00..100644
--- a/src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java
+++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogReader.java
@@@ -1,514 -1,0 +1,515 @@@
 +/*
 + * Licensed to the Apache Software Foundation (ASF) under one
 + * or more contributor license agreements.  See the NOTICE file
 + * distributed with this work for additional information
 + * regarding copyright ownership.  The ASF licenses this file
 + * to you under the Apache License, Version 2.0 (the
 + * "License"); you may not use this file except in compliance
 + * with the License.  You may obtain a copy of the License at
 + *
 + * http://www.apache.org/licenses/LICENSE-2.0
 + *
 + * Unless required by applicable law or agreed to in writing, software
 + * distributed under the License is distributed on an "AS IS" BASIS,
 + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 + * See the License for the specific language governing permissions and
 + * limitations under the License.
 + */
 +package org.apache.cassandra.db.commitlog;
 +
 +import java.io.*;
 +import java.util.*;
 +import java.util.concurrent.atomic.AtomicInteger;
 +import java.util.zip.CRC32;
 +
 +import com.google.common.annotations.VisibleForTesting;
 +import org.apache.commons.lang3.StringUtils;
 +import org.slf4j.Logger;
 +import org.slf4j.LoggerFactory;
 +
 +import org.apache.cassandra.config.DatabaseDescriptor;
 +import org.apache.cassandra.db.Mutation;
 +import org.apache.cassandra.db.UnknownColumnFamilyException;
 +import 
org.apache.cassandra.db.commitlog.CommitLogReadHandler.CommitLogReadErrorReason;
 +import 
org.apache.cassandra.db.commitlog.CommitLogReadHandler.CommitLogReadException;
 +import org.apache.cassandra.db.partitions.PartitionUpdate;
 +import org.apache.cassandra.db.rows.SerializationHelper;
 +import org.apache.cassandra.io.util.DataInputBuffer;
 +import org.apache.cassandra.io.util.FileDataInput;
 +import

[3/6] cassandra git commit: On log replay, the last segment header may be truncated (or null). We should tolerate failure to read the header the same way we tolerate truncation of any of the rest of t

On log replay, the last segment header may be truncated (or null). We should 
tolerate failure to read the header the same way we tolerate truncation of any 
of the rest of the final segment

Patch by Jeff Jirsa; Reviewed by Branimir Lambov for CASSANDRA-11995


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0493545d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0493545d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0493545d

Branch: refs/heads/trunk
Commit: 0493545dd08d29c34d757bb2f1d90052d03d24c6
Parents: ef5ac1a
Author: Jeff Jirsa 
Authored: Fri Mar 17 13:54:03 2017 -0700
Committer: Jeff Jirsa 
Committed: Tue Sep 5 11:46:03 2017 -0700

--
 CHANGES.txt|  1 +
 .../cassandra/db/commitlog/CommitLogReplayer.java  |  4 +++-
 .../apache/cassandra/db/commitlog/CommitLogTest.java   | 13 -
 3 files changed, 16 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/0493545d/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 853bf61..a3eccf2 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * Better handle corrupt final commitlog segment (CASSANDRA-11995)
  * StreamingHistogram is not thread safe (CASSANDRA-13756)
  * Fix MV timestamp issues (CASSANDRA-11500)
  * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0493545d/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java
--
diff --git a/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java 
b/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java
index 205c36a..b3b26dd 100644
--- a/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java
+++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java
@@ -355,7 +355,9 @@ public class CommitLogReplayer
 desc = null;
 }
 if (desc == null) {
-handleReplayError(false, "Could not read commit log descriptor 
in file %s", file);
+// Presumably a failed CRC or other IO error occurred, which 
may be ok if it's the last segment
+// where we tolerate (and expect) truncation
+handleReplayError(tolerateTruncation, "Could not read commit 
log descriptor in file %s", file);
 return;
 }
 if (segmentId != desc.id)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/0493545d/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java 
b/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java
index 90dc258..1543415 100644
--- a/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java
+++ b/test/unit/org/apache/cassandra/db/commitlog/CommitLogTest.java
@@ -117,13 +117,24 @@ public class CommitLogTest
 @Test
 public void testRecoveryWithEmptyLog() throws Exception
 {
+// The first empty file we expect to throw as it's invalid
+// We need to pass the second as well, because allowTruncation will be 
set to true for the final segment
 runExpecting(() -> {
-CommitLog.instance.recover(new File[]{ 
tmpFile(CommitLogDescriptor.current_version) });
+CommitLog.instance.recover(new File[]{
+tmpFile(CommitLogDescriptor.current_version),
+tmpFile(CommitLogDescriptor.current_version)  });
 return null;
 }, CommitLogReplayException.class);
 }
 
 @Test
+public void testRecoveryWithEmptyFinalLog() throws Exception
+{
+// Even though it's empty, it's the last commitlog segment, so 
allowTruncation=true should allow it to pass
+CommitLog.instance.recover(new File[]{ 
tmpFile(CommitLogDescriptor.current_version)  });
+}
+
+@Test
 public void testRecoveryWithEmptyLog20() throws Exception
 {
 CommitLog.instance.recover(new File[]{ 
tmpFile(CommitLogDescriptor.VERSION_20) });


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13756) StreamingHistogram is not thread safe


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13756:
---
   Resolution: Fixed
Fix Version/s: (was: 3.11.x)
   (was: 3.0.x)
   3.11.1
   3.0.15
   Status: Resolved  (was: Ready to Commit)

Thanks all. Marcus, I've addressed your comments on commit. Committed as 
{{ef5ac1a4abe4fb5f407c0a24f4bc808932c5d7a2}} to 3.0 and 3.11, and {{merge -s 
ours}} to trunk (no changes but CHANGES.txt) with Jason and Marcus both listed 
as reviewers. 

> StreamingHistogram is not thread safe
> -
>
> Key: CASSANDRA-13756
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13756
> Project: Cassandra
>  Issue Type: Bug
>Reporter: xiangzhou xia
>Assignee: Jeff Jirsa
> Fix For: 3.0.15, 3.11.1
>
>
> When we test C*3 in shadow cluster, we notice after a period of time, several 
> data node suddenly run into 100% cpu and stop process query anymore.
> After investigation, we found that threads are stuck on the sum() in 
> streaminghistogram class. Those are jmx threads that working on expose 
> getTombStoneRatio metrics (since jmx is kicked off every 3 seconds, there is 
> a chance that multiple jmx thread is access streaminghistogram at the same 
> time).  
> After further investigation, we find that the optimization in CASSANDRA-13038 
> led to a spool flush every time when we call sum(). Since TreeMap is not 
> thread safe, threads will be stuck when multiple threads visit sum() at the 
> same time.
> There are two approaches to solve this issue. 
> The first one is to add a lock to the flush in sum() which will introduce 
> some extra overhead to streaminghistogram.
> The second one is to avoid streaminghistogram to be access by multiple 
> threads. For our specific case, is to remove the metrics we added.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d44a0d25
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d44a0d25
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d44a0d25

Branch: refs/heads/trunk
Commit: d44a0d25bb9a07f5e43a9951bd13f8259dfccd36
Parents: 03155b0 ef5ac1a
Author: Jeff Jirsa 
Authored: Tue Sep 5 09:54:36 2017 -0700
Committer: Jeff Jirsa 
Committed: Tue Sep 5 09:55:26 2017 -0700

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |   2 +-
 .../io/sstable/metadata/MetadataCollector.java  |  10 +-
 .../cassandra/utils/StreamingHistogram.java | 333 ++-
 .../microbench/StreamingHistogramBench.java | 192 ++-
 .../cassandra/utils/StreamingHistogramTest.java |  43 +--
 6 files changed, 306 insertions(+), 275 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d44a0d25/CHANGES.txt
--
diff --cc CHANGES.txt
index 7056075,853bf61..0faed3d
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,17 -1,9 +1,18 @@@
 -3.0.15
 +3.11.1
 + * Add a compaction option to TWCS to ignore sstables overlapping checks 
(CASSANDRA-13418)
 + * BTree.Builder memory leak (CASSANDRA-13754)
 + * Revert CASSANDRA-10368 of supporting non-pk column filtering due to 
correctness (CASSANDRA-13798)
 + * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)
 + * Better bootstrap failure message when blocked by (potential) range 
movement (CASSANDRA-13744)
 + * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
 + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
 + * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
 + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
 +Merged from 3.0:
+  * StreamingHistogram is not thread safe (CASSANDRA-13756)
   * Fix MV timestamp issues (CASSANDRA-11500)
 - * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) 
 + * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626)
   * Fix race condition in read command serialization (CASSANDRA-13363)
 - * Enable segement creation before recovering commitlogs (CASSANDRA-13587)
   * Fix AssertionError in short read protection (CASSANDRA-13747)
   * Don't skip corrupted sstables on startup (CASSANDRA-13620)
   * Fix the merging of cells with different user type versions 
(CASSANDRA-13776)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d44a0d25/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d44a0d25/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d44a0d25/src/java/org/apache/cassandra/utils/StreamingHistogram.java
--
diff --cc src/java/org/apache/cassandra/utils/StreamingHistogram.java
index 9114c7d,6500a1a..df49d8d
--- a/src/java/org/apache/cassandra/utils/StreamingHistogram.java
+++ b/src/java/org/apache/cassandra/utils/StreamingHistogram.java
@@@ -24,6 -24,6 +24,7 @@@ import com.google.common.base.Objects
  
  import org.apache.cassandra.db.TypeSizes;
  import org.apache.cassandra.io.ISerializer;
++import org.apache.cassandra.io.sstable.SSTable;
  import org.apache.cassandra.io.util.DataInputPlus;
  import org.apache.cassandra.io.util.DataOutputPlus;
  
@@@ -39,172 -39,15 +40,33 @@@ public class StreamingHistogra
  public static final StreamingHistogramSerializer serializer = new 
StreamingHistogramSerializer();
  
  // TreeMap to hold bins of histogram.
 -private final TreeMap bin;
 +// The key is a numeric type so we can avoid boxing/unboxing streams of 
different key types
 +// The value is a unboxed long array always of length == 1
 +// Serialized Histograms always writes with double keys for backwards 
compatibility
 +private final TreeMap bin;
 +
- // Keep a second, larger buffer to spool data in, before finalizing it 
into `bin`
- private final TreeMap spool;
- 
 +// maximum bin size for this histogram
  private final int maxBinSize;
  
- // maximum size of the spool
- private final int maxSpoolSize;
- 
- // voluntarily give up resolution for speed
- private final int roundSeconds;
 -private StreamingHistogram(int maxBinSize, Map

[2/6] cassandra git commit: StreamingHistogram is not thread safe

StreamingHistogram is not thread safe

Patch by Jeff Jirsa; Reviewed by Jason Brown, Marcus Eriksson  for 
CASSANDRA-13756


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ef5ac1a4
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ef5ac1a4
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ef5ac1a4

Branch: refs/heads/cassandra-3.11
Commit: ef5ac1a4abe4fb5f407c0a24f4bc808932c5d7a2
Parents: 1e80e35
Author: Jeff Jirsa 
Authored: Tue Aug 29 10:31:16 2017 -0700
Committer: Jeff Jirsa 
Committed: Tue Sep 5 09:54:27 2017 -0700

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |   2 +-
 .../io/sstable/metadata/MetadataCollector.java  |  10 +-
 .../cassandra/utils/StreamingHistogram.java | 284 ++-
 .../microbench/StreamingHistogramBench.java | 192 +++--
 .../cassandra/utils/StreamingHistogramTest.java |  31 +-
 6 files changed, 263 insertions(+), 257 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/ef5ac1a4/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 1ed3fe6..853bf61 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * StreamingHistogram is not thread safe (CASSANDRA-13756)
  * Fix MV timestamp issues (CASSANDRA-11500)
  * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) 
  * Fix race condition in read command serialization (CASSANDRA-13363)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ef5ac1a4/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 183176c..983d6b1 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -2415,7 +2415,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 long allColumns = 0;
 int localTime = (int)(System.currentTimeMillis()/1000);
 
-for (SSTableReader sstable : getSSTables(SSTableSet.LIVE))
+for (SSTableReader sstable : getSSTables(SSTableSet.CANONICAL))
 {
 allDroppable += sstable.getDroppableTombstonesBefore(localTime - 
sstable.metadata.params.gcGraceSeconds);
 allColumns += sstable.getEstimatedColumnCount().mean() * 
sstable.getEstimatedColumnCount().count();

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ef5ac1a4/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
--
diff --git 
a/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java 
b/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
index 3b13cf4..487a932 100644
--- a/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
+++ b/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
@@ -59,9 +59,9 @@ public class MetadataCollector implements 
PartitionStatisticsCollector
 return new EstimatedHistogram(150);
 }
 
-static StreamingHistogram defaultTombstoneDropTimeHistogram()
+static StreamingHistogram.StreamingHistogramBuilder 
defaultTombstoneDropTimeHistogramBuilder()
 {
-return new StreamingHistogram(SSTable.TOMBSTONE_HISTOGRAM_BIN_SIZE, 
SSTable.TOMBSTONE_HISTOGRAM_SPOOL_SIZE, 
SSTable.TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS);
+return new 
StreamingHistogram.StreamingHistogramBuilder(SSTable.TOMBSTONE_HISTOGRAM_BIN_SIZE,
 SSTable.TOMBSTONE_HISTOGRAM_SPOOL_SIZE, 
SSTable.TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS);
 }
 
 public static StatsMetadata defaultStatsMetadata()
@@ -76,7 +76,7 @@ public class MetadataCollector implements 
PartitionStatisticsCollector
  0,
  Integer.MAX_VALUE,
  NO_COMPRESSION_RATIO,
- defaultTombstoneDropTimeHistogram(),
+ 
defaultTombstoneDropTimeHistogramBuilder().build(),
  0,
  Collections.emptyList(),
  Collections.emptyList(),
@@ -94,7 +94,7 @@ public class MetadataCollector implements 
PartitionStatisticsCollector
 protected final MinMaxIntTracker localDeletionTimeTracker = new 
MinMaxIntTracker(Cell.NO_DELETION_TIME, Cell.NO_DELETION_TIME);
 protected final MinMaxIntTracker ttlTracker = new 
MinMaxIntTracker(Cell.NO_TTL, Cell.NO_TTL);
 protected

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d44a0d25
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d44a0d25
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d44a0d25

Branch: refs/heads/cassandra-3.11
Commit: d44a0d25bb9a07f5e43a9951bd13f8259dfccd36
Parents: 03155b0 ef5ac1a
Author: Jeff Jirsa 
Authored: Tue Sep 5 09:54:36 2017 -0700
Committer: Jeff Jirsa 
Committed: Tue Sep 5 09:55:26 2017 -0700

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |   2 +-
 .../io/sstable/metadata/MetadataCollector.java  |  10 +-
 .../cassandra/utils/StreamingHistogram.java | 333 ++-
 .../microbench/StreamingHistogramBench.java | 192 ++-
 .../cassandra/utils/StreamingHistogramTest.java |  43 +--
 6 files changed, 306 insertions(+), 275 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d44a0d25/CHANGES.txt
--
diff --cc CHANGES.txt
index 7056075,853bf61..0faed3d
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,17 -1,9 +1,18 @@@
 -3.0.15
 +3.11.1
 + * Add a compaction option to TWCS to ignore sstables overlapping checks 
(CASSANDRA-13418)
 + * BTree.Builder memory leak (CASSANDRA-13754)
 + * Revert CASSANDRA-10368 of supporting non-pk column filtering due to 
correctness (CASSANDRA-13798)
 + * Fix cassandra-stress hang issues when an error during cluster connection 
happens (CASSANDRA-12938)
 + * Better bootstrap failure message when blocked by (potential) range 
movement (CASSANDRA-13744)
 + * "ignore" option is ignored in sstableloader (CASSANDRA-13721)
 + * Deadlock in AbstractCommitLogSegmentManager (CASSANDRA-13652)
 + * Duplicate the buffer before passing it to analyser in SASI operation 
(CASSANDRA-13512)
 + * Properly evict pstmts from prepared statements cache (CASSANDRA-13641)
 +Merged from 3.0:
+  * StreamingHistogram is not thread safe (CASSANDRA-13756)
   * Fix MV timestamp issues (CASSANDRA-11500)
 - * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) 
 + * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626)
   * Fix race condition in read command serialization (CASSANDRA-13363)
 - * Enable segement creation before recovering commitlogs (CASSANDRA-13587)
   * Fix AssertionError in short read protection (CASSANDRA-13747)
   * Don't skip corrupted sstables on startup (CASSANDRA-13620)
   * Fix the merging of cells with different user type versions 
(CASSANDRA-13776)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d44a0d25/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d44a0d25/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d44a0d25/src/java/org/apache/cassandra/utils/StreamingHistogram.java
--
diff --cc src/java/org/apache/cassandra/utils/StreamingHistogram.java
index 9114c7d,6500a1a..df49d8d
--- a/src/java/org/apache/cassandra/utils/StreamingHistogram.java
+++ b/src/java/org/apache/cassandra/utils/StreamingHistogram.java
@@@ -24,6 -24,6 +24,7 @@@ import com.google.common.base.Objects
  
  import org.apache.cassandra.db.TypeSizes;
  import org.apache.cassandra.io.ISerializer;
++import org.apache.cassandra.io.sstable.SSTable;
  import org.apache.cassandra.io.util.DataInputPlus;
  import org.apache.cassandra.io.util.DataOutputPlus;
  
@@@ -39,172 -39,15 +40,33 @@@ public class StreamingHistogra
  public static final StreamingHistogramSerializer serializer = new 
StreamingHistogramSerializer();
  
  // TreeMap to hold bins of histogram.
 -private final TreeMap bin;
 +// The key is a numeric type so we can avoid boxing/unboxing streams of 
different key types
 +// The value is a unboxed long array always of length == 1
 +// Serialized Histograms always writes with double keys for backwards 
compatibility
 +private final TreeMap bin;
 +
- // Keep a second, larger buffer to spool data in, before finalizing it 
into `bin`
- private final TreeMap spool;
- 
 +// maximum bin size for this histogram
  private final int maxBinSize;
  
- // maximum size of the spool
- private final int maxSpoolSize;
- 
- // voluntarily give up resolution for speed
- private final int roundSeconds;
 -private StreamingHistogram(int maxBinSize,

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/733ea88f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/733ea88f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/733ea88f

Branch: refs/heads/trunk
Commit: 733ea88faafa343eb6991a691f0bf36b2d5f0ed5
Parents: 33ab490 d44a0d2
Author: Jeff Jirsa 
Authored: Tue Sep 5 09:55:43 2017 -0700
Committer: Jeff Jirsa 
Committed: Tue Sep 5 09:56:05 2017 -0700

--
 CHANGES.txt | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/733ea88f/CHANGES.txt
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[3/6] cassandra git commit: StreamingHistogram is not thread safe

StreamingHistogram is not thread safe

Patch by Jeff Jirsa; Reviewed by Jason Brown, Marcus Eriksson  for 
CASSANDRA-13756


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ef5ac1a4
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ef5ac1a4
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ef5ac1a4

Branch: refs/heads/trunk
Commit: ef5ac1a4abe4fb5f407c0a24f4bc808932c5d7a2
Parents: 1e80e35
Author: Jeff Jirsa 
Authored: Tue Aug 29 10:31:16 2017 -0700
Committer: Jeff Jirsa 
Committed: Tue Sep 5 09:54:27 2017 -0700

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |   2 +-
 .../io/sstable/metadata/MetadataCollector.java  |  10 +-
 .../cassandra/utils/StreamingHistogram.java | 284 ++-
 .../microbench/StreamingHistogramBench.java | 192 +++--
 .../cassandra/utils/StreamingHistogramTest.java |  31 +-
 6 files changed, 263 insertions(+), 257 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/ef5ac1a4/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 1ed3fe6..853bf61 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * StreamingHistogram is not thread safe (CASSANDRA-13756)
  * Fix MV timestamp issues (CASSANDRA-11500)
  * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) 
  * Fix race condition in read command serialization (CASSANDRA-13363)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ef5ac1a4/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 183176c..983d6b1 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -2415,7 +2415,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 long allColumns = 0;
 int localTime = (int)(System.currentTimeMillis()/1000);
 
-for (SSTableReader sstable : getSSTables(SSTableSet.LIVE))
+for (SSTableReader sstable : getSSTables(SSTableSet.CANONICAL))
 {
 allDroppable += sstable.getDroppableTombstonesBefore(localTime - 
sstable.metadata.params.gcGraceSeconds);
 allColumns += sstable.getEstimatedColumnCount().mean() * 
sstable.getEstimatedColumnCount().count();

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ef5ac1a4/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
--
diff --git 
a/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java 
b/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
index 3b13cf4..487a932 100644
--- a/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
+++ b/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
@@ -59,9 +59,9 @@ public class MetadataCollector implements 
PartitionStatisticsCollector
 return new EstimatedHistogram(150);
 }
 
-static StreamingHistogram defaultTombstoneDropTimeHistogram()
+static StreamingHistogram.StreamingHistogramBuilder 
defaultTombstoneDropTimeHistogramBuilder()
 {
-return new StreamingHistogram(SSTable.TOMBSTONE_HISTOGRAM_BIN_SIZE, 
SSTable.TOMBSTONE_HISTOGRAM_SPOOL_SIZE, 
SSTable.TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS);
+return new 
StreamingHistogram.StreamingHistogramBuilder(SSTable.TOMBSTONE_HISTOGRAM_BIN_SIZE,
 SSTable.TOMBSTONE_HISTOGRAM_SPOOL_SIZE, 
SSTable.TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS);
 }
 
 public static StatsMetadata defaultStatsMetadata()
@@ -76,7 +76,7 @@ public class MetadataCollector implements 
PartitionStatisticsCollector
  0,
  Integer.MAX_VALUE,
  NO_COMPRESSION_RATIO,
- defaultTombstoneDropTimeHistogram(),
+ 
defaultTombstoneDropTimeHistogramBuilder().build(),
  0,
  Collections.emptyList(),
  Collections.emptyList(),
@@ -94,7 +94,7 @@ public class MetadataCollector implements 
PartitionStatisticsCollector
 protected final MinMaxIntTracker localDeletionTimeTracker = new 
MinMaxIntTracker(Cell.NO_DELETION_TIME, Cell.NO_DELETION_TIME);
 protected final MinMaxIntTracker ttlTracker = new 
MinMaxIntTracker(Cell.NO_TTL, Cell.NO_TTL);
 protected double

[1/6] cassandra git commit: StreamingHistogram is not thread safe

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 1e80e3539 -> ef5ac1a4a
  refs/heads/cassandra-3.11 03155b021 -> d44a0d25b
  refs/heads/trunk 33ab4902a -> 733ea88fa


StreamingHistogram is not thread safe

Patch by Jeff Jirsa; Reviewed by Jason Brown, Marcus Eriksson  for 
CASSANDRA-13756


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ef5ac1a4
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ef5ac1a4
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ef5ac1a4

Branch: refs/heads/cassandra-3.0
Commit: ef5ac1a4abe4fb5f407c0a24f4bc808932c5d7a2
Parents: 1e80e35
Author: Jeff Jirsa 
Authored: Tue Aug 29 10:31:16 2017 -0700
Committer: Jeff Jirsa 
Committed: Tue Sep 5 09:54:27 2017 -0700

--
 CHANGES.txt |   1 +
 .../apache/cassandra/db/ColumnFamilyStore.java  |   2 +-
 .../io/sstable/metadata/MetadataCollector.java  |  10 +-
 .../cassandra/utils/StreamingHistogram.java | 284 ++-
 .../microbench/StreamingHistogramBench.java | 192 +++--
 .../cassandra/utils/StreamingHistogramTest.java |  31 +-
 6 files changed, 263 insertions(+), 257 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/ef5ac1a4/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 1ed3fe6..853bf61 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.15
+ * StreamingHistogram is not thread safe (CASSANDRA-13756)
  * Fix MV timestamp issues (CASSANDRA-11500)
  * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) 
  * Fix race condition in read command serialization (CASSANDRA-13363)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ef5ac1a4/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java 
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index 183176c..983d6b1 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -2415,7 +2415,7 @@ public class ColumnFamilyStore implements 
ColumnFamilyStoreMBean
 long allColumns = 0;
 int localTime = (int)(System.currentTimeMillis()/1000);
 
-for (SSTableReader sstable : getSSTables(SSTableSet.LIVE))
+for (SSTableReader sstable : getSSTables(SSTableSet.CANONICAL))
 {
 allDroppable += sstable.getDroppableTombstonesBefore(localTime - 
sstable.metadata.params.gcGraceSeconds);
 allColumns += sstable.getEstimatedColumnCount().mean() * 
sstable.getEstimatedColumnCount().count();

http://git-wip-us.apache.org/repos/asf/cassandra/blob/ef5ac1a4/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
--
diff --git 
a/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java 
b/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
index 3b13cf4..487a932 100644
--- a/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
+++ b/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
@@ -59,9 +59,9 @@ public class MetadataCollector implements 
PartitionStatisticsCollector
 return new EstimatedHistogram(150);
 }
 
-static StreamingHistogram defaultTombstoneDropTimeHistogram()
+static StreamingHistogram.StreamingHistogramBuilder 
defaultTombstoneDropTimeHistogramBuilder()
 {
-return new StreamingHistogram(SSTable.TOMBSTONE_HISTOGRAM_BIN_SIZE, 
SSTable.TOMBSTONE_HISTOGRAM_SPOOL_SIZE, 
SSTable.TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS);
+return new 
StreamingHistogram.StreamingHistogramBuilder(SSTable.TOMBSTONE_HISTOGRAM_BIN_SIZE,
 SSTable.TOMBSTONE_HISTOGRAM_SPOOL_SIZE, 
SSTable.TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS);
 }
 
 public static StatsMetadata defaultStatsMetadata()
@@ -76,7 +76,7 @@ public class MetadataCollector implements 
PartitionStatisticsCollector
  0,
  Integer.MAX_VALUE,
  NO_COMPRESSION_RATIO,
- defaultTombstoneDropTimeHistogram(),
+ 
defaultTombstoneDropTimeHistogramBuilder().build(),
  0,
  Collections.emptyList(),
  Collections.emptyList(),
@@ -94,7 +94,7 @@ public class MetadataCollector implements 
PartitionStatisticsCollector
 protected final MinMaxIntTracker localDeletionTimeTracker =

[jira] [Commented] (CASSANDRA-13818) Add support for --hosts, --force, and subrange repair to incremental repair

2017-09-05 Thread Blake Eggleston (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153945#comment-16153945
 ] 

Blake Eggleston commented on CASSANDRA-13818:
-

The reason I didn't go that way was because it would mean adding another repair 
path for a repair option that shouldn't be used very often.

> Add support for --hosts, --force, and subrange repair to incremental repair
> ---
>
> Key: CASSANDRA-13818
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13818
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
> Fix For: 4.0
>
>
> It should be possible to run incremental repair with nodes down, we just 
> shouldn't promote the data to repaired afterwards



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13655) Range deletes in a CAS batch are ignored


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147711#comment-16147711
 ] 

Jeff Jirsa edited comment on CASSANDRA-13655 at 9/5/17 4:30 PM:


Unit tests are happy. DTests have the typical flakey tests + some failures that 
look environmental in 3.0 and 3.11, but trunk didnt run at all. The cqlsh copy 
dtests failed on both 3.0 and 3.11, and they both show a single failure in the 
last dozen runs, so that seems odd and I'll check that on the next run.

URLs will be:

Edit: rebuilding again ( no tests ran )
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/276/testReport/
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/277/testReport/
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/278/testReport/





was (Author: jjirsa):
Unit tests are happy. DTests have the typical flakey tests + some failures that 
look environmental in 3.0 and 3.11, but trunk didnt run at all. The cqlsh copy 
dtests failed on both 3.0 and 3.11, and they both show a single failure in the 
last dozen runs, so that seems odd and I'll check that on the next run.

URLs will be:

https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/247/testReport/
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/248/testReport/
https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/249/testReport/




> Range deletes in a CAS batch are ignored
> 
>
> Key: CASSANDRA-13655
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13655
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Jeff Jirsa
>Assignee: Jeff Jirsa
>Priority: Blocker
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> Range deletes in a CAS batch are ignored 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13837) Hanging threads in BulkLoader

2017-09-05 Thread Joel Knighton (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153917#comment-16153917
 ] 

Joel Knighton commented on CASSANDRA-13837:
---

I pointed [~mshuler] here prematurely - my bad. It looks like the System.exit 
uncommented in [CASSANDRA-13836] resolves the hang we were seeing on that test. 
The remaining hangs seem to be repair-related and unrelated to this ticket.

> Hanging threads in BulkLoader
> -
>
> Key: CASSANDRA-13837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13837
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
>
> [~krummas] discovered some threads that were not closing correctly when he 
> fixed CASSANDRA-13836. We suspect this is due to 
> CASSANDRA-8457/CASSANDRA-12229.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13530) GroupCommitLogService

2017-09-05 Thread Ariel Weisberg (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153918#comment-16153918
 ] 

Ariel Weisberg commented on CASSANDRA-13530:


That documentation in the YAML looks wrong to me. There is no default value it 
requires you to supply one apparently.

There is a [common bit of 
functionality|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/BatchCommitLogService.java#L26]
 with both periodic and batch commit log where it will run syncs periodically 
and that is in the shared superclass. Then batch itself [requests an extra 
sync|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/BatchCommitLogService.java#L33]
 every time it accepts a write so the periodic one does nothing productive as 
far as I can tell. Two seconds should be fine. The periodic one will never fire 
because every time the sync thread wakes up it recalculates the next wakeup 
time so it will never reach the periodic wakeup.

> GroupCommitLogService
> -
>
> Key: CASSANDRA-13530
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13530
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Yuji Ito
>Assignee: Yuji Ito
> Fix For: 2.2.x, 3.0.x, 3.11.x
>
> Attachments: groupCommit22.patch, groupCommit30.patch, 
> groupCommit3x.patch, groupCommitLog_noSerial_result.xlsx, 
> groupCommitLog_result.xlsx, GuavaRequestThread.java, MicroRequestThread.java
>
>
> I propose a new CommitLogService, GroupCommitLogService, to improve the 
> throughput when lots of requests are received.
> It improved the throughput by maximum 94%.
> I'd like to discuss about this CommitLogService.
> Currently, we can select either 2 CommitLog services; Periodic and Batch.
> In Periodic, we might lose some commit log which hasn't written to the disk.
> In Batch, we can write commit log to the disk every time. The size of commit 
> log to write is too small (< 4KB). When high concurrency, these writes are 
> gathered and persisted to the disk at once. But, when insufficient 
> concurrency, many small writes are issued and the performance decreases due 
> to the latency of the disk. Even if you use SSD, processes of many IO 
> commands decrease the performance.
> GroupCommitLogService writes some commitlog to the disk at once.
> The patch adds GroupCommitLogService (It is enabled by setting 
> `commitlog_sync` and `commitlog_sync_group_window_in_ms` in cassandra.yaml).
> The difference from Batch is just only waiting for the semaphore.
> By waiting for the semaphore, some writes for commit logs are executed at the 
> same time.
> In GroupCommitLogService, the latency becomes worse if the there is no 
> concurrency.
> I measured the performance with my microbench (MicroRequestThread.java) by 
> increasing the number of threads.The cluster has 3 nodes (Replication factor: 
> 3). Each nodes is AWS EC2 m4.large instance + 200IOPS io1 volume.
> The result is as below. The GroupCommitLogService with 10ms window improved 
> update with Paxos by 94% and improved select with Paxos by 76%.
> h6. SELECT / sec
> ||\# of threads||Batch 2ms||Group 10ms||
> |1|192|103|
> |2|163|212|
> |4|264|416|
> |8|454|800|
> |16|744|1311|
> |32|1151|1481|
> |64|1767|1844|
> |128|2949|3011|
> |256|4723|5000|
> h6. UPDATE / sec
> ||\# of threads||Batch 2ms||Group 10ms||
> |1|45|26|
> |2|39|51|
> |4|58|102|
> |8|102|198|
> |16|167|213|
> |32|289|295|
> |64|544|548|
> |128|1046|1058|
> |256|2020|2061|



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13844) sstableloader doesn't support non default storage_port and ssl_storage_port


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eduard Tudenhoefner updated CASSANDRA-13844:

Reproduced In: 2.2.10, 2.1.18  (was: 2.1.18, 2.2.10)
 Reviewer: Alex Petrov

> sstableloader doesn't support non default storage_port and ssl_storage_port
> ---
>
> Key: CASSANDRA-13844
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13844
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Eduard Tudenhoefner
>Assignee: Eduard Tudenhoefner
> Fix For: 2.2.11, 2.1.19
>
>
> Currently *storage_port* and *ssl_storage_port* are hardcoded to the 
> defaults. The problem was already fixed in CASSANDRA-13518 for C* 3.0+, so 
> this here is just backporting it to C* 2.1/2.2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13844) sstableloader doesn't support non default storage_port and ssl_storage_port


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eduard Tudenhoefner updated CASSANDRA-13844:

Reproduced In: 2.2.10, 2.1.18
   Status: Patch Available  (was: In Progress)

> sstableloader doesn't support non default storage_port and ssl_storage_port
> ---
>
> Key: CASSANDRA-13844
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13844
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Eduard Tudenhoefner
>Assignee: Eduard Tudenhoefner
> Fix For: 2.2.11, 2.1.19
>
>
> Currently *storage_port* and *ssl_storage_port* are hardcoded to the 
> defaults. The problem was already fixed in CASSANDRA-13518 for C* 3.0+, so 
> this here is just backporting it to C* 2.1/2.2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13844) sstableloader doesn't support non default storage_port and ssl_storage_port


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153893#comment-16153893
 ] 

Eduard Tudenhoefner edited comment on CASSANDRA-13844 at 9/5/17 4:21 PM:
-

2.1 Branch: https://github.com/nastra/cassandra/tree/13844-21
2.1 Test: https://circleci.com/gh/nastra/cassandra/18

2.2 Branch: https://github.com/nastra/cassandra/tree/13844-22
2.2 Test: https://circleci.com/gh/nastra/cassandra/19

2.2 upstream merges should be all empty


was (Author: eduard.tudenhoefner):
Branch: https://github.com/nastra/cassandra/tree/13844-21
Test: https://circleci.com/gh/nastra/cassandra/17

> sstableloader doesn't support non default storage_port and ssl_storage_port
> ---
>
> Key: CASSANDRA-13844
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13844
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Eduard Tudenhoefner
>Assignee: Eduard Tudenhoefner
> Fix For: 2.2.11, 2.1.19
>
>
> Currently *storage_port* and *ssl_storage_port* are hardcoded to the 
> defaults. The problem was already fixed in CASSANDRA-13518 for C* 3.0+, so 
> this here is just backporting it to C* 2.1/2.2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13844) sstableloader doesn't support non default storage_port and ssl_storage_port


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153893#comment-16153893
 ] 

Eduard Tudenhoefner commented on CASSANDRA-13844:
-

Branch: https://github.com/nastra/cassandra/tree/13844-21
Test: https://circleci.com/gh/nastra/cassandra/17

> sstableloader doesn't support non default storage_port and ssl_storage_port
> ---
>
> Key: CASSANDRA-13844
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13844
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Eduard Tudenhoefner
>Assignee: Eduard Tudenhoefner
> Fix For: 2.2.11, 2.1.19
>
>
> Currently *storage_port* and *ssl_storage_port* are hardcoded to the 
> defaults. The problem was already fixed in CASSANDRA-13518 for C* 3.0+, so 
> this here is just backporting it to C* 2.1/2.2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-13844) sstableloader doesn't support non default storage_port and ssl_storage_port

Eduard Tudenhoefner created CASSANDRA-13844:
---

 Summary: sstableloader doesn't support non default storage_port 
and ssl_storage_port
 Key: CASSANDRA-13844
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13844
 Project: Cassandra
  Issue Type: Improvement
  Components: Tools
Reporter: Eduard Tudenhoefner
Assignee: Eduard Tudenhoefner
 Fix For: 2.2.11, 2.1.19


Currently *storage_port* and *ssl_storage_port* are hardcoded to the defaults. 
The problem was already fixed in CASSANDRA-13518 for C* 3.0+, so this here is 
just backporting it to C* 2.1/2.2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13837) Hanging threads in BulkLoader

2017-09-05 Thread Michael Shuler (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153868#comment-16153868
 ] 

Michael Shuler commented on CASSANDRA-13837:


This appears to be hanging up the trunk dtest jobs in random tests. We've been 
looking for a single dtest that might be problematic, but there doesn't seem to 
be a clear pattern on a particular test.

Any way we can bump the priority on this from Minor?

> Hanging threads in BulkLoader
> -
>
> Key: CASSANDRA-13837
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13837
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
>
> [~krummas] discovered some threads that were not closing correctly when he 
> fixed CASSANDRA-13836. We suspect this is due to 
> CASSANDRA-8457/CASSANDRA-12229.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13843) Debian init shadows CASSANDRA_HEAPDUMP_DIR


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Jorgensen updated CASSANDRA-13843:
-
   Labels: newbie patch  (was: )
Fix Version/s: 4.x
   3.11.x
Reproduced In: 3.11.x, 4.x
   Status: Patch Available  (was: Open)

> Debian init shadows CASSANDRA_HEAPDUMP_DIR
> --
>
> Key: CASSANDRA-13843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13843
> Project: Cassandra
>  Issue Type: Bug
>  Components: Observability
>Reporter: Andrew Jorgensen
>Priority: Minor
>  Labels: newbie, patch
> Fix For: 3.11.x, 4.x
>
> Attachments: 0001-Remove-debian-init-setting-heap-dump-file.patch
>
>
> The debian init script sets the heap dump file directly using the cassandra 
> users home directory and the -H flag to the cassandra 
> process[1|https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/debian/init#L76].
>  The cassandra heap dump location can also be set in the cassandra-env.sh 
> file using CASSANDRA_HEAPDUMP_DIR. Unfortunately the debian init heap dump 
> location is based off the home directory of the cassandra user and cannot 
> easily be changed. Also unfortunately if you do `ps aux | grep casandra` you 
> can clearly see that the -H flag takes precedent over the value found in 
> cassandra-env.sh. This makes it difficult to change the heap dump location 
> for cassandra and is non-intuitive when the value is set in cassandra-env.sh 
> why the heap dump does not actually end up in the correct place.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13843) Debian init shadows CASSANDRA_HEAPDUMP_DIR


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Jorgensen updated CASSANDRA-13843:
-
Attachment: 0001-Remove-debian-init-setting-heap-dump-file.patch

> Debian init shadows CASSANDRA_HEAPDUMP_DIR
> --
>
> Key: CASSANDRA-13843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13843
> Project: Cassandra
>  Issue Type: Bug
>  Components: Observability
>Reporter: Andrew Jorgensen
>Priority: Minor
> Attachments: 0001-Remove-debian-init-setting-heap-dump-file.patch
>
>
> The debian init script sets the heap dump file directly using the cassandra 
> users home directory and the -H flag to the cassandra 
> process[1|https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/debian/init#L76].
>  The cassandra heap dump location can also be set in the cassandra-env.sh 
> file using CASSANDRA_HEAPDUMP_DIR. Unfortunately the debian init heap dump 
> location is based off the home directory of the cassandra user and cannot 
> easily be changed. Also unfortunately if you do `ps aux | grep casandra` you 
> can clearly see that the -H flag takes precedent over the value found in 
> cassandra-env.sh. This makes it difficult to change the heap dump location 
> for cassandra and is non-intuitive when the value is set in cassandra-env.sh 
> why the heap dump does not actually end up in the correct place.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13662) Remove unsupported CREDENTIALS message


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-13662:
---
   Resolution: Fixed
Fix Version/s: (was: 4.x)
   4.0
   Status: Resolved  (was: Ready to Commit)

Thanks Jeremiah! Merged as 33ab4902a.


> Remove unsupported CREDENTIALS message
> --
>
> Key: CASSANDRA-13662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13662
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Auth, CQL
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: security
> Fix For: 4.0
>
>
> Remove CREDENTIAL message, as protocol v1 isn't supported anyways. Let's try 
> not to keep unused legacy classes around for any security relevant features. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

cassandra git commit: Remove obsoleted CredentialsMessage

Repository: cassandra
Updated Branches:
  refs/heads/trunk c1f623902 -> 33ab4902a


Remove obsoleted CredentialsMessage

patch by Stefan Podkowinski; reviewed by Jeremiah Jordan for CASSANDRA-13662


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/33ab4902
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/33ab4902
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/33ab4902

Branch: refs/heads/trunk
Commit: 33ab4902a1bef5aa662b80b89b5dd9a318f67db5
Parents: c1f6239
Author: Stefan Podkowinski 
Authored: Tue Jul 4 10:05:51 2017 +0200
Committer: Stefan Podkowinski 
Committed: Tue Sep 5 16:11:42 2017 +0200

--
 .../org/apache/cassandra/transport/Client.java  |  7 --
 .../org/apache/cassandra/transport/Frame.java   | 37 +---
 .../org/apache/cassandra/transport/Message.java |  2 +-
 .../cassandra/transport/SimpleClient.java   |  8 --
 .../transport/messages/CredentialsMessage.java  | 97 
 .../messages/UnsupportedMessageCodec.java   | 56 +++
 .../cassandra/transport/ProtocolErrorTest.java  | 21 +
 7 files changed, 102 insertions(+), 126 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/33ab4902/src/java/org/apache/cassandra/transport/Client.java
--
diff --git a/src/java/org/apache/cassandra/transport/Client.java 
b/src/java/org/apache/cassandra/transport/Client.java
index 9a76e03..7fec473 100644
--- a/src/java/org/apache/cassandra/transport/Client.java
+++ b/src/java/org/apache/cassandra/transport/Client.java
@@ -175,13 +175,6 @@ public class Client extends SimpleClient
 {
 return new OptionsMessage();
 }
-else if (msgType.equals("CREDENTIALS"))
-{
-System.err.println("[WARN] CREDENTIALS command is deprecated, use 
AUTHENTICATE instead");
-CredentialsMessage msg = new CredentialsMessage();
-msg.credentials.putAll(readCredentials(iter));
-return msg;
-}
 else if (msgType.equals("AUTHENTICATE"))
 {
 Map credentials = readCredentials(iter);

http://git-wip-us.apache.org/repos/asf/cassandra/blob/33ab4902/src/java/org/apache/cassandra/transport/Frame.java
--
diff --git a/src/java/org/apache/cassandra/transport/Frame.java 
b/src/java/org/apache/cassandra/transport/Frame.java
index 6cd8b1e..41e64f9 100644
--- a/src/java/org/apache/cassandra/transport/Frame.java
+++ b/src/java/org/apache/cassandra/transport/Frame.java
@@ -22,6 +22,8 @@ import java.io.IOException;
 import java.util.EnumSet;
 import java.util.List;
 
+import com.google.common.annotations.VisibleForTesting;
+
 import io.netty.buffer.ByteBuf;
 import io.netty.channel.*;
 import io.netty.handler.codec.ByteToMessageDecoder;
@@ -146,8 +148,8 @@ public class Frame
 this.factory = factory;
 }
 
-@Override
-protected void decode(ChannelHandlerContext ctx, ByteBuf buffer, 
List results)
+@VisibleForTesting
+Frame decodeFrame(ByteBuf buffer)
 throws Exception
 {
 if (discardingTooLongFrame)
@@ -156,12 +158,12 @@ public class Frame
 // If we have discarded everything, throw the exception
 if (bytesToDiscard <= 0)
 fail();
-return;
+return null;
 }
 
 int readableBytes = buffer.readableBytes();
 if (readableBytes == 0)
-return;
+return null;
 
 int idx = buffer.readerIndex();
 
@@ -174,7 +176,7 @@ public class Frame
 
 // Wait until we have the complete header
 if (readableBytes < Header.LENGTH)
-return;
+return null;
 
 int flags = buffer.getByte(idx++);
 EnumSet decodedFlags = Header.Flag.deserialize(flags);
@@ -210,11 +212,11 @@ public class Frame
 bytesToDiscard = discard(buffer, frameLength);
 if (bytesToDiscard <= 0)
 fail();
-return;
+return null;
 }
 
 if (buffer.readableBytes() < frameLength)
-return;
+return null;
 
 // extract body
 ByteBuf body = buffer.slice(idx, (int) bodyLength);
@@ -223,24 +225,33 @@ public class Frame
 idx += bodyLength;
 buffer.readerIndex(idx);
 
+return new Frame(new Header(version, decodedFlags, streamId, 
type), body);
+}
+
+@Override
+protected void

[jira] [Commented] (CASSANDRA-13662) Remove unsupported CREDENTIALS message

2017-09-05 Thread Jeremiah Jordan (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153700#comment-16153700
 ] 

Jeremiah Jordan commented on CASSANDRA-13662:
-

https://github.com/spodkowinski/cassandra/commit/c23d00808ac37d5d6f71810b17c7da2a2e73831b
 looks good to me.

> Remove unsupported CREDENTIALS message
> --
>
> Key: CASSANDRA-13662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13662
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Auth, CQL
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: security
> Fix For: 4.x
>
>
> Remove CREDENTIAL message, as protocol v1 isn't supported anyways. Let's try 
> not to keep unused legacy classes around for any security relevant features. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Comment Edited] (CASSANDRA-13703) Using min_compress_ratio <= 1 causes corruption


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153675#comment-16153675
 ] 

Dimitar Dimitrov edited comment on CASSANDRA-13703 at 9/5/17 1:56 PM:
--

+1 - from my still largely layman perspective, the change looks good.

A couple of small nits:
* {{CompressedChunkReader.maybeCheckCrc()}} (renamed in this patch to 
{{shouldCheckCrc()}}) may be removed or at least its visibility could be 
reduced. It seems to be used solely in CompressedChunkReader.java, lines 158 
and 204 in this patch.
* The no-argument / single-argument factory methods for Snappy and LZ4 
compressions in CompressionParams.java seem to differ in the values that they 
default to for min compression ratio and max compressed length.
** For Snappy, if nothing is specified, or only chunk length is specified, a 
default min compression ratio of 1.1 is used, and therefore max compressed 
length ends up somewhere roughly around 90% of chunk length.
** For LZ4, if nothing is specified, or only chunk length is specified, a 
default max compressed length of chunk length is used, and therefore min 
compression ratio ends up at 1.0 (I'm not sure if a precision error is possible 
there).

Edit: Of course, take this review with the appropriate rock-sized grain of salt.


was (Author: dimitarndimitrov):
+1 - from my still largely layman perspective, the change looks good.

A couple of small nits:
* {{CompressedChunkReader.maybeCheckCrc()}} (renamed in this patch to 
{{shouldCheckCrc()}}) may be removed or at least its visibility could be 
reduced. It seems to be used solely in CompressedChunkReader.java, lines 158 
and 204 in this patch.
* The no-argument / single-argument factory methods for Snappy and LZ4 
compressions in CompressionParams.java seem to differ in the values that they 
default to for min compression ratio and max compressed length.
** For Snappy, if nothing is specified, or only chunk length is specified, a 
default min compression ratio of 1.1 is used, and therefore max compressed 
length ends up somewhere roughly around 90% of chunk length.
** For LZ4, if nothing is specified, or only chunk length is specified, a 
default max compressed length of chunk length is used, and therefore min 
compression ratio ends up at 1.0 (I'm not sure if a precision error is possible 
there).

> Using min_compress_ratio <= 1 causes corruption
> ---
>
> Key: CASSANDRA-13703
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13703
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Branimir Lambov
>Assignee: Branimir Lambov
>Priority: Blocker
> Fix For: 4.x
>
> Attachments: patch
>
>
> This is because chunks written uncompressed end up below the compressed size 
> threshold. Demonstrated by applying the attached patch meant to improve the 
> testing of the 10520 changes, and running 
> {{CompressedSequentialWriterTest.testLZ4Writer}}.
> The default {{min_compress_ratio: 0}} is not affected as it never writes 
> uncompressed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13703) Using min_compress_ratio <= 1 causes corruption


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153675#comment-16153675
 ] 

Dimitar Dimitrov commented on CASSANDRA-13703:
--

+1 - from my still largely layman perspective, the change looks good.

A couple of small nits:
* {{CompressedChunkReader.maybeCheckCrc()}} (renamed in this patch to 
{{shouldCheckCrc()}}) may be removed or at least its visibility could be 
reduced. It seems to be used solely in CompressedChunkReader.java, lines 158 
and 204 in this patch.
* The no-argument / single-argument factory methods for Snappy and LZ4 
compressions in CompressionParams.java seem to differ in the values that they 
default to for min compression ratio and max compressed length.
** For Snappy, if nothing is specified, or only chunk length is specified, a 
default min compression ratio of 1.1 is used, and therefore max compressed 
length ends up somewhere roughly around 90% of chunk length.
** For LZ4, if nothing is specified, or only chunk length is specified, a 
default max compressed length of chunk length is used, and therefore min 
compression ratio ends up at 1.0 (I'm not sure if a precision error is possible 
there).

> Using min_compress_ratio <= 1 causes corruption
> ---
>
> Key: CASSANDRA-13703
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13703
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Branimir Lambov
>Assignee: Branimir Lambov
>Priority: Blocker
> Fix For: 4.x
>
> Attachments: patch
>
>
> This is because chunks written uncompressed end up below the compressed size 
> threshold. Demonstrated by applying the attached patch meant to improve the 
> testing of the 10520 changes, and running 
> {{CompressedSequentialWriterTest.testLZ4Writer}}.
> The default {{min_compress_ratio: 0}} is not affected as it never writes 
> uncompressed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13843) Debian init shadows CASSANDRA_HEAPDUMP_DIR


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Jorgensen updated CASSANDRA-13843:
-
Description: The debian init script sets the heap dump file directly using 
the cassandra users home directory and the -H flag to the cassandra 
process[1|https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/debian/init#L76].
 The cassandra heap dump location can also be set in the cassandra-env.sh file 
using CASSANDRA_HEAPDUMP_DIR. Unfortunately the debian init heap dump location 
is based off the home directory of the cassandra user and cannot easily be 
changed. Also unfortunately if you do `ps aux | grep casandra` you can clearly 
see that the -H flag takes precedent over the value found in cassandra-env.sh. 
This makes it difficult to change the heap dump location for cassandra and is 
non-intuitive when the value is set in cassandra-env.sh why the heap dump does 
not actually end up in the correct place.  (was: The debian init script sets 
the heap dump file directly using the cassandra users home directory and the -H 
flag to the cassandra process[1 
title|https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/debian/init#L76].
 The cassandra heap dump location can also be set in the cassandra-env.sh file 
using CASSANDRA_HEAPDUMP_DIR. Unfortunately the debian init heap dump location 
is based off the home directory of the cassandra user and cannot easily be 
changed. Also unfortunately if you do `ps aux | grep casandra` you can clearly 
see that the -H flag takes precedent over the value found in cassandra-env.sh. 
This makes it difficult to change the heap dump location for cassandra and is 
non-intuitive when the value is set in cassandra-env.sh why the heap dump does 
not actually end up in the correct place.)

> Debian init shadows CASSANDRA_HEAPDUMP_DIR
> --
>
> Key: CASSANDRA-13843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13843
> Project: Cassandra
>  Issue Type: Bug
>  Components: Observability
>Reporter: Andrew Jorgensen
>Priority: Minor
>
> The debian init script sets the heap dump file directly using the cassandra 
> users home directory and the -H flag to the cassandra 
> process[1|https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/debian/init#L76].
>  The cassandra heap dump location can also be set in the cassandra-env.sh 
> file using CASSANDRA_HEAPDUMP_DIR. Unfortunately the debian init heap dump 
> location is based off the home directory of the cassandra user and cannot 
> easily be changed. Also unfortunately if you do `ps aux | grep casandra` you 
> can clearly see that the -H flag takes precedent over the value found in 
> cassandra-env.sh. This makes it difficult to change the heap dump location 
> for cassandra and is non-intuitive when the value is set in cassandra-env.sh 
> why the heap dump does not actually end up in the correct place.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-13843) Debian init shadows CASSANDRA_HEAPDUMP_DIR

Andrew Jorgensen created CASSANDRA-13843:


 Summary: Debian init shadows CASSANDRA_HEAPDUMP_DIR
 Key: CASSANDRA-13843
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13843
 Project: Cassandra
  Issue Type: Bug
  Components: Observability
Reporter: Andrew Jorgensen
Priority: Minor


The debian init script sets the heap dump file directly using the cassandra 
users home directory and the -H flag to the cassandra process[1 
title|https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/debian/init#L76].
 The cassandra heap dump location can also be set in the cassandra-env.sh file 
using CASSANDRA_HEAPDUMP_DIR. Unfortunately the debian init heap dump location 
is based off the home directory of the cassandra user and cannot easily be 
changed. Also unfortunately if you do `ps aux | grep casandra` you can clearly 
see that the -H flag takes precedent over the value found in cassandra-env.sh. 
This makes it difficult to change the heap dump location for cassandra and is 
non-intuitive when the value is set in cassandra-env.sh why the heap dump does 
not actually end up in the correct place.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-12696) Allow to change logging levels based on components


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-12696:
---
Resolution: Fixed
  Reviewer: Chris Lohfink
Status: Resolved  (was: Ready to Commit)

Thanks! Merged as c1f6239. 

I don't think keeping the list of classes in sync will be that much of an issue 
in practice. I've went through the list again that I've created a year ago and 
all classes still exist and seem to be still relevant. But patching the list 
should not be an issue in case something's missing.

> Allow to change logging levels based on components
> --
>
> Key: CASSANDRA-12696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12696
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Observability
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: lhf
> Attachments: 12696-trunk.patch
>
>
> Currently users are able to dynamically change logging configuration by using 
> {{nodetool setlogginglevel  }}. Unfortunately this requires to 
> know a bit about the Cassandra package hierarchy and gathering all the 
> involved packages/classes can be tedious, especially in troubleshooting 
> situations. What I'd like to have is a way to tell a user to "_when X 
> happens, enable debug logs for bootstrapping/repair/compactions/.._" by 
> simply running e.g. {{nodetool setlogginglevel bootstrap DEBUG}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

cassandra git commit: Allow to change logging levels based on components

Repository: cassandra
Updated Branches:
  refs/heads/trunk e38015ed1 -> c1f623902


Allow to change logging levels based on components

patch by Stefan Podkowinski; reviewed by Chris Lohfink for CASSANDRA-12696


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c1f62390
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c1f62390
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c1f62390

Branch: refs/heads/trunk
Commit: c1f62390239f792f010feb07b02218062e34cee0
Parents: e38015e
Author: Stefan Podkowinski 
Authored: Fri Sep 23 10:56:43 2016 +0200
Committer: Stefan Podkowinski 
Committed: Tue Sep 5 15:16:40 2017 +0200

--
 CHANGES.txt |  1 +
 .../tools/nodetool/SetLoggingLevel.java | 68 ++--
 2 files changed, 65 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c1f62390/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 845fced..3fd8ab0 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 4.0
+ * Allow changing log levels via nodetool for related classes (CASSANDRA-12696)
  * Add stress profile yaml with LWT (CASSANDRA-7960)
  * Reduce memory copies and object creations when acting on ByteBufs 
(CASSANDRA-13789)
  * simplify mx4j configuration (Cassandra-13578)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c1f62390/src/java/org/apache/cassandra/tools/nodetool/SetLoggingLevel.java
--
diff --git a/src/java/org/apache/cassandra/tools/nodetool/SetLoggingLevel.java 
b/src/java/org/apache/cassandra/tools/nodetool/SetLoggingLevel.java
index 94e4e3b..8d9ad90 100644
--- a/src/java/org/apache/cassandra/tools/nodetool/SetLoggingLevel.java
+++ b/src/java/org/apache/cassandra/tools/nodetool/SetLoggingLevel.java
@@ -22,22 +22,82 @@ import io.airlift.airline.Arguments;
 import io.airlift.airline.Command;
 
 import java.util.ArrayList;
+import java.util.Collections;
 import java.util.List;
 
+import com.google.common.collect.Lists;
+
 import org.apache.cassandra.tools.NodeProbe;
 import org.apache.cassandra.tools.NodeTool.NodeToolCmd;
 
-@Command(name = "setlogginglevel", description = "Set the log level threshold 
for a given class. If both class and level are empty/null, it will reset to the 
initial configuration")
+@Command(name = "setlogginglevel", description = "Set the log level threshold 
for a given component or class. Will reset to the initial configuration if 
called with no parameters.")
 public class SetLoggingLevel extends NodeToolCmd
 {
-@Arguments(usage = " ", description = "The class to change 
the level for and the log level threshold to set (can be empty)")
+@Arguments(usage = " ", description = "The 
component or class to change the level for and the log level threshold to set. 
Will reset to initial level if omitted. "
++ "Available components:  bootstrap, compaction, repair, streaming, 
cql, ring")
 private List args = new ArrayList<>();
 
 @Override
 public void execute(NodeProbe probe)
 {
-String classQualifier = args.size() >= 1 ? args.get(0) : EMPTY;
+String target = args.size() >= 1 ? args.get(0) : EMPTY;
 String level = args.size() == 2 ? args.get(1) : EMPTY;
-probe.setLoggingLevel(classQualifier, level);
+
+List classQualifiers = Collections.singletonList(target);
+if (target.equals("bootstrap"))
+{
+classQualifiers = Lists.newArrayList(
+"org.apache.cassandra.gms",
+"org.apache.cassandra.hints",
+"org.apache.cassandra.schema",
+"org.apache.cassandra.service.StorageService",
+"org.apache.cassandra.db.SystemKeyspace",
+"org.apache.cassandra.batchlog.BatchlogManager",
+"org.apache.cassandra.net.MessagingService");
+}
+else if (target.equals("repair"))
+{
+classQualifiers = Lists.newArrayList(
+"org.apache.cassandra.repair",
+"org.apache.cassandra.db.compaction.CompactionManager",
+"org.apache.cassandra.service.SnapshotVerbHandler");
+}
+else if (target.equals("streaming"))
+{
+classQualifiers = Lists.newArrayList(
+"org.apache.cassandra.streaming",
+"org.apache.cassandra.dht.RangeStreamer");
+}
+else if (target.equals("compaction"))
+{
+classQualifiers = Lists.newArrayList(
+

[jira] [Commented] (CASSANDRA-13787) RangeTombstoneMarker and ParitionDeletion is not properly included in MV

2017-09-05 Thread ZhaoYang (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153603#comment-16153603
 ] 

ZhaoYang commented on CASSANDRA-13787:
--

Thanks for reviewing.

I will prepare branches for 3.0/3.1/trunk. I have removed the code for second 
issue since 11500 is merged..

> RangeTombstoneMarker and ParitionDeletion is not properly included in MV
> 
>
> Key: CASSANDRA-13787
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13787
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths, Materialized Views
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>
> Found two problems related to MV tombstone. 
> 1. Range-tombstone-Marker being ignored after shadowing first row, subsequent 
> base rows are not shadowed in TableViews.
> If the range tombstone was not flushed, it was used as deleted row to 
> shadow new updates. It works correctly.
> After range tombstone was flushed, it was used as RangeTombstoneMarker 
> and being skipped after shadowing first update. The bound of 
> RangeTombstoneMarker seems wrong, it contained full clustering, but it should 
> contain range or it should be multiple RangeTombstoneMarkers for multiple 
> slices(aka. new updates)
> -2. Partition tombstone is not used when no existing live data, it will 
> resurrect deleted cells. It was found in 11500 and included in that patch.- 
> (Merged in CASSANDRA-11500)
> In order not to make 11500 patch more complicated, I will try fix 
> range/partition tombstone issue here.
> {code:title=Tests to reproduce}
> @Test
> public void testExistingRangeTombstoneWithFlush() throws Throwable
> {
> testExistingRangeTombstone(true);
> }
> @Test
> public void testExistingRangeTombstoneWithoutFlush() throws Throwable
> {
> testExistingRangeTombstone(false);
> }
> public void testExistingRangeTombstone(boolean flush) throws Throwable
> {
> createTable("CREATE TABLE %s (k1 int, c1 int, c2 int, v1 int, v2 int, 
> PRIMARY KEY (k1, c1, c2))");
> execute("USE " + keyspace());
> executeNet(protocolVersion, "USE " + keyspace());
> createView("view1",
>"CREATE MATERIALIZED VIEW view1 AS SELECT * FROM %%s WHERE 
> k1 IS NOT NULL AND c1 IS NOT NULL AND c2 IS NOT NULL PRIMARY KEY (k1, c2, 
> c1)");
> updateView("DELETE FROM %s USING TIMESTAMP 10 WHERE k1 = 1 and c1=1");
> if (flush)
> 
> Keyspace.open(keyspace()).getColumnFamilyStore(currentTable()).forceBlockingFlush();
> String table = KEYSPACE + "." + currentTable();
> updateView("BEGIN BATCH " +
> "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 0, 
> 0, 0, 0) USING TIMESTAMP 5; " +
> "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 0, 
> 1, 0, 1) USING TIMESTAMP 5; " +
> "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, 
> 0, 1, 0) USING TIMESTAMP 5; " +
> "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, 
> 1, 1, 1) USING TIMESTAMP 5; " +
> "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, 
> 2, 1, 2) USING TIMESTAMP 5; " +
> "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, 
> 3, 1, 3) USING TIMESTAMP 5; " +
> "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 2, 
> 0, 2, 0) USING TIMESTAMP 5; " +
> "APPLY BATCH");
> assertRowsIgnoringOrder(execute("select * from %s"),
> row(1, 0, 0, 0, 0),
> row(1, 0, 1, 0, 1),
> row(1, 2, 0, 2, 0));
> assertRowsIgnoringOrder(execute("select k1,c1,c2,v1,v2 from view1"),
> row(1, 0, 0, 0, 0),
> row(1, 0, 1, 0, 1),
> row(1, 2, 0, 2, 0));
> }
> @Test
> public void testExistingParitionDeletionWithFlush() throws Throwable
> {
> testExistingParitionDeletion(true);
> }
> @Test
> public void testExistingParitionDeletionWithoutFlush() throws Throwable
> {
> testExistingParitionDeletion(false);
> }
> public void testExistingParitionDeletion(boolean flush) throws Throwable
> {
> // for partition range deletion, need to know that existing row is 
> shadowed instead of not existed.
> createTable("CREATE TABLE %s (a int, b int, c int, d int, PRIMARY KEY 
> (a))");
> execute("USE " + keyspace());
> executeNet(protocolVersion, "USE " + keyspace());
> createView("mv_test1",
>"CREATE MATERIALIZED VIEW %s AS SELECT * FROM %%s WHERE a 
> IS NOT

[jira] [Commented] (CASSANDRA-13787) RangeTombstoneMarker and ParitionDeletion is not properly included in MV

2017-09-05 Thread Sylvain Lebresne (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153596#comment-16153596
 ] 

Sylvain Lebresne commented on CASSANDRA-13787:
--

Sorry for the delay reviewing, but patch lgtm (though, to make extract sure 
we're on the same page, this appear to only include the fix for the "first 
problem" since the "second problem" seems to have been included in 
CASSANDRA-11500).

Have you run CI on this? Also, if you could provide branches for all branches, 
that would be amazing (the fix itself should merge up cleanly, but I suspect 
the tests may require a few minor updates).

> RangeTombstoneMarker and ParitionDeletion is not properly included in MV
> 
>
> Key: CASSANDRA-13787
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13787
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths, Materialized Views
>Reporter: ZhaoYang
>Assignee: ZhaoYang
>
> Found two problems related to MV tombstone. 
> 1. Range-tombstone-Marker being ignored after shadowing first row, subsequent 
> base rows are not shadowed in TableViews.
> If the range tombstone was not flushed, it was used as deleted row to 
> shadow new updates. It works correctly.
> After range tombstone was flushed, it was used as RangeTombstoneMarker 
> and being skipped after shadowing first update. The bound of 
> RangeTombstoneMarker seems wrong, it contained full clustering, but it should 
> contain range or it should be multiple RangeTombstoneMarkers for multiple 
> slices(aka. new updates)
> -2. Partition tombstone is not used when no existing live data, it will 
> resurrect deleted cells. It was found in 11500 and included in that patch.- 
> (Merged in CASSANDRA-11500)
> In order not to make 11500 patch more complicated, I will try fix 
> range/partition tombstone issue here.
> {code:title=Tests to reproduce}
> @Test
> public void testExistingRangeTombstoneWithFlush() throws Throwable
> {
> testExistingRangeTombstone(true);
> }
> @Test
> public void testExistingRangeTombstoneWithoutFlush() throws Throwable
> {
> testExistingRangeTombstone(false);
> }
> public void testExistingRangeTombstone(boolean flush) throws Throwable
> {
> createTable("CREATE TABLE %s (k1 int, c1 int, c2 int, v1 int, v2 int, 
> PRIMARY KEY (k1, c1, c2))");
> execute("USE " + keyspace());
> executeNet(protocolVersion, "USE " + keyspace());
> createView("view1",
>"CREATE MATERIALIZED VIEW view1 AS SELECT * FROM %%s WHERE 
> k1 IS NOT NULL AND c1 IS NOT NULL AND c2 IS NOT NULL PRIMARY KEY (k1, c2, 
> c1)");
> updateView("DELETE FROM %s USING TIMESTAMP 10 WHERE k1 = 1 and c1=1");
> if (flush)
> 
> Keyspace.open(keyspace()).getColumnFamilyStore(currentTable()).forceBlockingFlush();
> String table = KEYSPACE + "." + currentTable();
> updateView("BEGIN BATCH " +
> "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 0, 
> 0, 0, 0) USING TIMESTAMP 5; " +
> "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 0, 
> 1, 0, 1) USING TIMESTAMP 5; " +
> "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, 
> 0, 1, 0) USING TIMESTAMP 5; " +
> "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, 
> 1, 1, 1) USING TIMESTAMP 5; " +
> "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, 
> 2, 1, 2) USING TIMESTAMP 5; " +
> "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, 
> 3, 1, 3) USING TIMESTAMP 5; " +
> "INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 2, 
> 0, 2, 0) USING TIMESTAMP 5; " +
> "APPLY BATCH");
> assertRowsIgnoringOrder(execute("select * from %s"),
> row(1, 0, 0, 0, 0),
> row(1, 0, 1, 0, 1),
> row(1, 2, 0, 2, 0));
> assertRowsIgnoringOrder(execute("select k1,c1,c2,v1,v2 from view1"),
> row(1, 0, 0, 0, 0),
> row(1, 0, 1, 0, 1),
> row(1, 2, 0, 2, 0));
> }
> @Test
> public void testExistingParitionDeletionWithFlush() throws Throwable
> {
> testExistingParitionDeletion(true);
> }
> @Test
> public void testExistingParitionDeletionWithoutFlush() throws Throwable
> {
> testExistingParitionDeletion(false);
> }
> public void testExistingParitionDeletion(boolean flush) throws Throwable
> {
> // for partition range deletion, need to know that existing row is 
> shadowed instead of

[jira] [Comment Edited] (CASSANDRA-13831) NettyFactoryTest is failing in trunk on MacOS


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153542#comment-16153542
 ] 

Jason Brown edited comment on CASSANDRA-13831 at 9/5/17 12:10 PM:
--

rebased and pushed a change that adds an additional check to see if the OS is 
linux, and only then log that epoll is unavailable.

UPDATE: just saw your comment about {{NativeTransportService}}. Will move the 
logging there, from {{NettyFactory}}


was (Author: jasobrown):
rebased and pushed a change that adds an additional check to see if the OS is 
linux, and only then log that epoll is unavailable.

> NettyFactoryTest is failing in trunk on MacOS
> -
>
> Key: CASSANDRA-13831
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13831
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Aleksey Yeschenko
>Assignee: Jason Brown
>Priority: Minor
> Fix For: 4.x
>
>
> Example failure:
> {code}
> [junit] Testcase: 
> getEventLoopGroup_EpollWithoutIoRatioBoost(org.apache.cassandra.net.async.NettyFactoryTest):
> Caused an ERROR
> [junit] failed to load the required native library
> [junit] java.lang.UnsatisfiedLinkError: failed to load the required 
> native library
> [junit]   at 
> io.netty.channel.epoll.Epoll.ensureAvailability(Epoll.java:78)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoop.(EpollEventLoop.java:53)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:134)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35)
> [junit]   at 
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:84)
> [junit]   at 
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:58)
> [junit]   at 
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:47)
> [junit]   at 
> io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:59)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:104)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:91)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:68)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactory.getEventLoopGroup(NettyFactory.java:175)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_Epoll(NettyFactoryTest.java:187)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_EpollWithoutIoRatioBoost(NettyFactoryTest.java:205)
> [junit] Caused by: java.lang.ExceptionInInitializerError
> [junit]   at io.netty.channel.epoll.Epoll.(Epoll.java:33)
> [junit]   at 
> org.apache.cassandra.service.NativeTransportService.useEpoll(NativeTransportService.java:162)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactory.(NettyFactory.java:94)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_Nio(NettyFactoryTest.java:216)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_NioWithoutIoRatioBoost(NettyFactoryTest.java:211)
> [junit] Caused by: java.lang.IllegalStateException: Only supported on 
> Linux
> [junit]   at 
> io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:189)
> [junit]   at io.netty.channel.epoll.Native.(Native.java:61)
> {code}
> It's obviously caused by epoll being unavailable on MacOS, but the tests 
> should still be made to pass.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13831) NettyFactoryTest is failing in trunk on MacOS


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153542#comment-16153542
 ] 

Jason Brown commented on CASSANDRA-13831:
-

rebased and pushed a change that adds an additional check to see if the OS is 
linux, and only then log that epoll is unavailable.

> NettyFactoryTest is failing in trunk on MacOS
> -
>
> Key: CASSANDRA-13831
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13831
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Aleksey Yeschenko
>Assignee: Jason Brown
>Priority: Minor
> Fix For: 4.x
>
>
> Example failure:
> {code}
> [junit] Testcase: 
> getEventLoopGroup_EpollWithoutIoRatioBoost(org.apache.cassandra.net.async.NettyFactoryTest):
> Caused an ERROR
> [junit] failed to load the required native library
> [junit] java.lang.UnsatisfiedLinkError: failed to load the required 
> native library
> [junit]   at 
> io.netty.channel.epoll.Epoll.ensureAvailability(Epoll.java:78)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoop.(EpollEventLoop.java:53)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:134)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35)
> [junit]   at 
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:84)
> [junit]   at 
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:58)
> [junit]   at 
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:47)
> [junit]   at 
> io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:59)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:104)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:91)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:68)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactory.getEventLoopGroup(NettyFactory.java:175)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_Epoll(NettyFactoryTest.java:187)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_EpollWithoutIoRatioBoost(NettyFactoryTest.java:205)
> [junit] Caused by: java.lang.ExceptionInInitializerError
> [junit]   at io.netty.channel.epoll.Epoll.(Epoll.java:33)
> [junit]   at 
> org.apache.cassandra.service.NativeTransportService.useEpoll(NativeTransportService.java:162)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactory.(NettyFactory.java:94)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_Nio(NettyFactoryTest.java:216)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_NioWithoutIoRatioBoost(NettyFactoryTest.java:211)
> [junit] Caused by: java.lang.IllegalStateException: Only supported on 
> Linux
> [junit]   at 
> io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:189)
> [junit]   at io.netty.channel.epoll.Native.(Native.java:61)
> {code}
> It's obviously caused by epoll being unavailable on MacOS, but the tests 
> should still be made to pass.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13831) NettyFactoryTest is failing in trunk on MacOS


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153534#comment-16153534
 ] 

Jason Brown commented on CASSANDRA-13831:
-

Printing the error is useful on linux to know why epoll might not be available, 
but I agree it's not necessary on other platforms. Fixing.

> NettyFactoryTest is failing in trunk on MacOS
> -
>
> Key: CASSANDRA-13831
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13831
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Aleksey Yeschenko
>Assignee: Jason Brown
>Priority: Minor
> Fix For: 4.x
>
>
> Example failure:
> {code}
> [junit] Testcase: 
> getEventLoopGroup_EpollWithoutIoRatioBoost(org.apache.cassandra.net.async.NettyFactoryTest):
> Caused an ERROR
> [junit] failed to load the required native library
> [junit] java.lang.UnsatisfiedLinkError: failed to load the required 
> native library
> [junit]   at 
> io.netty.channel.epoll.Epoll.ensureAvailability(Epoll.java:78)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoop.(EpollEventLoop.java:53)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:134)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35)
> [junit]   at 
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:84)
> [junit]   at 
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:58)
> [junit]   at 
> io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:47)
> [junit]   at 
> io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:59)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:104)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:91)
> [junit]   at 
> io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:68)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactory.getEventLoopGroup(NettyFactory.java:175)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_Epoll(NettyFactoryTest.java:187)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_EpollWithoutIoRatioBoost(NettyFactoryTest.java:205)
> [junit] Caused by: java.lang.ExceptionInInitializerError
> [junit]   at io.netty.channel.epoll.Epoll.(Epoll.java:33)
> [junit]   at 
> org.apache.cassandra.service.NativeTransportService.useEpoll(NativeTransportService.java:162)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactory.(NettyFactory.java:94)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_Nio(NettyFactoryTest.java:216)
> [junit]   at 
> org.apache.cassandra.net.async.NettyFactoryTest.getEventLoopGroup_NioWithoutIoRatioBoost(NettyFactoryTest.java:211)
> [junit] Caused by: java.lang.IllegalStateException: Only supported on 
> Linux
> [junit]   at 
> io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:189)
> [junit]   at io.netty.channel.epoll.Native.(Native.java:61)
> {code}
> It's obviously caused by epoll being unavailable on MacOS, but the tests 
> should still be made to pass.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-8457) nio MessagingService


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153528#comment-16153528
 ] 

Jason Brown commented on CASSANDRA-8457:


[~spo...@gmail.com] ahh! you are correct. ninja-fixed as sha 
{{e38015ed1fcf7d95694683601b121c918c5f6976}}

> nio MessagingService
> 
>
> Key: CASSANDRA-8457
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8457
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Jonathan Ellis
>Assignee: Jason Brown
>Priority: Minor
>  Labels: netty, performance
> Fix For: 4.0
>
> Attachments: 8457-load.tgz
>
>
> Thread-per-peer (actually two each incoming and outbound) is a big 
> contributor to context switching, especially for larger clusters.  Let's look 
> at switching to nio, possibly via Netty.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

cassandra git commit: ninja-fix build.xml netty version to 4.1.14 as jar is already updated in lib dir

2017-09-05 Thread jasobrown

Repository: cassandra
Updated Branches:
  refs/heads/trunk 07c90c436 -> e38015ed1


ninja-fix build.xml netty version to 4.1.14 as jar is already updated in lib dir


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e38015ed
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e38015ed
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e38015ed

Branch: refs/heads/trunk
Commit: e38015ed1fcf7d95694683601b121c918c5f6976
Parents: 07c90c4
Author: Jason Brown 
Authored: Tue Sep 5 04:52:43 2017 -0700
Committer: Jason Brown 
Committed: Tue Sep 5 04:52:43 2017 -0700

--
 build.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e38015ed/build.xml
--
diff --git a/build.xml b/build.xml
index ee22921..63518c5 100644
--- a/build.xml
+++ b/build.xml
@@ -420,7 +420,7 @@
   
   
   
-  
+  
   
   
   


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13418) Allow TWCS to ignore overlaps when dropping fully expired sstables

2017-09-05 Thread Romain GERARD (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153498#comment-16153498
 ] 

Romain GERARD commented on CASSANDRA-13418:
---

Champagne ! 
Thanks for the commitment :)

> Allow TWCS to ignore overlaps when dropping fully expired sstables
> --
>
> Key: CASSANDRA-13418
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13418
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Corentin Chary
>Assignee: Romain GERARD
>  Labels: twcs
> Fix For: 3.11.1, 4.0
>
> Attachments: twcs-cleanup.png
>
>
> http://thelastpickle.com/blog/2016/12/08/TWCS-part1.html explains it well. If 
> you really want read-repairs you're going to have sstables blocking the 
> expiration of other fully expired SSTables because they overlap.
> You can set unchecked_tombstone_compaction = true or tombstone_threshold to a 
> very low value and that will purge the blockers of old data that should 
> already have expired, thus removing the overlaps and allowing the other 
> SSTables to expire.
> The thing is that this is rather CPU intensive and not optimal. If you have 
> time series, you might not care if all your data doesn't exactly expire at 
> the right time, or if data re-appears for some time, as long as it gets 
> deleted as soon as it can. And in this situation I believe it would be really 
> beneficial to allow users to simply ignore overlapping SSTables when looking 
> for fully expired ones.
> To the question: why would you need read-repairs ?
> - Full repairs basically take longer than the TTL of the data on my dataset, 
> so this isn't really effective.
> - Even with a 10% chances of doing a repair, we found out that this would be 
> enough to greatly reduce entropy of the most used data (and if you have 
> timeseries, you're likely to have a dashboard doing the same important 
> queries over and over again).
> - LOCAL_QUORUM is too expensive (need >3 replicas), QUORUM is too slow.
> I'll try to come up with a patch demonstrating how this would work, try it on 
> our system and report the effects.
> cc: [~adejanovski], [~rgerard] as I know you worked on similar issues already.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Assigned] (CASSANDRA-13842) dtest failure: batch_test.TestBatch.batchlog_replay_compatibility_?_test

2017-09-05 Thread Aleksey Yeschenko (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko reassigned CASSANDRA-13842:
-

Assignee: Aleksey Yeschenko

> dtest failure: batch_test.TestBatch.batchlog_replay_compatibility_?_test
> 
>
> Key: CASSANDRA-13842
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13842
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Aleksey Yeschenko
>
> batch_test.TestBatch.batchlog_replay_compatibility_1_test and 
> batch_test.TestBatch.batchlog_replay_compatibility_4_test are failing:
> http://cassci.datastax.com/view/cassandra-3.11/job/cassandra-3.11_dtest/160/testReport/batch_test/TestBatch/batchlog_replay_compatibility_1_test/
> http://cassci.datastax.com/view/cassandra-3.11/job/cassandra-3.11_dtest/160/testReport/batch_test/TestBatch/batchlog_replay_compatibility_4_test/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13842) dtest failure: batch_test.TestBatch.batchlog_replay_compatibility_?_test

2017-09-05 Thread Aleksey Yeschenko (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153492#comment-16153492
 ] 

Aleksey Yeschenko commented on CASSANDRA-13842:
---

The test passes 100% reliably on MacOS, but it does fail for me in my Ubuntu VM 
just as it does with Jenkins.

Don't yet know why the difference is there, but I'll deal with it.

> dtest failure: batch_test.TestBatch.batchlog_replay_compatibility_?_test
> 
>
> Key: CASSANDRA-13842
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13842
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>
> batch_test.TestBatch.batchlog_replay_compatibility_1_test and 
> batch_test.TestBatch.batchlog_replay_compatibility_4_test are failing:
> http://cassci.datastax.com/view/cassandra-3.11/job/cassandra-3.11_dtest/160/testReport/batch_test/TestBatch/batchlog_replay_compatibility_1_test/
> http://cassci.datastax.com/view/cassandra-3.11/job/cassandra-3.11_dtest/160/testReport/batch_test/TestBatch/batchlog_replay_compatibility_4_test/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13808) Integer overflows with Amazon Elastic File System (EFS)

2017-09-05 Thread Vitalii Ishchenko (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vitalii Ishchenko updated CASSANDRA-13808:
--
Description: 
Integer overflow issue was fixed for cassandra 2.2, but in 3.8 new property was 
introduced in config that also derives from disk size  
{{cdc_total_space_in_mb}}, see CASSANDRA-8844

It should be updated too 
https://github.com/apache/cassandra/blob/6b7d73a49695c0ceb78bc7a003ace606a806c13a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java#L484

  was:
Integer overflow issue was fixed for cassandra 2.2, but since then new property 
was introduced in config that also derives from disk size  
{{cdc_total_space_in_mb}}

It should be updated too 
https://github.com/apache/cassandra/blob/6b7d73a49695c0ceb78bc7a003ace606a806c13a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java#L484


> Integer overflows with Amazon Elastic File System (EFS)
> ---
>
> Key: CASSANDRA-13808
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13808
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Vitalii Ishchenko
>Assignee: Benjamin Lerer
> Fix For: 3.11.1
>
>
> Integer overflow issue was fixed for cassandra 2.2, but in 3.8 new property 
> was introduced in config that also derives from disk size  
> {{cdc_total_space_in_mb}}, see CASSANDRA-8844
> It should be updated too 
> https://github.com/apache/cassandra/blob/6b7d73a49695c0ceb78bc7a003ace606a806c13a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java#L484



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13808) Integer overflows with Amazon Elastic File System (EFS)

2017-09-05 Thread Vitalii Ishchenko (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vitalii Ishchenko updated CASSANDRA-13808:
--
Fix Version/s: 3.11.1

> Integer overflows with Amazon Elastic File System (EFS)
> ---
>
> Key: CASSANDRA-13808
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13808
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Vitalii Ishchenko
>Assignee: Benjamin Lerer
> Fix For: 3.11.1
>
>
> Integer overflow issue was fixed for cassandra 2.2, but since then new 
> property was introduced in config that also derives from disk size  
> {{cdc_total_space_in_mb}}
> It should be updated too 
> https://github.com/apache/cassandra/blob/6b7d73a49695c0ceb78bc7a003ace606a806c13a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java#L484



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13263) Incorrect ComplexColumnData hashCode implementation


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-13263:
---
   Resolution: Fixed
Fix Version/s: 4.0
   3.11.1
   3.0.15
   Status: Resolved  (was: Patch Available)

I'm pretty sure that fixing a hashCode function that isn't even used yet will 
cause much harm. Merged as 1e80e35 for 3.0 upwards with your suggested changes. 
Thanks for reviewing.

> Incorrect ComplexColumnData hashCode implementation
> ---
>
> Key: CASSANDRA-13263
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13263
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
> Fix For: 3.0.15, 3.11.1, 4.0
>
>
> I went through some of the logs from CASSANDRA-13175 and one of the more 
> serious issues that we should address seem to be the 
> {{ComplexColumnData.hashCode()}} implementation. As Objects.hashCode is not 
> using deep hashing for array arguments, hashed will be based on the identity 
> instead of the array's content. See patch for simple fix.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[1/6] cassandra git commit: Fix ComplexColumnData hashCode implementation

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 fb92a5262 -> 1e80e3539
  refs/heads/cassandra-3.11 5a09fafea -> 03155b021
  refs/heads/trunk 9717cd67b -> 07c90c436


Fix ComplexColumnData hashCode implementation

patch by Stefan Podkowinski; reviewed by Sylvain Lebresne for CASSANDRA-13263


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1e80e353
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1e80e353
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1e80e353

Branch: refs/heads/cassandra-3.0
Commit: 1e80e35396e701bde97594bd49bb730656e02ce8
Parents: fb92a52
Author: Stefan Podkowinski 
Authored: Fri Feb 24 12:52:17 2017 +0100
Committer: Stefan Podkowinski 
Committed: Tue Sep 5 12:48:48 2017 +0200

--
 src/java/org/apache/cassandra/db/rows/ComplexColumnData.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1e80e353/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java
--
diff --git a/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java 
b/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java
index 2a8e843..e768769 100644
--- a/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java
+++ b/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java
@@ -218,7 +218,7 @@ public class ComplexColumnData extends ColumnData 
implements Iterable
 @Override
 public int hashCode()
 {
-return Objects.hash(column(), complexDeletion(), cells);
+return Objects.hash(column(), complexDeletion(), 
BTree.hashCode(cells));
 }
 
 public static Builder builder()


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/03155b02
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/03155b02
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/03155b02

Branch: refs/heads/trunk
Commit: 03155b021a1232a0ef852ed836b68cde5175094f
Parents: 5a09faf 1e80e35
Author: Stefan Podkowinski 
Authored: Tue Sep 5 12:51:08 2017 +0200
Committer: Stefan Podkowinski 
Committed: Tue Sep 5 12:52:17 2017 +0200

--
 src/java/org/apache/cassandra/db/rows/ComplexColumnData.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/03155b02/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/03155b02
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/03155b02
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/03155b02

Branch: refs/heads/cassandra-3.11
Commit: 03155b021a1232a0ef852ed836b68cde5175094f
Parents: 5a09faf 1e80e35
Author: Stefan Podkowinski 
Authored: Tue Sep 5 12:51:08 2017 +0200
Committer: Stefan Podkowinski 
Committed: Tue Sep 5 12:52:17 2017 +0200

--
 src/java/org/apache/cassandra/db/rows/ComplexColumnData.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/03155b02/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/07c90c43
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/07c90c43
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/07c90c43

Branch: refs/heads/trunk
Commit: 07c90c4360c8a1558810013ddd1a26d10af15084
Parents: 9717cd6 03155b0
Author: Stefan Podkowinski 
Authored: Tue Sep 5 12:52:37 2017 +0200
Committer: Stefan Podkowinski 
Committed: Tue Sep 5 12:52:52 2017 +0200

--
 src/java/org/apache/cassandra/db/rows/ComplexColumnData.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/07c90c43/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java
--


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[3/6] cassandra git commit: Fix ComplexColumnData hashCode implementation

Fix ComplexColumnData hashCode implementation

patch by Stefan Podkowinski; reviewed by Sylvain Lebresne for CASSANDRA-13263


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1e80e353
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1e80e353
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1e80e353

Branch: refs/heads/trunk
Commit: 1e80e35396e701bde97594bd49bb730656e02ce8
Parents: fb92a52
Author: Stefan Podkowinski 
Authored: Fri Feb 24 12:52:17 2017 +0100
Committer: Stefan Podkowinski 
Committed: Tue Sep 5 12:48:48 2017 +0200

--
 src/java/org/apache/cassandra/db/rows/ComplexColumnData.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1e80e353/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java
--
diff --git a/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java 
b/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java
index 2a8e843..e768769 100644
--- a/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java
+++ b/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java
@@ -218,7 +218,7 @@ public class ComplexColumnData extends ColumnData 
implements Iterable
 @Override
 public int hashCode()
 {
-return Objects.hash(column(), complexDeletion(), cells);
+return Objects.hash(column(), complexDeletion(), 
BTree.hashCode(cells));
 }
 
 public static Builder builder()


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[2/6] cassandra git commit: Fix ComplexColumnData hashCode implementation

Fix ComplexColumnData hashCode implementation

patch by Stefan Podkowinski; reviewed by Sylvain Lebresne for CASSANDRA-13263


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1e80e353
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1e80e353
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1e80e353

Branch: refs/heads/cassandra-3.11
Commit: 1e80e35396e701bde97594bd49bb730656e02ce8
Parents: fb92a52
Author: Stefan Podkowinski 
Authored: Fri Feb 24 12:52:17 2017 +0100
Committer: Stefan Podkowinski 
Committed: Tue Sep 5 12:48:48 2017 +0200

--
 src/java/org/apache/cassandra/db/rows/ComplexColumnData.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/1e80e353/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java
--
diff --git a/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java 
b/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java
index 2a8e843..e768769 100644
--- a/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java
+++ b/src/java/org/apache/cassandra/db/rows/ComplexColumnData.java
@@ -218,7 +218,7 @@ public class ComplexColumnData extends ColumnData 
implements Iterable
 @Override
 public int hashCode()
 {
-return Objects.hash(column(), complexDeletion(), cells);
+return Objects.hash(column(), complexDeletion(), 
BTree.hashCode(cells));
 }
 
 public static Builder builder()


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13662) Remove unsupported CREDENTIALS message


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153458#comment-16153458
 ] 

Stefan Podkowinski commented on CASSANDRA-13662:


I gave this another look and came up with a more reasonable test, but had to 
refactor Frame a bit in the process. Changes should be non-invasive enough for 
4.0, wdyt?

* [branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13662]
* [testall|https://circleci.com/gh/spodkowinski/cassandra/119]


> Remove unsupported CREDENTIALS message
> --
>
> Key: CASSANDRA-13662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13662
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Auth, CQL
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: security
> Fix For: 4.x
>
>
> Remove CREDENTIAL message, as protocol v1 isn't supported anyways. Let's try 
> not to keep unused legacy classes around for any security relevant features. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13787) RangeTombstoneMarker and ParitionDeletion is not properly included in MV

2017-09-05 Thread ZhaoYang (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ZhaoYang updated CASSANDRA-13787:
-
Description: 
Found two problems related to MV tombstone. 

1. Range-tombstone-Marker being ignored after shadowing first row, subsequent 
base rows are not shadowed in TableViews.

If the range tombstone was not flushed, it was used as deleted row to 
shadow new updates. It works correctly.
After range tombstone was flushed, it was used as RangeTombstoneMarker and 
being skipped after shadowing first update. The bound of RangeTombstoneMarker 
seems wrong, it contained full clustering, but it should contain range or it 
should be multiple RangeTombstoneMarkers for multiple slices(aka. new updates)

-2. Partition tombstone is not used when no existing live data, it will 
resurrect deleted cells. It was found in 11500 and included in that patch.- 
(Merged in CASSANDRA-11500)


In order not to make 11500 patch more complicated, I will try fix 
range/partition tombstone issue here.


{code:title=Tests to reproduce}
@Test
public void testExistingRangeTombstoneWithFlush() throws Throwable
{
testExistingRangeTombstone(true);
}

@Test
public void testExistingRangeTombstoneWithoutFlush() throws Throwable
{
testExistingRangeTombstone(false);
}

public void testExistingRangeTombstone(boolean flush) throws Throwable
{
createTable("CREATE TABLE %s (k1 int, c1 int, c2 int, v1 int, v2 int, 
PRIMARY KEY (k1, c1, c2))");

execute("USE " + keyspace());
executeNet(protocolVersion, "USE " + keyspace());

createView("view1",
   "CREATE MATERIALIZED VIEW view1 AS SELECT * FROM %%s WHERE 
k1 IS NOT NULL AND c1 IS NOT NULL AND c2 IS NOT NULL PRIMARY KEY (k1, c2, c1)");

updateView("DELETE FROM %s USING TIMESTAMP 10 WHERE k1 = 1 and c1=1");


if (flush)

Keyspace.open(keyspace()).getColumnFamilyStore(currentTable()).forceBlockingFlush();

String table = KEYSPACE + "." + currentTable();
updateView("BEGIN BATCH " +
"INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 0, 
0, 0, 0) USING TIMESTAMP 5; " +
"INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 0, 
1, 0, 1) USING TIMESTAMP 5; " +
"INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, 
0, 1, 0) USING TIMESTAMP 5; " +
"INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, 
1, 1, 1) USING TIMESTAMP 5; " +
"INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, 
2, 1, 2) USING TIMESTAMP 5; " +
"INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 1, 
3, 1, 3) USING TIMESTAMP 5; " +
"INSERT INTO " + table + " (k1, c1, c2, v1, v2) VALUES (1, 2, 
0, 2, 0) USING TIMESTAMP 5; " +
"APPLY BATCH");

assertRowsIgnoringOrder(execute("select * from %s"),
row(1, 0, 0, 0, 0),
row(1, 0, 1, 0, 1),
row(1, 2, 0, 2, 0));
assertRowsIgnoringOrder(execute("select k1,c1,c2,v1,v2 from view1"),
row(1, 0, 0, 0, 0),
row(1, 0, 1, 0, 1),
row(1, 2, 0, 2, 0));
}

@Test
public void testExistingParitionDeletionWithFlush() throws Throwable
{
testExistingParitionDeletion(true);
}

@Test
public void testExistingParitionDeletionWithoutFlush() throws Throwable
{
testExistingParitionDeletion(false);
}

public void testExistingParitionDeletion(boolean flush) throws Throwable
{
// for partition range deletion, need to know that existing row is 
shadowed instead of not existed.
createTable("CREATE TABLE %s (a int, b int, c int, d int, PRIMARY KEY 
(a))");

execute("USE " + keyspace());
executeNet(protocolVersion, "USE " + keyspace());

createView("mv_test1",
   "CREATE MATERIALIZED VIEW %s AS SELECT * FROM %%s WHERE a IS 
NOT NULL AND b IS NOT NULL PRIMARY KEY (a, b)");

Keyspace ks = Keyspace.open(keyspace());
ks.getColumnFamilyStore("mv_test1").disableAutoCompaction();

execute("INSERT INTO %s (a, b, c, d) VALUES (?, ?, ?, ?) using 
timestamp 0", 1, 1, 1, 1);
if (flush)
FBUtilities.waitOnFutures(ks.flush());

assertRowsIgnoringOrder(execute("SELECT * FROM mv_test1"), row(1, 1, 1, 
1));

// remove view row
updateView("UPDATE %s using timestamp 1 set b = null WHERE a=1");
if (flush)
FBUtilities.waitOnFutures(ks.flush());

assertRowsIgnoringOrder(execute("SELECT * FROM mv_test1"));
// remove base row, no view updated generated.
updateView("DELETE FROM %s using timestamp 2

[jira] [Commented] (CASSANDRA-12373) 3.0 breaks CQL compatibility with super columns families

2017-09-05 Thread Sylvain Lebresne (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-12373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153401#comment-16153401
 ] 

Sylvain Lebresne commented on CASSANDRA-12373:
--

Thanks for the update and great job on the much needed additional testing. I 
think we're getting there but I still have a bunch of remarks :)

* In {{CFMetaData}}
** In the ctor, for {{isDense}}, we seem to force recomputation for non-SC 
non-dense tables (which happens to include non-compact tables). I don't think 
thats what we want (and the {{recalculateIsDense}} is obviously meant only for 
SC). But for I'm also not 100% sure this work in all upgrade scenario. 
Typically, if a dense SCF is upgraded to a current 3.x version, I believe it 
will end up with 2 regular column definition: the first coming from the fact 
2.x will have had a "COMPACT_VALUE" column definition, which will become a 
REGULAR when upgraded, and second being the "super column map" column that is 
force-added to all SCF. And so that re-computation code will not detect things 
properly. Note that I haven't actually tested this tbh so I could be wrong, but 
this highlight that we really need to be running some SC upgrade dtest both for 
{{2.x \-> 3.x+}} and {{2.x \-> current 3.x \-> 3.x+}} 
because things are imo too complex to rely on reasoning alone.  Unfortunately, 
while we were supposed to have a [SC upgrade 
dtests|https://github.com/apache/cassandra-dtest/blob/master/upgrade_tests/upgrade_supercolumns_test.py],
 it appears [this 
commit|https://github.com/apache/cassandra-dtest/commit/7f47d4d82d6becdb9fd267bd3844d32aee6acdea]
 has made this test useless by making it (confusingly) not about super column 
families. So we'll need to revert that commit somehow and the improve the test 
to include both dense and sparse SC.
** In {{rebuild()}}, why not move the SC code after {{columnMetadata}} has been 
populated so it can be simplified? Nothing seems to rely on it being done 
_before_ {{columnMetadata}} is populated.
* I'm not entirely sure if/how column renaming works for the SC key and value 
"fake" column (would have assumed {{AlterTableStatement}} would need a least a 
bit of special casing to recognize those "fake" column). It would be nice to 
add at test to check that renames still work like in 2.2 for those columns.
* In {{UpdateParameters#addCounter}}, not sure the change from 
{{CounterContext#createUpdate}} to {{CounterContext#createLocal}} is corret. 
Probably a bad post-CASSANDRA-13691 rebase?
* In {{SuperColumnCompatibility}:
** The first line of the javadoc states that it's only for "dense" SC, but it's 
not 100% true as things like {{SUPER_COLUMN_MAP_COLUMN}} are used for all SC. 
And the rest of the javadoc describing the internal layout also applies to all 
SC. Bit of a detail I suppose, but I'd rephrase things to avoid any confusion.  
** In {{getSuperCfKeyColumn}}, not sure what "3.x-created supercolumn family" 
refers to (is that SC tables created from thrift in 3.x? or SC tables whose 
schema had been upgraded by a previous 3.x version?) and more importantly, why 
it would differ in number of clustering columns?
** I'm confused by the code of {{getSuperCfValueColumn}}: dense tables are 
supposed to have only a single regular column, and {{cfm.compactValueColumn()}} 
is supposed to alias it. So I'm not sure what the loop is trying to do, except 
basically returning {{cfm.compactValueColumn()}}, nor when that loop wouldn't 
return (all SC should have the "super column map" internally.  Further, due to 
what's above, said loop appears to return a definition whose type is the 
{{MapType}}, while the code following it clearly return a definition whose type 
is the values of said {{MapType}}, so it feels wrong the method would return 
different kind of types depending on the path taken. I'm most likely missing 
something since your tests seem to be working, but I'm not sure what.
** In {{processPartition}}:
*** There is a typo in a comment, where in "... can't be change to support 
inclusive slice ...", I believe you want "exclusive".
*** In the {{REGULAR}} branch of the {{switch}}:
 in the 2nd branch ({{else if ...}}), we should use {{result.add(cell)}} 
directly.  Not only does that already handle counters, but more importantly, if 
we don't do that, things like {{ttl(v)}} will not work on the compact column 
value.
 the last {{else}} is confusing: by definition we shouldn't be exposing any 
other column than the "compact value" column, so that branch should probably 
not be there (or be an assertion that we shouldn't get there).
** In the {{updates}} loop in {{prepareUpdateOperations}}, not sure why the 2 
first cases seem to assume that {{cfm.isSuperColumnValueColumn(def)}}, but the 
3rd case tests it explicitely (but after having assigned {{superColumnValue}}). 
 In general, not sure why the 3rd case looks different from the other 2. Also, 
I find it

[jira] [Commented] (CASSANDRA-13809) Make BatchlogManagerMBean.forceBatchlogReplay() blocking


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153365#comment-16153365
 ] 

Marcus Eriksson commented on CASSANDRA-13809:
-

created CASSANDRA-13842

> Make BatchlogManagerMBean.forceBatchlogReplay() blocking
> 
>
> Key: CASSANDRA-13809
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13809
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Coordination
>Reporter: Aleksey Yeschenko
>Assignee: Aleksey Yeschenko
>Priority: Trivial
> Fix For: 2.2.11
>
>
> In 3.0, {{BatchlogManagerMBean.forceBatchlogReplay()}} blocks until 
> completion. In 2.2 it just submits a runnable and instaexits, which makes it 
> impossible to create non-flaky dtests that rely on batchlog replay.
> [here|https://github.com/iamaleksey/cassandra/commits/13809-2.2] is a small 
> 2.2-only commit that make the behaviour consistent between 2.2 and 3.0.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Created] (CASSANDRA-13842) dtest failure: batch_test.TestBatch.batchlog_replay_compatibility_?_test

Marcus Eriksson created CASSANDRA-13842:
---

 Summary: dtest failure: 
batch_test.TestBatch.batchlog_replay_compatibility_?_test
 Key: CASSANDRA-13842
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13842
 Project: Cassandra
  Issue Type: Bug
Reporter: Marcus Eriksson


batch_test.TestBatch.batchlog_replay_compatibility_1_test and 
batch_test.TestBatch.batchlog_replay_compatibility_4_test are failing:
http://cassci.datastax.com/view/cassandra-3.11/job/cassandra-3.11_dtest/160/testReport/batch_test/TestBatch/batchlog_replay_compatibility_1_test/
http://cassci.datastax.com/view/cassandra-3.11/job/cassandra-3.11_dtest/160/testReport/batch_test/TestBatch/batchlog_replay_compatibility_4_test/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13809) Make BatchlogManagerMBean.forceBatchlogReplay() blocking

2017-09-05 Thread Aleksey Yeschenko (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153360#comment-16153360
 ] 

Aleksey Yeschenko commented on CASSANDRA-13809:
---

The dtests aren't for it, it's the other way around.

I have no idea how and why are they failing, however. Ran each of them dozens 
of times locally after I started seeing the failures on jenkins, and they never 
fail.

> Make BatchlogManagerMBean.forceBatchlogReplay() blocking
> 
>
> Key: CASSANDRA-13809
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13809
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Coordination
>Reporter: Aleksey Yeschenko
>Assignee: Aleksey Yeschenko
>Priority: Trivial
> Fix For: 2.2.11
>
>
> In 3.0, {{BatchlogManagerMBean.forceBatchlogReplay()}} blocks until 
> completion. In 2.2 it just submits a runnable and instaexits, which makes it 
> impossible to create non-flaky dtests that rely on batchlog replay.
> [here|https://github.com/iamaleksey/cassandra/commits/13809-2.2] is a small 
> 2.2-only commit that make the behaviour consistent between 2.2 and 3.0.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

2017-09-05 Thread ZhaoYang (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153337#comment-16153337
 ] 

ZhaoYang commented on CASSANDRA-11500:
--

[~pauloricardomg] (y) thanks for the feedback, review, improvements.

> Obsolete MV entry may not be properly deleted
> -
>
> Key: CASSANDRA-11500
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11500
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: Sylvain Lebresne
>Assignee: ZhaoYang
> Fix For: 3.0.15, 3.11.1, 4.0
>
>
> When a Materialized View uses a non-PK base table column in its PK, if an 
> update changes that column value, we add the new view entry and remove the 
> old one. When doing that removal, the current code uses the same timestamp 
> than for the liveness info of the new entry, which is the max timestamp for 
> any columns participating to the view PK. This is not correct for the 
> deletion as the old view entry could have other columns with higher timestamp 
> which won't be deleted as can easily shown by the failing of the following 
> test:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS 
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 4 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1;
> SELECT * FROM mv WHERE k = 1; // This currently return 2 entries, the old 
> (invalid) and the new one
> {noformat}
> So the correct timestamp to use for the deletion is the biggest timestamp in 
> the old view entry (which we know since we read the pre-existing base row), 
> and that is what CASSANDRA-11475 does (the test above thus doesn't fail on 
> that branch).
> Unfortunately, even then we can still have problems if further updates 
> requires us to overide the old entry. Consider the following case:
> {noformat}
> CREATE TABLE t (k int PRIMARY KEY, a int, b int);
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS 
> NOT NULL PRIMARY KEY (k, a);
> INSERT INTO t(k, a, b) VALUES (1, 1, 1) USING TIMESTAMP 0;
> UPDATE t USING TIMESTAMP 10 SET b = 2 WHERE k = 1;
> UPDATE t USING TIMESTAMP 2 SET a = 2 WHERE k = 1; // This will delete the 
> entry for a=1 with timestamp 10
> UPDATE t USING TIMESTAMP 3 SET a = 1 WHERE k = 1; // This needs to re-insert 
> an entry for a=1 but shouldn't be deleted by the prior deletion
> UPDATE t USING TIMESTAMP 4 SET a = 2 WHERE k = 1; // ... and we can play this 
> game more than once
> UPDATE t USING TIMESTAMP 5 SET a = 1 WHERE k = 1;
> ...
> {noformat}
> In a way, this is saying that the "shadowable" deletion mechanism is not 
> general enough: we need to be able to re-insert an entry when a prior one had 
> been deleted before, but we can't rely on timestamps being strictly bigger on 
> the re-insert. In that sense, this can be though as a similar problem than 
> CASSANDRA-10965, though the solution there of a single flag is not enough 
> since we can have to replace more than once.
> I think the proper solution would be to ship enough information to always be 
> able to decide when a view deletion is shadowed. Which means that both 
> liveness info (for updates) and shadowable deletion would need to ship the 
> timestamp of any base table column that is part the view PK (so {{a}} in the 
> example below).  It's doable (and not that hard really), but it does require 
> a change to the sstable and intra-node protocol, which makes this a bit 
> painful right now.
> But I'll also note that as CASSANDRA-1096 shows, the timestamp is not even 
> enough since on equal timestamp the value can be the deciding factor. So in 
> theory we'd have to ship the value of those columns (in the case of a 
> deletion at least since we have it in the view PK for updates). That said, on 
> that last problem, my preference would be that we start prioritizing 
> CASSANDRA-6123 seriously so we don't have to care about conflicting timestamp 
> anymore, which would make this problem go away.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13738) Load is over calculated after each IndexSummaryRedistribution


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-13738:

   Resolution: Fixed
Fix Version/s: (was: 3.11.x)
   (was: 4.x)
   (was: 3.0.x)
   (was: 2.2.x)
   4.0
   3.11.1
   3.0.15
   2.2.11
   Status: Resolved  (was: Patch Available)

> Load is over calculated after each IndexSummaryRedistribution
> -
>
> Key: CASSANDRA-13738
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13738
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
> Fix For: 2.2.11, 3.0.15, 3.11.1, 4.0
>
> Attachments: sizeIssue.png
>
>
> For example, here is one of our cluster with about 500GB per node, but 
> {{nodetool status}} shows far more load than it actually is and keeps 
> increasing, restarting the process will reset the load, but keeps increasing 
> afterwards:
> {noformat}
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  AddressLoad   Tokens   Owns (effective)  Host ID  
>  Rack
> UN  IP1*   13.52 TB   256  100.0%
> c4c31e0a-3f01-49f7-8a22-33043737975d  rac1
> UN  IP2*   14.25 TB   256  100.0%
> efec4980-ec9e-4424-8a21-ce7ddaf80aa0  rac1
> UN  IP3*   13.52 TB   256  100.0%
> 7dbcfdfc-9c07-4b1a-a4b9-970b715ebed8  rac1
> UN  IP4*   22.13 TB   256  100.0%
> 8879e6c4-93e3-4cc5-b957-f999c6b9b563  rac1
> UN  IP5*   18.02 TB   256  100.0%
> 4a1eaf22-4a83-4736-9e1c-12f898d685fa  rac1
> UN  IP6*   11.68 TB   256  100.0%
> d633c591-28af-42cc-bc5e-47d1c8bcf50f  rac1
> {noformat}
> !sizeIssue.png|test!
> The root cause is if the SSTable index summary is redistributed (typically 
> executes hourly), the updated SSTable size is added again.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13738) Load is over calculated after each IndexSummaryRedistribution


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153283#comment-16153283
 ] 

Marcus Eriksson commented on CASSANDRA-13738:
-

failing dtests pass locally

committed as {{4e834c53ca57910e8c4}}, thanks!

> Load is over calculated after each IndexSummaryRedistribution
> -
>
> Key: CASSANDRA-13738
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13738
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
> Fix For: 2.2.x, 3.0.x, 3.11.x, 4.x
>
> Attachments: sizeIssue.png
>
>
> For example, here is one of our cluster with about 500GB per node, but 
> {{nodetool status}} shows far more load than it actually is and keeps 
> increasing, restarting the process will reset the load, but keeps increasing 
> afterwards:
> {noformat}
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  AddressLoad   Tokens   Owns (effective)  Host ID  
>  Rack
> UN  IP1*   13.52 TB   256  100.0%
> c4c31e0a-3f01-49f7-8a22-33043737975d  rac1
> UN  IP2*   14.25 TB   256  100.0%
> efec4980-ec9e-4424-8a21-ce7ddaf80aa0  rac1
> UN  IP3*   13.52 TB   256  100.0%
> 7dbcfdfc-9c07-4b1a-a4b9-970b715ebed8  rac1
> UN  IP4*   22.13 TB   256  100.0%
> 8879e6c4-93e3-4cc5-b957-f999c6b9b563  rac1
> UN  IP5*   18.02 TB   256  100.0%
> 4a1eaf22-4a83-4736-9e1c-12f898d685fa  rac1
> UN  IP6*   11.68 TB   256  100.0%
> d633c591-28af-42cc-bc5e-47d1c8bcf50f  rac1
> {noformat}
> !sizeIssue.png|test!
> The root cause is if the SSTable index summary is redistributed (typically 
> executes hourly), the updated SSTable size is added again.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[03/10] cassandra git commit: Fix load over calculated issue in IndexSummaryRedistribution

Fix load over calculated issue in IndexSummaryRedistribution

patch by Jay Zhuang; reviewed by Marcus Eriksson for CASSANDRA-13738


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4e834c53
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4e834c53
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4e834c53

Branch: refs/heads/cassandra-3.11
Commit: 4e834c53ca57910e8c44cb1edf2a780df9390556
Parents: e80ede6
Author: Jay Zhuang 
Authored: Wed Aug 2 21:44:39 2017 -0700
Committer: Marcus Eriksson 
Committed: Tue Sep 5 10:33:39 2017 +0200

--
 CHANGES.txt |   1 +
 .../io/sstable/format/SSTableReader.java|   6 +-
 .../sstable/IndexSummaryRedistributionTest.java | 145 +++
 3 files changed, 149 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 03a78fd..1abd7de 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.11
+ * Fix load over calculated issue in IndexSummaryRedistribution 
(CASSANDRA-13738)
  * Fix compaction and flush exception not captured (CASSANDRA-13833)
  * Make BatchlogManagerMBean.forceBatchlogReplay() blocking (CASSANDRA-13809)
  * Uncaught exceptions in Netty pipeline (CASSANDRA-13649)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java 
b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
index ba060d4..885 100644
--- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
@@ -1174,9 +1174,9 @@ public abstract class SSTableReader extends SSTable 
implements SelfRefCountedhttp://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java 
b/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java
new file mode 100644
index 000..77fd69a
--- /dev/null
+++ 
b/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.io.sstable;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import org.apache.cassandra.SchemaLoader;
+import org.apache.cassandra.Util;
+import org.apache.cassandra.cache.CachingOptions;
+import org.apache.cassandra.config.KSMetaData;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.DecoratedKey;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.Mutation;
+import org.apache.cassandra.exceptions.ConfigurationException;
+import org.apache.cassandra.io.sstable.format.SSTableReader;
+import org.apache.cassandra.locator.SimpleStrategy;
+import org.apache.cassandra.metrics.RestorableMeter;
+import org.apache.cassandra.metrics.StorageMetrics;
+
+import static org.junit.Assert.assertEquals;
+
+public class IndexSummaryRedistributionTest
+{
+private static final String KEYSPACE1 = "IndexSummaryRedistributionTest";
+private static final String CF_STANDARD = "Standard";
+
+@BeforeClass
+public static void defineSchema() throws ConfigurationException
+{
+SchemaLoader.prepareServer();
+SchemaLoader.createKeyspace(KEYSPACE1,
+

[08/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5a09fafe
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5a09fafe
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5a09fafe

Branch: refs/heads/cassandra-3.11
Commit: 5a09fafea043d472f2d46b1e17062201f7baa38a
Parents: 2b29f8a fb92a52
Author: Marcus Eriksson 
Authored: Tue Sep 5 10:39:02 2017 +0200
Committer: Marcus Eriksson 
Committed: Tue Sep 5 10:39:02 2017 +0200

--
 CHANGES.txt |   1 +
 .../io/sstable/format/SSTableReader.java|   8 +-
 .../sstable/IndexSummaryRedistributionTest.java | 142 +++
 3 files changed, 147 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5a09fafe/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5a09fafe/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
--
diff --cc src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
index 03af2bb,25ba212..5d76f75
--- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
@@@ -1143,13 -1116,17 +1143,13 @@@ public abstract class SSTableReader ext
  "no adjustments to min/max_index_interval");
  }
  
 -//Always save the resampled index
 -try(SegmentedFile.Builder ibuilder = 
SegmentedFile.getBuilder(DatabaseDescriptor.getIndexAccessMode(), false);
 -SegmentedFile.Builder dbuilder = 
SegmentedFile.getBuilder(DatabaseDescriptor.getDiskAccessMode(), compression))
 -{
 -saveSummary(ibuilder, dbuilder, newSummary);
 -}
 +// Always save the resampled index
 +saveSummary(newSummary);
  
- long newSize = bytesOnDisk();
- StorageMetrics.load.inc(newSize - oldSize);
- parent.metric.liveDiskSpaceUsed.inc(newSize - oldSize);
- parent.metric.totalDiskSpaceUsed.inc(newSize - oldSize);
+ // The new size will be added in Transactional.commit() as an 
updated SSTable, more details: CASSANDRA-13738
+ StorageMetrics.load.dec(oldSize);
+ parent.metric.liveDiskSpaceUsed.dec(oldSize);
+ parent.metric.totalDiskSpaceUsed.dec(oldSize);
  
  return cloneAndReplace(first, OpenReason.METADATA_CHANGE, 
newSummary);
  }


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[07/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fb92a526
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fb92a526
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fb92a526

Branch: refs/heads/trunk
Commit: fb92a52624c13969386760b967b7168a87e857bb
Parents: f4014bb 4e834c5
Author: Marcus Eriksson 
Authored: Tue Sep 5 10:38:39 2017 +0200
Committer: Marcus Eriksson 
Committed: Tue Sep 5 10:38:39 2017 +0200

--
 CHANGES.txt |   1 +
 .../io/sstable/format/SSTableReader.java|   8 +-
 .../sstable/IndexSummaryRedistributionTest.java | 142 +++
 3 files changed, 147 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/fb92a526/CHANGES.txt
--
diff --cc CHANGES.txt
index 870351c,1abd7de..1ed3fe6
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,39 -1,9 +1,40 @@@
 -2.2.11
 +3.0.15
 + * Fix MV timestamp issues (CASSANDRA-11500)
 + * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) 
 + * Fix race condition in read command serialization (CASSANDRA-13363)
 + * Enable segement creation before recovering commitlogs (CASSANDRA-13587)
 + * Fix AssertionError in short read protection (CASSANDRA-13747)
 + * Don't skip corrupted sstables on startup (CASSANDRA-13620)
 + * Fix the merging of cells with different user type versions 
(CASSANDRA-13776)
 + * Copy session properties on cqlsh.py do_login (CASSANDRA-13640)
 + * Potential AssertionError during ReadRepair of range tombstone and 
partition deletions (CASSANDRA-13719)
 + * Don't let stress write warmup data if n=0 (CASSANDRA-13773)
 + * Gossip thread slows down when using batch commit log (CASSANDRA-12966)
 + * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)
 + * Fix digest calculation for counter cells (CASSANDRA-13750)
 + * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)
 + * Skip materialized view addition if the base table doesn't exist 
(CASSANDRA-13737)
 + * Drop table should remove corresponding entries in dropped_columns table 
(CASSANDRA-13730)
 + * Log warn message until legacy auth tables have been migrated 
(CASSANDRA-13371)
 + * Fix incorrect [2.1 <- 3.0] serialization of counter cells created in 2.0 
(CASSANDRA-13691)
 + * Fix invalid writetime for null cells (CASSANDRA-13711)
 + * Fix ALTER TABLE statement to atomically propagate changes to the table and 
its MVs (CASSANDRA-12952)
 + * Fixed ambiguous output of nodetool tablestats command (CASSANDRA-13722)
 + * JMXEnabledThreadPoolExecutor with corePoolSize equal to maxPoolSize 
(Backport CASSANDRA-13329)
 + * Fix Digest mismatch Exception if hints file has UnknownColumnFamily 
(CASSANDRA-13696)
 + * Purge tombstones created by expired cells (CASSANDRA-13643)
 + * Make concat work with iterators that have different subsets of columns 
(CASSANDRA-13482)
 + * Set test.runners based on cores and memory size (CASSANDRA-13078)
 + * Allow different NUMACTL_ARGS to be passed in (CASSANDRA-13557)
 + * Allow native function calls in CQLSSTableWriter (CASSANDRA-12606)
 + * Fix secondary index queries on COMPACT tables (CASSANDRA-13627)
 + * Nodetool listsnapshots output is missing a newline, if there are no 
snapshots (CASSANDRA-13568)
 + * sstabledump reports incorrect usage for argument order (CASSANDRA-13532)
 +Merged from 2.2:
+  * Fix load over calculated issue in IndexSummaryRedistribution 
(CASSANDRA-13738)
   * Fix compaction and flush exception not captured (CASSANDRA-13833)
 - * Make BatchlogManagerMBean.forceBatchlogReplay() blocking (CASSANDRA-13809)
   * Uncaught exceptions in Netty pipeline (CASSANDRA-13649)
 - * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 
 + * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067)
   * Fix queries with LIMIT and filtering on clustering columns 
(CASSANDRA-11223)
   * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272)
   * Fix toJSONString for the UDT, tuple and collection types (CASSANDRA-13592)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/fb92a526/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
--
diff --cc src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
index d56b3e7,885..25ba212
--- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
@@@ -1123,10 -1169,14 +1123,10 @@@ public abstract class SSTableReader ext

[01/10] cassandra git commit: Fix load over calculated issue in IndexSummaryRedistribution

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.2 e80ede6d3 -> 4e834c53c
  refs/heads/cassandra-3.0 f4014bb0f -> fb92a5262
  refs/heads/cassandra-3.11 2b29f8a78 -> 5a09fafea
  refs/heads/trunk 65f7216c6 -> 9717cd67b


Fix load over calculated issue in IndexSummaryRedistribution

patch by Jay Zhuang; reviewed by Marcus Eriksson for CASSANDRA-13738


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4e834c53
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4e834c53
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4e834c53

Branch: refs/heads/cassandra-2.2
Commit: 4e834c53ca57910e8c44cb1edf2a780df9390556
Parents: e80ede6
Author: Jay Zhuang 
Authored: Wed Aug 2 21:44:39 2017 -0700
Committer: Marcus Eriksson 
Committed: Tue Sep 5 10:33:39 2017 +0200

--
 CHANGES.txt |   1 +
 .../io/sstable/format/SSTableReader.java|   6 +-
 .../sstable/IndexSummaryRedistributionTest.java | 145 +++
 3 files changed, 149 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 03a78fd..1abd7de 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.11
+ * Fix load over calculated issue in IndexSummaryRedistribution 
(CASSANDRA-13738)
  * Fix compaction and flush exception not captured (CASSANDRA-13833)
  * Make BatchlogManagerMBean.forceBatchlogReplay() blocking (CASSANDRA-13809)
  * Uncaught exceptions in Netty pipeline (CASSANDRA-13649)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java 
b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
index ba060d4..885 100644
--- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
@@ -1174,9 +1174,9 @@ public abstract class SSTableReader extends SSTable 
implements SelfRefCountedhttp://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java 
b/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java
new file mode 100644
index 000..77fd69a
--- /dev/null
+++ 
b/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.io.sstable;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import org.apache.cassandra.SchemaLoader;
+import org.apache.cassandra.Util;
+import org.apache.cassandra.cache.CachingOptions;
+import org.apache.cassandra.config.KSMetaData;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.DecoratedKey;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.Mutation;
+import org.apache.cassandra.exceptions.ConfigurationException;
+import org.apache.cassandra.io.sstable.format.SSTableReader;
+import org.apache.cassandra.locator.SimpleStrategy;
+import org.apache.cassandra.metrics.RestorableMeter;
+import org.apache.cassandra.metrics.StorageMetrics;
+
+import static org.junit.Assert.assertEquals;
+
+public class IndexSummaryRedistributionTest
+{
+private static final String KEYSPACE1 = "IndexSummaryRedistributionTest";
+

[04/10] cassandra git commit: Fix load over calculated issue in IndexSummaryRedistribution

Fix load over calculated issue in IndexSummaryRedistribution

patch by Jay Zhuang; reviewed by Marcus Eriksson for CASSANDRA-13738


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4e834c53
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4e834c53
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4e834c53

Branch: refs/heads/trunk
Commit: 4e834c53ca57910e8c44cb1edf2a780df9390556
Parents: e80ede6
Author: Jay Zhuang 
Authored: Wed Aug 2 21:44:39 2017 -0700
Committer: Marcus Eriksson 
Committed: Tue Sep 5 10:33:39 2017 +0200

--
 CHANGES.txt |   1 +
 .../io/sstable/format/SSTableReader.java|   6 +-
 .../sstable/IndexSummaryRedistributionTest.java | 145 +++
 3 files changed, 149 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 03a78fd..1abd7de 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.11
+ * Fix load over calculated issue in IndexSummaryRedistribution 
(CASSANDRA-13738)
  * Fix compaction and flush exception not captured (CASSANDRA-13833)
  * Make BatchlogManagerMBean.forceBatchlogReplay() blocking (CASSANDRA-13809)
  * Uncaught exceptions in Netty pipeline (CASSANDRA-13649)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java 
b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
index ba060d4..885 100644
--- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
@@ -1174,9 +1174,9 @@ public abstract class SSTableReader extends SSTable 
implements SelfRefCountedhttp://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java 
b/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java
new file mode 100644
index 000..77fd69a
--- /dev/null
+++ 
b/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.io.sstable;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import org.apache.cassandra.SchemaLoader;
+import org.apache.cassandra.Util;
+import org.apache.cassandra.cache.CachingOptions;
+import org.apache.cassandra.config.KSMetaData;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.DecoratedKey;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.Mutation;
+import org.apache.cassandra.exceptions.ConfigurationException;
+import org.apache.cassandra.io.sstable.format.SSTableReader;
+import org.apache.cassandra.locator.SimpleStrategy;
+import org.apache.cassandra.metrics.RestorableMeter;
+import org.apache.cassandra.metrics.StorageMetrics;
+
+import static org.junit.Assert.assertEquals;
+
+public class IndexSummaryRedistributionTest
+{
+private static final String KEYSPACE1 = "IndexSummaryRedistributionTest";
+private static final String CF_STANDARD = "Standard";
+
+@BeforeClass
+public static void defineSchema() throws ConfigurationException
+{
+SchemaLoader.prepareServer();
+SchemaLoader.createKeyspace(KEYSPACE1,
+

[06/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fb92a526
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fb92a526
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fb92a526

Branch: refs/heads/cassandra-3.0
Commit: fb92a52624c13969386760b967b7168a87e857bb
Parents: f4014bb 4e834c5
Author: Marcus Eriksson 
Authored: Tue Sep 5 10:38:39 2017 +0200
Committer: Marcus Eriksson 
Committed: Tue Sep 5 10:38:39 2017 +0200

--
 CHANGES.txt |   1 +
 .../io/sstable/format/SSTableReader.java|   8 +-
 .../sstable/IndexSummaryRedistributionTest.java | 142 +++
 3 files changed, 147 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/fb92a526/CHANGES.txt
--
diff --cc CHANGES.txt
index 870351c,1abd7de..1ed3fe6
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,39 -1,9 +1,40 @@@
 -2.2.11
 +3.0.15
 + * Fix MV timestamp issues (CASSANDRA-11500)
 + * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) 
 + * Fix race condition in read command serialization (CASSANDRA-13363)
 + * Enable segement creation before recovering commitlogs (CASSANDRA-13587)
 + * Fix AssertionError in short read protection (CASSANDRA-13747)
 + * Don't skip corrupted sstables on startup (CASSANDRA-13620)
 + * Fix the merging of cells with different user type versions 
(CASSANDRA-13776)
 + * Copy session properties on cqlsh.py do_login (CASSANDRA-13640)
 + * Potential AssertionError during ReadRepair of range tombstone and 
partition deletions (CASSANDRA-13719)
 + * Don't let stress write warmup data if n=0 (CASSANDRA-13773)
 + * Gossip thread slows down when using batch commit log (CASSANDRA-12966)
 + * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)
 + * Fix digest calculation for counter cells (CASSANDRA-13750)
 + * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)
 + * Skip materialized view addition if the base table doesn't exist 
(CASSANDRA-13737)
 + * Drop table should remove corresponding entries in dropped_columns table 
(CASSANDRA-13730)
 + * Log warn message until legacy auth tables have been migrated 
(CASSANDRA-13371)
 + * Fix incorrect [2.1 <- 3.0] serialization of counter cells created in 2.0 
(CASSANDRA-13691)
 + * Fix invalid writetime for null cells (CASSANDRA-13711)
 + * Fix ALTER TABLE statement to atomically propagate changes to the table and 
its MVs (CASSANDRA-12952)
 + * Fixed ambiguous output of nodetool tablestats command (CASSANDRA-13722)
 + * JMXEnabledThreadPoolExecutor with corePoolSize equal to maxPoolSize 
(Backport CASSANDRA-13329)
 + * Fix Digest mismatch Exception if hints file has UnknownColumnFamily 
(CASSANDRA-13696)
 + * Purge tombstones created by expired cells (CASSANDRA-13643)
 + * Make concat work with iterators that have different subsets of columns 
(CASSANDRA-13482)
 + * Set test.runners based on cores and memory size (CASSANDRA-13078)
 + * Allow different NUMACTL_ARGS to be passed in (CASSANDRA-13557)
 + * Allow native function calls in CQLSSTableWriter (CASSANDRA-12606)
 + * Fix secondary index queries on COMPACT tables (CASSANDRA-13627)
 + * Nodetool listsnapshots output is missing a newline, if there are no 
snapshots (CASSANDRA-13568)
 + * sstabledump reports incorrect usage for argument order (CASSANDRA-13532)
 +Merged from 2.2:
+  * Fix load over calculated issue in IndexSummaryRedistribution 
(CASSANDRA-13738)
   * Fix compaction and flush exception not captured (CASSANDRA-13833)
 - * Make BatchlogManagerMBean.forceBatchlogReplay() blocking (CASSANDRA-13809)
   * Uncaught exceptions in Netty pipeline (CASSANDRA-13649)
 - * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 
 + * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067)
   * Fix queries with LIMIT and filtering on clustering columns 
(CASSANDRA-11223)
   * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272)
   * Fix toJSONString for the UDT, tuple and collection types (CASSANDRA-13592)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/fb92a526/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
--
diff --cc src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
index d56b3e7,885..25ba212
--- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
@@@ -1123,10 -1169,14 +1123,10 @@@ public abstract class

[02/10] cassandra git commit: Fix load over calculated issue in IndexSummaryRedistribution

Fix load over calculated issue in IndexSummaryRedistribution

patch by Jay Zhuang; reviewed by Marcus Eriksson for CASSANDRA-13738


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4e834c53
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4e834c53
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4e834c53

Branch: refs/heads/cassandra-3.0
Commit: 4e834c53ca57910e8c44cb1edf2a780df9390556
Parents: e80ede6
Author: Jay Zhuang 
Authored: Wed Aug 2 21:44:39 2017 -0700
Committer: Marcus Eriksson 
Committed: Tue Sep 5 10:33:39 2017 +0200

--
 CHANGES.txt |   1 +
 .../io/sstable/format/SSTableReader.java|   6 +-
 .../sstable/IndexSummaryRedistributionTest.java | 145 +++
 3 files changed, 149 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 03a78fd..1abd7de 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 2.2.11
+ * Fix load over calculated issue in IndexSummaryRedistribution 
(CASSANDRA-13738)
  * Fix compaction and flush exception not captured (CASSANDRA-13833)
  * Make BatchlogManagerMBean.forceBatchlogReplay() blocking (CASSANDRA-13809)
  * Uncaught exceptions in Netty pipeline (CASSANDRA-13649)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java 
b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
index ba060d4..885 100644
--- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
@@ -1174,9 +1174,9 @@ public abstract class SSTableReader extends SSTable 
implements SelfRefCountedhttp://git-wip-us.apache.org/repos/asf/cassandra/blob/4e834c53/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java 
b/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java
new file mode 100644
index 000..77fd69a
--- /dev/null
+++ 
b/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.io.sstable;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import org.apache.cassandra.SchemaLoader;
+import org.apache.cassandra.Util;
+import org.apache.cassandra.cache.CachingOptions;
+import org.apache.cassandra.config.KSMetaData;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.DecoratedKey;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.Mutation;
+import org.apache.cassandra.exceptions.ConfigurationException;
+import org.apache.cassandra.io.sstable.format.SSTableReader;
+import org.apache.cassandra.locator.SimpleStrategy;
+import org.apache.cassandra.metrics.RestorableMeter;
+import org.apache.cassandra.metrics.StorageMetrics;
+
+import static org.junit.Assert.assertEquals;
+
+public class IndexSummaryRedistributionTest
+{
+private static final String KEYSPACE1 = "IndexSummaryRedistributionTest";
+private static final String CF_STANDARD = "Standard";
+
+@BeforeClass
+public static void defineSchema() throws ConfigurationException
+{
+SchemaLoader.prepareServer();
+SchemaLoader.createKeyspace(KEYSPACE1,
+

[05/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fb92a526
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fb92a526
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fb92a526

Branch: refs/heads/cassandra-3.11
Commit: fb92a52624c13969386760b967b7168a87e857bb
Parents: f4014bb 4e834c5
Author: Marcus Eriksson 
Authored: Tue Sep 5 10:38:39 2017 +0200
Committer: Marcus Eriksson 
Committed: Tue Sep 5 10:38:39 2017 +0200

--
 CHANGES.txt |   1 +
 .../io/sstable/format/SSTableReader.java|   8 +-
 .../sstable/IndexSummaryRedistributionTest.java | 142 +++
 3 files changed, 147 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/fb92a526/CHANGES.txt
--
diff --cc CHANGES.txt
index 870351c,1abd7de..1ed3fe6
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,39 -1,9 +1,40 @@@
 -2.2.11
 +3.0.15
 + * Fix MV timestamp issues (CASSANDRA-11500)
 + * Better tolerate improperly formatted bcrypt hashes (CASSANDRA-13626) 
 + * Fix race condition in read command serialization (CASSANDRA-13363)
 + * Enable segement creation before recovering commitlogs (CASSANDRA-13587)
 + * Fix AssertionError in short read protection (CASSANDRA-13747)
 + * Don't skip corrupted sstables on startup (CASSANDRA-13620)
 + * Fix the merging of cells with different user type versions 
(CASSANDRA-13776)
 + * Copy session properties on cqlsh.py do_login (CASSANDRA-13640)
 + * Potential AssertionError during ReadRepair of range tombstone and 
partition deletions (CASSANDRA-13719)
 + * Don't let stress write warmup data if n=0 (CASSANDRA-13773)
 + * Gossip thread slows down when using batch commit log (CASSANDRA-12966)
 + * Randomize batchlog endpoint selection with only 1 or 2 racks 
(CASSANDRA-12884)
 + * Fix digest calculation for counter cells (CASSANDRA-13750)
 + * Fix ColumnDefinition.cellValueType() for non-frozen collection and change 
SSTabledump to use type.toJSONString() (CASSANDRA-13573)
 + * Skip materialized view addition if the base table doesn't exist 
(CASSANDRA-13737)
 + * Drop table should remove corresponding entries in dropped_columns table 
(CASSANDRA-13730)
 + * Log warn message until legacy auth tables have been migrated 
(CASSANDRA-13371)
 + * Fix incorrect [2.1 <- 3.0] serialization of counter cells created in 2.0 
(CASSANDRA-13691)
 + * Fix invalid writetime for null cells (CASSANDRA-13711)
 + * Fix ALTER TABLE statement to atomically propagate changes to the table and 
its MVs (CASSANDRA-12952)
 + * Fixed ambiguous output of nodetool tablestats command (CASSANDRA-13722)
 + * JMXEnabledThreadPoolExecutor with corePoolSize equal to maxPoolSize 
(Backport CASSANDRA-13329)
 + * Fix Digest mismatch Exception if hints file has UnknownColumnFamily 
(CASSANDRA-13696)
 + * Purge tombstones created by expired cells (CASSANDRA-13643)
 + * Make concat work with iterators that have different subsets of columns 
(CASSANDRA-13482)
 + * Set test.runners based on cores and memory size (CASSANDRA-13078)
 + * Allow different NUMACTL_ARGS to be passed in (CASSANDRA-13557)
 + * Allow native function calls in CQLSSTableWriter (CASSANDRA-12606)
 + * Fix secondary index queries on COMPACT tables (CASSANDRA-13627)
 + * Nodetool listsnapshots output is missing a newline, if there are no 
snapshots (CASSANDRA-13568)
 + * sstabledump reports incorrect usage for argument order (CASSANDRA-13532)
 +Merged from 2.2:
+  * Fix load over calculated issue in IndexSummaryRedistribution 
(CASSANDRA-13738)
   * Fix compaction and flush exception not captured (CASSANDRA-13833)
 - * Make BatchlogManagerMBean.forceBatchlogReplay() blocking (CASSANDRA-13809)
   * Uncaught exceptions in Netty pipeline (CASSANDRA-13649)
 - * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067) 
 + * Prevent integer overflow on exabyte filesystems (CASSANDRA-13067)
   * Fix queries with LIMIT and filtering on clustering columns 
(CASSANDRA-11223)
   * Fix potential NPE when resume bootstrap fails (CASSANDRA-13272)
   * Fix toJSONString for the UDT, tuple and collection types (CASSANDRA-13592)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/fb92a526/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
--
diff --cc src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
index d56b3e7,885..25ba212
--- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
@@@ -1123,10 -1169,14 +1123,10 @@@ public abstract class

[10/10] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9717cd67
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9717cd67
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9717cd67

Branch: refs/heads/trunk
Commit: 9717cd67bd327f527bd97775057d356b674f50ec
Parents: 65f7216 5a09faf
Author: Marcus Eriksson 
Authored: Tue Sep 5 10:41:24 2017 +0200
Committer: Marcus Eriksson 
Committed: Tue Sep 5 10:41:50 2017 +0200

--
 CHANGES.txt |   1 +
 .../io/sstable/format/SSTableReader.java|   8 +-
 .../sstable/IndexSummaryRedistributionTest.java | 143 +++
 3 files changed, 148 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/9717cd67/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9717cd67/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/9717cd67/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java
--
diff --cc 
test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java
index 000,31a57e1..07a2212
mode 00,100644..100644
--- 
a/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java
+++ 
b/test/unit/org/apache/cassandra/io/sstable/IndexSummaryRedistributionTest.java
@@@ -1,0 -1,142 +1,143 @@@
+ /*
+  * Licensed to the Apache Software Foundation (ASF) under one
+  * or more contributor license agreements.  See the NOTICE file
+  * distributed with this work for additional information
+  * regarding copyright ownership.  The ASF licenses this file
+  * to you under the Apache License, Version 2.0 (the
+  * "License"); you may not use this file except in compliance
+  * with the License.  You may obtain a copy of the License at
+  *
+  * http://www.apache.org/licenses/LICENSE-2.0
+  *
+  * Unless required by applicable law or agreed to in writing, software
+  * distributed under the License is distributed on an "AS IS" BASIS,
+  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  * See the License for the specific language governing permissions and
+  * limitations under the License.
+  */
+ 
+ package org.apache.cassandra.io.sstable;
+ 
+ import java.io.IOException;
+ import java.nio.ByteBuffer;
+ import java.util.ArrayList;
+ import java.util.List;
+ import java.util.concurrent.ExecutionException;
+ import java.util.concurrent.Future;
+ 
+ import org.junit.BeforeClass;
+ import org.junit.Test;
+ 
+ import org.apache.cassandra.SchemaLoader;
+ import org.apache.cassandra.db.ColumnFamilyStore;
+ import org.apache.cassandra.db.Keyspace;
+ import org.apache.cassandra.db.RowUpdateBuilder;
+ import org.apache.cassandra.exceptions.ConfigurationException;
+ import org.apache.cassandra.io.sstable.format.SSTableReader;
+ import org.apache.cassandra.metrics.RestorableMeter;
+ import org.apache.cassandra.metrics.StorageMetrics;
+ import org.apache.cassandra.schema.CachingParams;
+ import org.apache.cassandra.schema.KeyspaceParams;
++import org.apache.cassandra.schema.MigrationManager;
+ 
+ import static org.junit.Assert.assertEquals;
+ 
+ public class IndexSummaryRedistributionTest
+ {
+ private static final String KEYSPACE1 = "IndexSummaryRedistributionTest";
+ private static final String CF_STANDARD = "Standard";
+ 
+ @BeforeClass
+ public static void defineSchema() throws ConfigurationException
+ {
+ SchemaLoader.prepareServer();
+ SchemaLoader.createKeyspace(KEYSPACE1,
+ KeyspaceParams.simple(1),
+ SchemaLoader.standardCFMD(KEYSPACE1, 
CF_STANDARD)
+ .minIndexInterval(8)
+ .maxIndexInterval(256)
+ 
.caching(CachingParams.CACHE_NOTHING));
+ }
+ 
+ @Test
+ public void testMetricsLoadAfterRedistribution() throws IOException
+ {
+ String ksname = KEYSPACE1;
+ String cfname = CF_STANDARD;
+ Keyspace keyspace = Keyspace.open(ksname);
+ ColumnFamilyStore cfs = keyspace.getColumnFamilyStore(cfname);
+ int numSSTables = 1;
+ int numRows = 1024 * 10;
+ long load = StorageMetrics.load.getCount();
+ StorageMetrics.load.dec(load); // reset the load metric
+ createSSTables(ksname, cfname, numSSTables,

[09/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5a09fafe
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5a09fafe
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5a09fafe

Branch: refs/heads/trunk
Commit: 5a09fafea043d472f2d46b1e17062201f7baa38a
Parents: 2b29f8a fb92a52
Author: Marcus Eriksson 
Authored: Tue Sep 5 10:39:02 2017 +0200
Committer: Marcus Eriksson 
Committed: Tue Sep 5 10:39:02 2017 +0200

--
 CHANGES.txt |   1 +
 .../io/sstable/format/SSTableReader.java|   8 +-
 .../sstable/IndexSummaryRedistributionTest.java | 142 +++
 3 files changed, 147 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/5a09fafe/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/5a09fafe/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
--
diff --cc src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
index 03af2bb,25ba212..5d76f75
--- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
@@@ -1143,13 -1116,17 +1143,13 @@@ public abstract class SSTableReader ext
  "no adjustments to min/max_index_interval");
  }
  
 -//Always save the resampled index
 -try(SegmentedFile.Builder ibuilder = 
SegmentedFile.getBuilder(DatabaseDescriptor.getIndexAccessMode(), false);
 -SegmentedFile.Builder dbuilder = 
SegmentedFile.getBuilder(DatabaseDescriptor.getDiskAccessMode(), compression))
 -{
 -saveSummary(ibuilder, dbuilder, newSummary);
 -}
 +// Always save the resampled index
 +saveSummary(newSummary);
  
- long newSize = bytesOnDisk();
- StorageMetrics.load.inc(newSize - oldSize);
- parent.metric.liveDiskSpaceUsed.inc(newSize - oldSize);
- parent.metric.totalDiskSpaceUsed.inc(newSize - oldSize);
+ // The new size will be added in Transactional.commit() as an 
updated SSTable, more details: CASSANDRA-13738
+ StorageMetrics.load.dec(oldSize);
+ parent.metric.liveDiskSpaceUsed.dec(oldSize);
+ parent.metric.totalDiskSpaceUsed.dec(oldSize);
  
  return cloneAndReplace(first, OpenReason.METADATA_CHANGE, 
newSummary);
  }


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-12793) invalid jvm type and architecture [cassandra-env.sh]

2017-09-05 Thread Ali Ebrahiminejad (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-12793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153259#comment-16153259
 ] 

Ali Ebrahiminejad commented on CASSANDRA-12793:
---


{code:java}
jvmver=`echo "$java_ver_output" | grep '[openjdk|java] version' | awk -F'"' 
'NR==1 {print $2}' | cut -d\- -f1`
JVM_VERSION=${jvmver%_*}
JVM_PATCH_VERSION=${jvmver#*_}

if [ "$JVM_VERSION" \< "1.8" ] ; then
echo "Cassandra 3.0 and later require Java 8u40 or later."
exit 1;
fi

if [ "$JVM_VERSION" \< "1.8" ] && [ "$JVM_PATCH_VERSION" -lt 40 ] ; then
echo "Cassandra 3.0 and later require Java 8u40 or later."
exit 1;
fi

jvm=`echo "$java_ver_output" | grep -A 1 'java version' | awk 'NR==2 {print 
$1}'`
{code}

"jvmver" is just fine. 
The problem is with "jvm" which is a few lines below.

> invalid jvm type and architecture [cassandra-env.sh]
> 
>
> Key: CASSANDRA-12793
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12793
> Project: Cassandra
>  Issue Type: Bug
>  Components: Configuration
> Environment: ubuntu 16.04, openjdk 1.8.0_91
>Reporter: Ali Ebrahiminejad
>Priority: Minor
> Fix For: 3.7
>
>
> In cassandra-env.sh the part that determines the type of JVM we'll be running 
> on doesn't provide the right answer for openjdk 1.8.0_91.
> value of java_ver_output is "openjdk version "1.8.0_91" OpenJDK Runtime 
> Environment (build 1.8.0_91-8u91-b14-3ubuntu1~16.04.1-b14) OpenJDK 64-Bit 
> Server VM (build 25.91-b14, mixed mode)", yet the command looks for "java 
> version" (jvm=`echo "$java_ver_output" | grep -A 1 'java version' ...) which 
> does not exist.
> I guess it should be replaced with jvm=`echo "$java_ver_output" | grep -A 1 
> '[openjdk|java] version' | awk 'NR==2 {print $1}'`



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Resolved] (CASSANDRA-13657) Materialized Views: Index MV on TTL'ed column produces orphanized view entry if another column keeps entry live


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta resolved CASSANDRA-13657.
-
Resolution: Duplicate

Fixed on CASSANDRA-11500, closing as duplicate.

> Materialized Views: Index MV on TTL'ed column produces orphanized view entry 
> if another column keeps entry live
> ---
>
> Key: CASSANDRA-13657
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13657
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: Fridtjof Sander
>Assignee: Krishna Dattu Koneru
>  Labels: materializedviews, ttl
>
> {noformat}
> CREATE TABLE t (k int, a int, b int, PRIMARY KEY (k));
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS 
> NOT NULL PRIMARY KEY (a, k);
> INSERT INTO t (k) VALUES (1);
> UPDATE t USING TTL 5 SET a = 10 WHERE k = 1;
> UPDATE t SET b = 100 WHERE k = 1;
> SELECT * from t; SELECT * from mv;
>  k | a  | b
> ---++-
>  1 | 10 | 100
> (1 rows)
>  a  | k | b
> +---+-
>  10 | 1 | 100
> (1 rows)
> -- 5 seconds later
> SELECT * from t; SELECT * from mv;
>  k | a| b
> ---+--+-
>  1 | null | 100
> (1 rows)
>  a  | k | b
> +---+-
>  10 | 1 | 100
> (1 rows)
> -- that view entry's liveness-info is (probably) dead, but the entry is kept 
> alive by b=100
> DELETE b FROM t WHERE k=1;
> SELECT * from t; SELECT * from mv;
>  k | a| b
> ---+--+--
>  1 | null | null
> (1 rows)
>  a  | k | b
> +---+-
>  10 | 1 | 100
> (1 rows)
> DELETE FROM t WHERE k=1;
> cqlsh:test> SELECT * from t; SELECT * from mv;
>  k | a | b
> ---+---+---
> (0 rows)
>  a  | k | b
> +---+-
>  10 | 1 | 100
> (1 rows)
> -- deleting the base-entry doesn't help, because the view-key can not be 
> constructed anymore (a=10 already expired)
> {noformat}
> The problem here is that although the view-entry's liveness-info (probably) 
> expired correctly a regular column (`b`) keeps the view-entry live. It should 
> have disappeared since it's indexed column (`a`) expired in the corresponding 
> base-row. This is pretty severe, since that view-entry is now orphanized.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13657) Materialized Views: Index MV on TTL'ed column produces orphanized view entry if another column keeps entry live


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-13657:

Status: Open  (was: Patch Available)

> Materialized Views: Index MV on TTL'ed column produces orphanized view entry 
> if another column keeps entry live
> ---
>
> Key: CASSANDRA-13657
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13657
> Project: Cassandra
>  Issue Type: Bug
>  Components: Materialized Views
>Reporter: Fridtjof Sander
>Assignee: Krishna Dattu Koneru
>  Labels: materializedviews, ttl
>
> {noformat}
> CREATE TABLE t (k int, a int, b int, PRIMARY KEY (k));
> CREATE MATERIALIZED VIEW mv AS SELECT * FROM t WHERE k IS NOT NULL AND a IS 
> NOT NULL PRIMARY KEY (a, k);
> INSERT INTO t (k) VALUES (1);
> UPDATE t USING TTL 5 SET a = 10 WHERE k = 1;
> UPDATE t SET b = 100 WHERE k = 1;
> SELECT * from t; SELECT * from mv;
>  k | a  | b
> ---++-
>  1 | 10 | 100
> (1 rows)
>  a  | k | b
> +---+-
>  10 | 1 | 100
> (1 rows)
> -- 5 seconds later
> SELECT * from t; SELECT * from mv;
>  k | a| b
> ---+--+-
>  1 | null | 100
> (1 rows)
>  a  | k | b
> +---+-
>  10 | 1 | 100
> (1 rows)
> -- that view entry's liveness-info is (probably) dead, but the entry is kept 
> alive by b=100
> DELETE b FROM t WHERE k=1;
> SELECT * from t; SELECT * from mv;
>  k | a| b
> ---+--+--
>  1 | null | null
> (1 rows)
>  a  | k | b
> +---+-
>  10 | 1 | 100
> (1 rows)
> DELETE FROM t WHERE k=1;
> cqlsh:test> SELECT * from t; SELECT * from mv;
>  k | a | b
> ---+---+---
> (0 rows)
>  a  | k | b
> +---+-
>  10 | 1 | 100
> (1 rows)
> -- deleting the base-entry doesn't help, because the view-key can not be 
> constructed anymore (a=10 already expired)
> {noformat}
> The problem here is that although the view-entry's liveness-info (probably) 
> expired correctly a regular column (`b`) keeps the view-entry live. It should 
> have disappeared since it's indexed column (`a`) expired in the corresponding 
> base-row. This is pretty severe, since that view-entry is now orphanized.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted

[
https://issues.apache.org/jira/browse/CASSANDRA-11500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153239#comment-16153239
]

Paulo Motta commented on CASSANDRA-11500:
-

bq. As we discussed offline, we need to make sure the raw data including
tombstone, expired liveness are shipped to the coordinator side. Enforcing
strict liveness in ReadCommand.executeLocally() would remove the row before
digest or data response. Instead, we add enforceStrictLiveness to Row.purge to
get the same result but less interfaces changes for Row.

Excellent catch, my bad for overlooking this, but gladly you added a new dtest
for it:
[test_base_column_in_view_pk_complex_timestamp_with_flush|https://github.com/apache/cassandra-dtest/commit/6d77ace5361f020ba182072ade9f4ab98025c213#diff-62ba429edee6a4681782f078246c9893R993].
Enforcing strict liveness during reconciliation on the coordinator is the
correct approach to differentiate between non-existing and removed rows via
strict liveness. Ideally strict liveness would be a row property but as we saw
in the previous version, this requires changes a bunch of other classes and
interfaces, so let's keep it as a flag to Row.purge for now.

bq. Shadowable tombstone will be deprecated and use expired livenessInfo if the
deletion time is greater than merged-row deletion to avoid uncessary expired
livenessInfo.

LGTM, this will prevent using expired liveness info in most cases. We should
probably remove this as part of CASSANDRA-13826 since we're overlading
ExpiringLivenessInfo for a different purpose but it should be fine for now.

Final patch and CI results look good and I'm confident we've exercised all edge
cases properly with unit and dtests. Committed dtests to master as
{{6d77ace5361f020ba182072ade9f4ab98025c213}}. Commited to cassandra-3.0 branch
as {{1b36740ebe66b8ed4c3d6cb64eb2419a9279dfbf}} and merged up to cassandra-3.11
and trunk. Great job Zhao!

*Summary of improvements and fixed issues*

This patch address most outstanding timestamp problems with MVs on the 3.x
series without changing the binary or storage protocol. The remaining issues
which requires storage engine and binary protocol changes will be addressed on
CASSANDRA-13826 on trunk. Below is a summary of the main changes and fixes this
patch introduces.

{noformat}
* View same PK components as base

* DELETE of selected column should not affect out of order updates
(testPartialDeleteSelectedColumn)
* DELETE of unselected column/collection should not affect ordered updates
(testUpdateColumnNotInView, testPartialUpdateWithUnselectedCollections -
CASSANDRA-13127)
* Unselected columns should keep view row alive when other columns expire
(testUnselectedColumnsTTL - CASSANDRA-13127)

* Extra column on view PK

* View row should expire when view PK column expires in base
(testUpdateColumnInViewPKWithTTL - CASSANDRA-13657)
* Commutative row deletion (testCommutativeRowDeletion,
testCellTombstoneAndShadowableTombstones - CASSANDRA-13409)
* Out of order updates to extra column on view PK
(testUpdateWithColumnTimestampBiggerThanPk,
testUpdateWithColumnTimestampSmallerThanPk, CASSANDRA-11500)

* Unsupported scenarios

* DELETE of unselected column should not affect out of order updates
(testPartialDeleteUnselectedColumn - CASSANDRA-13127 - Added )
* Filtering by non-PK base column (Only on C* 3.11+, disallowed on
CASSANDRA-13798)

* Additional MV tests

* testNonBaseColumnInViewPk
* testRangeDeletion
* testStrictLivenessTombstone
* testFrozenCollectionsWithComplexInnerType
* testMVWithDifferentColumns
* testBaseTTLWithSameTimestampTest
{noformat}

*Summary of code changes*

{noformat}
1. Using expired livenessInfo if computed deletion time is greater than merged
row deletion. There are only 2 cases:
a. non-pk base column used in view pk is removed by partial update or
partial delete
b. unselected base column is removed by partial update or partial delete

Current shadowable tombstone is not used to avoid the issue of resurrecting
deleted cells (CASSANDRA-13049). We will expired-livenessInfo and merged base
row deletion instead.

2. It's strict-liveness iff there is non-key base column in view-pk. The
existence of view row is solely base on this non-key base column.

3. If there is no non-base PK column in view-pk, the view's liveness/deletion
is using max of base livenessIno + unselected column. unselected column's ttl
is used only when it affects view row liveness. Selected columns won't
contribute to livenessInfo or row deletion.
* this wouldn't support complex cases as explained above. eg. c/d
unselected, update c@10, delete c@11, update d@5. view row should be alive but
dead - We added a note to NEWS.txt as well as documentation about this
unsupported case.
* since we change the way the deletion timestamp is computed in this case,
an older deletion to the non-base PK may not

[jira] [Updated] (CASSANDRA-11500) Obsolete MV entry may not be properly deleted