[jira] [Commented] (CASSANDRA-11176) SSTableRewriter.InvalidateKeys should have a weak reference to cache

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15166809#comment-15166809
 ] 

Marcus Eriksson commented on CASSANDRA-11176:
-

uh start with -Dcassandra.debugrefcount=true, run stress and wait it seems

> SSTableRewriter.InvalidateKeys should have a weak reference to cache
> 
>
> Key: CASSANDRA-11176
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11176
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Jeremiah Jordan
>Assignee: Marcus Eriksson
> Fix For: 3.0.x
>
>
> From [~aweisberg]
> bq. The SSTableReader.DropPageCache runnable references 
> SSTableRewriter.InvalidateKeys which references the cache. The cache 
> reference should be a WeakReference.
> {noformat}
> ERROR [Strong-Reference-Leak-Detector:1] 2016-02-17 14:51:52,111  
> NoSpamLogger.java:97 - Strong self-ref loop detected 
> [/var/lib/cassandra/data/keyspace1/standard1-990bc741d56411e591d5590d7a7ad312/ma-20-big,
> private java.lang.Runnable 
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.runOnClose-org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache,
> final java.lang.Runnable 
> org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache.andThen-org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys,
> final org.apache.cassandra.cache.InstrumentingCache 
> org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys.cache-org.apache.cassandra.cache.AutoSavingCache,
> protected volatile java.util.concurrent.ScheduledFuture 
> org.apache.cassandra.cache.AutoSavingCache.saveTask-java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask,
> final java.util.concurrent.ScheduledThreadPoolExecutor 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.this$0-org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor,
> private final java.util.concurrent.BlockingQueue 
> java.util.concurrent.ThreadPoolExecutor.workQueue-java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue,
> private final java.util.concurrent.BlockingQueue 
> java.util.concurrent.ThreadPoolExecutor.workQueue-java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask,
> private java.util.concurrent.Callable 
> java.util.concurrent.FutureTask.callable-java.util.concurrent.Executors$RunnableAdapter,
> final java.lang.Runnable 
> java.util.concurrent.Executors$RunnableAdapter.task-org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable,
> private final java.lang.Runnable 
> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.runnable-org.apache.cassandra.db.ColumnFamilyStore$3,
> final org.apache.cassandra.db.ColumnFamilyStore 
> org.apache.cassandra.db.ColumnFamilyStore$3.this$0-org.apache.cassandra.db.ColumnFamilyStore,
> public final org.apache.cassandra.db.Keyspace 
> org.apache.cassandra.db.ColumnFamilyStore.keyspace-org.apache.cassandra.db.Keyspace,
> private final java.util.concurrent.ConcurrentMap 
> org.apache.cassandra.db.Keyspace.columnFamilyStores-java.util.concurrent.ConcurrentHashMap,
> private final java.util.concurrent.ConcurrentMap 
> org.apache.cassandra.db.Keyspace.columnFamilyStores-org.apache.cassandra.db.ColumnFamilyStore,
> private final org.apache.cassandra.db.lifecycle.Tracker 
> org.apache.cassandra.db.ColumnFamilyStore.data-org.apache.cassandra.db.lifecycle.Tracker,
> final java.util.concurrent.atomic.AtomicReference 
> org.apache.cassandra.db.lifecycle.Tracker.view-java.util.concurrent.atomic.AtomicReference,
> private volatile java.lang.Object 
> java.util.concurrent.atomic.AtomicReference.value-org.apache.cassandra.db.lifecycle.View,
> public final java.util.List 
> org.apache.cassandra.db.lifecycle.View.liveMemtables-com.google.common.collect.SingletonImmutableList,
> final transient java.lang.Object 
> com.google.common.collect.SingletonImmutableList.element-org.apache.cassandra.db.Memtable,
> private final org.apache.cassandra.utils.memory.MemtableAllocator 
> org.apache.cassandra.db.Memtable.allocator-org.apache.cassandra.utils.memory.SlabAllocator,
> private final 
> org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator 
> org.apache.cassandra.utils.memory.MemtableAllocator.onHeap-org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator,
> private final org.apache.cassandra.utils.memory.MemtablePool$SubPool 
> org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator.parent-org.apache.cassandra.utils.memory.MemtablePool$SubPool,
> final org.apache.cassandra.utils.memory.MemtablePool 
> org.apache.cassandra.utils.memory.MemtablePool$SubPool.this$0-org.apache.cassandra.utils.memory.SlabPool,
> final 

[jira] [Commented] (CASSANDRA-11176) SSTableRewriter.InvalidateKeys should have a weak reference to cache

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15166766#comment-15166766
 ] 

Marcus Eriksson commented on CASSANDRA-11176:
-

How do I reproduce this?

> SSTableRewriter.InvalidateKeys should have a weak reference to cache
> 
>
> Key: CASSANDRA-11176
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11176
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Jeremiah Jordan
>Assignee: Marcus Eriksson
> Fix For: 3.0.x
>
>
> From [~aweisberg]
> bq. The SSTableReader.DropPageCache runnable references 
> SSTableRewriter.InvalidateKeys which references the cache. The cache 
> reference should be a WeakReference.
> {noformat}
> ERROR [Strong-Reference-Leak-Detector:1] 2016-02-17 14:51:52,111  
> NoSpamLogger.java:97 - Strong self-ref loop detected 
> [/var/lib/cassandra/data/keyspace1/standard1-990bc741d56411e591d5590d7a7ad312/ma-20-big,
> private java.lang.Runnable 
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier.runOnClose-org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache,
> final java.lang.Runnable 
> org.apache.cassandra.io.sstable.format.SSTableReader$DropPageCache.andThen-org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys,
> final org.apache.cassandra.cache.InstrumentingCache 
> org.apache.cassandra.io.sstable.SSTableRewriter$InvalidateKeys.cache-org.apache.cassandra.cache.AutoSavingCache,
> protected volatile java.util.concurrent.ScheduledFuture 
> org.apache.cassandra.cache.AutoSavingCache.saveTask-java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask,
> final java.util.concurrent.ScheduledThreadPoolExecutor 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.this$0-org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor,
> private final java.util.concurrent.BlockingQueue 
> java.util.concurrent.ThreadPoolExecutor.workQueue-java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue,
> private final java.util.concurrent.BlockingQueue 
> java.util.concurrent.ThreadPoolExecutor.workQueue-java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask,
> private java.util.concurrent.Callable 
> java.util.concurrent.FutureTask.callable-java.util.concurrent.Executors$RunnableAdapter,
> final java.lang.Runnable 
> java.util.concurrent.Executors$RunnableAdapter.task-org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable,
> private final java.lang.Runnable 
> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.runnable-org.apache.cassandra.db.ColumnFamilyStore$3,
> final org.apache.cassandra.db.ColumnFamilyStore 
> org.apache.cassandra.db.ColumnFamilyStore$3.this$0-org.apache.cassandra.db.ColumnFamilyStore,
> public final org.apache.cassandra.db.Keyspace 
> org.apache.cassandra.db.ColumnFamilyStore.keyspace-org.apache.cassandra.db.Keyspace,
> private final java.util.concurrent.ConcurrentMap 
> org.apache.cassandra.db.Keyspace.columnFamilyStores-java.util.concurrent.ConcurrentHashMap,
> private final java.util.concurrent.ConcurrentMap 
> org.apache.cassandra.db.Keyspace.columnFamilyStores-org.apache.cassandra.db.ColumnFamilyStore,
> private final org.apache.cassandra.db.lifecycle.Tracker 
> org.apache.cassandra.db.ColumnFamilyStore.data-org.apache.cassandra.db.lifecycle.Tracker,
> final java.util.concurrent.atomic.AtomicReference 
> org.apache.cassandra.db.lifecycle.Tracker.view-java.util.concurrent.atomic.AtomicReference,
> private volatile java.lang.Object 
> java.util.concurrent.atomic.AtomicReference.value-org.apache.cassandra.db.lifecycle.View,
> public final java.util.List 
> org.apache.cassandra.db.lifecycle.View.liveMemtables-com.google.common.collect.SingletonImmutableList,
> final transient java.lang.Object 
> com.google.common.collect.SingletonImmutableList.element-org.apache.cassandra.db.Memtable,
> private final org.apache.cassandra.utils.memory.MemtableAllocator 
> org.apache.cassandra.db.Memtable.allocator-org.apache.cassandra.utils.memory.SlabAllocator,
> private final 
> org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator 
> org.apache.cassandra.utils.memory.MemtableAllocator.onHeap-org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator,
> private final org.apache.cassandra.utils.memory.MemtablePool$SubPool 
> org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator.parent-org.apache.cassandra.utils.memory.MemtablePool$SubPool,
> final org.apache.cassandra.utils.memory.MemtablePool 
> org.apache.cassandra.utils.memory.MemtablePool$SubPool.this$0-org.apache.cassandra.utils.memory.SlabPool,
> final org.apache.cassandra.utils.memory.MemtableCleanerThread 
> 

[jira] [Assigned] (CASSANDRA-10331) Establish and implement canonical bulk reading workload(s)

[Stefania (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania reassigned CASSANDRA-10331:


Assignee: Stefania

> Establish and implement canonical bulk reading workload(s)
> --
>
> Key: CASSANDRA-10331
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10331
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Ariel Weisberg
>Assignee: Stefania
> Fix For: 3.x
>
>
> Implement a client, use stress, or extend stress to a bulk reading workload 
> that is indicative of the performance we are trying to improve.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (CASSANDRA-11062) BatchlogManager May Attempt to Flush Hints After HintsService is Shutdown

[Caleb Rackliffe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Caleb Rackliffe reopened CASSANDRA-11062:
-
Reproduced In: 3.0.0

It appears my fix was probably necessary but not sufficient to fully address 
the issue, and I've been able to reproduce again, although more sporadically. 
When we drain, BatchlogManager is actually shut down before the HintsService 
and so we never see the error. However, when the StorageService shutdown hook 
runs, we don't shut down the BatchlogManager, and so if we're unlucky, it's 
scheduled task runs and tries to hit the already shut down HintsService.

I'll work on a (properly formed) set of PRs for this...

> BatchlogManager May Attempt to Flush Hints After HintsService is Shutdown
> -
>
> Key: CASSANDRA-11062
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11062
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local Write-Read Paths
>Reporter: Caleb Rackliffe
>Assignee: Caleb Rackliffe
>Priority: Minor
> Fix For: 3.0.4, 3.4
>
> Attachments: 11062-3.0.txt
>
>
> {{ScheduledThreadPoolExecutor}}'s default behavior is to keep running delayed 
> tasks after shutdown, so I think that means {{BatchlogManager}} is trying to 
> call {{replayFailedBatches()}} after drain has instructed both it and the 
> {{HintsService}} to shut down. When this happens, we get an exception when 
> that tries to submit a task to the executor in {{HintsWriteExecutor}}:
> {noformat}
> ERROR [BatchlogTasks:1] 2016-01-22 17:01:38,936  CassandraDaemon.java:195 - 
> Exception in thread Thread[BatchlogTasks:1,5,main]
> java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut 
> down
>   at 
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61)
>  ~[cassandra-all-3.0.1.816.jar:3.0.1.816]
>   at 
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) 
> [na:1.8.0_65]
>   at 
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) 
> [na:1.8.0_65]
>   at 
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:165)
>  ~[cassandra-all-3.0.1.816.jar:3.0.1.816]
>   at 
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
>  ~[na:1.8.0_65]
>   at 
> org.apache.cassandra.hints.HintsWriteExecutor.flushBufferPool(HintsWriteExecutor.java:89)
>  ~[cassandra-all-3.0.1.816.jar:3.0.1.816]
>   at 
> org.apache.cassandra.hints.HintsService.flushAndFsyncBlockingly(HintsService.java:177)
>  ~[cassandra-all-3.0.1.816.jar:3.0.1.816]
>   at 
> org.apache.cassandra.batchlog.BatchlogManager.processBatchlogEntries(BatchlogManager.java:259)
>  ~[cassandra-all-3.0.1.816.jar:3.0.1.816]
>   at 
> org.apache.cassandra.batchlog.BatchlogManager.replayFailedBatches(BatchlogManager.java:200)
>  ~[cassandra-all-3.0.1.816.jar:3.0.1.816]
>   at 
> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118)
>  ~[cassandra-all-3.0.1.816.jar:3.0.1.816]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_65]
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
> [na:1.8.0_65]
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  [na:1.8.0_65]
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  [na:1.8.0_65]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_65]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_65]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-9259) Bulk Reading from Cassandra

[Stefania (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania reassigned CASSANDRA-9259:
---

Assignee: Stefania

> Bulk Reading from Cassandra
> ---
>
> Key: CASSANDRA-9259
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9259
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Compaction, CQL, Local Write-Read Paths, Streaming and 
> Messaging, Testing
>Reporter:  Brian Hess
>Assignee: Stefania
>Priority: Critical
> Fix For: 3.x
>
>
> This ticket is following on from the 2015 NGCC.  This ticket is designed to 
> be a place for discussing and designing an approach to bulk reading.
> The goal is to have a bulk reading path for Cassandra.  That is, a path 
> optimized to grab a large portion of the data for a table (potentially all of 
> it).  This is a core element in the Spark integration with Cassandra, and the 
> speed at which Cassandra can deliver bulk data to Spark is limiting the 
> performance of Spark-plus-Cassandra operations.  This is especially of 
> importance as Cassandra will (likely) leverage Spark for internal operations 
> (for example CASSANDRA-8234).
> The core CQL to consider is the following:
> SELECT a, b, c FROM myKs.myTable WHERE Token(partitionKey) > X AND 
> Token(partitionKey) <= Y
> Here, we choose X and Y to be contained within one token range (perhaps 
> considering the primary range of a node without vnodes, for example).  This 
> query pushes 50K-100K rows/sec, which is not very fast if we are doing bulk 
> operations via Spark (or other processing frameworks - ETL, etc).  There are 
> a few causes (e.g., inefficient paging).
> There are a few approaches that could be considered.  First, we consider a 
> new "Streaming Compaction" approach.  The key observation here is that a bulk 
> read from Cassandra is a lot like a major compaction, though instead of 
> outputting a new SSTable we would output CQL rows to a stream/socket/etc.  
> This would be similar to a CompactionTask, but would strip out some 
> unnecessary things in there (e.g., some of the indexing, etc). Predicates and 
> projections could also be encapsulated in this new "StreamingCompactionTask", 
> for example.
> Another approach would be an alternate storage format.  For example, we might 
> employ Parquet (just as an example) to store the same data as in the primary 
> Cassandra storage (aka SSTables).  This is akin to Global Indexes (an 
> alternate storage of the same data optimized for a particular query).  Then, 
> Cassandra can choose to leverage this alternate storage for particular CQL 
> queries (e.g., range scans).
> These are just 2 suggestions to get the conversation going.
> One thing to note is that it will be useful to have this storage segregated 
> by token range so that when you extract via these mechanisms you do not get 
> replications-factor numbers of copies of the data.  That will certainly be an 
> issue for some Spark operations (e.g., counting).  Thus, we will want 
> per-token-range storage (even for single disks), so this will likely leverage 
> CASSANDRA-6696 (though, we'll want to also consider the single disk case).
> It is also worth discussing what the success criteria is here.  It is 
> unlikely to be as fast as EDW or HDFS performance (though, that is still a 
> good goal), but being within some percentage of that performance should be 
> set as success.  For example, 2x as long as doing bulk operations on HDFS 
> with similar node count/size/etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10458) cqlshrc: add option to always use ssl

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15166484#comment-15166484
 ] 

Stefania commented on CASSANDRA-10458:
--

Updated status to PA; this can now be rebased since CASSANDRA-11124 has just 
been committed.

> cqlshrc: add option to always use ssl
> -
>
> Key: CASSANDRA-10458
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10458
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Matt Wringe
>Assignee: Stefan Podkowinski
>  Labels: lhf
>
> I am currently running on a system in which my cassandra cluster is only 
> accessible over tls.
> The cqlshrc file is used to specify the host, the certificates and other 
> configurations, but one option its missing is to always connect over ssl.
> I would like to be able to call 'cqlsh' instead of always having to specify 
> 'cqlsh --ssl'



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11053) COPY FROM on large datasets: fix progress report and debug performance

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15166414#comment-15166414
 ] 

Stefania commented on CASSANDRA-11053:
--

bq. Can you help me understand what the issue was? It should not change return 
values. I think I'm missing something.

With cython extensions this line in cqlsh no longer works:

{code}
cassandra.cqltypes.BytesType.deserialize = staticmethod(lambda byts, 
protocol_version: bytearray(byts))
{code}

So it no longer returns byte arrays for blobs and I needed a way to distinguish 
between blob and ascii types; they are both of type {{str}} and it may well be 
that a blob contains only valid ascii bytes. At least for the time being I 
decided to look directly into the CQL type name and hence the parsing for 
composite types. We may want to add a new hook in the driver moving forward but 
I am no so sure how it would be possible with the cython extensions.

> COPY FROM on large datasets: fix progress report and debug performance
> --
>
> Key: CASSANDRA-11053
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11053
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
> Attachments: copy_from_large_benchmark.txt, 
> copy_from_large_benchmark_2.txt, parent_profile.txt, parent_profile_2.txt, 
> worker_profiles.txt, worker_profiles_2.txt
>
>
> Running COPY from on a large dataset (20G divided in 20M records) revealed 
> two issues:
> * The progress report is incorrect, it is very slow until almost the end of 
> the test at which point it catches up extremely quickly.
> * The performance in rows per second is similar to running smaller tests with 
> a smaller cluster locally (approx 35,000 rows per second). As a comparison, 
> cassandra-stress manages 50,000 rows per second under the same set-up, 
> therefore resulting 1.5 times faster. 
> See attached file _copy_from_large_benchmark.txt_ for the benchmark details.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11231) dtest failure in bootstrap_test.TestBootstrap.simple_bootstrap_test_nodata

[Russ Hatch (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15166384#comment-15166384
 ] 

Russ Hatch commented on CASSANDRA-11231:


looks to be a test problem of some kind
{noformat}
('Unable to connect to any servers', {'127.0.0.3': error(111, "Tried connecting 
to [('127.0.0.3', 9042)]. Last error: Connection refused")})
{noformat}

> dtest failure in bootstrap_test.TestBootstrap.simple_bootstrap_test_nodata
> --
>
> Key: CASSANDRA-11231
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11231
> Project: Cassandra
>  Issue Type: Test
>Reporter: Russ Hatch
>Assignee: DS Test Eng
>  Labels: dtest
>
> example failure:
> http://cassci.datastax.com/job/cassandra-2.2_dtest/526/testReport/bootstrap_test/TestBootstrap/simple_bootstrap_test_nodata
> Failed on CassCI build cassandra-2.2_dtest #526



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11231) dtest failure in bootstrap_test.TestBootstrap.simple_bootstrap_test_nodata

[Russ Hatch (JIRA)]

Russ Hatch created CASSANDRA-11231:
--

 Summary: dtest failure in 
bootstrap_test.TestBootstrap.simple_bootstrap_test_nodata
 Key: CASSANDRA-11231
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11231
 Project: Cassandra
  Issue Type: Test
Reporter: Russ Hatch
Assignee: DS Test Eng


example failure:

http://cassci.datastax.com/job/cassandra-2.2_dtest/526/testReport/bootstrap_test/TestBootstrap/simple_bootstrap_test_nodata

Failed on CassCI build cassandra-2.2_dtest #526



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11230) dtest failure in consistency_test.TestConsistency.short_read_reversed_test

[Russ Hatch (JIRA)]

Russ Hatch created CASSANDRA-11230:
--

 Summary: dtest failure in 
consistency_test.TestConsistency.short_read_reversed_test
 Key: CASSANDRA-11230
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11230
 Project: Cassandra
  Issue Type: Test
Reporter: Russ Hatch
Assignee: DS Test Eng


example failure:

http://cassci.datastax.com/job/cassandra-2.2_dtest/524/testReport/consistency_test/TestConsistency/short_read_reversed_test

Failed on CassCI build cassandra-2.2_dtest #524

flaps intermittently on earlier builds, most recent symptom:
{noformat}
[Unavailable exception] message="Cannot achieve consistency level QUORUM" 
info={'required_replicas': 2, 'alive_replicas': 1, 'consistency': 'QUORUM'}
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11229) dtest failure in repair_test.TestRepair.local_dc_repair_test

[Russ Hatch (JIRA)]

Russ Hatch created CASSANDRA-11229:
--

 Summary: dtest failure in 
repair_test.TestRepair.local_dc_repair_test
 Key: CASSANDRA-11229
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11229
 Project: Cassandra
  Issue Type: Test
Reporter: Russ Hatch
Assignee: DS Test Eng


example failure:

http://cassci.datastax.com/job/cassandra-2.2_dtest/519/testReport/repair_test/TestRepair/local_dc_repair_test

Failed on CassCI build cassandra-2.2_dtest #519

Appears to be flapping occasionally, most recent error:
{noformat}
[Unavailable exception] message="Cannot achieve consistency level ALL" 
info={'required_replicas': 4, 'alive_replicas': 3, 'consistency': 'ALL'
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10990) Support streaming of older version sstables in 3.0

[Paulo Motta (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15165679#comment-15165679
 ] 

Paulo Motta commented on CASSANDRA-10990:
-

Thanks for the review and comments [~yukim]! Follow-up below.

bq.  Isn't it simpler to just have CachedInputStream that first fills on heap 
buffer and writes out to file as buffer gets full?

+1, merged {{MemoryCachedInputStream}} and {{FileCachedInputStream}} into a 
single class {{RewindableInputStream}}, and also made some simplifications to 
cover only our needed use case. Before I was trying to make it too general 
that's why it became unnecessarily complex. Also updated tests to reflect 
changes.

bq. I think we can just set to be more sane value (few hundreds kilobytes?), 
the use case here is for static columns only.

Made 128KB initial capacity and 1MB max memory capacity to store read 
partitions. After that, read bytes are spilled to disk. Also removed 
{{ByteArrayOutputStream}} and {{ByteArrayInputStream}}, now working with a  
growing {{byte[]}} buffer.

Besides that, addressed github comments, finished previous TODO list and fixed 
all tests. Patch and tests available below.

||trunk||dtest||
|[branch|https://github.com/apache/cassandra/compare/trunk...pauloricardomg:10990]|[branch|https://github.com/riptano/cassandra-dtest/compare/master...pauloricardomg:10990]|
|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-10990-testall/lastCompletedBuild/testReport/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-10990-dtest/lastCompletedBuild/testReport/]|

I will provide 3.0 patch after review, as well as update CHANGEs, etc. 

[~philipthompson] if I used the same C* and dtests branches as before, do you 
need to update the cassci job or can I just resubmit?

> Support streaming of older version sstables in 3.0
> --
>
> Key: CASSANDRA-10990
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10990
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Jeremy Hanna
>Assignee: Paulo Motta
>
> In 2.0 we introduced support for streaming older versioned sstables 
> (CASSANDRA-5772).  In 3.0, because of the rewrite of the storage layer, this 
> became no longer supported.  So currently, while 3.0 can read sstables in the 
> 2.1/2.2 format, it cannot stream the older versioned sstables.  We should do 
> some work to make this still possible to be consistent with what 
> CASSANDRA-5772 provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11228) dtest failure in counter_tests.TestCounters.upgrade_test

[Russ Hatch (JIRA)]

Russ Hatch created CASSANDRA-11228:
--

 Summary: dtest failure in counter_tests.TestCounters.upgrade_test
 Key: CASSANDRA-11228
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11228
 Project: Cassandra
  Issue Type: Test
Reporter: Russ Hatch
Assignee: DS Test Eng


example failure:

http://cassci.datastax.com/job/cassandra-2.1_offheap_dtest/301/testReport/counter_tests/TestCounters/upgrade_test

Failed on CassCI build cassandra-2.1_offheap_dtest #301

Looks like perhaps a test timing issue, 2 failures in the past with variants of:
{noformat}
[Unavailable exception] message="Cannot achieve consistency level QUORUM"
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11053) COPY FROM on large datasets: fix progress report and debug performance

[Adam Holmberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15165662#comment-15165662
 ] 

Adam Holmberg commented on CASSANDRA-11053:
---

Just starting on this. There are a lot of formatting changes 
([96fb58|https://github.com/apache/cassandra/commit/96fb585574199b84646c1d97cc8dd689f32d4687],
 
[ce1504b|https://github.com/apache/cassandra/commit/ce1504b78e64597a1a2251ef3ecedf0452609694],
 
[8553e1|https://github.com/apache/cassandra/commit/8553e1fee4069cec6c65090e9774b68ef0de2e6c])
 that refer to the Cythonized driver. Can you help me understand what the issue 
was? It should not change return values. I think I'm missing something.

> COPY FROM on large datasets: fix progress report and debug performance
> --
>
> Key: CASSANDRA-11053
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11053
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
> Attachments: copy_from_large_benchmark.txt, 
> copy_from_large_benchmark_2.txt, parent_profile.txt, parent_profile_2.txt, 
> worker_profiles.txt, worker_profiles_2.txt
>
>
> Running COPY from on a large dataset (20G divided in 20M records) revealed 
> two issues:
> * The progress report is incorrect, it is very slow until almost the end of 
> the test at which point it catches up extremely quickly.
> * The performance in rows per second is similar to running smaller tests with 
> a smaller cluster locally (approx 35,000 rows per second). As a comparison, 
> cassandra-stress manages 50,000 rows per second under the same set-up, 
> therefore resulting 1.5 times faster. 
> See attached file _copy_from_large_benchmark.txt_ for the benchmark details.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11227) dtest failure in replication_test.SnitchConfigurationUpdateTest.test_failed_snitch_update_property_file_snitch

[Russ Hatch (JIRA)]

Russ Hatch created CASSANDRA-11227:
--

 Summary: dtest failure in 
replication_test.SnitchConfigurationUpdateTest.test_failed_snitch_update_property_file_snitch
 Key: CASSANDRA-11227
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11227
 Project: Cassandra
  Issue Type: Test
Reporter: Russ Hatch
Assignee: DS Test Eng


example failure:

http://cassci.datastax.com/job/cassandra-2.1_offheap_dtest/301/testReport/replication_test/SnitchConfigurationUpdateTest/test_failed_snitch_update_property_file_snitch

Failed on CassCI build cassandra-2.1_offheap_dtest #301
Failed on CassCI build cassandra-2.1_offheap_dtest #286

Looks to be a timeout issue on both historical failures:
{noformat}
Ran out of time waiting for topology to change on node 0
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11226) nodetool tablestats' keyspace-level metrics are wrong/misleading

[Tyler Hobbs (JIRA)]

Tyler Hobbs created CASSANDRA-11226:
---

 Summary: nodetool tablestats' keyspace-level metrics are 
wrong/misleading
 Key: CASSANDRA-11226
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11226
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
Reporter: Tyler Hobbs
Priority: Minor
 Fix For: 2.2.x, 3.0.x, 3.x


In the nodetool tablestats output (formerly cfstats), we display "keyspace" 
level metrics before the table-level metrics:

{noformat}
Keyspace: testks
Read Count: 14772528
Read Latency: 0.14456651623879135 ms.
Write Count: 4761283
Write Latency: 0.062120404521218336 ms.
Pending Flushes: 0
Table: processes
SSTable count: 7
Space used (live): 496.76 MB
Space used (total): 496.76 MB
Space used by snapshots (total): 0 bytes
Off heap memory used (total): 285.76 KB
SSTable Compression Ratio: 0.2318241570710227
Number of keys (estimate): 3027
Memtable cell count: 2140
Memtable data size: 1.66 MB
Memtable off heap memory used: 0 bytes
Memtable switch count: 967
Local read count: 14772528
Local read latency: 0.159 ms
Local write count: 4761283
Local write latency: 0.068 ms
{noformat}

However, the keyspace-level metrics are misleading, at best.  They are 
aggregate metrics for every table in the keyspace _that is included in the 
command line filters_.  So, if you run {{tablestats}} for a single table, the 
keyspace-level stats will only reflect that table's stats.

I see two possible fixes:
# If the command line options don't include the entire keyspace, skip the 
keyspace-level stats
# Ignore the command line options, and always make the keyspace-level stats an 
aggregate of all tables in the keyspace

My only concern with option 2 is that performance may suffer a bit on keyspaces 
with many tables.  However, this is a command line tool, so as long as the 
response time is reasonable, I don't think it's a big deal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-9692) Print sensible units for all log messages

[Giampaolo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Giampaolo updated CASSANDRA-9692:
-
Attachment: Cassandra9692-trunk-giampaolo-trapasso-at-radicalbit-io.diff

The patch as separate file for the records.

> Print sensible units for all log messages
> -
>
> Key: CASSANDRA-9692
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9692
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Benedict
>Assignee: Giampaolo
>Priority: Minor
>  Labels: lhf
> Fix For: 3.x
>
> Attachments: 
> Cassandra9692-trunk-giampaolo-trapasso-at-radicalbit-io.diff
>
>
> Like CASSANDRA-9691, this has bugged me too long. it also adversely impacts 
> log analysis. I've introduced some improvements to the bits I touched for 
> CASSANDRA-9681, but we should do this across the codebase. It's a small 
> investment for a lot of long term clarity in the logs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11085) cassandra-3.3 eclipse-warnings

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163739#comment-15163739
 ] 

Jason Brown commented on CASSANDRA-11085:
-

+1

> cassandra-3.3 eclipse-warnings
> --
>
> Key: CASSANDRA-11085
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11085
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Michael Shuler
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.x
>
> Attachments: 11085-3.3.txt
>
>
> REF = origin/cassandra-3.3 
> COMMIT = 1a31958bfa2adb04ec965e6e2776862eee30ecf4
> {noformat}
> # 1/27/16 9:03:40 PM UTC
> # Eclipse Compiler for Java(TM) v20150120-1634, 3.10.2, Copyright IBM Corp 
> 2000, 2013. All rights reserved.
> --
> 1. ERROR in 
> /var/lib/jenkins/workspace/cassandra-3.3_eclipse-warnings/src/java/org/apache/cassandra/db/compaction/writers/MaxSSTableSizeWriter.java
>  (at line 86)
>   SSTableWriter writer = 
> SSTableWriter.create(Descriptor.fromFilename(cfs.getSSTablePath(getDirectories().getLocationForDisk(sstableDirectory))),
> ^^
> Potential resource leak: 'writer' may not be closed
> --
> --
> 2. ERROR in 
> /var/lib/jenkins/workspace/cassandra-3.3_eclipse-warnings/src/java/org/apache/cassandra/db/compaction/writers/SplittingSizeTieredCompactionWriter.java
>  (at line 108)
>   SSTableWriter writer = 
> SSTableWriter.create(Descriptor.fromFilename(cfs.getSSTablePath(getDirectories().getLocationForDisk(location))),
> ^^
> Potential resource leak: 'writer' may not be closed
> --
> --
> 3. ERROR in 
> /var/lib/jenkins/workspace/cassandra-3.3_eclipse-warnings/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
>  (at line 1073)
>   SSTableMultiWriter writer = writerIterator.next();
>  ^^
> Potential resource leak: 'writer' may not be closed
> --
> --
> 4. ERROR in 
> /var/lib/jenkins/workspace/cassandra-3.3_eclipse-warnings/src/java/org/apache/cassandra/net/OutboundTcpConnectionPool.java
>  (at line 141)
>   return channel.socket();
>   
> Potential resource leak: 'channel' may not be closed at this location
> --
> --
> 5. ERROR in 
> /var/lib/jenkins/workspace/cassandra-3.3_eclipse-warnings/src/java/org/apache/cassandra/metrics/TableMetrics.java
>  (at line 335)
>   return 
> computeCompressionRatio(Iterables.concat(Iterables.transform(Keyspace.all(),
>  ^^^
> The method computeCompressionRatio(Iterable) in the type 
> TableMetrics is not applicable for the arguments (Iterable)
> --
> --
> 6. ERROR in 
> /var/lib/jenkins/workspace/cassandra-3.3_eclipse-warnings/src/java/org/apache/cassandra/hints/CompressedChecksummedDataInput.java
>  (at line 156)
>   return builder.build();
>   ^^^
> Potential resource leak: '' may not 
> be closed at this location
> --
> 6 problems (6 errors)
> {noformat}
> Check latest build on 
> http://cassci.datastax.com/job/cassandra-3.3_eclipse-warnings/ for the most 
> current warnings



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11085) cassandra-3.3 eclipse-warnings

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown updated CASSANDRA-11085:

Reviewer: Jason Brown

> cassandra-3.3 eclipse-warnings
> --
>
> Key: CASSANDRA-11085
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11085
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Michael Shuler
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.x
>
> Attachments: 11085-3.3.txt
>
>
> REF = origin/cassandra-3.3 
> COMMIT = 1a31958bfa2adb04ec965e6e2776862eee30ecf4
> {noformat}
> # 1/27/16 9:03:40 PM UTC
> # Eclipse Compiler for Java(TM) v20150120-1634, 3.10.2, Copyright IBM Corp 
> 2000, 2013. All rights reserved.
> --
> 1. ERROR in 
> /var/lib/jenkins/workspace/cassandra-3.3_eclipse-warnings/src/java/org/apache/cassandra/db/compaction/writers/MaxSSTableSizeWriter.java
>  (at line 86)
>   SSTableWriter writer = 
> SSTableWriter.create(Descriptor.fromFilename(cfs.getSSTablePath(getDirectories().getLocationForDisk(sstableDirectory))),
> ^^
> Potential resource leak: 'writer' may not be closed
> --
> --
> 2. ERROR in 
> /var/lib/jenkins/workspace/cassandra-3.3_eclipse-warnings/src/java/org/apache/cassandra/db/compaction/writers/SplittingSizeTieredCompactionWriter.java
>  (at line 108)
>   SSTableWriter writer = 
> SSTableWriter.create(Descriptor.fromFilename(cfs.getSSTablePath(getDirectories().getLocationForDisk(location))),
> ^^
> Potential resource leak: 'writer' may not be closed
> --
> --
> 3. ERROR in 
> /var/lib/jenkins/workspace/cassandra-3.3_eclipse-warnings/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
>  (at line 1073)
>   SSTableMultiWriter writer = writerIterator.next();
>  ^^
> Potential resource leak: 'writer' may not be closed
> --
> --
> 4. ERROR in 
> /var/lib/jenkins/workspace/cassandra-3.3_eclipse-warnings/src/java/org/apache/cassandra/net/OutboundTcpConnectionPool.java
>  (at line 141)
>   return channel.socket();
>   
> Potential resource leak: 'channel' may not be closed at this location
> --
> --
> 5. ERROR in 
> /var/lib/jenkins/workspace/cassandra-3.3_eclipse-warnings/src/java/org/apache/cassandra/metrics/TableMetrics.java
>  (at line 335)
>   return 
> computeCompressionRatio(Iterables.concat(Iterables.transform(Keyspace.all(),
>  ^^^
> The method computeCompressionRatio(Iterable) in the type 
> TableMetrics is not applicable for the arguments (Iterable)
> --
> --
> 6. ERROR in 
> /var/lib/jenkins/workspace/cassandra-3.3_eclipse-warnings/src/java/org/apache/cassandra/hints/CompressedChecksummedDataInput.java
>  (at line 156)
>   return builder.build();
>   ^^^
> Potential resource leak: '' may not 
> be closed at this location
> --
> 6 problems (6 errors)
> {noformat}
> Check latest build on 
> http://cassci.datastax.com/job/cassandra-3.3_eclipse-warnings/ for the most 
> current warnings



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11225) dtest failure in consistency_test.TestAccuracy.test_simple_strategy_counters

[Russ Hatch (JIRA)]

Russ Hatch created CASSANDRA-11225:
--

 Summary: dtest failure in 
consistency_test.TestAccuracy.test_simple_strategy_counters
 Key: CASSANDRA-11225
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11225
 Project: Cassandra
  Issue Type: Test
Reporter: Russ Hatch
Assignee: DS Test Eng


example failure:

http://cassci.datastax.com/job/cassandra-2.1_novnode_dtest/209/testReport/consistency_test/TestAccuracy/test_simple_strategy_counters

Failed on CassCI build cassandra-2.1_novnode_dtest #209

error: "AssertionError: Failed to read value from sufficient number of nodes, 
required 2 but got 1 - [574, 2]"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11224) dtest failure in bootstrap_test.TestBootstrap.failed_bootstrap_wiped_node_can_join_test

[Russ Hatch (JIRA)]

Russ Hatch created CASSANDRA-11224:
--

 Summary: dtest failure in 
bootstrap_test.TestBootstrap.failed_bootstrap_wiped_node_can_join_test
 Key: CASSANDRA-11224
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11224
 Project: Cassandra
  Issue Type: Test
Reporter: Russ Hatch
Assignee: DS Test Eng


example failure:

http://cassci.datastax.com/job/cassandra-2.1_novnode_dtest/209/testReport/bootstrap_test/TestBootstrap/failed_bootstrap_wiped_node_can_join_test

Failed on CassCI build cassandra-2.1_novnode_dtest #209

Happening only intermittently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10392) Allow Cassandra to trace to custom tracing implementations

[mck (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163692#comment-15163692
 ] 

mck commented on CASSANDRA-10392:
-

sweet, was just running the same patch through tests. Thanks [~tjake]!

> Allow Cassandra to trace to custom tracing implementations 
> ---
>
> Key: CASSANDRA-10392
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10392
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: mck
>Assignee: mck
> Fix For: 3.4
>
> Attachments: 10392-trunk.txt, cassandra-dtest_master-10392.txt
>
>
> It can be possible to use an external tracing solution in Cassandra by 
> abstracting out the writing of tracing to system_traces tables in the tracing 
> package to separate implementation classes and leaving abstract classes in 
> place that define the interface and behaviour otherwise of C* tracing.
> Then via a system property "cassandra.custom_tracing_class" the Tracing class 
> implementation could be swapped out with something third party.
> An example of this is adding Zipkin tracing into Cassandra in the Summit 
> [presentation|http://thelastpickle.com/files/2015-09-24-using-zipkin-for-full-stack-tracing-including-cassandra/presentation/tlp-reveal.js/tlp-cassandra-zipkin.html].
>  Code for the implemented Zipkin plugin can be found at 
> https://github.com/thelastpickle/cassandra-zipkin-tracing/
> In addition this patch passes the custom payload through into the tracing 
> session allowing a third party tracing solution like Zipkin to do full-stack 
> tracing from clients through and into Cassandra.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10707) Add support for Group By to Select statement

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163608#comment-15163608
 ] 

Sylvain Lebresne commented on CASSANDRA-10707:
--

The first point that needs discussing is that this patch changes the inter-node 
protocol and this without bumping the protocol version, which is something 
we've never done.

On the one side bumping the protocol version is something we've kept for major 
releases and so far planned not to do before 4.0. But on the other side, not 
being able to add anything to the inter-node messages limits a lot new features 
(like this one) and doesn't make sense in a tick-tock world.

We could accept this kind of change without bumping the version on the argument 
that it's only "additive" to the serialization format and that it breaks 
nothing if the new feature is not used, and we'd document that "you shall not 
use GROUP BY until all nodes in the cluster are updated to a version that 
supports it". But it's not something to treat lightly because if a user don't 
respect that rule, older node will fail deserialization of the message, which 
will drop the connection, which can impact other on-ongoing queries, which is 
not good.

Alternatively, we could simply bump the protocol version, forgetting the "not 
before 4.0" rule. The advantage is that if we do so we can have the sending 
node (instead of the receiving one) throw if a GROUP BY is used but not all 
nodes are up to date, which is cleaner and avoid the connection dropped 
problem.  

So something we need to decide first, and maybe we need a separate ticket to 
have that discussion.

Anyway, outside of this question, some general remarks on the patch:
* In the parser, we have the {{GROUP BY}} before the {{ORDER BY}}. On the one 
side it's more consistent with {{SQL}} but on the other side, we do happen to 
order before we group. And we kind of know that post-query ordering is not 
really something we can do due to paging (we do happen to do it in a few case 
when paging is not involved out of backward compatibility but given the 
limitation, I don't think we'll want to ever extend that) so it does feel the 
{{ORDER BY}} should come before {{GROUP BY}}.
* The {{forPagingByQueryPager}} and {{forGroupByInternalPaging}} methods are 
not very intuitive: it's unclear when just looking at {{DataLimits}} why they 
exist and why you should use them or not. I'm also not entirely sure we need 
the difference between the two. If I understand correctly, the problem is that 
when a replica ends a page in the middle of a group, the coordinator don't know 
if said replica has stopped because it reaches the end of a group which was the 
last one to return (in which case we're done), or if the replica stopped 
because of the page limit (but the group itself is not finished). But the 
coordinator should be able to figure that out by checking it's row counts: if 
after having consumed the page from the replica we've reached the group limit 
_or_ there was less row in the page than the page limit, then we're done, 
otherwise we should ask for more.  Granted, that does mean in the rare case 
where we've exhausted all data exactly on the page limit, we'll do an 
additional query (that will be empty), but pretty sure you have that problem in 
one form or another whatever you do.  Am I missing something here?
* Not sure how I feel about {{GroupSelector}}. It's a bit confusing as of this 
patch and while I get that its for allowing functions later, it might not be as 
generic as we may need. For instance, if we ever want to support function on 
multiple columns (which would be possible, assuming those are consecutive PK 
columns), this won't work, at least not as implemented. I think I'd slightly 
prefer removing {{GroupSelector}} for now and just rely on extending 
{{GroupBySpecification}} for future extension (typically we'll add a 
{{withFunctionGroupBySpec}} which would have all liberty as to how it is 
implemented).  Doing so also mean we can simplify the probably common and only 
currently supported case as we just need to keep the length of the prefix we 
group by rather than a list of (dumb) selector.
* I feel keeping/passing the {{lastPartitionKey}} and {{lastClustering}} in 
{{DataLimits.CQLGroupByLimits}} and {{GroupMaker}} is not as future 
proof/generic as the patch is otherwise trying to be. Typically, if we allow 
functions, the {{lastClustering}} won't really be a clustering. Granted, it'll 
still fit into a {{ByteBuffer[]}} but it'll be a bit of an abuse. So I would 
suggest adding {{GroupMaker.State}} concept whose implementation would be 
specific to the {{GroupMaker/GroupBySpecification}} but would be transparent to 
other code. Imo that would make some code cleaner (for instance, 
{{GroupMaker.GROUP_EVERYTHING}} wouldn't need to store anything, which makes 
sense).
* {{GroupBySpeccification.NO_GROUPING}} feels 

[jira] [Updated] (CASSANDRA-11203) Improve 'nothing to repair' message when RF=1

[Jason Kania (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Kania updated CASSANDRA-11203:

Summary: Improve 'nothing to repair' message when RF=1  (was: Improve 
nothing to repair message when RF=1)

> Improve 'nothing to repair' message when RF=1
> -
>
> Key: CASSANDRA-11203
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11203
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
> Environment: debian jesse up to date content
>Reporter: Jason Kania
>Priority: Trivial
>  Labels: lhf
>
> When nodetool repair is run, it indicates that no repair is needed on some 
> keyspaces but on others it attempts repair. However, when run multiple times, 
> the output seems to indicate that the same triggering conditions still 
> persists that indicate a problem. Alternatively, the output could indicate 
> that the underlying condition has not been resolved.
> root@marble:/var/lib/cassandra/data/sensordb/periodicReading# nodetool repair
> [2016-02-21 23:33:10,356] Nothing to repair for keyspace 'sensordb'
> [2016-02-21 23:33:10,364] Nothing to repair for keyspace 'system_auth'
> [2016-02-21 23:33:10,402] Starting repair command #1, repairing keyspace 
> system_traces with repair options (parallelism: parallel, primary range: 
> false, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: 
> [], hosts: [], # of ranges: 256)
> [2016-02-21 23:33:12,144] Repair completed successfully
> [2016-02-21 23:33:12,157] Repair command #1 finished in 1 second
> root@marble:/var/lib/cassandra/data/sensordb/periodicReading# nodetool repair
> [2016-02-21 23:33:31,683] Nothing to repair for keyspace 'sensordb'
> [2016-02-21 23:33:31,689] Nothing to repair for keyspace 'system_auth'
> [2016-02-21 23:33:31,713] Starting repair command #2, repairing keyspace 
> system_traces with repair options (parallelism: parallel, primary range: 
> false, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: 
> [], hosts: [], # of ranges: 256)
> [2016-02-21 23:33:33,324] Repair completed successfully
> [2016-02-21 23:33:33,334] Repair command #2 finished in 1 second



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-10392) Allow Cassandra to trace to custom tracing implementations

[T Jake Luciani (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163470#comment-15163470
 ] 

T Jake Luciani edited comment on CASSANDRA-10392 at 2/24/16 6:31 PM:
-

committed in {{bf25e668f416b1c279986d1b23fee2a0192d8022}}

Thanks!


was (Author: tjake):
committed in {bf25e668f416b1c279986d1b23fee2a0192d8022}

Thanks!

> Allow Cassandra to trace to custom tracing implementations 
> ---
>
> Key: CASSANDRA-10392
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10392
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: mck
>Assignee: mck
> Fix For: 3.4
>
> Attachments: 10392-trunk.txt, cassandra-dtest_master-10392.txt
>
>
> It can be possible to use an external tracing solution in Cassandra by 
> abstracting out the writing of tracing to system_traces tables in the tracing 
> package to separate implementation classes and leaving abstract classes in 
> place that define the interface and behaviour otherwise of C* tracing.
> Then via a system property "cassandra.custom_tracing_class" the Tracing class 
> implementation could be swapped out with something third party.
> An example of this is adding Zipkin tracing into Cassandra in the Summit 
> [presentation|http://thelastpickle.com/files/2015-09-24-using-zipkin-for-full-stack-tracing-including-cassandra/presentation/tlp-reveal.js/tlp-cassandra-zipkin.html].
>  Code for the implemented Zipkin plugin can be found at 
> https://github.com/thelastpickle/cassandra-zipkin-tracing/
> In addition this patch passes the custom payload through into the tracing 
> session allowing a third party tracing solution like Zipkin to do full-stack 
> tracing from clients through and into Cassandra.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-10392) Allow Cassandra to trace to custom tracing implementations

[T Jake Luciani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

T Jake Luciani resolved CASSANDRA-10392.

   Resolution: Fixed
Fix Version/s: (was: 3.x)
   3.4

committed in {bf25e668f416b1c279986d1b23fee2a0192d8022}

Thanks!

> Allow Cassandra to trace to custom tracing implementations 
> ---
>
> Key: CASSANDRA-10392
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10392
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: mck
>Assignee: mck
> Fix For: 3.4
>
> Attachments: 10392-trunk.txt, cassandra-dtest_master-10392.txt
>
>
> It can be possible to use an external tracing solution in Cassandra by 
> abstracting out the writing of tracing to system_traces tables in the tracing 
> package to separate implementation classes and leaving abstract classes in 
> place that define the interface and behaviour otherwise of C* tracing.
> Then via a system property "cassandra.custom_tracing_class" the Tracing class 
> implementation could be swapped out with something third party.
> An example of this is adding Zipkin tracing into Cassandra in the Summit 
> [presentation|http://thelastpickle.com/files/2015-09-24-using-zipkin-for-full-stack-tracing-including-cassandra/presentation/tlp-reveal.js/tlp-cassandra-zipkin.html].
>  Code for the implemented Zipkin plugin can be found at 
> https://github.com/thelastpickle/cassandra-zipkin-tracing/
> In addition this patch passes the custom payload through into the tracing 
> session allowing a third party tracing solution like Zipkin to do full-stack 
> tracing from clients through and into Cassandra.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

cassandra git commit: Add support for custom tracing implementations

[jake]

Repository: cassandra
Updated Branches:
  refs/heads/trunk c94a9f236 -> bf25e668f


Add support for custom tracing implementations

Patch by Mick Semb Wever; reviewed by tjake for CASSANDRA-10392


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bf25e668
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bf25e668
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bf25e668

Branch: refs/heads/trunk
Commit: bf25e668f416b1c279986d1b23fee2a0192d8022
Parents: c94a9f2
Author: Mick Semb Wever 
Authored: Wed Feb 24 10:26:44 2016 -0500
Committer: T Jake Luciani 
Committed: Wed Feb 24 13:28:18 2016 -0500

--
 CHANGES.txt |   1 +
 .../org/apache/cassandra/net/MessageOut.java|   7 +-
 .../cassandra/net/OutboundTcpConnection.java|   2 +-
 .../apache/cassandra/service/QueryState.java|  12 +-
 .../cassandra/tracing/ExpiredTraceState.java|  24 ++-
 .../apache/cassandra/tracing/TraceState.java|  61 +--
 .../cassandra/tracing/TraceStateImpl.java   |  74 
 .../org/apache/cassandra/tracing/Tracing.java   | 105 ++-
 .../apache/cassandra/tracing/TracingImpl.java   |  89 ++
 .../transport/messages/ExecuteMessage.java  |   2 +-
 .../apache/cassandra/tracing/TracingTest.java   | 173 +++
 11 files changed, 437 insertions(+), 113 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/bf25e668/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 50a298e..bc21818 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.4
+ * Allow custom tracing implementations (CASSANDRA-10392)
  * Extract LoaderOptions to be able to be used from outside (CASSANDRA-10637)
  * fix OnDiskIndexTest to properly treat empty ranges (CASSANDRA-11205)
  * fix TrackerTest to handle new notifications (CASSANDRA-11178)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bf25e668/src/java/org/apache/cassandra/net/MessageOut.java
--
diff --git a/src/java/org/apache/cassandra/net/MessageOut.java 
b/src/java/org/apache/cassandra/net/MessageOut.java
index a524e7a..bc5c41b 100644
--- a/src/java/org/apache/cassandra/net/MessageOut.java
+++ b/src/java/org/apache/cassandra/net/MessageOut.java
@@ -33,10 +33,6 @@ import org.apache.cassandra.io.IVersionedSerializer;
 import org.apache.cassandra.io.util.DataOutputPlus;
 import org.apache.cassandra.tracing.Tracing;
 import org.apache.cassandra.utils.FBUtilities;
-import org.apache.cassandra.utils.UUIDGen;
-
-import static org.apache.cassandra.tracing.Tracing.TRACE_HEADER;
-import static org.apache.cassandra.tracing.Tracing.TRACE_TYPE;
 import static org.apache.cassandra.tracing.Tracing.isTracing;
 
 public class MessageOut
@@ -61,8 +57,7 @@ public class MessageOut
  payload,
  serializer,
  isTracing()
- ? ImmutableMap.of(TRACE_HEADER, 
UUIDGen.decompose(Tracing.instance.getSessionId()),
-   TRACE_TYPE, new byte[] { 
Tracing.TraceType.serialize(Tracing.instance.getTraceType()) })
+ ? Tracing.instance.getTraceHeaders()
  : Collections.emptyMap());
 }
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bf25e668/src/java/org/apache/cassandra/net/OutboundTcpConnection.java
--
diff --git a/src/java/org/apache/cassandra/net/OutboundTcpConnection.java 
b/src/java/org/apache/cassandra/net/OutboundTcpConnection.java
index 7b6e26e..8b1ecf3 100644
--- a/src/java/org/apache/cassandra/net/OutboundTcpConnection.java
+++ b/src/java/org/apache/cassandra/net/OutboundTcpConnection.java
@@ -284,7 +284,7 @@ public class OutboundTcpConnection extends Thread
 {
 byte[] traceTypeBytes = 
qm.message.parameters.get(Tracing.TRACE_TYPE);
 Tracing.TraceType traceType = traceTypeBytes == null ? 
Tracing.TraceType.QUERY : Tracing.TraceType.deserialize(traceTypeBytes[0]);
-
TraceState.mutateWithTracing(ByteBuffer.wrap(sessionBytes), message, -1, 
traceType.getTTL());
+Tracing.instance.trace(ByteBuffer.wrap(sessionBytes), 
message, traceType.getTTL());
 }
 else
 {

http://git-wip-us.apache.org/repos/asf/cassandra/blob/bf25e668/src/java/org/apache/cassandra/service/QueryState.java
--
diff --git a/src/java/org/apache/cassandra/service/QueryState.java 
b/src/java/org/apache/cassandra/service/QueryState.java
index 

[jira] [Commented] (CASSANDRA-10070) Automatic repair scheduling

[Marcus Olsson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163462#comment-15163462
 ] 

Marcus Olsson commented on CASSANDRA-10070:
---

For the basic implementation I think the tasks could be broken down as:
- Resource locking API & implementation
-- Maintenance scheduling API & basic repair scheduling
--- Rejection policy interface & default implementations
--- Configuration support
 Table configuration
 Global configuration (for pausing repairs in the basic implementation)
 Node configuration
--- Aborting/interrupting repairs (Requires CASSANDRA-3486,CASSANDRA-11190)
 Polling and monitoring module
 Failure handling and retry 

So that we start with the resource locking and then move on to the maintenance 
scheduling API. And after that I think most tasks could be discussed in 
parallel. Also I removed the task for management commands since I think it 
would be easier to add them while implementing the features. WDYT?

> Automatic repair scheduling
> ---
>
> Key: CASSANDRA-10070
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10070
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Marcus Olsson
>Assignee: Marcus Olsson
>Priority: Minor
> Fix For: 3.x
>
> Attachments: Distributed Repair Scheduling.doc
>
>
> Scheduling and running repairs in a Cassandra cluster is most often a 
> required task, but this can both be hard for new users and it also requires a 
> bit of manual configuration. There are good tools out there that can be used 
> to simplify things, but wouldn't this be a good feature to have inside of 
> Cassandra? To automatically schedule and run repairs, so that when you start 
> up your cluster it basically maintains itself in terms of normal 
> anti-entropy, with the possibility for manual configuration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10637) Extract LoaderOptions and refactor BulkLoader to be able to be used from within existing Java code instead of just through main()

[Eric Fenderbosch (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163452#comment-15163452
 ] 

Eric Fenderbosch commented on CASSANDRA-10637:
--

Thanks! Glad I could contribute.

> Extract LoaderOptions and refactor BulkLoader to be able to be used from 
> within existing Java code instead of just through main()
> -
>
> Key: CASSANDRA-10637
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10637
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Eric Fenderbosch
>Assignee: Eric Fenderbosch
>Priority: Minor
> Fix For: 3.4
>
>
> We are writing a service to migrate data from various RDMBS tables in to 
> Cassandra. We write out a CSV from the source system, use CQLSSTableWriter to 
> write sstables to disk, then call sstableloader to stream to the Cassandra 
> cluster.
> Right now, we either have to:
> * return a CSV location from one Java process to a wrapper script which then 
> kicks off sstableloader
> * or call sstableloader via Runtime.getRuntime().exec
> * or call BulkLoader.main from within our Java code, using a custom 
> SecurityManager to trap the System.exit calls
> * or subclass BulkLoader putting the subclass in the 
> org.apache.cassandra.tools package in order to access the package scoped 
> inner classes
> None of these solutions are ideal. Ideally, we should be able to use the 
> functionality of BulkLoader.main directly. I've extracted LoaderOptions to a 
> top level class that uses the builder pattern so that it can be used as part 
> of a Java migration service directly.
> Creating the builder can now be performed with a fluent builder interface:
> LoaderOptions options = LoaderOptions.builder(). //
> connectionsPerHost(2). //
> directory(directory). //
> hosts(hosts). //
> build();
> Or used to parse command line arguments:
> LoaderOptions options = LoaderOptions.builder().parseArgs(args).build();
> A new load method takes a LoaderOptions parameter and throws 
> BulkLoadException instead of System.exit(1).
> Fork on github can be found here:
> https://github.com/efenderbosch/cassandra



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-7464) Replace sstable2json

[Yuki Morishita (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-7464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuki Morishita updated CASSANDRA-7464:
--
Summary: Replace sstable2json  (was: Replace sstable2json and json2sstable)

> Replace sstable2json
> 
>
> Key: CASSANDRA-7464
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7464
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Sylvain Lebresne
>Assignee: Chris Lohfink
>Priority: Minor
>  Labels: docs-impacting
> Fix For: 3.0.4, 3.4
>
> Attachments: sstable-only.patch, sstabledump.patch
>
>
> Both tools are pretty awful. They are primarily meant for debugging (there is 
> much more efficient and convenient ways to do import/export data), but their 
> output manage to be hard to handle both for humans and for tools (especially 
> as soon as you have modern stuff like composites).
> There is value to having tools to export sstable contents into a format that 
> is easy to manipulate by human and tools for debugging, small hacks and 
> general tinkering, but sstable2json and json2sstable are not that.  
> So I propose that we deprecate those tools and consider writing better 
> replacements. It shouldn't be too hard to come up with an output format that 
> is more aware of modern concepts like composites, UDTs, 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-7464) Replace sstable2json

[Yuki Morishita (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-7464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yuki Morishita updated CASSANDRA-7464:
--
Labels: docs-impacting  (was: )

> Replace sstable2json
> 
>
> Key: CASSANDRA-7464
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7464
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Sylvain Lebresne
>Assignee: Chris Lohfink
>Priority: Minor
>  Labels: docs-impacting
> Fix For: 3.0.4, 3.4
>
> Attachments: sstable-only.patch, sstabledump.patch
>
>
> Both tools are pretty awful. They are primarily meant for debugging (there is 
> much more efficient and convenient ways to do import/export data), but their 
> output manage to be hard to handle both for humans and for tools (especially 
> as soon as you have modern stuff like composites).
> There is value to having tools to export sstable contents into a format that 
> is easy to manipulate by human and tools for debugging, small hacks and 
> general tinkering, but sstable2json and json2sstable are not that.  
> So I propose that we deprecate those tools and consider writing better 
> replacements. It shouldn't be too hard to come up with an output format that 
> is more aware of modern concepts like composites, UDTs, 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/3] cassandra git commit: Add sstabledump tool

[yukim]

Add sstabledump tool

patch by Chris Lohfink; reviewed by yukim for CASSANDRA-7464


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/71b1c4a6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/71b1c4a6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/71b1c4a6

Branch: refs/heads/trunk
Commit: 71b1c4a63187f746c0caecc41bc07b42dedd3488
Parents: a623977
Author: Chris Lohfink 
Authored: Tue Feb 23 11:22:26 2016 -0600
Committer: Yuki Morishita 
Committed: Wed Feb 24 11:48:53 2016 -0600

--
 CHANGES.txt |   1 +
 NEWS.txt|  14 +
 .../org/apache/cassandra/config/CFMetaData.java |   8 +-
 .../cassandra/db/SerializationHeader.java   |  25 +
 .../db/rows/AbstractRangeTombstoneMarker.java   |   4 +
 .../apache/cassandra/db/rows/AbstractRow.java   |  12 +-
 .../apache/cassandra/db/rows/Unfiltered.java|   1 +
 .../io/sstable/format/SSTableReader.java|   8 +
 .../io/sstable/format/big/BigTableReader.java   |  13 +-
 .../io/sstable/format/big/BigTableScanner.java  |   5 +
 .../apache/cassandra/tools/JsonTransformer.java | 501 +++
 .../apache/cassandra/tools/SSTableExport.java   | 242 +
 .../io/sstable/SSTableScannerTest.java  |   8 +-
 tools/bin/sstabledump   |  52 ++
 tools/bin/sstabledump.bat   |  48 ++
 15 files changed, 933 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/71b1c4a6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 4549ded..aefc02e 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.4
+ * Add sstabledump tool (CASSANDRA-7464)
  * Introduce backpressure for hints (CASSANDRA-10972)
  * Fix ClusteringPrefix not being able to read tombstone range boundaries 
(CASSANDRA-11158)
  * Prevent logging in sandboxed state (CASSANDRA-11033)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/71b1c4a6/NEWS.txt
--
diff --git a/NEWS.txt b/NEWS.txt
index 89fc4a7..5fca578 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -13,6 +13,20 @@ restore snapshots created with the previous major version 
using the
 'sstableloader' tool. You can upgrade the file format of your snapshots
 using the provided 'sstableupgrade' tool.
 
+3.0.4
+=
+
+New features
+
+   - sstabledump tool is added to be 3.0 version of former sstable2json. The 
tool only
+ supports v3.0+ SSTables. See tool's help for more detail.
+
+Upgrading
+-
+   - Nothing specific to this release, but please see previous versions 
upgrading section,
+ especially if you are upgrading from 2.2.
+
+
 3.0.3
 =
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/71b1c4a6/src/java/org/apache/cassandra/config/CFMetaData.java
--
diff --git a/src/java/org/apache/cassandra/config/CFMetaData.java 
b/src/java/org/apache/cassandra/config/CFMetaData.java
index cb6d3b8..79cd779 100644
--- a/src/java/org/apache/cassandra/config/CFMetaData.java
+++ b/src/java/org/apache/cassandra/config/CFMetaData.java
@@ -1132,7 +1132,7 @@ public final class CFMetaData
 private final boolean isSuper;
 private final boolean isCounter;
 private final boolean isView;
-private IPartitioner partitioner;
+private Optional partitioner;
 
 private UUID tableId;
 
@@ -1150,7 +1150,7 @@ public final class CFMetaData
 this.isSuper = isSuper;
 this.isCounter = isCounter;
 this.isView = isView;
-this.partitioner = DatabaseDescriptor.getPartitioner();
+this.partitioner = Optional.empty();
 }
 
 public static Builder create(String keyspace, String table)
@@ -1185,7 +1185,7 @@ public final class CFMetaData
 
 public Builder withPartitioner(IPartitioner partitioner)
 {
-this.partitioner = partitioner;
+this.partitioner = Optional.ofNullable(partitioner);
 return this;
 }
 
@@ -1296,7 +1296,7 @@ public final class CFMetaData
   partitions,
   clusterings,
   builder.build(),
-  partitioner);
+  
partitioner.orElseGet(DatabaseDescriptor::getPartitioner));
 }
 }
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/71b1c4a6/src/java/org/apache/cassandra/db/SerializationHeader.java

[1/3] cassandra git commit: Add sstabledump tool

[yukim]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 a62397782 -> 71b1c4a63
  refs/heads/trunk 6d15a6da7 -> c94a9f236


Add sstabledump tool

patch by Chris Lohfink; reviewed by yukim for CASSANDRA-7464


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/71b1c4a6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/71b1c4a6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/71b1c4a6

Branch: refs/heads/cassandra-3.0
Commit: 71b1c4a63187f746c0caecc41bc07b42dedd3488
Parents: a623977
Author: Chris Lohfink 
Authored: Tue Feb 23 11:22:26 2016 -0600
Committer: Yuki Morishita 
Committed: Wed Feb 24 11:48:53 2016 -0600

--
 CHANGES.txt |   1 +
 NEWS.txt|  14 +
 .../org/apache/cassandra/config/CFMetaData.java |   8 +-
 .../cassandra/db/SerializationHeader.java   |  25 +
 .../db/rows/AbstractRangeTombstoneMarker.java   |   4 +
 .../apache/cassandra/db/rows/AbstractRow.java   |  12 +-
 .../apache/cassandra/db/rows/Unfiltered.java|   1 +
 .../io/sstable/format/SSTableReader.java|   8 +
 .../io/sstable/format/big/BigTableReader.java   |  13 +-
 .../io/sstable/format/big/BigTableScanner.java  |   5 +
 .../apache/cassandra/tools/JsonTransformer.java | 501 +++
 .../apache/cassandra/tools/SSTableExport.java   | 242 +
 .../io/sstable/SSTableScannerTest.java  |   8 +-
 tools/bin/sstabledump   |  52 ++
 tools/bin/sstabledump.bat   |  48 ++
 15 files changed, 933 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/71b1c4a6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 4549ded..aefc02e 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.4
+ * Add sstabledump tool (CASSANDRA-7464)
  * Introduce backpressure for hints (CASSANDRA-10972)
  * Fix ClusteringPrefix not being able to read tombstone range boundaries 
(CASSANDRA-11158)
  * Prevent logging in sandboxed state (CASSANDRA-11033)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/71b1c4a6/NEWS.txt
--
diff --git a/NEWS.txt b/NEWS.txt
index 89fc4a7..5fca578 100644
--- a/NEWS.txt
+++ b/NEWS.txt
@@ -13,6 +13,20 @@ restore snapshots created with the previous major version 
using the
 'sstableloader' tool. You can upgrade the file format of your snapshots
 using the provided 'sstableupgrade' tool.
 
+3.0.4
+=
+
+New features
+
+   - sstabledump tool is added to be 3.0 version of former sstable2json. The 
tool only
+ supports v3.0+ SSTables. See tool's help for more detail.
+
+Upgrading
+-
+   - Nothing specific to this release, but please see previous versions 
upgrading section,
+ especially if you are upgrading from 2.2.
+
+
 3.0.3
 =
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/71b1c4a6/src/java/org/apache/cassandra/config/CFMetaData.java
--
diff --git a/src/java/org/apache/cassandra/config/CFMetaData.java 
b/src/java/org/apache/cassandra/config/CFMetaData.java
index cb6d3b8..79cd779 100644
--- a/src/java/org/apache/cassandra/config/CFMetaData.java
+++ b/src/java/org/apache/cassandra/config/CFMetaData.java
@@ -1132,7 +1132,7 @@ public final class CFMetaData
 private final boolean isSuper;
 private final boolean isCounter;
 private final boolean isView;
-private IPartitioner partitioner;
+private Optional partitioner;
 
 private UUID tableId;
 
@@ -1150,7 +1150,7 @@ public final class CFMetaData
 this.isSuper = isSuper;
 this.isCounter = isCounter;
 this.isView = isView;
-this.partitioner = DatabaseDescriptor.getPartitioner();
+this.partitioner = Optional.empty();
 }
 
 public static Builder create(String keyspace, String table)
@@ -1185,7 +1185,7 @@ public final class CFMetaData
 
 public Builder withPartitioner(IPartitioner partitioner)
 {
-this.partitioner = partitioner;
+this.partitioner = Optional.ofNullable(partitioner);
 return this;
 }
 
@@ -1296,7 +1296,7 @@ public final class CFMetaData
   partitions,
   clusterings,
   builder.build(),
-  partitioner);
+  
partitioner.orElseGet(DatabaseDescriptor::getPartitioner));
 }
 }
 

[3/3] cassandra git commit: Merge branch 'cassandra-3.0' into trunk

[yukim]

Merge branch 'cassandra-3.0' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c94a9f23
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c94a9f23
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c94a9f23

Branch: refs/heads/trunk
Commit: c94a9f2362a8f87e9b1ec048a54ec570c61b29fe
Parents: 6d15a6d 71b1c4a
Author: Yuki Morishita 
Authored: Wed Feb 24 11:55:18 2016 -0600
Committer: Yuki Morishita 
Committed: Wed Feb 24 11:55:18 2016 -0600

--
 CHANGES.txt |   1 +
 NEWS.txt|   2 +
 .../org/apache/cassandra/config/CFMetaData.java |   8 +-
 .../db/rows/AbstractRangeTombstoneMarker.java   |   4 +
 .../apache/cassandra/db/rows/AbstractRow.java   |  12 +-
 .../apache/cassandra/db/rows/Unfiltered.java|   1 +
 .../io/sstable/format/SSTableReader.java|   8 +
 .../io/sstable/format/big/BigTableReader.java   |  13 +-
 .../io/sstable/format/big/BigTableScanner.java  |   5 +
 .../apache/cassandra/tools/JsonTransformer.java | 501 +++
 .../apache/cassandra/tools/SSTableExport.java   | 242 +
 .../io/sstable/SSTableScannerTest.java  |   8 +-
 tools/bin/sstabledump   |  52 ++
 tools/bin/sstabledump.bat   |  48 ++
 14 files changed, 896 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/c94a9f23/CHANGES.txt
--
diff --cc CHANGES.txt
index 3aa62ae,aefc02e..50a298e
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,33 -1,5 +1,34 @@@
 -3.0.4
 +3.4
 + * Extract LoaderOptions to be able to be used from outside (CASSANDRA-10637)
 + * fix OnDiskIndexTest to properly treat empty ranges (CASSANDRA-11205)
 + * fix TrackerTest to handle new notifications (CASSANDRA-11178)
 + * add SASI validation for partitioner and complex columns (CASSANDRA-11169)
 + * Add caching of encrypted credentials in PasswordAuthenticator 
(CASSANDRA-7715)
 + * fix SASI memtable switching on flush (CASSANDRA-11159)
 + * Remove duplicate offline compaction tracking (CASSANDRA-11148)
 + * fix EQ semantics of analyzed SASI indexes (CASSANDRA-11130)
 + * Support long name output for nodetool commands (CASSANDRA-7950)
 + * Encrypted hints (CASSANDRA-11040)
 + * SASI index options validation (CASSANDRA-11136)
 + * Optimize disk seek using min/max column name meta data when the LIMIT 
clause is used
 +   (CASSANDRA-8180)
 + * Add LIKE support to CQL3 (CASSANDRA-11067)
 + * Generic Java UDF types (CASSANDRA-10819)
 + * cqlsh: Include sub-second precision in timestamps by default 
(CASSANDRA-10428)
 + * Set javac encoding to utf-8 (CASSANDRA-11077)
 + * Integrate SASI index into Cassandra (CASSANDRA-10661)
 + * Add --skip-flush option to nodetool snapshot
 + * Skip values for non-queried columns (CASSANDRA-10657)
 + * Add support for secondary indexes on static columns (CASSANDRA-8103)
 + * CommitLogUpgradeTestMaker creates broken commit logs (CASSANDRA-11051)
 + * Add metric for number of dropped mutations (CASSANDRA-10866)
 + * Simplify row cache invalidation code (CASSANDRA-10396)
 + * Support user-defined compaction through nodetool (CASSANDRA-10660)
 + * Stripe view locks by key and table ID to reduce contention 
(CASSANDRA-10981)
 + * Add nodetool gettimeout and settimeout commands (CASSANDRA-10953)
 + * Add 3.0 metadata to sstablemetadata output (CASSANDRA-10838)
 +Merged from 3.0:
+  * Add sstabledump tool (CASSANDRA-7464)
   * Introduce backpressure for hints (CASSANDRA-10972)
   * Fix ClusteringPrefix not being able to read tombstone range boundaries 
(CASSANDRA-11158)
   * Prevent logging in sandboxed state (CASSANDRA-11033)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/c94a9f23/NEWS.txt
--
diff --cc NEWS.txt
index 68ccefb,5fca578..d554ac2
--- a/NEWS.txt
+++ b/NEWS.txt
@@@ -18,10 -18,8 +18,12 @@@ using the provided 'sstableupgrade' too
  
  New features
  
 -   - sstabledump tool is added to be 3.0 version of former sstable2json. The 
tool only
 - supports v3.0+ SSTables. See tool's help for more detail.
 +- Internal authentication now supports caching of encrypted credentials.
 +  Reference cassandra.yaml:credentials_validity_in_ms
 +- Remote configuration of auth caches via JMX can be disabled using the
 +  the system property cassandra.disable_auth_caches_remote_configuration
++- sstabledump tool is added to be 3.0 version of former sstable2json. The 
tool only
++  supports v3.0+ SSTables. See tool's help for more detail.
  
  Upgrading
  -

[jira] [Commented] (CASSANDRA-8890) Enhance cassandra-env.sh to handle Java version output in case of OpenJDK icedtea"

[Michael Shuler (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163412#comment-15163412
 ] 

Michael Shuler commented on CASSANDRA-8890:
---

8890-v2.txt patch works well. +1
{noformat}
mshuler@hana:~$ java -version   

 
openjdk version "1.8.0_72-internal" 

 
OpenJDK Runtime Environment (build 1.8.0_72-internal-b15)   

 
OpenJDK 64-Bit Server VM (build 25.72-b15, mixed mode)  

 

mshuler@hana:~$ java_ver_output=`"${JAVA:-java}" -version 2>&1` 

 
mshuler@hana:~$ jvmver=`echo "$java_ver_output" | grep '[openjdk|java] version' 
| awk -F'"' 'NR==1 {print $2}'`
mshuler@hana:~$ echo $jvmver
1.8.0_72-internal
mshuler@hana:~$ jvmver=`echo "$java_ver_output" | grep '[openjdk|java] version' 
| awk -F'"' 'NR==1 {print $2}' | cut -d\- -f1`
mshuler@hana:~$ echo $jvmver
1.8.0_72
{noformat}

> Enhance cassandra-env.sh to handle Java version output in case of OpenJDK 
> icedtea"
> --
>
> Key: CASSANDRA-8890
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8890
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
> Environment: Red Hat Enterprise Linux Server release 6.4 (Santiago)
>Reporter: Sumod Pawgi
>Assignee: Brandon Williams
>Priority: Minor
>  Labels: conf, icedtea
> Fix For: 3.x
>
> Attachments: 8890-v2.txt, trunk-8890.patch, trunk-8890.txt
>
>
> Where observed - 
> Cassandra node has OpenJDK - 
> java version "1.7.0_09-icedtea"
> In some situations, external agents trying to monitor a C* cluster would need 
> to run cassandra -v command to determine the Cassandra version and would 
> expect a numerical output e.g. java version "1.7.0_75" as in case of Oracle 
> JDK. But if the cluster has OpenJDK IcedTea installed, then this condition is 
> not satisfied and the agents will not work correctly as the output from 
> "cassandra -v" is 
> /opt/apache/cassandra/bin/../conf/cassandra-env.sh: line 102: [: 09-icedtea: 
> integer expression expected
> Cause - 
> The line which is causing this behavior is -
> jvmver=`echo "$java_ver_output" | grep '[openjdk|java] version' | awk -F'"' 
> 'NR==1 {print $2}'`
> Suggested enhancement -
> If we change the line to -
>  jvmver=`echo "$java_ver_output" | grep '[openjdk|java] version' | awk -F'"' 
> 'NR==1 {print $2}' | awk 'BEGIN {FS="-"};{print $1}'`,
> it will give $jvmver as - 1.7.0_09 for the above case. 
> Can we add this enhancement in the cassandra-env.sh? I would like to add it 
> myself and submit for review, but I am not familiar with C* check in process. 
> There might be better ways to do this, but I thought of this to be simplest 
> and as the edition is at the end of the line, it will be easy to reverse if 
> needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10637) Extract LoaderOptions and refactor BulkLoader to be able to be used from within existing Java code instead of just through main()

[Tyler Hobbs (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tyler Hobbs updated CASSANDRA-10637:

Assignee: Eric Fenderbosch

> Extract LoaderOptions and refactor BulkLoader to be able to be used from 
> within existing Java code instead of just through main()
> -
>
> Key: CASSANDRA-10637
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10637
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Eric Fenderbosch
>Assignee: Eric Fenderbosch
>Priority: Minor
> Fix For: 3.4
>
>
> We are writing a service to migrate data from various RDMBS tables in to 
> Cassandra. We write out a CSV from the source system, use CQLSSTableWriter to 
> write sstables to disk, then call sstableloader to stream to the Cassandra 
> cluster.
> Right now, we either have to:
> * return a CSV location from one Java process to a wrapper script which then 
> kicks off sstableloader
> * or call sstableloader via Runtime.getRuntime().exec
> * or call BulkLoader.main from within our Java code, using a custom 
> SecurityManager to trap the System.exit calls
> * or subclass BulkLoader putting the subclass in the 
> org.apache.cassandra.tools package in order to access the package scoped 
> inner classes
> None of these solutions are ideal. Ideally, we should be able to use the 
> functionality of BulkLoader.main directly. I've extracted LoaderOptions to a 
> top level class that uses the builder pattern so that it can be used as part 
> of a Java migration service directly.
> Creating the builder can now be performed with a fluent builder interface:
> LoaderOptions options = LoaderOptions.builder(). //
> connectionsPerHost(2). //
> directory(directory). //
> hosts(hosts). //
> build();
> Or used to parse command line arguments:
> LoaderOptions options = LoaderOptions.builder().parseArgs(args).build();
> A new load method takes a LoaderOptions parameter and throws 
> BulkLoadException instead of System.exit(1).
> Fork on github can be found here:
> https://github.com/efenderbosch/cassandra



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

cassandra git commit: Extract LoaderOptions to be able to be used from outside

[yukim]

Repository: cassandra
Updated Branches:
  refs/heads/trunk e856d8f81 -> 6d15a6da7


Extract LoaderOptions to be able to be used from outside

patch by Eric Fenderbosch; reviewed by yukim for CASSANDRA-10637


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6d15a6da
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6d15a6da
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6d15a6da

Branch: refs/heads/trunk
Commit: 6d15a6da7da8fc41f97d4346d65f6bf9d0e1637f
Parents: e856d8f
Author: Eric Fenderbosch 
Authored: Mon Nov 2 14:50:35 2015 -0500
Committer: Yuki Morishita 
Committed: Wed Feb 24 11:35:39 2016 -0600

--
 CHANGES.txt |   1 +
 .../cassandra/tools/BulkLoadException.java  |  13 +
 .../org/apache/cassandra/tools/BulkLoader.java  | 415 ++
 .../apache/cassandra/tools/LoaderOptions.java   | 537 +++
 4 files changed, 592 insertions(+), 374 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/6d15a6da/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 1d8664c..3aa62ae 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.4
+ * Extract LoaderOptions to be able to be used from outside (CASSANDRA-10637)
  * fix OnDiskIndexTest to properly treat empty ranges (CASSANDRA-11205)
  * fix TrackerTest to handle new notifications (CASSANDRA-11178)
  * add SASI validation for partitioner and complex columns (CASSANDRA-11169)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/6d15a6da/src/java/org/apache/cassandra/tools/BulkLoadException.java
--
diff --git a/src/java/org/apache/cassandra/tools/BulkLoadException.java 
b/src/java/org/apache/cassandra/tools/BulkLoadException.java
new file mode 100644
index 000..fb5d459
--- /dev/null
+++ b/src/java/org/apache/cassandra/tools/BulkLoadException.java
@@ -0,0 +1,13 @@
+package org.apache.cassandra.tools;
+
+public class BulkLoadException extends Exception
+{
+
+private static final long serialVersionUID = 1L;
+
+public BulkLoadException(Throwable cause)
+{
+super(cause);
+}
+
+}

http://git-wip-us.apache.org/repos/asf/cassandra/blob/6d15a6da/src/java/org/apache/cassandra/tools/BulkLoader.java
--
diff --git a/src/java/org/apache/cassandra/tools/BulkLoader.java 
b/src/java/org/apache/cassandra/tools/BulkLoader.java
index 63caae1..f19924e 100644
--- a/src/java/org/apache/cassandra/tools/BulkLoader.java
+++ b/src/java/org/apache/cassandra/tools/BulkLoader.java
@@ -17,26 +17,22 @@
  */
 package org.apache.cassandra.tools;
 
-import java.io.File;
 import java.io.IOException;
-import java.lang.reflect.Constructor;
-import java.lang.reflect.InvocationTargetException;
 import java.net.InetAddress;
-import java.net.MalformedURLException;
-import java.net.UnknownHostException;
-import java.util.*;
-
-import com.google.common.collect.HashMultimap;
-import com.google.common.collect.Multimap;
-import org.apache.commons.cli.*;
+import java.util.Set;
+import javax.net.ssl.SSLContext;
 
 import com.datastax.driver.core.AuthProvider;
 import com.datastax.driver.core.JdkSSLOptions;
-import com.datastax.driver.core.PlainTextAuthProvider;
 import com.datastax.driver.core.SSLOptions;
-import javax.net.ssl.SSLContext;
-import org.apache.cassandra.config.*;
-import org.apache.cassandra.exceptions.ConfigurationException;
+import com.google.common.collect.HashMultimap;
+import com.google.common.collect.Multimap;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+
+import org.apache.cassandra.config.Config;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.config.EncryptionOptions;
 import org.apache.cassandra.io.sstable.SSTableLoader;
 import org.apache.cassandra.security.SSLFactory;
 import org.apache.cassandra.streaming.*;
@@ -46,35 +42,15 @@ import org.apache.cassandra.utils.OutputHandler;
 
 public class BulkLoader
 {
-private static final String TOOL_NAME = "sstableloader";
-private static final String VERBOSE_OPTION  = "verbose";
-private static final String HELP_OPTION  = "help";
-private static final String NOPROGRESS_OPTION  = "no-progress";
-private static final String IGNORE_NODES_OPTION  = "ignore";
-private static final String INITIAL_HOST_ADDRESS_OPTION = "nodes";
-private static final String NATIVE_PORT_OPTION = "port";
-private static final String USER_OPTION = "username";
-private static final String PASSWD_OPTION = "password";
-private static final String AUTH_PROVIDER_OPTION 

[jira] [Commented] (CASSANDRA-10392) Allow Cassandra to trace to custom tracing implementations

[T Jake Luciani (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163309#comment-15163309
 ] 

T Jake Luciani commented on CASSANDRA-10392:


I pushed a fix ( ExpiredTraceState was passing in null)

Tests run clean now. will commit.

[trunk 
unittests|http://cassci.datastax.com/view/Dev/view/tjake/job/tjake-10392-tracing-testall/lastCompletedBuild/testReport/]
[trunk 
dtests|http://cassci.datastax.com/view/Dev/view/tjake/job/tjake-10392-tracing-dtest/lastCompletedBuild/testReport/]

> Allow Cassandra to trace to custom tracing implementations 
> ---
>
> Key: CASSANDRA-10392
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10392
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: mck
>Assignee: mck
> Fix For: 3.x
>
> Attachments: 10392-trunk.txt, cassandra-dtest_master-10392.txt
>
>
> It can be possible to use an external tracing solution in Cassandra by 
> abstracting out the writing of tracing to system_traces tables in the tracing 
> package to separate implementation classes and leaving abstract classes in 
> place that define the interface and behaviour otherwise of C* tracing.
> Then via a system property "cassandra.custom_tracing_class" the Tracing class 
> implementation could be swapped out with something third party.
> An example of this is adding Zipkin tracing into Cassandra in the Summit 
> [presentation|http://thelastpickle.com/files/2015-09-24-using-zipkin-for-full-stack-tracing-including-cassandra/presentation/tlp-reveal.js/tlp-cassandra-zipkin.html].
>  Code for the implemented Zipkin plugin can be found at 
> https://github.com/thelastpickle/cassandra-zipkin-tracing/
> In addition this patch passes the custom payload through into the tracing 
> session allowing a third party tracing solution like Zipkin to do full-stack 
> tracing from clients through and into Cassandra.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10956) Enable authentication of native protocol users via client certificates

[Stefan Podkowinski (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163275#comment-15163275
 ] 

Stefan Podkowinski commented on CASSANDRA-10956:


bq. Regarding anonymous authentication: would it be reasonable to make this 
behavior configurable? The intent is to enable operators to provide some level 
of access (perhaps read-only) to users who are not capable of authenticating.

I have a similar use case where we like to use different authenticators for 
application logins and employees. The idea would be to use 
{{PasswordAuthenticator}} for applications and LDAP for human users. One way to 
implement this would be to create a chain of individual authenticators that are 
to be checked by order, maybe with a default action if all of them fail 
(deny/anonymous). However, as you probably also want to use a different role 
manager depending on used authenticator, you'd need some kind of mapping for 
that as well and I don't like to end up creating iptables for the Cassandra 
login process.

Maybe the better option would be to allow configuring multiple native transport 
ports, each of them using a single authenticator and role manager. This should 
be relatively easy to implement and would also allow to restrict access on 
network level, e.g. setting up different firewall rules for the "applications" 
or "developer" ports. 


> Enable authentication of native protocol users via client certificates
> --
>
> Key: CASSANDRA-10956
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10956
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Samuel Klock
>Assignee: Samuel Klock
> Attachments: 10956.patch
>
>
> Currently, the native protocol only supports user authentication via SASL.  
> While this is adequate for many use cases, it may be superfluous in scenarios 
> where clients are required to present an SSL certificate to connect to the 
> server.  If the certificate presented by a client is sufficient by itself to 
> specify a user, then an additional (series of) authentication step(s) via 
> SASL merely add overhead.  Worse, for uses wherein it's desirable to obtain 
> the identity from the client's certificate, it's necessary to implement a 
> custom SASL mechanism to do so, which increases the effort required to 
> maintain both client and server and which also duplicates functionality 
> already provided via SSL/TLS.
> Cassandra should provide a means of using certificates for user 
> authentication in the native protocol without any effort above configuring 
> SSL on the client and server.  Here's a possible strategy:
> * Add a new authenticator interface that returns {{AuthenticatedUser}} 
> objects based on the certificate chain presented by the client.
> * If this interface is in use, the user is authenticated immediately after 
> the server receives the {{STARTUP}} message.  It then responds with a 
> {{READY}} message.
> * Otherwise, the existing flow of control is used (i.e., if the authenticator 
> requires authentication, then an {{AUTHENTICATE}} message is sent to the 
> client).
> One advantage of this strategy is that it is backwards-compatible with 
> existing schemes; current users of SASL/{{IAuthenticator}} are not impacted.  
> Moreover, it can function as a drop-in replacement for SASL schemes without 
> requiring code changes (or even config changes) on the client side.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-8713) Print out all current values of available configuration parameters at startup

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-8713:

Labels: lhf  (was: )

> Print out all current values of available configuration parameters at startup
> -
>
> Key: CASSANDRA-8713
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8713
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Wei Deng
>Priority: Minor
>  Labels: lhf
>
> I know we have https://issues.apache.org/jira/browse/CASSANDRA-6456 to print 
> out all parameters configured in cassandra.yaml at startup as in here 
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/config/YamlConfigurationLoader.java#L124-136,
>  however, how about those parameters that we don't configure in 
> cassandra.yaml? It will be helpful to troubleshooting if we can add a similar 
> logConfig() method for all parameters in 
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/config/Config.java.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-8727) nodetool command to get the status of things that can be enable/disable'd

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-8727:

Labels: lhf  (was: )

> nodetool command to get the status of things that can be enable/disable'd
> -
>
> Key: CASSANDRA-8727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8727
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter:  Brian Hess
>  Labels: lhf
>
> It is possible to say enableautocompaction or disableautocompaction via 
> nodetool.  But it doesn't appear that there is a nodetool command to display 
> the current state of autocompaction - enabled or disabled.  
> It would be good to be able to get the status of these binary commands so 
> that you can determine the status, possibly make a change to 
> troubleshoot/experiment/etc, and return the status to the way it was when you 
> arrived.
> This applies for autocompaction, backup, binary, gossip, handoff, and thrift.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11209) SSTable ancestor leaked reference

[Jose Fernandez (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163220#comment-15163220
 ] 

Jose Fernandez commented on CASSANDRA-11209:


Another thought: during repairs or anticompaction, are sstables "registered" 
right away as they get created or only at the end of the process? If its only 
at the end, then I'm wondering what happens if there is long running repair 
going on for days.

> SSTable ancestor leaked reference
> -
>
> Key: CASSANDRA-11209
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11209
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Jose Fernandez
>Assignee: Marcus Eriksson
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> We're running a fork of 2.1.13 that adds the TimeWindowCompactionStrategy 
> from [~jjirsa]. We've been running 4 clusters without any issues for many 
> months until a few weeks ago we started scheduling incremental repairs every 
> 24 hours (previously we didn't run any repairs at all).
> Since then we started noticing big discrepancies in the LiveDiskSpaceUsed, 
> TotalDiskSpaceUsed, and actual size of files on disk. The numbers are brought 
> back in sync by restarting the node. We also noticed that when this bug 
> happens there are several ancestors that don't get cleaned up. A restart will 
> queue up a lot of compactions that slowly eat away the ancestors.
> I looked at the code and noticed that we only decrease the LiveTotalDiskUsed 
> metric in the SSTableDeletingTask. Since we have no errors being logged, I'm 
> assuming that for some reason this task is not getting queued up. If I 
> understand correctly this only happens when the reference count for the 
> SStable reaches 0. So this is leading us to believe that something during 
> repairs and/or compactions is causing a reference leak to the ancestor table.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11209) SSTable ancestor leaked reference

[Jose Fernandez (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163209#comment-15163209
 ] 

Jose Fernandez commented on CASSANDRA-11209:


Yesterday we did the following on the misbehaving node:
- Turned off scheduled repairs
- Marked all tables as repaired, kicked off another incremental and watch it 
complete quickly.

This morning the node is not showing any of the issues posted above. The Live 
and Total metrics == the size on disk. So we're pretty confident now that the 
bug is somehow related to overlapping incremental repairs. We had them running 
every 24 hours, but noticed that on this node this was each one was taking 
longer than 24h, so they started overlapping.

Also, when the node was "fixed" by restarting it, we saw a big spike in 
SSTables marked as repaired... maybe overlapping anticompaction could cause 
this?

> SSTable ancestor leaked reference
> -
>
> Key: CASSANDRA-11209
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11209
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Jose Fernandez
>Assignee: Marcus Eriksson
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> We're running a fork of 2.1.13 that adds the TimeWindowCompactionStrategy 
> from [~jjirsa]. We've been running 4 clusters without any issues for many 
> months until a few weeks ago we started scheduling incremental repairs every 
> 24 hours (previously we didn't run any repairs at all).
> Since then we started noticing big discrepancies in the LiveDiskSpaceUsed, 
> TotalDiskSpaceUsed, and actual size of files on disk. The numbers are brought 
> back in sync by restarting the node. We also noticed that when this bug 
> happens there are several ancestors that don't get cleaned up. A restart will 
> queue up a lot of compactions that slowly eat away the ancestors.
> I looked at the code and noticed that we only decrease the LiveTotalDiskUsed 
> metric in the SSTableDeletingTask. Since we have no errors being logged, I'm 
> assuming that for some reason this task is not getting queued up. If I 
> understand correctly this only happens when the reference count for the 
> SStable reaches 0. So this is leading us to believe that something during 
> repairs and/or compactions is causing a reference leak to the ancestor table.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11209) SSTable ancestor leaked reference

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163200#comment-15163200
 ] 

Marcus Eriksson commented on CASSANDRA-11209:
-

I've been trying to reproduce (it is not related to CASSANDRA-11215 btw) with 
no luck

It looks a bit like CASSANDRA-9577 where individual nodes/machines go bad and 
show this behavior.

> SSTable ancestor leaked reference
> -
>
> Key: CASSANDRA-11209
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11209
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Jose Fernandez
>Assignee: Marcus Eriksson
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> We're running a fork of 2.1.13 that adds the TimeWindowCompactionStrategy 
> from [~jjirsa]. We've been running 4 clusters without any issues for many 
> months until a few weeks ago we started scheduling incremental repairs every 
> 24 hours (previously we didn't run any repairs at all).
> Since then we started noticing big discrepancies in the LiveDiskSpaceUsed, 
> TotalDiskSpaceUsed, and actual size of files on disk. The numbers are brought 
> back in sync by restarting the node. We also noticed that when this bug 
> happens there are several ancestors that don't get cleaned up. A restart will 
> queue up a lot of compactions that slowly eat away the ancestors.
> I looked at the code and noticed that we only decrease the LiveTotalDiskUsed 
> metric in the SSTableDeletingTask. Since we have no errors being logged, I'm 
> assuming that for some reason this task is not getting queued up. If I 
> understand correctly this only happens when the reference count for the 
> SStable reaches 0. So this is leading us to believe that something during 
> repairs and/or compactions is causing a reference leak to the ancestor table.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-8840) Classify our Assertions (like com.google.base.Preconditions)

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne resolved CASSANDRA-8840.
-
Resolution: Later

There doesn't seem to have too much interest in going through that so far so 
lets close for now.

> Classify our Assertions (like com.google.base.Preconditions)
> 
>
> Key: CASSANDRA-8840
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8840
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Benedict
>Priority: Minor
>
> I raised this on IRC, then dropped it due to opposition, but it's possible 
> the opposition was due to my conflation of the act of classification with the 
> disabling of some of the assertions. The two aren't wed, and I think it 
> _would_ improve readability significantly by itself, as Ariel reminded me 
> with his use of google's Preconditions class in CASSANDRA-8692.
> I would prefer to use our own version of this class, that we can force the 
> \@Inline compiler hint onto, so that we have no negative performance 
> implications. Also, we can then introduce a class of data corruption checks, 
> etc. I think this would aid readability, and also permit easier analysis of 
> the codebase via IDE (right now it's very hard to say what data corruption 
> checks we actually perform, for instance).
> Thoughts?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-8880) Add metrics to monitor the amount of tombstones created

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-8880:

Labels: lhf metrics  (was: metrics)

> Add metrics to monitor the amount of tombstones created
> ---
>
> Key: CASSANDRA-8880
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8880
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Michaël Figuière
>Priority: Minor
>  Labels: lhf, metrics
> Fix For: 2.1.x
>
> Attachments: cassandra-2.1-8880.patch
>
>
> AFAIK there's currently no way to monitor the amount of tombstones created on 
> a Cassandra node. CASSANDRA-6057 has made it possible for users to figure out 
> how many tombstones are scanned at read time, but in write mostly workloads, 
> it may not be possible to realize if some inappropriate queries are 
> generating too many tombstones.
> Therefore the following additional metrics should be added:
> * {{writtenCells}}: amount of cells that have been written
> * {{writtenTombstoneCells}}: amount of tombstone cells that have been written
> Alternatively these could be exposed as a single gauge such as 
> {{writtenTombstoneCellsRatio}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-8887) Direct (de)compression of internode communication

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko resolved CASSANDRA-8887.
--
   Resolution: Not A Problem
Fix Version/s: (was: 3.x)

As Benedict says, it should be irrelevant in modern versions of C*, and cannot 
be feasibly backported to older ones. Closing as Not A Problem unless you had 
something different in mind.

> Direct (de)compression of internode communication
> -
>
> Key: CASSANDRA-8887
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8887
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Matt Stump
>Priority: Minor
>
> Internode compression is on by default. Currently we allocate one set of 
> buffers for the raw data, and then compress which results in another set of 
> buffers. This greatly increases the GC load. We can decrease the GC load by 
> doing direct compression/decompression of the communication buffers. This is 
> the same work as done in CASSANDRA-8464 but applied to internode 
> communication.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-8888) Disable internode compression by default

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-:
-
Component/s: Streaming and Messaging

> Disable internode compression by default
> 
>
> Key: CASSANDRA-
> URL: https://issues.apache.org/jira/browse/CASSANDRA-
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Streaming and Messaging
>Reporter: Matt Stump
>  Labels: lhf
> Fix For: 3.x
>
>
> Internode compression increases GC load, and can cause high CPU utilization 
> for high throughput use cases. Very rarely are customers restricted by 
> intra-DC or cross-DC network bandwidth. I'de rather we optimize for the 75% 
> of cases where internode compression isn't needed and then selectively enable 
> it for customers where it would provide a benefit. Currently I'm advising all 
> field consultants disable compression by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-8888) Disable internode compression by default

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-:
-
Labels: lhf  (was: )

> Disable internode compression by default
> 
>
> Key: CASSANDRA-
> URL: https://issues.apache.org/jira/browse/CASSANDRA-
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Streaming and Messaging
>Reporter: Matt Stump
>  Labels: lhf
> Fix For: 3.x
>
>
> Internode compression increases GC load, and can cause high CPU utilization 
> for high throughput use cases. Very rarely are customers restricted by 
> intra-DC or cross-DC network bandwidth. I'de rather we optimize for the 75% 
> of cases where internode compression isn't needed and then selectively enable 
> it for customers where it would provide a benefit. Currently I'm advising all 
> field consultants disable compression by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8890) Enhance cassandra-env.sh to handle Java version output in case of OpenJDK icedtea"

[Joshua McKenzie (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163185#comment-15163185
 ] 

Joshua McKenzie commented on CASSANDRA-8890:


[~mshuler] - are you going to be able to get to this or should we dig up 
another reviewer?

Thanks!

> Enhance cassandra-env.sh to handle Java version output in case of OpenJDK 
> icedtea"
> --
>
> Key: CASSANDRA-8890
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8890
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
> Environment: Red Hat Enterprise Linux Server release 6.4 (Santiago)
>Reporter: Sumod Pawgi
>Assignee: Brandon Williams
>Priority: Minor
>  Labels: conf, icedtea
> Fix For: 3.x
>
> Attachments: 8890-v2.txt, trunk-8890.patch, trunk-8890.txt
>
>
> Where observed - 
> Cassandra node has OpenJDK - 
> java version "1.7.0_09-icedtea"
> In some situations, external agents trying to monitor a C* cluster would need 
> to run cassandra -v command to determine the Cassandra version and would 
> expect a numerical output e.g. java version "1.7.0_75" as in case of Oracle 
> JDK. But if the cluster has OpenJDK IcedTea installed, then this condition is 
> not satisfied and the agents will not work correctly as the output from 
> "cassandra -v" is 
> /opt/apache/cassandra/bin/../conf/cassandra-env.sh: line 102: [: 09-icedtea: 
> integer expression expected
> Cause - 
> The line which is causing this behavior is -
> jvmver=`echo "$java_ver_output" | grep '[openjdk|java] version' | awk -F'"' 
> 'NR==1 {print $2}'`
> Suggested enhancement -
> If we change the line to -
>  jvmver=`echo "$java_ver_output" | grep '[openjdk|java] version' | awk -F'"' 
> 'NR==1 {print $2}' | awk 'BEGIN {FS="-"};{print $1}'`,
> it will give $jvmver as - 1.7.0_09 for the above case. 
> Can we add this enhancement in the cassandra-env.sh? I would like to add it 
> myself and submit for review, but I am not familiar with C* check in process. 
> There might be better ways to do this, but I thought of this to be simplest 
> and as the edition is at the end of the line, it will be easy to reverse if 
> needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-8891) Conditionally compress data between nodes

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko resolved CASSANDRA-8891.
--
   Resolution: Later
Fix Version/s: (was: 3.x)

We aren't sure if this is feasible. If you can write a quick POC for viability, 
please reopen the ticket. Thanks!

> Conditionally compress data between nodes
> -
>
> Key: CASSANDRA-8891
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8891
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Rich Rein
>
> Conditionally compress internode traffic when the outbound connection is 
> congested.
> This would reduce unnecessary CPU usage and improve Cassandra's performance.
> The result should automatically adapts to a long-term and/or short term 
> bottlenecks in either CPU or internode connection. 
> The result should automatically make use of the most luxurious resource. 
> This would apply to inter-node within the data center and between the data 
> centers.
> Cassandra.yaml
> The property, internode_compression, could have a new "auto" value. As in:
> internode_compression: auto
> Optionally, a thresh-hold can be set for the definition of locally available 
> CPU bandwidth and the it compression can depend on locally available CPU 
> bandwidth.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-8898) sstableloader utility should allow loading of data from mounted filesystem

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko resolved CASSANDRA-8898.
--
Resolution: Cannot Reproduce

We've had many changes to that code in the last few releases, and it's likely 
have been resolved since then. If you can reproduce in latest 2.2.x or 3.0.x, 
please let us know, and reopen the JIRA.

> sstableloader utility should allow loading of data from mounted filesystem
> --
>
> Key: CASSANDRA-8898
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8898
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
> Environment: 2.0.12
>Reporter: Kenneth Failbus
>
> When trying to load data from a mounted filesystem onto a new cluster, 
> following exceptions is observed intermittently, and at some point the 
> sstableloader process gets hung without completing the loading process.
> Please note that in my case the scenario was loading the existing sstables 
> from an existing cluster to a brand new cluster.
> Finally, it was found that there were some hard assumptions been made by 
> sstableloader utility w.r.t response from the filesystem, which was not 
> working with mounted filesystem.
> The work-around was to copy each existing nodes sstable data files locally 
> and then point sstableloader to that local filesystem to then load data onto 
> new cluster.
> In case of restoring during disaster recovery from backups the data using 
> sstableloader, this copying to local filesystem of data files and then 
> loading would take a long time.
> It would be a good enhancement of the sstableloader utility to enable use of 
> mounted filesystem as copying data locally and then loading is time consuming.
> Below is the exception seen during the use of mounted filesystem.
> {code}
> java.lang.AssertionError: Reference counter -1 for 
> /opt/tmp/casapp-c1-c00053-g.ch.tvx.comcast.com/MinDataService/System/MinDataService-System-jb-5449-Data.db
>  
> at 
> org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:1146)
>  
> at 
> org.apache.cassandra.streaming.StreamTransferTask.complete(StreamTransferTask.java:74)
>  
> at 
> org.apache.cassandra.streaming.StreamSession.received(StreamSession.java:542) 
> at 
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:424)
>  
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245)
>  
> at java.lang.Thread.run(Thread.java:744) 
> WARN 21:07:16,853 [Stream #3e5a5ba0-bdef-11e4-a975-5777dbff0945] Stream failed
>   at 
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:59)
>  
> at 
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1406)
>  
> at 
> org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:55)
>  
> at 
> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:59)
>  
> at 
> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:42)
>  
> at 
> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45)
>  
> at 
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:339)
>  
> at 
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:311)
>  
> at java.lang.Thread.run(Thread.java:744) 
> Caused by: java.io.FileNotFoundException: 
> /opt/tmp/casapp-c1-c00055-g.ch.tvx.comcast.com/MinDataService/System/MinDataService-System-jb-5997-Data.db
>  (No such file or directory) 
> at java.io.RandomAccessFile.open(Native Method) 
> at java.io.RandomAccessFile.(RandomAccessFile.java:241) 
> at 
> org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58)
>  
> at 
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.(CompressedRandomAccessReader.java:76)
>  
> at 
> org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:55)
>  
> ... 8 more 
> Exception in thread "STREAM-OUT-/96.115.88.196" 
> java.lang.NullPointerException 
> at 
> org.apache.cassandra.streaming.ConnectionHandler$MessageHandler.signalCloseDone(ConnectionHandler.java:205)
>  
> at 
> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:331)
>  
> at java.lang.Thread.run(Thread.java:744) 
> ERROR 20:49:35,646 [Stream #d9fce650-bdf3-11e4-b6c0-252cb9b3e9f3] 

[jira] [Updated] (CASSANDRA-8937) add liveSSTableCount metrics at keyspace level

[Sylvain Lebresne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne updated CASSANDRA-8937:

Labels: lhf  (was: )

> add liveSSTableCount metrics at keyspace level
> --
>
> Key: CASSANDRA-8937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8937
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: yangwei
>Assignee: yangwei
>Priority: Minor
>  Labels: lhf
> Fix For: 2.1.x
>
> Attachments: 
> 0001-add-liveSSTableCount-metrics-at-keyspace-level-2.1.patch, 
> 0001-add-liveSSTableCount-metrics-at-keyspace-level.patch
>
>
> We currently don't have liveSSTableCount metrics aggregated at keyspace 
> level.  If many sstables exists and we can't realize it earlier,  cassandra 
> will oom while doing scanning operations. It would be nice and easy to 
> aggregate it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-8945) Error log is written on client malformed request

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko resolved CASSANDRA-8945.
--
Resolution: Not A Problem

> Error log is written on client malformed request
> 
>
> Key: CASSANDRA-8945
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8945
> Project: Cassandra
>  Issue Type: Improvement
> Environment: 2.1.3
>Reporter: Vovodroid
>Priority: Minor
>
> When client sends  wrong or malformed request error log is printed, though 
> it's not Cassandra bug or problem.
> {code:title=Message.java|borderStyle=solid}
>  @Override
> public boolean apply(Throwable exception)
> ...
> // Anything else is probably a bug in server of client binary protocol 
> handling
> logger.error(message, exception);
> ..
> {code}
> Assume public Cassandra server (for instance accepting connection from widely 
> distributed application, e.g. mobile). Badly written or hostile applications 
> can flood Cassandra log.
> I discovered two possible scenarios so far:
> 1. Connect with telnet to Cassandra and send any string.
> *exception* is _io.netty.handler.codec.DecoderException_
> *cause* is _org.apache.cassandra.transport.ProtocolException_
> *message* is _Invalid or unsupported protocol version_
> 2. Connect to Cassandra with password authentication with client that doesn't 
> recognize that (sending request as reply to AuthResponse)
> *exception* is _java.lang.AssertionError_
> *cause* is _org.apache.cassandra.exceptions.InvalidRequestException_
> *message* is _Protocol error: Unexpected message QUERY, expecting 
> SASL_RESPONSE_
> Thus I suggest that way to distinguish between Cassandra originated exception 
> and bad clients should be found not to print log in latter cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-11086) trunk eclipse-warnings

[Benjamin Lerer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer updated CASSANDRA-11086:
---
Reviewer: Benjamin Lerer

> trunk eclipse-warnings
> --
>
> Key: CASSANDRA-11086
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11086
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Michael Shuler
>Assignee: Jason Brown
>Priority: Minor
> Fix For: 3.x
>
>
> REF = origin/trunk 
> COMMIT = 7230a66318ce8add742959d095900d5870689f0c
> {noformat}
> # 1/27/16 8:50:25 PM UTC
> # Eclipse Compiler for Java(TM) v20150120-1634, 3.10.2, Copyright IBM Corp 
> 2000, 2013. All rights reserved.
> --
> 1. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/metrics/TableMetrics.java
>  (at line 338)
>   return 
> computeCompressionRatio(Iterables.concat(Iterables.transform(Keyspace.all(),
>  ^^^
> The method computeCompressionRatio(Iterable) in the type 
> TableMetrics is not applicable for the arguments (Iterable)
> --
> --
> 2. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/net/OutboundTcpConnectionPool.java
>  (at line 141)
>   return channel.socket();
>   
> Potential resource leak: 'channel' may not be closed at this location
> --
> --
> 3. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskBlock.java
>  (at line 93)
>   return cast(dup);
>   ^
> Potential resource leak: 'dup' may not be closed at this location
> --
> --
> 4. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskIndex.java
>  (at line 142)
>   indexFile = new MappedBuffer(new ChannelProxy(indexPath, 
> backingFile.getChannel()));
>
> ^
> Potential resource leak: '' may not be closed
> --
> 5. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskIndex.java
>  (at line 161)
>   throw new FSReadError(e, index);
>   
> Potential resource leak: 'backingFile' may not be closed at this location
> --
> 6. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskIndex.java
>  (at line 249)
>   RangeIterator range = searchRange(e);
>  ^
> Potential resource leak: 'range' may not be closed
> --
> --
> 7. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskIndexBuilder.java
>  (at line 286)
>   throw new FSWriteError(e, file);
>   
> Potential resource leak: 'out' may not be closed at this location
> --
> --
> 8. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/PerSSTableIndexWriter.java
>  (at line 284)
>   OnDiskIndex last = scheduleSegmentFlush(false).call();
>   
> Potential resource leak: 'last' may not be closed
> --
> 9. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/PerSSTableIndexWriter.java
>  (at line 293)
>   OnDiskIndex part = Futures.getUnchecked(f);
>   
> Potential resource leak: 'part' may not be closed
> --
> --
> 10. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/TermIterator.java
>  (at line 118)
>   RangeIterator keyIterator = index.search(e);
>  ^^^
> Potential resource leak: 'keyIterator' may not be closed
> --
> 11. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/TermIterator.java
>  (at line 148)
>   return ranges == null ? null : new TermIterator(e, ranges, 
> referencedIndexes);
>   
> ^^
> Potential resource leak: 'ranges' may not be closed at this location
> --
> 12. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/TermIterator.java
>  (at line 156)
>   throw ex;
>   ^
> Potential resource leak: 'memtableIterator' may not be closed at this location
> --
> --
> 13. ERROR in 
> 

[jira] [Commented] (CASSANDRA-11109) Cassandra process killed by OS due to out of memory issue

[Tony Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163116#comment-15163116
 ] 

Tony Xu commented on CASSANDRA-11109:
-

Is anyone looking into this?

> Cassandra process killed by OS due to out of memory issue
> -
>
> Key: CASSANDRA-11109
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11109
> Project: Cassandra
>  Issue Type: Bug
> Environment: Operating System: 
> Amazon Linux AMI release 2015.03
> Kernel \r on an \m
> Instance type: m3.2xlarge
> vCPUs: 8
> Memory: 30G
> Commitlog storage: 1 x 80 SSD Storage
> Data disks: EBS io1 300GB with 1000 IOPS
>Reporter: Tony Xu
> Fix For: 2.2.x
>
> Attachments: cassandra-env.txt, cassandra-system-log.txt, 
> cassandra.yaml, system-messages.txt
>
>
> After we upgraded Cassandra from 2.1.12 to 2.2.4 on one of our nodes (A three 
> nodes Cassandra cluster), we've been experiencing an og-going issue with 
> cassandra process running with continuously increasing memory util killed by 
> OOM. 
> {quote}
> Feb  1 23:53:10 kernel: [24135455.025185] [19862]   494 19862 133728623  
> 7379077  1390680 0 java
> Feb  1 23:53:10 kernel: [24135455.029678] Out of memory: Kill process 19862 
> (java) score 973 or sacrifice child
> Feb  1 23:53:10 kernel: [24135455.035434] Killed process 19862 (java) 
> total-vm:534918588kB, anon-rss:29413728kB, file-rss:102940kB
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-11223) Queries with LIMIT filtering on clustering columns can return less row than expected

[Benjamin Lerer (JIRA)]

Benjamin Lerer created CASSANDRA-11223:
--

 Summary: Queries with LIMIT filtering on clustering columns can 
return less row than expected
 Key: CASSANDRA-11223
 URL: https://issues.apache.org/jira/browse/CASSANDRA-11223
 Project: Cassandra
  Issue Type: Bug
  Components: Local Write-Read Paths
Reporter: Benjamin Lerer
Assignee: Benjamin Lerer


A query like {{SELECT * FROM %s WHERE b = 1 LIMIT 2 ALLOW FILTERING}} can 
return less row than expected if the table has some static columns and some of 
the partition have no rows matching b = 1.

The problem can be reproduced with the following unit test:
{code}
public void testFilteringOnClusteringColumnsWithLimitAndStaticColumns() 
throws Throwable
{
createTable("CREATE TABLE %s (a int, b int, s int static, c int, 
primary key (a, b))");

for (int i = 0; i < 3; i++)
{
execute("INSERT INTO %s (a, s) VALUES (?, ?)", i, i);
for (int j = 0; j < 3; j++)
if (!(i == 0 && j == 1))
execute("INSERT INTO %s (a, b, c) VALUES (?, ?, ?)", i, 
j, i + j);
}

assertRows(execute("SELECT * FROM %s"),
   row(1, 0, 1, 1),
   row(1, 1, 1, 2),
   row(1, 2, 1, 3),
   row(0, 0, 0, 0),
   row(0, 2, 0, 2),
   row(2, 0, 2, 2),
   row(2, 1, 2, 3),
   row(2, 2, 2, 4));

assertRows(execute("SELECT * FROM %s WHERE b = 1 ALLOW FILTERING"),
   row(1, 1, 1, 2),
   row(2, 1, 2, 3));

assertRows(execute("SELECT * FROM %s WHERE b = 1 LIMIT 2 ALLOW 
FILTERING"),
   row(1, 1, 1, 2),
   row(2, 1, 2, 3)); // < FAIL It returns only one row 
because the static row of partition 0 is counted and filtered out in SELECT 
statement
}
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (CASSANDRA-11086) trunk eclipse-warnings

[Jason Brown (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Brown reassigned CASSANDRA-11086:
---

Assignee: Jason Brown  (was: Benjamin Lerer)

> trunk eclipse-warnings
> --
>
> Key: CASSANDRA-11086
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11086
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Michael Shuler
>Assignee: Jason Brown
>Priority: Minor
> Fix For: 3.x
>
>
> REF = origin/trunk 
> COMMIT = 7230a66318ce8add742959d095900d5870689f0c
> {noformat}
> # 1/27/16 8:50:25 PM UTC
> # Eclipse Compiler for Java(TM) v20150120-1634, 3.10.2, Copyright IBM Corp 
> 2000, 2013. All rights reserved.
> --
> 1. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/metrics/TableMetrics.java
>  (at line 338)
>   return 
> computeCompressionRatio(Iterables.concat(Iterables.transform(Keyspace.all(),
>  ^^^
> The method computeCompressionRatio(Iterable) in the type 
> TableMetrics is not applicable for the arguments (Iterable)
> --
> --
> 2. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/net/OutboundTcpConnectionPool.java
>  (at line 141)
>   return channel.socket();
>   
> Potential resource leak: 'channel' may not be closed at this location
> --
> --
> 3. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskBlock.java
>  (at line 93)
>   return cast(dup);
>   ^
> Potential resource leak: 'dup' may not be closed at this location
> --
> --
> 4. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskIndex.java
>  (at line 142)
>   indexFile = new MappedBuffer(new ChannelProxy(indexPath, 
> backingFile.getChannel()));
>
> ^
> Potential resource leak: '' may not be closed
> --
> 5. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskIndex.java
>  (at line 161)
>   throw new FSReadError(e, index);
>   
> Potential resource leak: 'backingFile' may not be closed at this location
> --
> 6. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskIndex.java
>  (at line 249)
>   RangeIterator range = searchRange(e);
>  ^
> Potential resource leak: 'range' may not be closed
> --
> --
> 7. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskIndexBuilder.java
>  (at line 286)
>   throw new FSWriteError(e, file);
>   
> Potential resource leak: 'out' may not be closed at this location
> --
> --
> 8. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/PerSSTableIndexWriter.java
>  (at line 284)
>   OnDiskIndex last = scheduleSegmentFlush(false).call();
>   
> Potential resource leak: 'last' may not be closed
> --
> 9. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/PerSSTableIndexWriter.java
>  (at line 293)
>   OnDiskIndex part = Futures.getUnchecked(f);
>   
> Potential resource leak: 'part' may not be closed
> --
> --
> 10. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/TermIterator.java
>  (at line 118)
>   RangeIterator keyIterator = index.search(e);
>  ^^^
> Potential resource leak: 'keyIterator' may not be closed
> --
> 11. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/TermIterator.java
>  (at line 148)
>   return ranges == null ? null : new TermIterator(e, ranges, 
> referencedIndexes);
>   
> ^^
> Potential resource leak: 'ranges' may not be closed at this location
> --
> 12. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/TermIterator.java
>  (at line 156)
>   throw ex;
>   ^
> Potential resource leak: 'memtableIterator' may not be closed at this location
> --
> 

[jira] [Comment Edited] (CASSANDRA-11086) trunk eclipse-warnings

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163090#comment-15163090
 ] 

Jason Brown edited comment on CASSANDRA-11086 at 2/24/16 2:36 PM:
--

On the whole, the CommitLog and HInts related warning were fine, but I added a 
few {{SupressWarning}} annotations to where they are appropriate (and correct), 
as well as add try-with-resources blocks in two places.

||trunk||
|[branch|https://github.com/apache/cassandra/compare/trunk...jasobrown:11086_trunk]|
|[testall|http://cassci.datastax.com/view/Dev/view/jasobrown/job/jasobrown-11086_trunk-dtest/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/jasobrown/job/jasobrown-11086_trunk-testall/]|

Waiting for tests to finish execution...


was (Author: jasobrown):
Added a few {{SupressWarning}} annotations to where they are appropriate (and 
correct), as well as add try-with-resources blocks in two places.

||trunk||
|[branch|https://github.com/apache/cassandra/compare/trunk...jasobrown:11086_trunk]|
|[testall|http://cassci.datastax.com/view/Dev/view/jasobrown/job/jasobrown-11086_trunk-dtest/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/jasobrown/job/jasobrown-11086_trunk-testall/]|

Waiting for tests to finish execution...

> trunk eclipse-warnings
> --
>
> Key: CASSANDRA-11086
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11086
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Michael Shuler
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.x
>
>
> REF = origin/trunk 
> COMMIT = 7230a66318ce8add742959d095900d5870689f0c
> {noformat}
> # 1/27/16 8:50:25 PM UTC
> # Eclipse Compiler for Java(TM) v20150120-1634, 3.10.2, Copyright IBM Corp 
> 2000, 2013. All rights reserved.
> --
> 1. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/metrics/TableMetrics.java
>  (at line 338)
>   return 
> computeCompressionRatio(Iterables.concat(Iterables.transform(Keyspace.all(),
>  ^^^
> The method computeCompressionRatio(Iterable) in the type 
> TableMetrics is not applicable for the arguments (Iterable)
> --
> --
> 2. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/net/OutboundTcpConnectionPool.java
>  (at line 141)
>   return channel.socket();
>   
> Potential resource leak: 'channel' may not be closed at this location
> --
> --
> 3. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskBlock.java
>  (at line 93)
>   return cast(dup);
>   ^
> Potential resource leak: 'dup' may not be closed at this location
> --
> --
> 4. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskIndex.java
>  (at line 142)
>   indexFile = new MappedBuffer(new ChannelProxy(indexPath, 
> backingFile.getChannel()));
>
> ^
> Potential resource leak: '' may not be closed
> --
> 5. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskIndex.java
>  (at line 161)
>   throw new FSReadError(e, index);
>   
> Potential resource leak: 'backingFile' may not be closed at this location
> --
> 6. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskIndex.java
>  (at line 249)
>   RangeIterator range = searchRange(e);
>  ^
> Potential resource leak: 'range' may not be closed
> --
> --
> 7. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskIndexBuilder.java
>  (at line 286)
>   throw new FSWriteError(e, file);
>   
> Potential resource leak: 'out' may not be closed at this location
> --
> --
> 8. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/PerSSTableIndexWriter.java
>  (at line 284)
>   OnDiskIndex last = scheduleSegmentFlush(false).call();
>   
> Potential resource leak: 'last' may not be closed
> --
> 9. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/PerSSTableIndexWriter.java
>  (at line 293)
>   OnDiskIndex part = Futures.getUnchecked(f);
>   
> Potential resource leak: 

[jira] [Comment Edited] (CASSANDRA-11086) trunk eclipse-warnings

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163090#comment-15163090
 ] 

Jason Brown edited comment on CASSANDRA-11086 at 2/24/16 2:37 PM:
--

On the whole, the CommitLog and HInts related warning were not a problem, but I 
added a few {{SupressWarning}} annotations to where they are appropriate (and 
correct), as well as add try-with-resources blocks in two places.

||trunk||
|[branch|https://github.com/apache/cassandra/compare/trunk...jasobrown:11086_trunk]|
|[testall|http://cassci.datastax.com/view/Dev/view/jasobrown/job/jasobrown-11086_trunk-dtest/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/jasobrown/job/jasobrown-11086_trunk-testall/]|

Waiting for tests to finish execution...


was (Author: jasobrown):
On the whole, the CommitLog and HInts related warning were fine, but I added a 
few {{SupressWarning}} annotations to where they are appropriate (and correct), 
as well as add try-with-resources blocks in two places.

||trunk||
|[branch|https://github.com/apache/cassandra/compare/trunk...jasobrown:11086_trunk]|
|[testall|http://cassci.datastax.com/view/Dev/view/jasobrown/job/jasobrown-11086_trunk-dtest/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/jasobrown/job/jasobrown-11086_trunk-testall/]|

Waiting for tests to finish execution...

> trunk eclipse-warnings
> --
>
> Key: CASSANDRA-11086
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11086
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Michael Shuler
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.x
>
>
> REF = origin/trunk 
> COMMIT = 7230a66318ce8add742959d095900d5870689f0c
> {noformat}
> # 1/27/16 8:50:25 PM UTC
> # Eclipse Compiler for Java(TM) v20150120-1634, 3.10.2, Copyright IBM Corp 
> 2000, 2013. All rights reserved.
> --
> 1. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/metrics/TableMetrics.java
>  (at line 338)
>   return 
> computeCompressionRatio(Iterables.concat(Iterables.transform(Keyspace.all(),
>  ^^^
> The method computeCompressionRatio(Iterable) in the type 
> TableMetrics is not applicable for the arguments (Iterable)
> --
> --
> 2. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/net/OutboundTcpConnectionPool.java
>  (at line 141)
>   return channel.socket();
>   
> Potential resource leak: 'channel' may not be closed at this location
> --
> --
> 3. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskBlock.java
>  (at line 93)
>   return cast(dup);
>   ^
> Potential resource leak: 'dup' may not be closed at this location
> --
> --
> 4. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskIndex.java
>  (at line 142)
>   indexFile = new MappedBuffer(new ChannelProxy(indexPath, 
> backingFile.getChannel()));
>
> ^
> Potential resource leak: '' may not be closed
> --
> 5. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskIndex.java
>  (at line 161)
>   throw new FSReadError(e, index);
>   
> Potential resource leak: 'backingFile' may not be closed at this location
> --
> 6. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskIndex.java
>  (at line 249)
>   RangeIterator range = searchRange(e);
>  ^
> Potential resource leak: 'range' may not be closed
> --
> --
> 7. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskIndexBuilder.java
>  (at line 286)
>   throw new FSWriteError(e, file);
>   
> Potential resource leak: 'out' may not be closed at this location
> --
> --
> 8. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/PerSSTableIndexWriter.java
>  (at line 284)
>   OnDiskIndex last = scheduleSegmentFlush(false).call();
>   
> Potential resource leak: 'last' may not be closed
> --
> 9. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/PerSSTableIndexWriter.java
>  (at line 293)
>   OnDiskIndex part 

[jira] [Commented] (CASSANDRA-11086) trunk eclipse-warnings

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163090#comment-15163090
 ] 

Jason Brown commented on CASSANDRA-11086:
-

Added a few {{SupressWarning}} annotations to where they are appropriate (and 
correct), as well as add try-with-resources blocks in two places.

||trunk||
|[branch|https://github.com/apache/cassandra/compare/trunk...jasobrown:11086_trunk]|
|[testall|http://cassci.datastax.com/view/Dev/view/jasobrown/job/jasobrown-11086_trunk-dtest/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/jasobrown/job/jasobrown-11086_trunk-testall/]|

Waiting for tests to finish execution...

> trunk eclipse-warnings
> --
>
> Key: CASSANDRA-11086
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11086
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Michael Shuler
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.x
>
>
> REF = origin/trunk 
> COMMIT = 7230a66318ce8add742959d095900d5870689f0c
> {noformat}
> # 1/27/16 8:50:25 PM UTC
> # Eclipse Compiler for Java(TM) v20150120-1634, 3.10.2, Copyright IBM Corp 
> 2000, 2013. All rights reserved.
> --
> 1. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/metrics/TableMetrics.java
>  (at line 338)
>   return 
> computeCompressionRatio(Iterables.concat(Iterables.transform(Keyspace.all(),
>  ^^^
> The method computeCompressionRatio(Iterable) in the type 
> TableMetrics is not applicable for the arguments (Iterable)
> --
> --
> 2. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/net/OutboundTcpConnectionPool.java
>  (at line 141)
>   return channel.socket();
>   
> Potential resource leak: 'channel' may not be closed at this location
> --
> --
> 3. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskBlock.java
>  (at line 93)
>   return cast(dup);
>   ^
> Potential resource leak: 'dup' may not be closed at this location
> --
> --
> 4. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskIndex.java
>  (at line 142)
>   indexFile = new MappedBuffer(new ChannelProxy(indexPath, 
> backingFile.getChannel()));
>
> ^
> Potential resource leak: '' may not be closed
> --
> 5. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskIndex.java
>  (at line 161)
>   throw new FSReadError(e, index);
>   
> Potential resource leak: 'backingFile' may not be closed at this location
> --
> 6. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskIndex.java
>  (at line 249)
>   RangeIterator range = searchRange(e);
>  ^
> Potential resource leak: 'range' may not be closed
> --
> --
> 7. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/OnDiskIndexBuilder.java
>  (at line 286)
>   throw new FSWriteError(e, file);
>   
> Potential resource leak: 'out' may not be closed at this location
> --
> --
> 8. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/PerSSTableIndexWriter.java
>  (at line 284)
>   OnDiskIndex last = scheduleSegmentFlush(false).call();
>   
> Potential resource leak: 'last' may not be closed
> --
> 9. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/disk/PerSSTableIndexWriter.java
>  (at line 293)
>   OnDiskIndex part = Futures.getUnchecked(f);
>   
> Potential resource leak: 'part' may not be closed
> --
> --
> 10. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/TermIterator.java
>  (at line 118)
>   RangeIterator keyIterator = index.search(e);
>  ^^^
> Potential resource leak: 'keyIterator' may not be closed
> --
> 11. ERROR in 
> /var/lib/jenkins/jobs/trunk_eclipse-warnings/workspace/src/java/org/apache/cassandra/index/sasi/TermIterator.java
>  (at line 148)
>   return ranges == null ? null : new TermIterator(e, ranges, 
> referencedIndexes);

[jira] [Commented] (CASSANDRA-11215) Reference leak with parallel repairs on the same table

[Marcus Eriksson (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163028#comment-15163028
 ] 

Marcus Eriksson commented on CASSANDRA-11215:
-

I guess we could ignore the other errors in the test, but assert that we 
actually get the "Cannot start multiple ..." - error in the logs?

> Reference leak with parallel repairs on the same table
> --
>
> Key: CASSANDRA-11215
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11215
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Olsson
>Assignee: Marcus Olsson
>
> When starting multiple repairs on the same table Cassandra starts to log 
> about reference leak as:
> {noformat}
> ERROR [Reference-Reaper:1] 2016-02-23 15:02:05,516 Ref.java:187 - LEAK 
> DETECTED: a reference 
> (org.apache.cassandra.utils.concurrent.Ref$State@5213f926) to class 
> org.apache.cassandra.io.sstable.format.SSTableReader
> $InstanceTidier@605893242:.../testrepair/standard1-dcf311a0da3411e5a5c0c1a39c091431/la-30-big
>  was not released before the reference was garbage collected
> {noformat}
> Reproducible with:
> {noformat}
> ccm create repairtest -v 2.2.5 -n 3
> ccm start
> ccm stress write n=100 -schema 
> replication(strategy=SimpleStrategy,factor=3) keyspace=testrepair
> # And then perform two repairs concurrently with:
> ccm node1 nodetool repair testrepair
> {noformat}
> I know that starting multiple repairs in parallel on the same table isn't 
> very wise, but this shouldn't result in reference leaks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[5/7] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[jasobrown]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a6239778
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a6239778
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a6239778

Branch: refs/heads/cassandra-3.0
Commit: a62397782c31a686c2a148e3f6ee75e1d58cf6d4
Parents: d7e0c30 941d13d
Author: Jason Brown 
Authored: Wed Feb 24 05:52:10 2016 -0800
Committer: Jason Brown 
Committed: Wed Feb 24 05:52:53 2016 -0800

--
 bin/cqlsh| 11 +--
 bin/cqlsh.py |  4 ++--
 2 files changed, 11 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/a6239778/bin/cqlsh.py
--

[4/7] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[jasobrown]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a6239778
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a6239778
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a6239778

Branch: refs/heads/trunk
Commit: a62397782c31a686c2a148e3f6ee75e1d58cf6d4
Parents: d7e0c30 941d13d
Author: Jason Brown 
Authored: Wed Feb 24 05:52:10 2016 -0800
Committer: Jason Brown 
Committed: Wed Feb 24 05:52:53 2016 -0800

--
 bin/cqlsh| 11 +--
 bin/cqlsh.py |  4 ++--
 2 files changed, 11 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/a6239778/bin/cqlsh.py
--

[3/7] cassandra git commit: Update cqlsh python version checking to find python2.7

[jasobrown]

Update cqlsh python version checking to find python2.7

patch by Jeremiah Jordan; reviewed by Stefania Alborghetti for CASSANDRA-11212


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/941d13da
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/941d13da
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/941d13da

Branch: refs/heads/trunk
Commit: 941d13dac8f67bf959c3f1f66d1972a4750ba116
Parents: 66b0d3f
Author: Jeremiah D Jordan 
Authored: Mon Feb 22 23:31:34 2016 -0600
Committer: Jason Brown 
Committed: Wed Feb 24 05:51:42 2016 -0800

--
 bin/cqlsh| 11 +--
 bin/cqlsh.py |  4 ++--
 2 files changed, 11 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/941d13da/bin/cqlsh
--
diff --git a/bin/cqlsh b/bin/cqlsh
index 89d094f..518b986 100755
--- a/bin/cqlsh
+++ b/bin/cqlsh
@@ -15,5 +15,12 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-
-python "`python -c "import 
os;print(os.path.dirname(os.path.realpath('$0')))"`/cqlsh.py" "$@"
+# bash code here; finds a suitable python interpreter and execs this file.
+# prefer unqualified "python" if suitable:
+python -c 'import sys; sys.exit(not (0x020700b0 < sys.hexversion < 
0x0300))' 2>/dev/null \
+&& exec python "`python -c "import 
os;print(os.path.dirname(os.path.realpath('$0')))"`/cqlsh.py" "$@"
+for pyver in 2.7; do
+which python$pyver > /dev/null 2>&1 && exec python$pyver "`python$pyver -c 
"import os;print(os.path.dirname(os.path.realpath('$0')))"`/cqlsh.py" "$@"
+done
+echo "No appropriate python interpreter found." >&2
+exit 1
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/cassandra/blob/941d13da/bin/cqlsh.py
--
diff --git a/bin/cqlsh.py b/bin/cqlsh.py
index c82a294..b3ec1ac 100644
--- a/bin/cqlsh.py
+++ b/bin/cqlsh.py
@@ -20,9 +20,9 @@
 """:"
 # bash code here; finds a suitable python interpreter and execs this file.
 # prefer unqualified "python" if suitable:
-python -c 'import sys; sys.exit(not (0x020500b0 < sys.hexversion < 
0x0300))' 2>/dev/null \
+python -c 'import sys; sys.exit(not (0x020700b0 < sys.hexversion < 
0x0300))' 2>/dev/null \
 && exec python "$0" "$@"
-for pyver in 2.6 2.7 2.5; do
+for pyver in 2.7; do
 which python$pyver > /dev/null 2>&1 && exec python$pyver "$0" "$@"
 done
 echo "No appropriate python interpreter found." >&2

[1/7] cassandra git commit: Update cqlsh python version checking to find python2.7

[jasobrown]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.2 66b0d3f01 -> 941d13dac
  refs/heads/cassandra-3.0 d7e0c3018 -> a62397782
  refs/heads/trunk a12188cd4 -> e856d8f81


Update cqlsh python version checking to find python2.7

patch by Jeremiah Jordan; reviewed by Stefania Alborghetti for CASSANDRA-11212


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/941d13da
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/941d13da
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/941d13da

Branch: refs/heads/cassandra-2.2
Commit: 941d13dac8f67bf959c3f1f66d1972a4750ba116
Parents: 66b0d3f
Author: Jeremiah D Jordan 
Authored: Mon Feb 22 23:31:34 2016 -0600
Committer: Jason Brown 
Committed: Wed Feb 24 05:51:42 2016 -0800

--
 bin/cqlsh| 11 +--
 bin/cqlsh.py |  4 ++--
 2 files changed, 11 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/941d13da/bin/cqlsh
--
diff --git a/bin/cqlsh b/bin/cqlsh
index 89d094f..518b986 100755
--- a/bin/cqlsh
+++ b/bin/cqlsh
@@ -15,5 +15,12 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-
-python "`python -c "import 
os;print(os.path.dirname(os.path.realpath('$0')))"`/cqlsh.py" "$@"
+# bash code here; finds a suitable python interpreter and execs this file.
+# prefer unqualified "python" if suitable:
+python -c 'import sys; sys.exit(not (0x020700b0 < sys.hexversion < 
0x0300))' 2>/dev/null \
+&& exec python "`python -c "import 
os;print(os.path.dirname(os.path.realpath('$0')))"`/cqlsh.py" "$@"
+for pyver in 2.7; do
+which python$pyver > /dev/null 2>&1 && exec python$pyver "`python$pyver -c 
"import os;print(os.path.dirname(os.path.realpath('$0')))"`/cqlsh.py" "$@"
+done
+echo "No appropriate python interpreter found." >&2
+exit 1
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/cassandra/blob/941d13da/bin/cqlsh.py
--
diff --git a/bin/cqlsh.py b/bin/cqlsh.py
index c82a294..b3ec1ac 100644
--- a/bin/cqlsh.py
+++ b/bin/cqlsh.py
@@ -20,9 +20,9 @@
 """:"
 # bash code here; finds a suitable python interpreter and execs this file.
 # prefer unqualified "python" if suitable:
-python -c 'import sys; sys.exit(not (0x020500b0 < sys.hexversion < 
0x0300))' 2>/dev/null \
+python -c 'import sys; sys.exit(not (0x020700b0 < sys.hexversion < 
0x0300))' 2>/dev/null \
 && exec python "$0" "$@"
-for pyver in 2.6 2.7 2.5; do
+for pyver in 2.7; do
 which python$pyver > /dev/null 2>&1 && exec python$pyver "$0" "$@"
 done
 echo "No appropriate python interpreter found." >&2

[6/7] cassandra git commit: Merge branch 'cassandra-3.0' into trunk

[jasobrown]

Merge branch 'cassandra-3.0' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/65464492
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/65464492
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/65464492

Branch: refs/heads/trunk
Commit: 65464492ef522c29b6a2812ab64d751118b62924
Parents: a12188c a623977
Author: Jason Brown 
Authored: Wed Feb 24 05:53:52 2016 -0800
Committer: Jason Brown 
Committed: Wed Feb 24 05:53:52 2016 -0800

--

--

[7/7] cassandra git commit: Update cqlsh python version checking to find python2.7

[jasobrown]

Update cqlsh python version checking to find python2.7

patch by Jeremiah Jordan; reviewed by Stefania Alborghetti for CASSANDRA-11212


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e856d8f8
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e856d8f8
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e856d8f8

Branch: refs/heads/trunk
Commit: e856d8f81efa7d1e33400798eadaf9b23e8da4c0
Parents: 6546449
Author: Jeremiah D Jordan 
Authored: Mon Feb 22 23:31:34 2016 -0600
Committer: Jason Brown 
Committed: Wed Feb 24 05:54:20 2016 -0800

--
 bin/cqlsh| 11 +--
 bin/cqlsh.py |  4 ++--
 2 files changed, 11 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/e856d8f8/bin/cqlsh
--
diff --git a/bin/cqlsh b/bin/cqlsh
index 89d094f..518b986 100755
--- a/bin/cqlsh
+++ b/bin/cqlsh
@@ -15,5 +15,12 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-
-python "`python -c "import 
os;print(os.path.dirname(os.path.realpath('$0')))"`/cqlsh.py" "$@"
+# bash code here; finds a suitable python interpreter and execs this file.
+# prefer unqualified "python" if suitable:
+python -c 'import sys; sys.exit(not (0x020700b0 < sys.hexversion < 
0x0300))' 2>/dev/null \
+&& exec python "`python -c "import 
os;print(os.path.dirname(os.path.realpath('$0')))"`/cqlsh.py" "$@"
+for pyver in 2.7; do
+which python$pyver > /dev/null 2>&1 && exec python$pyver "`python$pyver -c 
"import os;print(os.path.dirname(os.path.realpath('$0')))"`/cqlsh.py" "$@"
+done
+echo "No appropriate python interpreter found." >&2
+exit 1
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/cassandra/blob/e856d8f8/bin/cqlsh.py
--
diff --git a/bin/cqlsh.py b/bin/cqlsh.py
index c174346..1c40900 100644
--- a/bin/cqlsh.py
+++ b/bin/cqlsh.py
@@ -20,9 +20,9 @@
 """:"
 # bash code here; finds a suitable python interpreter and execs this file.
 # prefer unqualified "python" if suitable:
-python -c 'import sys; sys.exit(not (0x020500b0 < sys.hexversion < 
0x0300))' 2>/dev/null \
+python -c 'import sys; sys.exit(not (0x020700b0 < sys.hexversion < 
0x0300))' 2>/dev/null \
 && exec python "$0" "$@"
-for pyver in 2.6 2.7 2.5; do
+for pyver in 2.7; do
 which python$pyver > /dev/null 2>&1 && exec python$pyver "$0" "$@"
 done
 echo "No appropriate python interpreter found." >&2

[2/7] cassandra git commit: Update cqlsh python version checking to find python2.7

[jasobrown]

Update cqlsh python version checking to find python2.7

patch by Jeremiah Jordan; reviewed by Stefania Alborghetti for CASSANDRA-11212


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/941d13da
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/941d13da
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/941d13da

Branch: refs/heads/cassandra-3.0
Commit: 941d13dac8f67bf959c3f1f66d1972a4750ba116
Parents: 66b0d3f
Author: Jeremiah D Jordan 
Authored: Mon Feb 22 23:31:34 2016 -0600
Committer: Jason Brown 
Committed: Wed Feb 24 05:51:42 2016 -0800

--
 bin/cqlsh| 11 +--
 bin/cqlsh.py |  4 ++--
 2 files changed, 11 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/941d13da/bin/cqlsh
--
diff --git a/bin/cqlsh b/bin/cqlsh
index 89d094f..518b986 100755
--- a/bin/cqlsh
+++ b/bin/cqlsh
@@ -15,5 +15,12 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-
-python "`python -c "import 
os;print(os.path.dirname(os.path.realpath('$0')))"`/cqlsh.py" "$@"
+# bash code here; finds a suitable python interpreter and execs this file.
+# prefer unqualified "python" if suitable:
+python -c 'import sys; sys.exit(not (0x020700b0 < sys.hexversion < 
0x0300))' 2>/dev/null \
+&& exec python "`python -c "import 
os;print(os.path.dirname(os.path.realpath('$0')))"`/cqlsh.py" "$@"
+for pyver in 2.7; do
+which python$pyver > /dev/null 2>&1 && exec python$pyver "`python$pyver -c 
"import os;print(os.path.dirname(os.path.realpath('$0')))"`/cqlsh.py" "$@"
+done
+echo "No appropriate python interpreter found." >&2
+exit 1
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/cassandra/blob/941d13da/bin/cqlsh.py
--
diff --git a/bin/cqlsh.py b/bin/cqlsh.py
index c82a294..b3ec1ac 100644
--- a/bin/cqlsh.py
+++ b/bin/cqlsh.py
@@ -20,9 +20,9 @@
 """:"
 # bash code here; finds a suitable python interpreter and execs this file.
 # prefer unqualified "python" if suitable:
-python -c 'import sys; sys.exit(not (0x020500b0 < sys.hexversion < 
0x0300))' 2>/dev/null \
+python -c 'import sys; sys.exit(not (0x020700b0 < sys.hexversion < 
0x0300))' 2>/dev/null \
 && exec python "$0" "$@"
-for pyver in 2.6 2.7 2.5; do
+for pyver in 2.7; do
 which python$pyver > /dev/null 2>&1 && exec python$pyver "$0" "$@"
 done
 echo "No appropriate python interpreter found." >&2

[jira] [Commented] (CASSANDRA-11212) cqlsh python version checking is out of date

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163015#comment-15163015
 ] 

Jason Brown commented on CASSANDRA-11212:
-

I will commit this.

> cqlsh python version checking is out of date
> 
>
> Key: CASSANDRA-11212
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11212
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Jeremiah Jordan
>Assignee: Jeremiah Jordan
> Fix For: 2.2.x, 3.0.x, 3.x
>
>
> cqlsh.py has python version checking code at the top, but it still says 
> python 2.5 is a valid version, which we then error out on a few lines down in 
> the file.  We should fix that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[7/7] cassandra git commit: cqlsh: change default encoding to UTF-8

[jasobrown]

cqlsh: change default encoding to UTF-8

patch by Paulo Motta; reviewed by Stefania Alborghetti for CASSANDRA-11124


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a12188cd
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a12188cd
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a12188cd

Branch: refs/heads/trunk
Commit: a12188cd4e6fcef496480fdb9127b26aeb33b589
Parents: 6be8346
Author: Paulo Motta 
Authored: Wed Feb 17 15:25:59 2016 -0300
Committer: Jason Brown 
Committed: Wed Feb 24 05:46:08 2016 -0800

--
 CHANGES.txt |  1 +
 bin/cqlsh.py| 12 
 conf/cqlshrc.sample |  3 ++-
 3 files changed, 7 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/a12188cd/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 361eedc..1d8664c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -62,6 +62,7 @@ Merged from 2.2:
  * Fix paging on DISTINCT queries repeats result when first row in partition 
changes
(CASSANDRA-10010)
  * (cqlsh) Support timezone conversion using pytz (CASSANDRA-10397)
+ * cqlsh: change default encoding to UTF-8 (CASSANDRA-11124)
 Merged from 2.1:
  * Don't remove FailureDetector history on removeEndpoint (CASSANDRA-10371)
  * Only notify if repair status changed (CASSANDRA-11172)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a12188cd/bin/cqlsh.py
--
diff --git a/bin/cqlsh.py b/bin/cqlsh.py
index 3d7ea5d..c174346 100644
--- a/bin/cqlsh.py
+++ b/bin/cqlsh.py
@@ -217,9 +217,8 @@ parser.add_option('-k', '--keyspace', help='Authenticate to 
the given keyspace.'
 parser.add_option("-f", "--file", help="Execute commands from FILE, then exit")
 parser.add_option('--debug', action='store_true',
   help='Show additional debugging information')
-parser.add_option("--encoding", help="Specify a non-default encoding for 
output.  If you are " +
-  "experiencing problems with unicode characters, using utf8 
may fix the problem." +
-  " (Default from system preferences: %s)" % 
(locale.getpreferredencoding(),))
+parser.add_option("--encoding", help="Specify a non-default encoding for 
output." +
+  " (Default: %s)" % (UTF8,))
 parser.add_option("--cqlshrc", help="Specify an alternative cqlshrc file 
location.")
 parser.add_option('--cqlversion', default=DEFAULT_CQLVER,
   help='Specify a particular CQL version (default: %default).'
@@ -760,10 +759,6 @@ class Shell(cmd.Cmd):
 self.session.max_trace_wait = max_trace_wait
 
 self.tty = tty
-if encoding is None:
-encoding = locale.getpreferredencoding()
-if encoding is None:
-encoding = UTF8
 self.encoding = encoding
 self.check_windows_encoding()
 
@@ -2446,7 +2441,7 @@ def read_options(cmdlineargs, environment):
 optvalues.debug = False
 optvalues.file = None
 optvalues.ssl = False
-optvalues.encoding = None
+optvalues.encoding = option_with_default(configs.get, 'ui', 'encoding', 
UTF8)
 
 optvalues.tty = option_with_default(configs.getboolean, 'ui', 'tty', 
sys.stdin.isatty())
 optvalues.cqlversion = option_with_default(configs.get, 'cql', 'version', 
DEFAULT_CQLVER)
@@ -2559,6 +2554,7 @@ def main(options, hostname, port):
 if options.debug:
 sys.stderr.write("Using CQL driver: %s\n" % (cassandra,))
 sys.stderr.write("Using connect timeout: %s seconds\n" % 
(options.connect_timeout,))
+sys.stderr.write("Using '%s' encoding\n" % (options.encoding,))
 
 # create timezone based on settings, environment or auto-detection
 timezone = None

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a12188cd/conf/cqlshrc.sample
--
diff --git a/conf/cqlshrc.sample b/conf/cqlshrc.sample
index a0012a3..462dcc6 100644
--- a/conf/cqlshrc.sample
+++ b/conf/cqlshrc.sample
@@ -42,7 +42,8 @@
 ;; Used for automatic completion and suggestions
 ; completekey = tab
 
-
+;; The encoding used for characters
+; encoding = utf8
 
 ; To use another than the system default browser for cqlsh HELP to open
 ; the CQL doc HTML, use the 'browser' preference.

[4/7] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[jasobrown]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d7e0c301
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d7e0c301
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d7e0c301

Branch: refs/heads/trunk
Commit: d7e0c301863858b913f234b158aae383ed8c4ae1
Parents: c4bd6d2 66b0d3f
Author: Jason Brown 
Authored: Wed Feb 24 05:43:31 2016 -0800
Committer: Jason Brown 
Committed: Wed Feb 24 05:44:41 2016 -0800

--
 CHANGES.txt |  1 +
 bin/cqlsh.py| 12 
 conf/cqlshrc.sample |  3 ++-
 3 files changed, 7 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d7e0c301/CHANGES.txt
--
diff --cc CHANGES.txt
index 9ca2f80,9162165..4549ded
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -33,6 -16,6 +33,7 @@@ Merged from 2.2
   * (cqlsh) Support utf-8/cp65001 encoding on Windows (CASSANDRA-11030)
   * Fix paging on DISTINCT queries repeats result when first row in partition 
changes
 (CASSANDRA-10010)
++ * cqlsh: change default encoding to UTF-8 (CASSANDRA-11124)
  Merged from 2.1:
   * Don't remove FailureDetector history on removeEndpoint (CASSANDRA-10371)
   * Only notify if repair status changed (CASSANDRA-11172)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d7e0c301/bin/cqlsh.py
--

[2/7] cassandra git commit: cqlsh: change default encoding to UTF-8

[jasobrown]

cqlsh: change default encoding to UTF-8

patch by Paulo Motta; reviewed by Stefania Alborghetti for CASSANDRA-11124


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/66b0d3f0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/66b0d3f0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/66b0d3f0

Branch: refs/heads/cassandra-3.0
Commit: 66b0d3f016c5ca3c167960a8f7bbef815c95363f
Parents: 77ff794
Author: Paulo Motta 
Authored: Wed Feb 17 15:25:59 2016 -0300
Committer: Jason Brown 
Committed: Wed Feb 24 05:43:00 2016 -0800

--
 CHANGES.txt |  2 +-
 bin/cqlsh.py| 12 
 conf/cqlshrc.sample |  3 ++-
 3 files changed, 7 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/66b0d3f0/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index e989e7f..9162165 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -30,7 +30,7 @@ Merged from 2.1:
  * Make it clear what DTCS timestamp_resolution is used for (CASSANDRA-11041)
  * test_bulk_round_trip_blogposts is failing occasionally (CASSANDRA-10938)
  * (cqlsh) Support timezone conversion using pytz (CASSANDRA-10397)
-
+ * cqlsh: change default encoding to UTF-8 (CASSANDRA-11124)
 
 2.2.5
  * maxPurgeableTimestamp needs to check memtables too (CASSANDRA-9949)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/66b0d3f0/bin/cqlsh.py
--
diff --git a/bin/cqlsh.py b/bin/cqlsh.py
index 6a464b0..c82a294 100644
--- a/bin/cqlsh.py
+++ b/bin/cqlsh.py
@@ -209,9 +209,8 @@ parser.add_option('-k', '--keyspace', help='Authenticate to 
the given keyspace.'
 parser.add_option("-f", "--file", help="Execute commands from FILE, then exit")
 parser.add_option('--debug', action='store_true',
   help='Show additional debugging information')
-parser.add_option("--encoding", help="Specify a non-default encoding for 
output.  If you are " +
-  "experiencing problems with unicode characters, using utf8 
may fix the problem." +
-  " (Default from system preferences: %s)" % 
(locale.getpreferredencoding(),))
+parser.add_option("--encoding", help="Specify a non-default encoding for 
output." +
+  " (Default: %s)" % (UTF8,))
 parser.add_option("--cqlshrc", help="Specify an alternative cqlshrc file 
location.")
 parser.add_option('--cqlversion', default=DEFAULT_CQLVER,
   help='Specify a particular CQL version (default: %default).'
@@ -737,10 +736,6 @@ class Shell(cmd.Cmd):
 self.session.max_trace_wait = max_trace_wait
 
 self.tty = tty
-if encoding is None:
-encoding = locale.getpreferredencoding()
-if encoding is None:
-encoding = UTF8
 self.encoding = encoding
 self.check_windows_encoding()
 
@@ -2358,7 +2353,7 @@ def read_options(cmdlineargs, environment):
 optvalues.debug = False
 optvalues.file = None
 optvalues.ssl = False
-optvalues.encoding = None
+optvalues.encoding = option_with_default(configs.get, 'ui', 'encoding', 
UTF8)
 
 optvalues.tty = option_with_default(configs.getboolean, 'ui', 'tty', 
sys.stdin.isatty())
 optvalues.cqlversion = option_with_default(configs.get, 'cql', 'version', 
DEFAULT_CQLVER)
@@ -2471,6 +2466,7 @@ def main(options, hostname, port):
 if options.debug:
 sys.stderr.write("Using CQL driver: %s\n" % (cassandra,))
 sys.stderr.write("Using connect timeout: %s seconds\n" % 
(options.connect_timeout,))
+sys.stderr.write("Using '%s' encoding\n" % (options.encoding,))
 
 # create timezone based on settings, environment or auto-detection
 timezone = None

http://git-wip-us.apache.org/repos/asf/cassandra/blob/66b0d3f0/conf/cqlshrc.sample
--
diff --git a/conf/cqlshrc.sample b/conf/cqlshrc.sample
index a0012a3..462dcc6 100644
--- a/conf/cqlshrc.sample
+++ b/conf/cqlshrc.sample
@@ -42,7 +42,8 @@
 ;; Used for automatic completion and suggestions
 ; completekey = tab
 
-
+;; The encoding used for characters
+; encoding = utf8
 
 ; To use another than the system default browser for cqlsh HELP to open
 ; the CQL doc HTML, use the 'browser' preference.

[6/7] cassandra git commit: Merge branch 'cassandra-3.0' into trunk

[jasobrown]

Merge branch 'cassandra-3.0' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6be83462
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6be83462
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6be83462

Branch: refs/heads/trunk
Commit: 6be83462c2faef7198694d3df44dcb8f2cf97fd2
Parents: ac8c8b2 d7e0c30
Author: Jason Brown 
Authored: Wed Feb 24 05:45:03 2016 -0800
Committer: Jason Brown 
Committed: Wed Feb 24 05:45:03 2016 -0800

--

--

[5/7] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0

[jasobrown]

Merge branch 'cassandra-2.2' into cassandra-3.0


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d7e0c301
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d7e0c301
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d7e0c301

Branch: refs/heads/cassandra-3.0
Commit: d7e0c301863858b913f234b158aae383ed8c4ae1
Parents: c4bd6d2 66b0d3f
Author: Jason Brown 
Authored: Wed Feb 24 05:43:31 2016 -0800
Committer: Jason Brown 
Committed: Wed Feb 24 05:44:41 2016 -0800

--
 CHANGES.txt |  1 +
 bin/cqlsh.py| 12 
 conf/cqlshrc.sample |  3 ++-
 3 files changed, 7 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d7e0c301/CHANGES.txt
--
diff --cc CHANGES.txt
index 9ca2f80,9162165..4549ded
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -33,6 -16,6 +33,7 @@@ Merged from 2.2
   * (cqlsh) Support utf-8/cp65001 encoding on Windows (CASSANDRA-11030)
   * Fix paging on DISTINCT queries repeats result when first row in partition 
changes
 (CASSANDRA-10010)
++ * cqlsh: change default encoding to UTF-8 (CASSANDRA-11124)
  Merged from 2.1:
   * Don't remove FailureDetector history on removeEndpoint (CASSANDRA-10371)
   * Only notify if repair status changed (CASSANDRA-11172)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d7e0c301/bin/cqlsh.py
--

[3/7] cassandra git commit: cqlsh: change default encoding to UTF-8

[jasobrown]

cqlsh: change default encoding to UTF-8

patch by Paulo Motta; reviewed by Stefania Alborghetti for CASSANDRA-11124


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/66b0d3f0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/66b0d3f0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/66b0d3f0

Branch: refs/heads/trunk
Commit: 66b0d3f016c5ca3c167960a8f7bbef815c95363f
Parents: 77ff794
Author: Paulo Motta 
Authored: Wed Feb 17 15:25:59 2016 -0300
Committer: Jason Brown 
Committed: Wed Feb 24 05:43:00 2016 -0800

--
 CHANGES.txt |  2 +-
 bin/cqlsh.py| 12 
 conf/cqlshrc.sample |  3 ++-
 3 files changed, 7 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/66b0d3f0/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index e989e7f..9162165 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -30,7 +30,7 @@ Merged from 2.1:
  * Make it clear what DTCS timestamp_resolution is used for (CASSANDRA-11041)
  * test_bulk_round_trip_blogposts is failing occasionally (CASSANDRA-10938)
  * (cqlsh) Support timezone conversion using pytz (CASSANDRA-10397)
-
+ * cqlsh: change default encoding to UTF-8 (CASSANDRA-11124)
 
 2.2.5
  * maxPurgeableTimestamp needs to check memtables too (CASSANDRA-9949)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/66b0d3f0/bin/cqlsh.py
--
diff --git a/bin/cqlsh.py b/bin/cqlsh.py
index 6a464b0..c82a294 100644
--- a/bin/cqlsh.py
+++ b/bin/cqlsh.py
@@ -209,9 +209,8 @@ parser.add_option('-k', '--keyspace', help='Authenticate to 
the given keyspace.'
 parser.add_option("-f", "--file", help="Execute commands from FILE, then exit")
 parser.add_option('--debug', action='store_true',
   help='Show additional debugging information')
-parser.add_option("--encoding", help="Specify a non-default encoding for 
output.  If you are " +
-  "experiencing problems with unicode characters, using utf8 
may fix the problem." +
-  " (Default from system preferences: %s)" % 
(locale.getpreferredencoding(),))
+parser.add_option("--encoding", help="Specify a non-default encoding for 
output." +
+  " (Default: %s)" % (UTF8,))
 parser.add_option("--cqlshrc", help="Specify an alternative cqlshrc file 
location.")
 parser.add_option('--cqlversion', default=DEFAULT_CQLVER,
   help='Specify a particular CQL version (default: %default).'
@@ -737,10 +736,6 @@ class Shell(cmd.Cmd):
 self.session.max_trace_wait = max_trace_wait
 
 self.tty = tty
-if encoding is None:
-encoding = locale.getpreferredencoding()
-if encoding is None:
-encoding = UTF8
 self.encoding = encoding
 self.check_windows_encoding()
 
@@ -2358,7 +2353,7 @@ def read_options(cmdlineargs, environment):
 optvalues.debug = False
 optvalues.file = None
 optvalues.ssl = False
-optvalues.encoding = None
+optvalues.encoding = option_with_default(configs.get, 'ui', 'encoding', 
UTF8)
 
 optvalues.tty = option_with_default(configs.getboolean, 'ui', 'tty', 
sys.stdin.isatty())
 optvalues.cqlversion = option_with_default(configs.get, 'cql', 'version', 
DEFAULT_CQLVER)
@@ -2471,6 +2466,7 @@ def main(options, hostname, port):
 if options.debug:
 sys.stderr.write("Using CQL driver: %s\n" % (cassandra,))
 sys.stderr.write("Using connect timeout: %s seconds\n" % 
(options.connect_timeout,))
+sys.stderr.write("Using '%s' encoding\n" % (options.encoding,))
 
 # create timezone based on settings, environment or auto-detection
 timezone = None

http://git-wip-us.apache.org/repos/asf/cassandra/blob/66b0d3f0/conf/cqlshrc.sample
--
diff --git a/conf/cqlshrc.sample b/conf/cqlshrc.sample
index a0012a3..462dcc6 100644
--- a/conf/cqlshrc.sample
+++ b/conf/cqlshrc.sample
@@ -42,7 +42,8 @@
 ;; Used for automatic completion and suggestions
 ; completekey = tab
 
-
+;; The encoding used for characters
+; encoding = utf8
 
 ; To use another than the system default browser for cqlsh HELP to open
 ; the CQL doc HTML, use the 'browser' preference.

[1/7] cassandra git commit: cqlsh: change default encoding to UTF-8

[jasobrown]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-2.2 77ff79473 -> 66b0d3f01
  refs/heads/cassandra-3.0 c4bd6d254 -> d7e0c3018
  refs/heads/trunk ac8c8b213 -> a12188cd4


cqlsh: change default encoding to UTF-8

patch by Paulo Motta; reviewed by Stefania Alborghetti for CASSANDRA-11124


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/66b0d3f0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/66b0d3f0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/66b0d3f0

Branch: refs/heads/cassandra-2.2
Commit: 66b0d3f016c5ca3c167960a8f7bbef815c95363f
Parents: 77ff794
Author: Paulo Motta 
Authored: Wed Feb 17 15:25:59 2016 -0300
Committer: Jason Brown 
Committed: Wed Feb 24 05:43:00 2016 -0800

--
 CHANGES.txt |  2 +-
 bin/cqlsh.py| 12 
 conf/cqlshrc.sample |  3 ++-
 3 files changed, 7 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/66b0d3f0/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index e989e7f..9162165 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -30,7 +30,7 @@ Merged from 2.1:
  * Make it clear what DTCS timestamp_resolution is used for (CASSANDRA-11041)
  * test_bulk_round_trip_blogposts is failing occasionally (CASSANDRA-10938)
  * (cqlsh) Support timezone conversion using pytz (CASSANDRA-10397)
-
+ * cqlsh: change default encoding to UTF-8 (CASSANDRA-11124)
 
 2.2.5
  * maxPurgeableTimestamp needs to check memtables too (CASSANDRA-9949)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/66b0d3f0/bin/cqlsh.py
--
diff --git a/bin/cqlsh.py b/bin/cqlsh.py
index 6a464b0..c82a294 100644
--- a/bin/cqlsh.py
+++ b/bin/cqlsh.py
@@ -209,9 +209,8 @@ parser.add_option('-k', '--keyspace', help='Authenticate to 
the given keyspace.'
 parser.add_option("-f", "--file", help="Execute commands from FILE, then exit")
 parser.add_option('--debug', action='store_true',
   help='Show additional debugging information')
-parser.add_option("--encoding", help="Specify a non-default encoding for 
output.  If you are " +
-  "experiencing problems with unicode characters, using utf8 
may fix the problem." +
-  " (Default from system preferences: %s)" % 
(locale.getpreferredencoding(),))
+parser.add_option("--encoding", help="Specify a non-default encoding for 
output." +
+  " (Default: %s)" % (UTF8,))
 parser.add_option("--cqlshrc", help="Specify an alternative cqlshrc file 
location.")
 parser.add_option('--cqlversion', default=DEFAULT_CQLVER,
   help='Specify a particular CQL version (default: %default).'
@@ -737,10 +736,6 @@ class Shell(cmd.Cmd):
 self.session.max_trace_wait = max_trace_wait
 
 self.tty = tty
-if encoding is None:
-encoding = locale.getpreferredencoding()
-if encoding is None:
-encoding = UTF8
 self.encoding = encoding
 self.check_windows_encoding()
 
@@ -2358,7 +2353,7 @@ def read_options(cmdlineargs, environment):
 optvalues.debug = False
 optvalues.file = None
 optvalues.ssl = False
-optvalues.encoding = None
+optvalues.encoding = option_with_default(configs.get, 'ui', 'encoding', 
UTF8)
 
 optvalues.tty = option_with_default(configs.getboolean, 'ui', 'tty', 
sys.stdin.isatty())
 optvalues.cqlversion = option_with_default(configs.get, 'cql', 'version', 
DEFAULT_CQLVER)
@@ -2471,6 +2466,7 @@ def main(options, hostname, port):
 if options.debug:
 sys.stderr.write("Using CQL driver: %s\n" % (cassandra,))
 sys.stderr.write("Using connect timeout: %s seconds\n" % 
(options.connect_timeout,))
+sys.stderr.write("Using '%s' encoding\n" % (options.encoding,))
 
 # create timezone based on settings, environment or auto-detection
 timezone = None

http://git-wip-us.apache.org/repos/asf/cassandra/blob/66b0d3f0/conf/cqlshrc.sample
--
diff --git a/conf/cqlshrc.sample b/conf/cqlshrc.sample
index a0012a3..462dcc6 100644
--- a/conf/cqlshrc.sample
+++ b/conf/cqlshrc.sample
@@ -42,7 +42,8 @@
 ;; Used for automatic completion and suggestions
 ; completekey = tab
 
-
+;; The encoding used for characters
+; encoding = utf8
 
 ; To use another than the system default browser for cqlsh HELP to open
 ; the CQL doc HTML, use the 'browser' preference.

[jira] [Commented] (CASSANDRA-11124) Change default cqlsh encoding to utf-8

[Jason Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15162997#comment-15162997
 ] 

Jason Brown commented on CASSANDRA-11124:
-

I will commit this

> Change default cqlsh encoding to utf-8
> --
>
> Key: CASSANDRA-11124
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11124
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Tools
>Reporter: Paulo Motta
>Assignee: Paulo Motta
>Priority: Trivial
>  Labels: cqlsh
>
> Strange things can happen when utf-8 is not the default cqlsh encoding (see 
> CASSANDRA-11030). This ticket proposes changing the default cqlsh encoding to 
> utf-8.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-10091) Align JMX authentication with internal authentication

[Sam Tunnicliffe (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15162912#comment-15162912
 ] 

Sam Tunnicliffe commented on CASSANDRA-10091:
-

[~Jan Karlsson] I've pushed a rebased & squashed patch to [my 
branch|https://github.com/beobal/cassandra/tree/10091-trunk] which incorporates 
the caching changes that went in with CASSANDRA-7715. I've also pulled the 
setup of the JMX connector server out of CassandraDaemon and refactored it a 
bit. A benefit of that is that is simplifies cassandra-env quite a bit, and we 
can do away with the JMX_MODE setting. Also, it makes authn/authz and ssl 
orthogonal to whether jmx is running in local-only mode or not. I also switched 
back to using CassandraLoginModule, and having the JMX authenticator use that, 
rather than going directly through the authenticator, because as you pointed 
out this makes it somewhat easier to extend. Authentication and authorization 
are also nicely independent, so operators have to options of using the various 
combinations of C*'s own authn (or some other LoginModule) or standard JMX file 
based authn in conjunction with C* authz, access file authz, or no authz.

Sorry it's taken so long, but I'd be glad to get your feedback at this point.


> Align JMX authentication with internal authentication
> -
>
> Key: CASSANDRA-10091
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10091
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Jan Karlsson
>Assignee: Jan Karlsson
>Priority: Minor
> Fix For: 3.x
>
>
> It would be useful to authenticate with JMX through Cassandra's internal 
> authentication. This would reduce the overhead of keeping passwords in files 
> on the machine and would consolidate passwords to one location. It would also 
> allow the possibility to handle JMX permissions in Cassandra.
> It could be done by creating our own JMX server and setting custom classes 
> for the authenticator and authorizer. We could then add some parameters where 
> the user could specify what authenticator and authorizer to use in case they 
> want to make their own.
> This could also be done by creating a premain method which creates a jmx 
> server. This would give us the feature without changing the Cassandra code 
> itself. However I believe this would be a good feature to have in Cassandra.
> I am currently working on a solution which creates a JMX server and uses a 
> custom authenticator and authorizer. It is currently build as a premain, 
> however it would be great if we could put this in Cassandra instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-11053) COPY FROM on large datasets: fix progress report and debug performance

[Stefania (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15162800#comment-15162800
 ] 

Stefania commented on CASSANDRA-11053:
--

The ticket is ready for review [~aholmber].

The patch: https://github.com/stef1927/cassandra/tree/11053-2.1
The dtest branch: https://github.com/stef1927/cassandra-dtest/tree/11053
The dtest results: 
http://cassci.datastax.com/job/stef1927-11053-2.1-dtest/lastCompletedBuild/testReport/

The reason the patch targets 2.1 is mostly to compare with cassandra loader. We 
must decide whether we want this in 2.1 or 2.2. Technically it should be in 2.2 
but the original COPY FROM performance ticket (CASSANDRA-9302) was new 
functionality in 2.1 and hence I am not entirely sure, cc [~jbellis].

h3. Performance results

The full results are available in this 
[spreadsheet|https://docs.google.com/spreadsheets/d/1XTE2fSDJkwHzpdaD5HI0HlsFuPCW1Kc1NeqauF6WX2s/edit#gid=0].

h4. Executive Summary

The optimizations introduced by this patch have increased the import speed 
(average rows per second) for the 1KB test (20,480,000 rows of 1kb, as 
specified in this 
[benchmark|https://github.com/brianmhess/DSEBulkLoadTest.git]) from *35k* to 
*117k* on an 8 core VM and to *190k* on a 16 core VM. To achieve this, the 
following points were adopted and are listed in order of importance:

* the driver must be installed with c extensions, cython and libev, and used by 
setting {{export CQLSH_NO_BUNDLED=TRUE}}
* the number of worker processes and the chunk size COPY FROM options must be 
adjusted for maximum performance by looking at performance stats ({{dstat -lvrn 
10}}) and minimizing the CPU idle time
* minimum and maximum batch size must be increased if close to cluster 
saturation or in the presence of VNODES
* Linux CPU scheduling must be set to {{SCHED_BATCH}} via {{schedtool -B}}
* the cqlsh copyutil module must also be cythonized via {{python setup.py 
build_ext --inplace}}
* the clock source must be set to "tsc"

For further details refer to the setup instructions in the spreadsheet.

h4. Setup

The spreadsheet above contains full setup details, including commands on how to 
launch and configure AWS instances. All tests were run on Ubuntu 14.04 with 
clock source set to tsc, using prepared statements. The client was running 
cassandra-2.1 with this ticket patch, and the servers were running 
cassandra-2.1 HEAD with the following customizations:

{code}
batch_size_warn_threshold_in_kb: 65
compaction_throughput_mb_per_sec: 0
auto_snapshot: false
{code}


h4. R3.2XLARGE client and 8 node I2.2XLARGE cluster

This is the set-up that has been used so far. These tests give the final 
numbers and highlight the impact of cythonizing modules:

||MODULE CYTHONIZED||NUM WORKER PROCESSES||CHUNK SIZE||AVG. ROWS / SEC||
|NONE|12|5,000|70k|
|DRIVER|12|5,000|110k|
|DRIVER + COPYUTIL|12|5,000|117k|

The number of worker processes and chunk size were chosen optimally after 
several trials. It was noted that the optimal number of worker processes is 
slightly higher than the number of cores. CPU usage was observed with {{dstat}} 
in order to minimize idle time, and increase network packets sent. 

Here is the impact of introducing VNODES with 256 tokens per node:

||MODULE CYTHONIZED||NUM WORKER PROCESSES||CHUNK SIZE||ADDITIONAL PARAMS||AVG. 
ROWS / SEC||
|DRIVER|12|5,000| |86k \(T\)|
|DRIVER|12|10,000| |68k \(T\)|
|DRIVER|12|20,000| |66k \(T\)|
|DRIVER|12|40,000| |63k \(T\)|
|DRIVER|12|20,000|"MIN_BATCH_SIZE=20 MAX_BATCH_SIZE=30"|95k \(B\)|
|DRIVER|12|30,000|"MIN_BATCH_SIZE=30 MAX_BATCH_SIZE=40"|101k \(B\)|
|DRIVER|12|40,000|"MIN_BATCH_SIZE=30 MAX_BATCH_SIZE=40"|105k \(B\)|

*\(T\)* indicates timeouts, which resulted in degraded average performance. 
*\(B\)* indicates batch sizes higher than default values (MIN_BATCH_SIZE=10 
MAX_BATCH_SIZE=20) and was a workaround adopted to mitigate the timeouts as 
indicated by the results.

Another workaround for the timeouts was to increase cluster size and was 
adopted in the next set of tests.

h4. C3.2XLARGE client and 15 node C3.2XLARGE cluster

To investigate scaling and improvements with larger clusters we switched to AWS 
instances with less memory (COPY FROM doesn't need it) and less storage (by 
truncating the table after every measurement and setting {{auto_snapshot}} to 
false in cassandra.yaml we were able to perform tests with the same dataset but 
less disk space). All tests were run with only the driver cythonized, not 
copyutil, and using prepared statements.

This is the impact of batch CPU scheduling ({{schedtool -B}}):

||NUM WORKER PROCESSES||CHUNK SIZE||MIN-MAX BATCH SIZE||{{schedtool -B}}||AVG. 
ROWS / SEC||
|12|5000|10-20 (default)|NO|99k|
|12|1|10-20 (default)|NO|97k|
|12|5000|20-40|NO|111k|
|12|1|20-40|NO|109k|
|12|5000|30-50|NO|114k|
|12|1|30-50|NO|113k|
|12|5000|10-20 (default)|YES|107k|
|12|1|10-20 (default)|YES|106k|

[jira] [Issue Comment Deleted] (CASSANDRA-11209) SSTable ancestor leaked reference

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-11209:
---
Comment: was deleted

(was: Similar to CASSANDRA-10510 as well ?)

> SSTable ancestor leaked reference
> -
>
> Key: CASSANDRA-11209
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11209
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Jose Fernandez
>Assignee: Marcus Eriksson
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> We're running a fork of 2.1.13 that adds the TimeWindowCompactionStrategy 
> from [~jjirsa]. We've been running 4 clusters without any issues for many 
> months until a few weeks ago we started scheduling incremental repairs every 
> 24 hours (previously we didn't run any repairs at all).
> Since then we started noticing big discrepancies in the LiveDiskSpaceUsed, 
> TotalDiskSpaceUsed, and actual size of files on disk. The numbers are brought 
> back in sync by restarting the node. We also noticed that when this bug 
> happens there are several ancestors that don't get cleaned up. A restart will 
> queue up a lot of compactions that slowly eat away the ancestors.
> I looked at the code and noticed that we only decrease the LiveTotalDiskUsed 
> metric in the SSTableDeletingTask. Since we have no errors being logged, I'm 
> assuming that for some reason this task is not getting queued up. If I 
> understand correctly this only happens when the reference count for the 
> SStable reaches 0. So this is leading us to believe that something during 
> repairs and/or compactions is causing a reference leak to the ancestor table.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)