[jira] [Commented] (CASSANDRA-2848) Make the Client API support passing down timeouts
[ https://issues.apache.org/jira/browse/CASSANDRA-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504494#comment-14504494 ] Sylvain Lebresne commented on CASSANDRA-2848: - bq. Would it make sense to use CASSANDRA-8553? I'd rather not. I'd prefer keeping the custom payload really custom and never use it for anything Cassandra. And honestly, the protocol change for this (using the remaining flag) is really not a big deal so it's not like we need to look for alternatives. One thing worth mentioning on this issue however is that currently C* doesn't respects timeouts too well. Typically, any query with a digest mismatch could take closer to twice the configured timeout, and with short read retries that can be even worth. And some practical experience with the java driver has shown that it's not a theoretical problem: it's relatively easy to have C* take way longer than the configured timeout to answer (as in multiple seconds more than the configured timeout). So I'm left wondering if we shouldn't fix that first, having timeouts really be a reasonably precise upper-bound on when C* will respond to a user query, before allowing per-query timeouts? Make the Client API support passing down timeouts - Key: CASSANDRA-2848 URL: https://issues.apache.org/jira/browse/CASSANDRA-2848 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Chris Goffinet Assignee: Sylvain Lebresne Priority: Minor Fix For: 3.1 Having a max server RPC timeout is good for worst case, but many applications that have middleware in front of Cassandra, might have higher timeout requirements. In a fail fast environment, if my application starting at say the front-end, only has 20ms to process a request, and it must connect to X services down the stack, by the time it hits Cassandra, we might only have 10ms. I propose we provide the ability to specify the timeout on each call we do optionally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-2848) Make the Client API support passing down timeouts
[
https://issues.apache.org/jira/browse/CASSANDRA-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504502#comment-14504502
]
Sylvain Lebresne commented on CASSANDRA-2848:
-
As a side note, the patch on CASSANDRA-5239 was actually fixing the timeout
precision problem mentioned above (by using a timer for the whole query and
timeouting the future if need be) but that's not directly applicable to the
current synchronous {{StorageProxy}}.
Make the Client API support passing down timeouts
-
Key: CASSANDRA-2848
URL: https://issues.apache.org/jira/browse/CASSANDRA-2848
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Chris Goffinet
Assignee: Sylvain Lebresne
Priority: Minor
Fix For: 3.1
Having a max server RPC timeout is good for worst case, but many applications
that have middleware in front of Cassandra, might have higher timeout
requirements. In a fail fast environment, if my application starting at say
the front-end, only has 20ms to process a request, and it must connect to X
services down the stack, by the time it hits Cassandra, we might only have
10ms. I propose we provide the ability to specify the timeout on each call we
do optionally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[1/2] cassandra git commit: Fix whitespace, unused imports and variables in cqlsh
Repository: cassandra
Updated Branches:
refs/heads/trunk b379e3260 - 840d9a663
Fix whitespace, unused imports and variables in cqlsh
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/49c1dbda
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/49c1dbda
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/49c1dbda
Branch: refs/heads/trunk
Commit: 49c1dbdad4bd36081ab79aee378c6a4ed71cb3cc
Parents: aeadda5
Author: Tyler Hobbs tylerlho...@gmail.com
Authored: Tue Apr 21 12:20:37 2015 -0500
Committer: Tyler Hobbs tylerlho...@gmail.com
Committed: Tue Apr 21 12:20:37 2015 -0500
--
bin/cqlsh | 84 --
1 file changed, 58 insertions(+), 26 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/49c1dbda/bin/cqlsh
--
diff --git a/bin/cqlsh b/bin/cqlsh
index e1bd312..05f3368 100755
--- a/bin/cqlsh
+++ b/bin/cqlsh
@@ -36,7 +36,7 @@ description = CQL Shell for Apache Cassandra
version = 5.0.1
from StringIO import StringIO
-from contextlib import contextmanager, closing
+from contextlib import contextmanager
from glob import glob
import cmd
@@ -79,6 +79,7 @@ if myplatform == 'Linux':
if os.environ.get('CQLSH_NO_BUNDLED', ''):
ZIPLIB_DIRS = ()
+
def find_zip(libprefix):
for ziplibdir in ZIPLIB_DIRS:
zips = glob(os.path.join(ziplibdir, libprefix + '*.zip'))
@@ -111,7 +112,6 @@ from cassandra.cluster import Cluster, PagedResult
from cassandra.query import SimpleStatement, ordered_dict_factory
from cassandra.policies import WhiteListRoundRobinPolicy
from cassandra.protocol import QueryMessage, ResultMessage
-from cassandra.marshal import int16_pack, int32_pack, uint64_pack
from cassandra.metadata import protect_name, protect_names, protect_value
from cassandra.auth import PlainTextAuthProvider
@@ -142,6 +142,9 @@ if readline is not None and readline.__doc__ is not None
and 'libedit' in readli
else:
DEFAULT_COMPLETEKEY = 'tab'
+cqldocs = None
+cqlruleset = None
+
epilog = Connects to %(DEFAULT_HOST)s:%(DEFAULT_PORT)d by default. These
defaults can be changed by setting $CQLSH_HOST and/or $CQLSH_PORT. When a
host (and optional port number) are given on the command line, they take
@@ -226,13 +229,17 @@ my_commands_ending_with_newline = (
'quit'
)
+
cqlsh_syntax_completers = []
+
+
def cqlsh_syntax_completer(rulename, termname):
def registrator(f):
cqlsh_syntax_completers.append((rulename, termname, f))
return f
return registrator
+
cqlsh_extra_syntax_rules = r'''
cqlshCommand ::= CQL_Statement
| specialCommand ( ; | \n )
@@ -324,13 +331,15 @@ cqlsh_extra_syntax_rules = r'''
qmark ::= ? ;
'''
+
@cqlsh_syntax_completer('helpCommand', 'topic')
def complete_help(ctxt, cqlsh):
-return sorted([ t.upper() for t in cqldocs.get_help_topics() +
cqlsh.get_help_topics() ])
+return sorted([t.upper() for t in cqldocs.get_help_topics() +
cqlsh.get_help_topics()])
+
def complete_source_quoted_filename(ctxt, cqlsh):
-partial = ctxt.get_binding('partial', '')
-head, tail = os.path.split(partial)
+partial_path = ctxt.get_binding('partial', '')
+head, tail = os.path.split(partial_path)
exhead = os.path.expanduser(head)
try:
contents = os.listdir(exhead or '.')
@@ -345,21 +354,22 @@ def complete_source_quoted_filename(ctxt, cqlsh):
annotated.append(match)
return annotated
-cqlsh_syntax_completer('sourceCommand', 'fname') \
-(complete_source_quoted_filename)
-cqlsh_syntax_completer('captureCommand', 'fname') \
-(complete_source_quoted_filename)
+
+cqlsh_syntax_completer('sourceCommand',
'fname')(complete_source_quoted_filename)
+cqlsh_syntax_completer('captureCommand',
'fname')(complete_source_quoted_filename)
+
@cqlsh_syntax_completer('copyCommand', 'fname')
def copy_fname_completer(ctxt, cqlsh):
lasttype = ctxt.get_binding('*LASTTYPE*')
if lasttype == 'unclosedString':
return complete_source_quoted_filename(ctxt, cqlsh)
-partial = ctxt.get_binding('partial')
-if partial == '':
+partial_path = ctxt.get_binding('partial')
+if partial_path == '':
return [']
return ()
+
@cqlsh_syntax_completer('copyCommand', 'colnames')
def complete_copy_column_names(ctxt, cqlsh):
existcols = map(cqlsh.cql_unprotect_name, ctxt.get_binding('colnames', ()))
@@ -370,8 +380,10 @@ def complete_copy_column_names(ctxt, cqlsh):
return [colnames[0]]
return set(colnames[1:]) - set(existcols)
+
COPY_OPTIONS = ('DELIMITER', 'QUOTE', 'ESCAPE', 'HEADER', 'ENCODING', 'NULL')
+
@cqlsh_syntax_completer('copyOption', 'optnames')
def
[jira] [Updated] (CASSANDRA-8798) don't throw TombstoneOverwhelmingException during bootstrap
[
https://issues.apache.org/jira/browse/CASSANDRA-8798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-8798:
Reviewer: Ariel Weisberg
don't throw TombstoneOverwhelmingException during bootstrap
---
Key: CASSANDRA-8798
URL: https://issues.apache.org/jira/browse/CASSANDRA-8798
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: mck
Assignee: Jeff Jirsa
Fix For: 2.0.15
Attachments: 8798.txt
During bootstrap honouring tombstone_failure_threshold seems
counter-productive as the node is not serving requests so not protecting
anything.
Instead what happens is bootstrap fails, and a cluster that obviously needs
an extra node isn't getting it...
**History**
When adding a new node bootstrap process looks complete in that streaming is
finished, compactions finished, and all disk and cpu activity is calm.
But the node is still stuck in joining status.
The last stage in the bootstrapping process is the rebuilding of secondary
indexes. grepping the logs confirmed it failed during this stage.
{code}grep SecondaryIndexManager cassandra/logs/*{code}
To see what secondary index rebuilding was initiated
{code}
grep index build of cassandra/logs/* | awk -F for data in '{print $1}'
INFO 13:18:11,252 Submitting index build of addresses.unobfuscatedIndex
INFO 13:18:11,352 Submitting index build of Inbox.FINNBOXID_INDEX
INFO 23:03:54,758 Submitting index build of [events.collected_tbIndex,
events.real_tbIndex]
{code}
To get an idea of successful secondary index rebuilding
{code}grep Index build of cassandra/logs/*
INFO 13:18:11,263 Index build of addresses.unobfuscatedIndex complete
INFO 13:18:11,355 Index build of Inbox.FINNBOXID_INDEX complete
{code}
Looking closer at {{[events.collected_tbIndex, events.real_tbIndex]}} showed
the following stacktrace
{code}
ERROR [StreamReceiveTask:121] 2015-02-12 05:54:47,768 CassandraDaemon.java
(line 199) Exception in thread Thread[StreamReceiveTask:121,5,main]
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.RuntimeException:
org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413)
at
org.apache.cassandra.db.index.SecondaryIndexManager.maybeBuildSecondaryIndexes(SecondaryIndexManager.java:142)
at
org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:130)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException:
java.lang.RuntimeException:
org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409)
... 7 more
Caused by: java.lang.RuntimeException:
org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at
org.apache.cassandra.service.pager.QueryPagers$1.next(QueryPagers.java:160)
at
org.apache.cassandra.service.pager.QueryPagers$1.next(QueryPagers.java:143)
at org.apache.cassandra.db.Keyspace.indexRow(Keyspace.java:406)
at
org.apache.cassandra.db.index.SecondaryIndexBuilder.build(SecondaryIndexBuilder.java:62)
at
org.apache.cassandra.db.compaction.CompactionManager$9.run(CompactionManager.java:834)
... 5 more
Caused by: org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at
org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:202)
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1547)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1376)
[2/2] cassandra git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk
Conflicts:
bin/cqlsh
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/840d9a66
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/840d9a66
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/840d9a66
Branch: refs/heads/trunk
Commit: 840d9a663295fa0bfb78fb9706aa844e0b30
Parents: b379e32 49c1dbd
Author: Tyler Hobbs tylerlho...@gmail.com
Authored: Tue Apr 21 12:24:47 2015 -0500
Committer: Tyler Hobbs tylerlho...@gmail.com
Committed: Tue Apr 21 12:24:47 2015 -0500
--
bin/cqlsh | 88 +++---
1 file changed, 60 insertions(+), 28 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/840d9a66/bin/cqlsh
--
diff --cc bin/cqlsh
index 2078107,05f3368..9dffa7f
--- a/bin/cqlsh
+++ b/bin/cqlsh
@@@ -111,8 -112,7 +112,7 @@@ from cassandra.cluster import Cluster,
from cassandra.query import SimpleStatement, ordered_dict_factory
from cassandra.policies import WhiteListRoundRobinPolicy
from cassandra.protocol import QueryMessage, ResultMessage
- from cassandra.marshal import int16_pack, int32_pack, uint64_pack
-from cassandra.metadata import protect_name, protect_names, protect_value
+from cassandra.metadata import protect_name, protect_names, protect_value,
KeyspaceMetadata, TableMetadata, ColumnMetadata
from cassandra.auth import PlainTextAuthProvider
# cqlsh should run correctly when run out of a Cassandra source tree,
@@@ -436,7 -456,8 +457,8 @@@ def full_cql_version(ver)
vertuple = tuple(map(int, ver_parts[0].split('.')) + [ver_parts[1]])
return ver, vertuple
+
-def format_value(val, output_encoding, addcolor=False, time_format=None,
+def format_value(val, output_encoding, addcolor=False, date_time_format=None,
float_precision=None, colormap=None, nullval=None):
if isinstance(val, DecodeError):
if addcolor:
@@@ -444,9 -465,10 +466,10 @@@
else:
return FormattedValue(repr(val.thebytes))
return format_by_type(type(val), val, output_encoding, colormap=colormap,
- addcolor=addcolor, nullval=nullval,
time_format=time_format,
+ addcolor=addcolor, nullval=nullval,
date_time_format=date_time_format,
float_precision=float_precision)
+
def show_warning_without_quoting_line(message, category, filename, lineno,
file=None, line=None):
if file is None:
file = sys.stderr
@@@ -769,32 -782,10 +795,32 @@@ class Shell(cmd.Cmd)
ksmeta = self.get_keyspace_meta(ksname)
if tablename not in ksmeta.tables:
- if ksname == 'system_auth' and tablename in
['roles','role_permissions']:
++if ksname == 'system_auth' and tablename in ['roles',
'role_permissions']:
+self.get_fake_auth_table_meta(ksname, tablename)
+else:
+raise ColumnFamilyNotFound(Column family %r not found %
tablename)
+else:
+return ksmeta.tables[tablename]
+
+def get_fake_auth_table_meta(self, ksname, tablename):
+# may be using external auth implementation so internal tables
+# aren't actually defined in schema. In this case, we'll fake
+# them up
+if tablename == 'roles':
+ks_meta = KeyspaceMetadata(ksname, True, None, None)
+table_meta = TableMetadata(ks_meta, 'roles')
+table_meta.columns['role'] = ColumnMetadata(table_meta, 'role',
cassandra.cqltypes.UTF8Type)
+table_meta.columns['is_superuser'] = ColumnMetadata(table_meta,
'is_superuser', cassandra.cqltypes.BooleanType)
+table_meta.columns['can_login'] = ColumnMetadata(table_meta,
'can_login', cassandra.cqltypes.BooleanType)
+elif tablename == 'role_permissions':
+ks_meta = KeyspaceMetadata(ksname, True, None, None)
+table_meta = TableMetadata(ks_meta, 'role_permissions')
+table_meta.columns['role'] = ColumnMetadata(table_meta, 'role',
cassandra.cqltypes.UTF8Type)
+table_meta.columns['resource'] = ColumnMetadata(table_meta,
'resource', cassandra.cqltypes.UTF8Type)
+table_meta.columns['permission'] = ColumnMetadata(table_meta,
'permission', cassandra.cqltypes.UTF8Type)
+else:
- raise ColumnFamilyNotFoundException(Column family %r not found
% tablename)
+ raise ColumnFamilyNotFound(Column family %r not found %
tablename)
-return ksmeta.tables[tablename]
-
def get_usertypes_meta(self):
data = self.session.execute(select * from
[jira] [Created] (CASSANDRA-9222) AssertionError after decommission
Brandon Williams created CASSANDRA-9222:
---
Summary: AssertionError after decommission
Key: CASSANDRA-9222
URL: https://issues.apache.org/jira/browse/CASSANDRA-9222
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Brandon Williams
Priority: Minor
Fix For: 3.0
Saw this on trunk while working on CASSANDRA-8072, but it may affect earlier
revisions as well:
{noformat}
INFO 17:48:57 MessagingService has terminated the accept() thread
INFO 17:48:58 DECOMMISSIONED
ERROR 17:52:25 Exception in thread Thread[OptionalTasks:1,5,main]
java.lang.AssertionError: -1011553757645129692 not found in
-9212067178699207814, -9200531256183869940, -9166030381776079682,
-9162013024688602642, -9151724494713671168, -9095828490921521759,
-9035494031488373110, -8993765846966048219, -8912013107131353260,
-8909000788978186800, -8879514397454962673, -8868628980500567099,
-8850730903031889070, -8810378752213886595, -8779200870214886308,
-8758215747589442842, -8751091270073031687, -8727034084505556969,
-8665197275159395069, -8656563059526305598, -8468078121019364990,
-8465001791134178844, -8442193507205463429, -8422069069190372219,
-8342133517826612505, -8341643847610190520, -8340770353573450569,
-8337671516798157281, -8299063757464280571, -8294397037816683529,
-8190643358275415766, -8125907580996325958, -8080821167493102683,
-8058428707430264364, -8033777866368709204, -8018079744052327023,
-8005568943124488030, -7911488756902729132, -7831006227012170930,
-7824529182957931950, -7807286997402075771, -7795080548612350344,
-7778629955912441437, -7771701686959718810, -7759250335393772671,
-7745731940317799541, -7703194536911509010, -7694764467260740698,
-7691909270364954632, -7687121918922986909, -7682707339911246942,
-7517133373189921954, -7482800574078120526, -7475897243891441451,
-7334307376946940271, -7326649207653179327, -7258677281263041990,
-7221843646683358238, -7193299656451825680, -7105256682000196035,
-7035269781687029457, -7024278722443497027, -7019197046707993025,
-7015131617238216508, -7003811999522811317, -6980314778696530567,
-6966235125715836473, -691530498397662, -6912703644363131398,
-6881456879008059927, -6861265076865721267, -6850740895102395611,
-6808435504617684311, -6785202117470372844, -6782573711981746574,
-6763604807975420855, -6738443719321921481, -6718513123799422576,
-6711670508127917469, -6709012720615571304, -6645945635050443947,
-6629420613692951932, -6542209628003661283, -6535684002637060628,
-6507671461487774245, -6423206762015678338, -6409843839148310789,
-6404011469157179029, -6381904465334594648, -6311911206861962333,
-6296991709696294898, -6264931794517958640, -6261574198670386500,
-6261382604358802512, -6252257370391459113, -6241897861580419597,
-6227245701986117982, -6199525755295090433, -6180934919369759659,
-6144605078172691818, -6126223305042342065, -6118447361839427651,
-6074679422903704861, -6053157348245110185, -6029489996808528900,
-5984211855143878285, -5976157876053718897, -5960786495011670628,
-5958735514226770035, -5899767639655442330, -5822684184303415148,
-5781417439294763637, -5751460432371890910, -5740166642636309327,
-5695626417612186310, -5640765045723408247, -5617181156049689169,
-5609533985177356591, -5601369236916580549, -5597950494887081576,
-5563417985168606424, -5544827346340456629, -5532661047516804641,
-5522839053491352218, -5515748028172318343, -5503681859719385351,
-5454037971834611841, -5391841126413524561, -5391486446881271229,
-5345799278441821500, -5334673760925625816, -5223383618739305156,
-5221923994481449381, -5201263557535069480, -5146266397250565218,
-5129908985877585855, -5105202808286786842, -5087879514740126453,
-5015647678958926683, -4956601765875516828, -4870012706573251068,
-4843165740363419346, -4785540557423875550, -4769272272470020667,
-4743838345902355963, -4652149714081482841, -4651813505681686208,
-4633498525751156636, -4617489888285113964, -4575171285024168183,
-4426852178336308913, -4426400792698710435, -4389286320937036309,
-4324528033603203034, -4310368852323145495, -4302216608677327172,
-4229528661709148440, -4207740831738287983, -4203528661247313570,
-3948641241721335982, -3946554569612854645, -3931865850800685387,
-3925635355333550077, -3834502440481769685, -3827908348147378297,
-3805680095754927988, -3804947918584815385, -3800995210938487618,
-3783564223836955070, -3775028120786497996, -3711629770355538643,
-3710182799291812403, -3643158926306968005, -3625334149683154824,
-3601333132746233576, -3525454189106118076, -3488846023499783916,
-3422858644002425188, -3404812947955923675, -3375380683958613573,
-3345579637207839123, -3311077311616228299, -3224730245077365292,
-3210580185929025170, -3154636995770046004, -3147060265746012341,
-3137703098005731598, -3018617509092964154, -2980990616707099169,
[jira] [Updated] (CASSANDRA-7410) Pig support for BulkOutputFormat as a parameter in url
[ https://issues.apache.org/jira/browse/CASSANDRA-7410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Liu updated CASSANDRA-7410: Attachment: CASSANDRA-7410-v5-2.0-branch.txt CASSANDRA-7410-v4-2.1-branch.txt Patches addressing comments are attached. Pig support for BulkOutputFormat as a parameter in url -- Key: CASSANDRA-7410 URL: https://issues.apache.org/jira/browse/CASSANDRA-7410 Project: Cassandra Issue Type: Improvement Components: Hadoop Reporter: Alex Liu Assignee: Alex Liu Priority: Minor Fix For: 2.0.15 Attachments: 7410-2.0-branch.txt, 7410-2.1-branch.txt, 7410-v2-2.0-branch.txt, 7410-v3-2.0-branch.txt, CASSANDRA-7410-v2-2.1-branch.txt, CASSANDRA-7410-v3-2.1-branch.txt, CASSANDRA-7410-v4-2.0-branch.txt, CASSANDRA-7410-v4-2.1-branch.txt, CASSANDRA-7410-v5-2.0-branch.txt Add BulkOutputFormat support in Pig url -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9223) ArithmeticException after decommission
Brandon Williams created CASSANDRA-9223:
---
Summary: ArithmeticException after decommission
Key: CASSANDRA-9223
URL: https://issues.apache.org/jira/browse/CASSANDRA-9223
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Brandon Williams
Priority: Minor
Fix For: 3.0
Also seen on trunk while working on CASSANDRA-8072:
{noformat}
ERROR 19:21:33 Exception in thread Thread[BatchlogTasks:1,5,main]
java.lang.ArithmeticException: / by zero
at
org.apache.cassandra.db.BatchlogManager.replayAllFailedBatches(BatchlogManager.java:173)
~[main/:na]
at org.apache.cassandra.db.BatchlogManager.access$000(BatchlogManager.java:61)
~[main/:na]
at
org.apache.cassandra.db.BatchlogManager$1.runMayThrow(BatchlogManager.java:91)
~[main/:na]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[main/:na]
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:82)
~[main/:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[na:1.7.0_76]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
[na:1.7.0_76]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[na:1.7.0_76]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.7.0_76]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_76]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_76]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_76]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8252) dtests that involve topology changes should verify system.peers on all nodes
[ https://issues.apache.org/jira/browse/CASSANDRA-8252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505497#comment-14505497 ] Shawn Kumar commented on CASSANDRA-8252: Seems to be a number of failures when I tried incorporating into existing code. To try and clarify/separate from other tests, I wrote up a separate [dtest|https://github.com/riptano/cassandra-dtest/blob/peerstest/peers_test.py] to test some basic actions (bootstrapping, removing nodes) and am still running into failures on all of these. dtests that involve topology changes should verify system.peers on all nodes Key: CASSANDRA-8252 URL: https://issues.apache.org/jira/browse/CASSANDRA-8252 Project: Cassandra Issue Type: Test Components: Tests Reporter: Brandon Williams Assignee: Shawn Kumar Fix For: 2.0.15, 2.1.5 This is especially true for replace where I've discovered it's wrong in 1.2.19, which is sad because now it's too late to fix. We've had a lot of problems with incorrect/null system.peers, so after any topology change we should verify it on every live node when everything is finished. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7212) Allow to switch user within CQLSH session
[
https://issues.apache.org/jira/browse/CASSANDRA-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carl Yeksigian updated CASSANDRA-7212:
--
Attachment: 7212-v3.txt
I've attached a patch that removes the requirement for the server-side grammar
change.
As I did this, I think it makes more sense to use a different command
altogether for this, so that any changes to {{USE}} won't cause problems with
our overload. {{LOGIN}} makes sense for this use case.
Allow to switch user within CQLSH session
-
Key: CASSANDRA-7212
URL: https://issues.apache.org/jira/browse/CASSANDRA-7212
Project: Cassandra
Issue Type: Improvement
Components: API
Environment: [cqlsh 4.1.1 | Cassandra 2.0.7.31 | CQL spec 3.1.1 |
Thrift protocol 19.39.0]
Reporter: Jose Martinez Poblete
Labels: cqlsh
Attachments: 7212-v3.txt, 7212_1.patch, 7212_v2.patch
Once a user is logged into CQLSH, it is not possible to switch to another
user without exiting and relaunch
This is a feature offered in postgres and probably other databases:
http://secure.encivasolutions.com/knowledgebase.php?action=displayarticleid=1126
Perhaps this could be implemented on CQLSH as part of the USE directive:
USE Keyspace [USER] [password]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7910) wildcard prepared statements are incorrect after a column is added to the table
[ https://issues.apache.org/jira/browse/CASSANDRA-7910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505485#comment-14505485 ] Itay Adler commented on CASSANDRA-7910: --- Is this also fixed for 2.0.14? wildcard prepared statements are incorrect after a column is added to the table --- Key: CASSANDRA-7910 URL: https://issues.apache.org/jira/browse/CASSANDRA-7910 Project: Cassandra Issue Type: Bug Components: Core Reporter: Oded Peer Assignee: Tyler Hobbs Priority: Minor Labels: client-impacting Fix For: 2.1.3 Attachments: 7910-2.1.txt, 7910-trunk.txt, PreparedStatementAfterAddColumnTest.java 1. Prepare a statement with a wildcard in the select clause. 2. Alter the table - add a column 3. execute the prepared statement Expected result - get all the columns including the new column Actual result - get the columns except the new column Attached a test using cassandra-unit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8798) don't throw TombstoneOverwhelmingException during bootstrap
[
https://issues.apache.org/jira/browse/CASSANDRA-8798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505550#comment-14505550
]
Jeff Jirsa commented on CASSANDRA-8798:
---
[~aweisberg] - Yes, respectTombstoneThresholds() would also effectively disable
the threshold during other streaming cases (such as rebuild and repair), even
outside of bootstrapping. The thought process was that if you added another DC
to the cluster, best practices (such as
http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html
) will cause you to bootstrap and then, once all new nodes in the new DC were
online, you'd issue a rebuild. That rebuild could fail due to
TombstoneOverwhelmingException in the same way (when building indexes for data
streamed over).
It is a scary patch, I agree, which is why I wanted someone else to agree with
the approach in theory before I wrote a full test. I'm open to other
suggestions, though.
don't throw TombstoneOverwhelmingException during bootstrap
---
Key: CASSANDRA-8798
URL: https://issues.apache.org/jira/browse/CASSANDRA-8798
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: mck
Assignee: Jeff Jirsa
Fix For: 2.0.15
Attachments: 8798.txt
During bootstrap honouring tombstone_failure_threshold seems
counter-productive as the node is not serving requests so not protecting
anything.
Instead what happens is bootstrap fails, and a cluster that obviously needs
an extra node isn't getting it...
**History**
When adding a new node bootstrap process looks complete in that streaming is
finished, compactions finished, and all disk and cpu activity is calm.
But the node is still stuck in joining status.
The last stage in the bootstrapping process is the rebuilding of secondary
indexes. grepping the logs confirmed it failed during this stage.
{code}grep SecondaryIndexManager cassandra/logs/*{code}
To see what secondary index rebuilding was initiated
{code}
grep index build of cassandra/logs/* | awk -F for data in '{print $1}'
INFO 13:18:11,252 Submitting index build of addresses.unobfuscatedIndex
INFO 13:18:11,352 Submitting index build of Inbox.FINNBOXID_INDEX
INFO 23:03:54,758 Submitting index build of [events.collected_tbIndex,
events.real_tbIndex]
{code}
To get an idea of successful secondary index rebuilding
{code}grep Index build of cassandra/logs/*
INFO 13:18:11,263 Index build of addresses.unobfuscatedIndex complete
INFO 13:18:11,355 Index build of Inbox.FINNBOXID_INDEX complete
{code}
Looking closer at {{[events.collected_tbIndex, events.real_tbIndex]}} showed
the following stacktrace
{code}
ERROR [StreamReceiveTask:121] 2015-02-12 05:54:47,768 CassandraDaemon.java
(line 199) Exception in thread Thread[StreamReceiveTask:121,5,main]
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.RuntimeException:
org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413)
at
org.apache.cassandra.db.index.SecondaryIndexManager.maybeBuildSecondaryIndexes(SecondaryIndexManager.java:142)
at
org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:130)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException:
java.lang.RuntimeException:
org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409)
... 7 more
Caused by: java.lang.RuntimeException:
org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at
org.apache.cassandra.service.pager.QueryPagers$1.next(QueryPagers.java:160)
at
org.apache.cassandra.service.pager.QueryPagers$1.next(QueryPagers.java:143)
at org.apache.cassandra.db.Keyspace.indexRow(Keyspace.java:406)
at
org.apache.cassandra.db.index.SecondaryIndexBuilder.build(SecondaryIndexBuilder.java:62)
at
org.apache.cassandra.db.compaction.CompactionManager$9.run(CompactionManager.java:834)
... 5 more
Caused by: org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at
cassandra git commit: Fix whitespace, unused imports and variables in cqlsh
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1 aeadda5ba - 49c1dbdad
Fix whitespace, unused imports and variables in cqlsh
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/49c1dbda
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/49c1dbda
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/49c1dbda
Branch: refs/heads/cassandra-2.1
Commit: 49c1dbdad4bd36081ab79aee378c6a4ed71cb3cc
Parents: aeadda5
Author: Tyler Hobbs tylerlho...@gmail.com
Authored: Tue Apr 21 12:20:37 2015 -0500
Committer: Tyler Hobbs tylerlho...@gmail.com
Committed: Tue Apr 21 12:20:37 2015 -0500
--
bin/cqlsh | 84 --
1 file changed, 58 insertions(+), 26 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/49c1dbda/bin/cqlsh
--
diff --git a/bin/cqlsh b/bin/cqlsh
index e1bd312..05f3368 100755
--- a/bin/cqlsh
+++ b/bin/cqlsh
@@ -36,7 +36,7 @@ description = CQL Shell for Apache Cassandra
version = 5.0.1
from StringIO import StringIO
-from contextlib import contextmanager, closing
+from contextlib import contextmanager
from glob import glob
import cmd
@@ -79,6 +79,7 @@ if myplatform == 'Linux':
if os.environ.get('CQLSH_NO_BUNDLED', ''):
ZIPLIB_DIRS = ()
+
def find_zip(libprefix):
for ziplibdir in ZIPLIB_DIRS:
zips = glob(os.path.join(ziplibdir, libprefix + '*.zip'))
@@ -111,7 +112,6 @@ from cassandra.cluster import Cluster, PagedResult
from cassandra.query import SimpleStatement, ordered_dict_factory
from cassandra.policies import WhiteListRoundRobinPolicy
from cassandra.protocol import QueryMessage, ResultMessage
-from cassandra.marshal import int16_pack, int32_pack, uint64_pack
from cassandra.metadata import protect_name, protect_names, protect_value
from cassandra.auth import PlainTextAuthProvider
@@ -142,6 +142,9 @@ if readline is not None and readline.__doc__ is not None
and 'libedit' in readli
else:
DEFAULT_COMPLETEKEY = 'tab'
+cqldocs = None
+cqlruleset = None
+
epilog = Connects to %(DEFAULT_HOST)s:%(DEFAULT_PORT)d by default. These
defaults can be changed by setting $CQLSH_HOST and/or $CQLSH_PORT. When a
host (and optional port number) are given on the command line, they take
@@ -226,13 +229,17 @@ my_commands_ending_with_newline = (
'quit'
)
+
cqlsh_syntax_completers = []
+
+
def cqlsh_syntax_completer(rulename, termname):
def registrator(f):
cqlsh_syntax_completers.append((rulename, termname, f))
return f
return registrator
+
cqlsh_extra_syntax_rules = r'''
cqlshCommand ::= CQL_Statement
| specialCommand ( ; | \n )
@@ -324,13 +331,15 @@ cqlsh_extra_syntax_rules = r'''
qmark ::= ? ;
'''
+
@cqlsh_syntax_completer('helpCommand', 'topic')
def complete_help(ctxt, cqlsh):
-return sorted([ t.upper() for t in cqldocs.get_help_topics() +
cqlsh.get_help_topics() ])
+return sorted([t.upper() for t in cqldocs.get_help_topics() +
cqlsh.get_help_topics()])
+
def complete_source_quoted_filename(ctxt, cqlsh):
-partial = ctxt.get_binding('partial', '')
-head, tail = os.path.split(partial)
+partial_path = ctxt.get_binding('partial', '')
+head, tail = os.path.split(partial_path)
exhead = os.path.expanduser(head)
try:
contents = os.listdir(exhead or '.')
@@ -345,21 +354,22 @@ def complete_source_quoted_filename(ctxt, cqlsh):
annotated.append(match)
return annotated
-cqlsh_syntax_completer('sourceCommand', 'fname') \
-(complete_source_quoted_filename)
-cqlsh_syntax_completer('captureCommand', 'fname') \
-(complete_source_quoted_filename)
+
+cqlsh_syntax_completer('sourceCommand',
'fname')(complete_source_quoted_filename)
+cqlsh_syntax_completer('captureCommand',
'fname')(complete_source_quoted_filename)
+
@cqlsh_syntax_completer('copyCommand', 'fname')
def copy_fname_completer(ctxt, cqlsh):
lasttype = ctxt.get_binding('*LASTTYPE*')
if lasttype == 'unclosedString':
return complete_source_quoted_filename(ctxt, cqlsh)
-partial = ctxt.get_binding('partial')
-if partial == '':
+partial_path = ctxt.get_binding('partial')
+if partial_path == '':
return [']
return ()
+
@cqlsh_syntax_completer('copyCommand', 'colnames')
def complete_copy_column_names(ctxt, cqlsh):
existcols = map(cqlsh.cql_unprotect_name, ctxt.get_binding('colnames', ()))
@@ -370,8 +380,10 @@ def complete_copy_column_names(ctxt, cqlsh):
return [colnames[0]]
return set(colnames[1:]) - set(existcols)
+
COPY_OPTIONS = ('DELIMITER', 'QUOTE', 'ESCAPE', 'HEADER', 'ENCODING', 'NULL')
+
@cqlsh_syntax_completer('copyOption',
[jira] [Commented] (CASSANDRA-8798) don't throw TombstoneOverwhelmingException during bootstrap
[
https://issues.apache.org/jira/browse/CASSANDRA-8798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505529#comment-14505529
]
Ariel Weisberg commented on CASSANDRA-8798:
---
[~jjirsa] Too scary without a test that shows that it transitions back to
respecting tombstone thresholds.
Can you make a dtest that reproduces the problem without your fix? You can set
a low threshold to make it easy.
Looking at the predicate for respectTombstoneThresholds() I am a little
concerned that it might be true after bootstrapping since it is all ||. It's
also a lot of predicates when maybe one should suffice since bootstrapping
nodes aren't supposed to serve reads at all anyways and that is the only thing
effected by the tombstone thresholds. Each extra predicate feels like an
opportunity to make a mistake transitioning back to honoring the threshold at
some point in the future.
don't throw TombstoneOverwhelmingException during bootstrap
---
Key: CASSANDRA-8798
URL: https://issues.apache.org/jira/browse/CASSANDRA-8798
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: mck
Assignee: Jeff Jirsa
Fix For: 2.0.15
Attachments: 8798.txt
During bootstrap honouring tombstone_failure_threshold seems
counter-productive as the node is not serving requests so not protecting
anything.
Instead what happens is bootstrap fails, and a cluster that obviously needs
an extra node isn't getting it...
**History**
When adding a new node bootstrap process looks complete in that streaming is
finished, compactions finished, and all disk and cpu activity is calm.
But the node is still stuck in joining status.
The last stage in the bootstrapping process is the rebuilding of secondary
indexes. grepping the logs confirmed it failed during this stage.
{code}grep SecondaryIndexManager cassandra/logs/*{code}
To see what secondary index rebuilding was initiated
{code}
grep index build of cassandra/logs/* | awk -F for data in '{print $1}'
INFO 13:18:11,252 Submitting index build of addresses.unobfuscatedIndex
INFO 13:18:11,352 Submitting index build of Inbox.FINNBOXID_INDEX
INFO 23:03:54,758 Submitting index build of [events.collected_tbIndex,
events.real_tbIndex]
{code}
To get an idea of successful secondary index rebuilding
{code}grep Index build of cassandra/logs/*
INFO 13:18:11,263 Index build of addresses.unobfuscatedIndex complete
INFO 13:18:11,355 Index build of Inbox.FINNBOXID_INDEX complete
{code}
Looking closer at {{[events.collected_tbIndex, events.real_tbIndex]}} showed
the following stacktrace
{code}
ERROR [StreamReceiveTask:121] 2015-02-12 05:54:47,768 CassandraDaemon.java
(line 199) Exception in thread Thread[StreamReceiveTask:121,5,main]
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.RuntimeException:
org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413)
at
org.apache.cassandra.db.index.SecondaryIndexManager.maybeBuildSecondaryIndexes(SecondaryIndexManager.java:142)
at
org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:130)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException:
java.lang.RuntimeException:
org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409)
... 7 more
Caused by: java.lang.RuntimeException:
org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at
org.apache.cassandra.service.pager.QueryPagers$1.next(QueryPagers.java:160)
at
org.apache.cassandra.service.pager.QueryPagers$1.next(QueryPagers.java:143)
at org.apache.cassandra.db.Keyspace.indexRow(Keyspace.java:406)
at
org.apache.cassandra.db.index.SecondaryIndexBuilder.build(SecondaryIndexBuilder.java:62)
at
org.apache.cassandra.db.compaction.CompactionManager$9.run(CompactionManager.java:834)
... 5 more
Caused by: org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at
[jira] [Created] (CASSANDRA-9221) No test coverage for TombstoneOverwhelmingException
Jeff Jirsa created CASSANDRA-9221: - Summary: No test coverage for TombstoneOverwhelmingException Key: CASSANDRA-9221 URL: https://issues.apache.org/jira/browse/CASSANDRA-9221 Project: Cassandra Issue Type: Test Reporter: Jeff Jirsa Priority: Minor While investigating CASSANDRA-8798 discovered there were no unit tests for TombstoneOverwhelmingException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9198) Deleting from an empty list produces an error
[
https://issues.apache.org/jira/browse/CASSANDRA-9198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Lerer updated CASSANDRA-9198:
--
Reviewer: Benjamin Lerer
Fix Version/s: (was: 3.0)
2.1.5
Assignee: Jeff Jirsa (was: Benjamin Lerer)
Deleting from an empty list produces an error
-
Key: CASSANDRA-9198
URL: https://issues.apache.org/jira/browse/CASSANDRA-9198
Project: Cassandra
Issue Type: Bug
Components: API
Reporter: Olivier Michallat
Assignee: Jeff Jirsa
Priority: Minor
Fix For: 2.1.5
Attachments: 9198.txt
While deleting an element from a list that does not contain it is a no-op,
deleting it from an empty list causes an error.
This edge case is a bit inconsistent, because it makes list deletion non
idempotent:
{code}
cqlsh:test create table foo (k int primary key, v listint);
cqlsh:test insert into foo(k,v) values (1, [1,2]);
cqlsh:test update foo set v = v - [1] where k = 1;
cqlsh:test update foo set v = v - [1] where k = 1;
cqlsh:test update foo set v = v - [2] where k = 1;
cqlsh:test update foo set v = v - [2] where k = 1;
InvalidRequest: code=2200 [Invalid query] message=Attempted to delete an
element from a list which is null
{code}
With speculative retries coming to the drivers, idempotency becomes more
important because it determines which query we might retry or not. So it
would be better if deleting from an empty list succeeded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9197) Startup slowdown due to preloading jemalloc
[
https://issues.apache.org/jira/browse/CASSANDRA-9197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504542#comment-14504542
]
Sylvain Lebresne commented on CASSANDRA-9197:
-
bq. hardcoding the path as you stated does improve the startup time
Out of curiosity, does it make it like without jemalloc? In other words, is it
the {{find_library}} method that improve the startup time, or is it both that
and loading jemalloc?
Startup slowdown due to preloading jemalloc
---
Key: CASSANDRA-9197
URL: https://issues.apache.org/jira/browse/CASSANDRA-9197
Project: Cassandra
Issue Type: Bug
Reporter: Sylvain Lebresne
Assignee: Philip Thompson
Priority: Minor
On my box, it seems that the jemalloc loading from CASSANDRA-8714 made the
process take ~10 seconds to even start (I have no explication for it). I
don't know if it's specific to my machine or not, so that ticket is mainly so
someone else can check if it sees the same, in particular for jenkins. If it
does sees the same slowness, we might want to at least disable jemalloc for
dtests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9198) Deleting from an empty list produces an error
[
https://issues.apache.org/jira/browse/CASSANDRA-9198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504581#comment-14504581
]
Benjamin Lerer commented on CASSANDRA-9198:
---
Could you also make sure that your patch apply to trunk? CASSANDRA-7304 had
some impact in the same area.
Deleting from an empty list produces an error
-
Key: CASSANDRA-9198
URL: https://issues.apache.org/jira/browse/CASSANDRA-9198
Project: Cassandra
Issue Type: Bug
Components: API
Reporter: Olivier Michallat
Assignee: Jeff Jirsa
Priority: Minor
Fix For: 2.1.5
Attachments: 9198.txt
While deleting an element from a list that does not contain it is a no-op,
deleting it from an empty list causes an error.
This edge case is a bit inconsistent, because it makes list deletion non
idempotent:
{code}
cqlsh:test create table foo (k int primary key, v listint);
cqlsh:test insert into foo(k,v) values (1, [1,2]);
cqlsh:test update foo set v = v - [1] where k = 1;
cqlsh:test update foo set v = v - [1] where k = 1;
cqlsh:test update foo set v = v - [2] where k = 1;
cqlsh:test update foo set v = v - [2] where k = 1;
InvalidRequest: code=2200 [Invalid query] message=Attempted to delete an
element from a list which is null
{code}
With speculative retries coming to the drivers, idempotency becomes more
important because it determines which query we might retry or not. So it
would be better if deleting from an empty list succeeded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9197) Startup slowdown due to preloading jemalloc
[ https://issues.apache.org/jira/browse/CASSANDRA-9197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504676#comment-14504676 ] Sylvain Lebresne commented on CASSANDRA-9197: - Yes, the attached patch does seem to solve the problem. Haven't reviewed the patch more than that however. Startup slowdown due to preloading jemalloc --- Key: CASSANDRA-9197 URL: https://issues.apache.org/jira/browse/CASSANDRA-9197 Project: Cassandra Issue Type: Bug Reporter: Sylvain Lebresne Assignee: Philip Thompson Priority: Minor Attachments: 9197.txt On my box, it seems that the jemalloc loading from CASSANDRA-8714 made the process take ~10 seconds to even start (I have no explication for it). I don't know if it's specific to my machine or not, so that ticket is mainly so someone else can check if it sees the same, in particular for jenkins. If it does sees the same slowness, we might want to at least disable jemalloc for dtests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7168) Add repair aware consistency levels
[ https://issues.apache.org/jira/browse/CASSANDRA-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504910#comment-14504910 ] Jonathan Ellis commented on CASSANDRA-7168: --- I'm fine with adding this as opt-in. But in the meantime I do think we should drop digest reads. It's an optimization that really doesn't buy us much, at the cost of a lot of complexity (that I'm still not entirely sure is bug-free) as well as significantly complicating this new optimization (that has much more potential to actually save work). Add repair aware consistency levels --- Key: CASSANDRA-7168 URL: https://issues.apache.org/jira/browse/CASSANDRA-7168 Project: Cassandra Issue Type: Improvement Components: Core Reporter: T Jake Luciani Labels: performance Fix For: 3.1 With CASSANDRA-5351 and CASSANDRA-2424 I think there is an opportunity to avoid a lot of extra disk I/O when running queries with higher consistency levels. Since repaired data is by definition consistent and we know which sstables are repaired, we can optimize the read path by having a REPAIRED_QUORUM which breaks reads into two phases: 1) Read from one replica the result from the repaired sstables. 2) Read from a quorum only the un-repaired data. For the node performing 1) we can pipeline the call so it's a single hop. In the long run (assuming data is repaired regularly) we will end up with much closer to CL.ONE performance while maintaining consistency. Some things to figure out: - If repairs fail on some nodes we can have a situation where we don't have a consistent repaired state across the replicas. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9183) Failure detector should detect and ignore local pauses
[ https://issues.apache.org/jira/browse/CASSANDRA-9183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504921#comment-14504921 ] Erik Forsberg commented on CASSANDRA-9183: -- As CASSANDRA-9218 was closed as a duplicate of this, I would like to add that I'm seeing a behaviour where the node that had a pause never recovers, you need to restart parts of your cluster to make it recover, as the gossip is waiting for an echo reply that never comes back, as network packets were dropped during the pause. Failure detector should detect and ignore local pauses -- Key: CASSANDRA-9183 URL: https://issues.apache.org/jira/browse/CASSANDRA-9183 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Brandon Williams Assignee: Brandon Williams Fix For: 3.0 Attachments: 9183-v2.txt, 9183.txt A local node can be paused for many reasons such as GC, and if the pause is long enough when it recovers it will think all the other nodes are dead until it gossips, causing UAE to be thrown to clients trying to use it as a coordinator. Instead, the FD can track the current time, and if the gap there becomes too large, skip marking the nodes down (reset the FD data perhaps) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7029) Investigate alternative transport protocols for both client and inter-server communications
[ https://issues.apache.org/jira/browse/CASSANDRA-7029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504924#comment-14504924 ] T Jake Luciani commented on CASSANDRA-7029: --- Also Areon https://github.com/real-logic/Aeron Investigate alternative transport protocols for both client and inter-server communications --- Key: CASSANDRA-7029 URL: https://issues.apache.org/jira/browse/CASSANDRA-7029 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Benedict Labels: performance Fix For: 3.0 There are a number of reasons to think we can do better than TCP for our communications: 1) We can actually tolerate sporadic small message losses, so guaranteed delivery isn't essential (although for larger messages it probably is) 2) As shown in \[1\] and \[2\], Linux can behave quite suboptimally with regard to TCP message delivery when the system is under load. Judging from the theoretical description, this is likely to apply even when the system-load is not high, but the number of processes to schedule is high. Cassandra generally has a lot of threads to schedule, so this is quite pertinent for us. UDP performs substantially better here. 3) Even when the system is not under load, UDP has a lower CPU burden, and that burden is constant regardless of the number of connections it processes. 4) On a simple benchmark on my local PC, using non-blocking IO for UDP and busy spinning on IO I can actually push 20-40% more throughput through loopback (where TCP should be optimal, as no latency), even for very small messages. Since we can see networking taking multiple CPUs' worth of time during a stress test, using a busy-spin for ~100micros after last message receipt is almost certainly acceptable, especially as we can (ultimately) process inter-server and client communications on the same thread/socket in this model. 5) We can optimise the threading model heavily: since we generally process very small messages (200 bytes not at all implausible), the thread signalling costs on the processing thread can actually dramatically impede throughput. In general it costs ~10micros to signal (and passing the message to another thread for processing in the current model requires signalling). For 200-byte messages this caps our throughput at 20MB/s. I propose to knock up a highly naive UDP-based connection protocol with super-trivial congestion control over the course of a few days, with the only initial goal being maximum possible performance (not fairness, reliability, or anything else), and trial it in Netty (possibly making some changes to Netty to mitigate thread signalling costs). The reason for knocking up our own here is to get a ceiling on what the absolute limit of potential for this approach is. Assuming this pans out with performance gains in C* proper, we then look to contributing to/forking the udt-java project and see how easy it is to bring performance in line with what we can get with our naive approach (I don't suggest starting here, as the project is using blocking old-IO, and modifying it with latency in mind may be challenging, and we won't know for sure what the best case scenario is). \[1\] http://test-docdb.fnal.gov/0016/001648/002/Potential%20Performance%20Bottleneck%20in%20Linux%20TCP.PDF \[2\] http://cd-docdb.fnal.gov/cgi-bin/RetrieveFile?docid=1968;filename=Performance%20Analysis%20of%20Linux%20Networking%20-%20Packet%20Receiving%20(Official).pdf;version=2 Further related reading: http://public.dhe.ibm.com/software/commerce/doc/mft/cdunix/41/UDTWhitepaper.pdf https://mospace.umsystem.edu/xmlui/bitstream/handle/10355/14482/ChoiUndPerTcp.pdf?sequence=1 https://access.redhat.com/site/documentation/en-US/JBoss_Enterprise_Web_Platform/5/html/Administration_And_Configuration_Guide/jgroups-perf-udpbuffer.html http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.153.3762rep=rep1type=pdf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7029) Investigate alternative transport protocols for both client and inter-server communications
[ https://issues.apache.org/jira/browse/CASSANDRA-7029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504938#comment-14504938 ] Benedict commented on CASSANDRA-7029: - Aeron is very cool, and in many ways the kind of thing I was talking about. My concern with it is that it is (AFAICT) designed for a small number of super-high-throuhgput (and low latency) connections. We need _many_ connections instead, which is a different profile. Put another way: we want the number of resources (memory and cpu) per connection to be as low as possible, whereas Aeron can use as much as it wants to achieve its goals. I haven't had time to read through the implementation to determine if it would efficiently scale out the number of such connections. If it can, it could be exactly what we are looking for. It is also possible we could minimally modify it to meet those criteria. Investigate alternative transport protocols for both client and inter-server communications --- Key: CASSANDRA-7029 URL: https://issues.apache.org/jira/browse/CASSANDRA-7029 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Benedict Labels: performance Fix For: 3.0 There are a number of reasons to think we can do better than TCP for our communications: 1) We can actually tolerate sporadic small message losses, so guaranteed delivery isn't essential (although for larger messages it probably is) 2) As shown in \[1\] and \[2\], Linux can behave quite suboptimally with regard to TCP message delivery when the system is under load. Judging from the theoretical description, this is likely to apply even when the system-load is not high, but the number of processes to schedule is high. Cassandra generally has a lot of threads to schedule, so this is quite pertinent for us. UDP performs substantially better here. 3) Even when the system is not under load, UDP has a lower CPU burden, and that burden is constant regardless of the number of connections it processes. 4) On a simple benchmark on my local PC, using non-blocking IO for UDP and busy spinning on IO I can actually push 20-40% more throughput through loopback (where TCP should be optimal, as no latency), even for very small messages. Since we can see networking taking multiple CPUs' worth of time during a stress test, using a busy-spin for ~100micros after last message receipt is almost certainly acceptable, especially as we can (ultimately) process inter-server and client communications on the same thread/socket in this model. 5) We can optimise the threading model heavily: since we generally process very small messages (200 bytes not at all implausible), the thread signalling costs on the processing thread can actually dramatically impede throughput. In general it costs ~10micros to signal (and passing the message to another thread for processing in the current model requires signalling). For 200-byte messages this caps our throughput at 20MB/s. I propose to knock up a highly naive UDP-based connection protocol with super-trivial congestion control over the course of a few days, with the only initial goal being maximum possible performance (not fairness, reliability, or anything else), and trial it in Netty (possibly making some changes to Netty to mitigate thread signalling costs). The reason for knocking up our own here is to get a ceiling on what the absolute limit of potential for this approach is. Assuming this pans out with performance gains in C* proper, we then look to contributing to/forking the udt-java project and see how easy it is to bring performance in line with what we can get with our naive approach (I don't suggest starting here, as the project is using blocking old-IO, and modifying it with latency in mind may be challenging, and we won't know for sure what the best case scenario is). \[1\] http://test-docdb.fnal.gov/0016/001648/002/Potential%20Performance%20Bottleneck%20in%20Linux%20TCP.PDF \[2\] http://cd-docdb.fnal.gov/cgi-bin/RetrieveFile?docid=1968;filename=Performance%20Analysis%20of%20Linux%20Networking%20-%20Packet%20Receiving%20(Official).pdf;version=2 Further related reading: http://public.dhe.ibm.com/software/commerce/doc/mft/cdunix/41/UDTWhitepaper.pdf https://mospace.umsystem.edu/xmlui/bitstream/handle/10355/14482/ChoiUndPerTcp.pdf?sequence=1 https://access.redhat.com/site/documentation/en-US/JBoss_Enterprise_Web_Platform/5/html/Administration_And_Configuration_Guide/jgroups-perf-udpbuffer.html http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.153.3762rep=rep1type=pdf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-2848) Make the Client API support passing down timeouts
[ https://issues.apache.org/jira/browse/CASSANDRA-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504909#comment-14504909 ] Jonathan Ellis commented on CASSANDRA-2848: --- Getting rid of digests (CASSANDRA-7168) would fix the timeout problem without having to do weird things like we'll halve the timeout when digests are involved in case we need to redo it. :) Make the Client API support passing down timeouts - Key: CASSANDRA-2848 URL: https://issues.apache.org/jira/browse/CASSANDRA-2848 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Chris Goffinet Assignee: Sylvain Lebresne Priority: Minor Fix For: 3.1 Having a max server RPC timeout is good for worst case, but many applications that have middleware in front of Cassandra, might have higher timeout requirements. In a fail fast environment, if my application starting at say the front-end, only has 20ms to process a request, and it must connect to X services down the stack, by the time it hits Cassandra, we might only have 10ms. I propose we provide the ability to specify the timeout on each call we do optionally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-9218) Node thinks other nodes are down after heavy GC
[ https://issues.apache.org/jira/browse/CASSANDRA-9218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis resolved CASSANDRA-9218. --- Resolution: Duplicate Node thinks other nodes are down after heavy GC --- Key: CASSANDRA-9218 URL: https://issues.apache.org/jira/browse/CASSANDRA-9218 Project: Cassandra Issue Type: Bug Reporter: Erik Forsberg I have a few troublesome nodes which often end up doing very long GC pauses. The root cause of this is yet to be found, but it's causing another problem - the affected node(s) mark other nodes as down, and they never recover. Here's how it goes: 1. Node goes into troublesome mode, doing heavy GC with long (10+ seconds) GC pauses. 2. While this happens, node will mark other nodes as down. 3. Once the overload situation resolves, the node still thinks the other nodes are down (they are not). It's also quite common that other nodes think the affected node is down. So we often end up with node A thinking there's some 30 nodes down, then a bunch of other nodes beliving node A is down. This in a cluster with 56 nodes. The only way to get out of the situation is to restart node A, and sometimes a few other nodes. And while node A is in this state, any queries that use node A as coordinator have a high risk of getting errors about not enough replicas being available. I have enabled TRACE level gossip debugging while this happens, and on node A, there will be multiple messages about, has already a pending echo, skipping it - i.e the debug line in Gossiper.java line 882. I have also observed while this was happening that other nodes were trying to establish connections (SYN packets sent) but the trouble node (A) were not picking up the line (no accept()). Not knowing exactly how Gossiper works here but it looks like node A is sending out some gossiper echo messages, but then is too busy to get the replies, and never retries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7168) Add repair aware consistency levels
[ https://issues.apache.org/jira/browse/CASSANDRA-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504915#comment-14504915 ] Aleksey Yeschenko commented on CASSANDRA-7168: -- There has been quite some momentum behind dropping digests for good, and I want that to happen too. That said, we should probably do some heavy benchmarking first at see getting rid of them won't introduce a regression. Do we have any spare resources to do the testing prior to 3.0 release? Add repair aware consistency levels --- Key: CASSANDRA-7168 URL: https://issues.apache.org/jira/browse/CASSANDRA-7168 Project: Cassandra Issue Type: Improvement Components: Core Reporter: T Jake Luciani Labels: performance Fix For: 3.1 With CASSANDRA-5351 and CASSANDRA-2424 I think there is an opportunity to avoid a lot of extra disk I/O when running queries with higher consistency levels. Since repaired data is by definition consistent and we know which sstables are repaired, we can optimize the read path by having a REPAIRED_QUORUM which breaks reads into two phases: 1) Read from one replica the result from the repaired sstables. 2) Read from a quorum only the un-repaired data. For the node performing 1) we can pipeline the call so it's a single hop. In the long run (assuming data is repaired regularly) we will end up with much closer to CL.ONE performance while maintaining consistency. Some things to figure out: - If repairs fail on some nodes we can have a situation where we don't have a consistent repaired state across the replicas. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7029) Investigate alternative transport protocols for both client and inter-server communications
[ https://issues.apache.org/jira/browse/CASSANDRA-7029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504931#comment-14504931 ] Benedict commented on CASSANDRA-7029: - QUIC is exactly the kind of thing I'm talking about, although it may be more complex than we need (and we may not want to wait for a Java implementation to appear for general consumption). bq. Do we even have an idea of how much overhead the transport is for us? If I were to guess I'd say probably around 10% which is in not worth optimizing yet territory for me. I've yet to obtain datapoints on a server, but when benchmarking CASSANDRA-4718 locally I found networking calls to be a really significant portion of the CPU time (30%) for the small in-memory workloads we were testing. It is likely this would be lower on server-grade hardware since some of the work would be offloaded to the NIC, but: we know we need many netty threads to reach peak performance (most likely for interrupt queues, but it seems also because the cost of the kernel calls to manage connection states overload a single thread) - if we can drive the entire networking of the box from a single thread (which should be possible over a UDP protocol) I am optimistic we could see really significant dividends. The problem with that is that the highest yield improvements for benchmarking are likely to come by replacing the client connections, since we generally test small clusters, and refactoring that to support multiple protocols is trickier than for MessagingService. It is probably worth doing, though, so that we can easily explore all of the ideas we have for efficient transport. Investigate alternative transport protocols for both client and inter-server communications --- Key: CASSANDRA-7029 URL: https://issues.apache.org/jira/browse/CASSANDRA-7029 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Benedict Labels: performance Fix For: 3.0 There are a number of reasons to think we can do better than TCP for our communications: 1) We can actually tolerate sporadic small message losses, so guaranteed delivery isn't essential (although for larger messages it probably is) 2) As shown in \[1\] and \[2\], Linux can behave quite suboptimally with regard to TCP message delivery when the system is under load. Judging from the theoretical description, this is likely to apply even when the system-load is not high, but the number of processes to schedule is high. Cassandra generally has a lot of threads to schedule, so this is quite pertinent for us. UDP performs substantially better here. 3) Even when the system is not under load, UDP has a lower CPU burden, and that burden is constant regardless of the number of connections it processes. 4) On a simple benchmark on my local PC, using non-blocking IO for UDP and busy spinning on IO I can actually push 20-40% more throughput through loopback (where TCP should be optimal, as no latency), even for very small messages. Since we can see networking taking multiple CPUs' worth of time during a stress test, using a busy-spin for ~100micros after last message receipt is almost certainly acceptable, especially as we can (ultimately) process inter-server and client communications on the same thread/socket in this model. 5) We can optimise the threading model heavily: since we generally process very small messages (200 bytes not at all implausible), the thread signalling costs on the processing thread can actually dramatically impede throughput. In general it costs ~10micros to signal (and passing the message to another thread for processing in the current model requires signalling). For 200-byte messages this caps our throughput at 20MB/s. I propose to knock up a highly naive UDP-based connection protocol with super-trivial congestion control over the course of a few days, with the only initial goal being maximum possible performance (not fairness, reliability, or anything else), and trial it in Netty (possibly making some changes to Netty to mitigate thread signalling costs). The reason for knocking up our own here is to get a ceiling on what the absolute limit of potential for this approach is. Assuming this pans out with performance gains in C* proper, we then look to contributing to/forking the udt-java project and see how easy it is to bring performance in line with what we can get with our naive approach (I don't suggest starting here, as the project is using blocking old-IO, and modifying it with latency in mind may be challenging, and we won't know for sure what the best case scenario is). \[1\]
[jira] [Commented] (CASSANDRA-7523) add date and time types
[ https://issues.apache.org/jira/browse/CASSANDRA-7523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504504#comment-14504504 ] Sylvain Lebresne commented on CASSANDRA-7523: - I'm fine with reverting and making it 3.0 only as well. add date and time types --- Key: CASSANDRA-7523 URL: https://issues.apache.org/jira/browse/CASSANDRA-7523 Project: Cassandra Issue Type: New Feature Components: API Reporter: Jonathan Ellis Assignee: Joshua McKenzie Priority: Minor Labels: client-impacting, docs Fix For: 2.1.5 http://www.postgresql.org/docs/9.1/static/datatype-datetime.html (we already have timestamp; interval is out of scope for now, and see CASSANDRA-6350 for discussion on timestamp-with-time-zone. but date/time should be pretty easy to add.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9197) Startup slowdown due to preloading jemalloc
[
https://issues.apache.org/jira/browse/CASSANDRA-9197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504642#comment-14504642
]
Robert Stupp commented on CASSANDRA-9197:
-
It seems that the {{find_library}} method causes the slowdown. I've got some
patch ready but want to check it first.
Startup slowdown due to preloading jemalloc
---
Key: CASSANDRA-9197
URL: https://issues.apache.org/jira/browse/CASSANDRA-9197
Project: Cassandra
Issue Type: Bug
Reporter: Sylvain Lebresne
Assignee: Philip Thompson
Priority: Minor
On my box, it seems that the jemalloc loading from CASSANDRA-8714 made the
process take ~10 seconds to even start (I have no explication for it). I
don't know if it's specific to my machine or not, so that ticket is mainly so
someone else can check if it sees the same, in particular for jenkins. If it
does sees the same slowness, we might want to at least disable jemalloc for
dtests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9193) Facility to write dynamic code to selectively trigger trace or log for queries
[
https://issues.apache.org/jira/browse/CASSANDRA-9193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504597#comment-14504597
]
Sylvain Lebresne commented on CASSANDRA-9193:
-
I don't know, I'm sorry to be difficult but I don't really like the idea of
having people write functions that manipulate internal data structures (like
ReadCommand). Those stuffs are way too volatile to make a user-facing API
depend on it (please don't get me wrong, I'm not saying we haven't done such
things in the past, I'm saying that every single time we've done it, we've come
to regret it and it has been a bad user experience).
I also really agree that this should reuse stuffs from CASSANDRA-7526 as much
as possible rather than being JS specific.
Anyway, I'm going to go out on a limb here since I haven't though too much
about this, but the kind of interface I would prefer would be something along
the lines of:
{noformat}
CREATE FUNCTION myFunction(r request) LANGUAGE JAVA AS
$$
if (!r.isSelect())
return;
if (r.consistency().equals(ALL))
logger.warn(WHAT!?);
if (r.size() 50)
logger.warn(Ouch!);
// If the request has 'c1 = 3' in the WHERE clause
if (r.selects(c1, 3))
r.trace();
$$;
INJECT myFunction ON REQUEST TO myKs.myCf;
{noformat}
I'm not attached to any of the specifics here, it's just to say that it would
be ideal in my book if 1) we could reuse UDFs and 2) having it directly doable
from CQL. I'm also her implying we'd add some custom {{request}} object that
we'd pass to the function and would also to get the different parameters of a
request.
Facility to write dynamic code to selectively trigger trace or log for queries
--
Key: CASSANDRA-9193
URL: https://issues.apache.org/jira/browse/CASSANDRA-9193
Project: Cassandra
Issue Type: New Feature
Reporter: Matt Stump
I want the equivalent of dtrace for Cassandra. I want the ability to
intercept a query with a dynamic script (assume JS) and based on logic in
that script trigger the statement for trace or logging.
Examples
- Trace only INSERT statements to a particular CF.
- Trace statements for a particular partition or consistency level.
- Log statements that fail to reach the desired consistency for read or write.
- Log If the request size for read or write exceeds some threshold
At some point in the future it would be helpful to also do things such as log
partitions greater than X bytes or Z cells when performing compaction.
Essentially be able to inject custom code dynamically without a reboot to the
different stages of C*.
The code should be executed synchronously as part of the monitored task, but
we should provide the ability to log or execute CQL asynchronously from the
provided API.
Further down the line we could use this functionality to modify/rewrite
requests or tasks dynamically.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9197) Startup slowdown due to preloading jemalloc
[
https://issues.apache.org/jira/browse/CASSANDRA-9197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Stupp updated CASSANDRA-9197:
Attachment: 9197.txt
Patch removes probably expensive looping using shell {{while}}/{{for}} and
replaces it with {{find}}.
[~slebresne] can you check whether this solves the problem? I've tested it on
some Linux VMs (Debian + CentOS).
Startup slowdown due to preloading jemalloc
---
Key: CASSANDRA-9197
URL: https://issues.apache.org/jira/browse/CASSANDRA-9197
Project: Cassandra
Issue Type: Bug
Reporter: Sylvain Lebresne
Assignee: Philip Thompson
Priority: Minor
Attachments: 9197.txt
On my box, it seems that the jemalloc loading from CASSANDRA-8714 made the
process take ~10 seconds to even start (I have no explication for it). I
don't know if it's specific to my machine or not, so that ticket is mainly so
someone else can check if it sees the same, in particular for jenkins. If it
does sees the same slowness, we might want to at least disable jemalloc for
dtests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-9218) Node thinks other nodes are down after heavy GC
Erik Forsberg created CASSANDRA-9218: Summary: Node thinks other nodes are down after heavy GC Key: CASSANDRA-9218 URL: https://issues.apache.org/jira/browse/CASSANDRA-9218 Project: Cassandra Issue Type: Bug Reporter: Erik Forsberg I have a few troublesome nodes which often end up doing very long GC pauses. The root cause of this is yet to be found, but it's causing another problem - the affected node(s) mark other nodes as down, and they never recover. Here's how it goes: 1. Node goes into troublesome mode, doing heavy GC with long (10+ seconds) GC pauses. 2. While this happens, node will mark other nodes as down. 3. Once the overload situation resolves, the node still thinks the other nodes are down (they are not). It's also quite common that other nodes think the affected node is down. So we often end up with node A thinking there's some 30 nodes down, then a bunch of other nodes beliving node A is down. This in a cluster with 56 nodes. The only way to get out of the situation is to restart node A, and sometimes a few other nodes. And while node A is in this state, any queries that use node A as coordinator have a high risk of getting errors about not enough replicas being available. I have enabled TRACE level gossip debugging while this happens, and on node A, there will be multiple messages about, has already a pending echo, skipping it - i.e the debug line in Gossiper.java line 882. I have also observed while this was happening that other nodes were trying to establish connections (SYN packets sent) but the trouble node (A) were not picking up the line (no accept()). Not knowing exactly how Gossiper works here but it looks like node A is sending out some gossiper echo messages, but then is too busy to get the replies, and never retries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9198) Deleting from an empty list produces an error
[
https://issues.apache.org/jira/browse/CASSANDRA-9198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504616#comment-14504616
]
Sylvain Lebresne commented on CASSANDRA-9198:
-
bq. Could you also make sure that your patch apply to trunk?
Just so we agree, we should push this to 2.1 (and for the record, if only a
patch for 2.1 is provided, I'm fine dealing with merge conflict on trunk so you
don't really have to bother providing patches for both 2.1 and trunk).
Deleting from an empty list produces an error
-
Key: CASSANDRA-9198
URL: https://issues.apache.org/jira/browse/CASSANDRA-9198
Project: Cassandra
Issue Type: Bug
Components: API
Reporter: Olivier Michallat
Assignee: Jeff Jirsa
Priority: Minor
Fix For: 2.1.5
Attachments: 9198.txt
While deleting an element from a list that does not contain it is a no-op,
deleting it from an empty list causes an error.
This edge case is a bit inconsistent, because it makes list deletion non
idempotent:
{code}
cqlsh:test create table foo (k int primary key, v listint);
cqlsh:test insert into foo(k,v) values (1, [1,2]);
cqlsh:test update foo set v = v - [1] where k = 1;
cqlsh:test update foo set v = v - [1] where k = 1;
cqlsh:test update foo set v = v - [2] where k = 1;
cqlsh:test update foo set v = v - [2] where k = 1;
InvalidRequest: code=2200 [Invalid query] message=Attempted to delete an
element from a list which is null
{code}
With speculative retries coming to the drivers, idempotency becomes more
important because it determines which query we might retry or not. So it
would be better if deleting from an empty list succeeded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-9197) Startup slowdown due to preloading jemalloc
[ https://issues.apache.org/jira/browse/CASSANDRA-9197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Stupp reassigned CASSANDRA-9197: --- Assignee: Robert Stupp (was: Philip Thompson) Startup slowdown due to preloading jemalloc --- Key: CASSANDRA-9197 URL: https://issues.apache.org/jira/browse/CASSANDRA-9197 Project: Cassandra Issue Type: Bug Reporter: Sylvain Lebresne Assignee: Robert Stupp Priority: Minor Attachments: 9197.txt On my box, it seems that the jemalloc loading from CASSANDRA-8714 made the process take ~10 seconds to even start (I have no explication for it). I don't know if it's specific to my machine or not, so that ticket is mainly so someone else can check if it sees the same, in particular for jenkins. If it does sees the same slowness, we might want to at least disable jemalloc for dtests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7168) Add repair aware consistency levels
[ https://issues.apache.org/jira/browse/CASSANDRA-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504539#comment-14504539 ] Sylvain Lebresne commented on CASSANDRA-7168: - bq. What makes you uncomfortable about relying on repair time? The problem is, what if we screw up the repair time (even temporarily)? Or what if for some reason a sstable get deleted from one node without the user realizing right away (you could argue that in that case we can already break CL guarantees and that's true, but this would make it a lot worth since in practice reading from all replicas does give us a reasonably good protection against this)? The fact that we'll be reading only one node (for the repaired data at least) makes it a lot easier imo to screw up consistency guarantees that if we actually read the data on every node (even if just to send digests). In a way, data/digest reads is a bit brute-force, but that's what make it pretty reliable a mechanism. Relying too heavily on the repair time feels fragile in comparison, and being fragile when it comes to consistency guarantees makes me uncomfortable. bq. What would make you more comfortable? I'm not sure. I would probably like to see this added as an opt-in feature first (ideally with some granularity, either per-query or per-table) so we can slowly built some confidence that our handling of the repair time is solid and we have fail-safes around that mechanism for when things go badly. Add repair aware consistency levels --- Key: CASSANDRA-7168 URL: https://issues.apache.org/jira/browse/CASSANDRA-7168 Project: Cassandra Issue Type: Improvement Components: Core Reporter: T Jake Luciani Labels: performance Fix For: 3.1 With CASSANDRA-5351 and CASSANDRA-2424 I think there is an opportunity to avoid a lot of extra disk I/O when running queries with higher consistency levels. Since repaired data is by definition consistent and we know which sstables are repaired, we can optimize the read path by having a REPAIRED_QUORUM which breaks reads into two phases: 1) Read from one replica the result from the repaired sstables. 2) Read from a quorum only the un-repaired data. For the node performing 1) we can pipeline the call so it's a single hop. In the long run (assuming data is repaired regularly) we will end up with much closer to CL.ONE performance while maintaining consistency. Some things to figure out: - If repairs fail on some nodes we can have a situation where we don't have a consistent repaired state across the replicas. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9198) Deleting from an empty list produces an error
[
https://issues.apache.org/jira/browse/CASSANDRA-9198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504573#comment-14504573
]
Benjamin Lerer commented on CASSANDRA-9198:
---
My remarks on the patch:
* I still prefer the previous error messages. {{List index 0 out of bound,
list has size 0}} will not makes a lot of sense if the user as never created
any list. The fact that C* has no empty list can be confusing. In my opinion,
we should be carefull about the error messages and not make people believe that
you can have some empty list.
* It would be nice if you could add some similar unit tests for sets and maps.
* The standard in C* is to use ternarys operator when you have a a simple {{if
return else return}} (even if I am not a fan of it). Nested ternary operators
should be avoided.
Side remark: It makes full sense that you provide a patch for that issue as you
already worked on CASSANDRA-9077. Just make sure, that if the ticket is
assigned to somebody else, you checked with the person before that he is fine
with you providing a patch (which I am ;-)). We never know the person might
have started working on it or be really interested in fixing the ticket.
Deleting from an empty list produces an error
-
Key: CASSANDRA-9198
URL: https://issues.apache.org/jira/browse/CASSANDRA-9198
Project: Cassandra
Issue Type: Bug
Components: API
Reporter: Olivier Michallat
Assignee: Jeff Jirsa
Priority: Minor
Fix For: 2.1.5
Attachments: 9198.txt
While deleting an element from a list that does not contain it is a no-op,
deleting it from an empty list causes an error.
This edge case is a bit inconsistent, because it makes list deletion non
idempotent:
{code}
cqlsh:test create table foo (k int primary key, v listint);
cqlsh:test insert into foo(k,v) values (1, [1,2]);
cqlsh:test update foo set v = v - [1] where k = 1;
cqlsh:test update foo set v = v - [1] where k = 1;
cqlsh:test update foo set v = v - [2] where k = 1;
cqlsh:test update foo set v = v - [2] where k = 1;
InvalidRequest: code=2200 [Invalid query] message=Attempted to delete an
element from a list which is null
{code}
With speculative retries coming to the drivers, idempotency becomes more
important because it determines which query we might retry or not. So it
would be better if deleting from an empty list succeeded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8603) Cut tombstone memory footprint in half for cql deletes
[ https://issues.apache.org/jira/browse/CASSANDRA-8603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504587#comment-14504587 ] Benjamin Lerer commented on CASSANDRA-8603: --- [~dominicletz] could you provide another patch which takes my previous remarks into account? Cut tombstone memory footprint in half for cql deletes -- Key: CASSANDRA-8603 URL: https://issues.apache.org/jira/browse/CASSANDRA-8603 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Dominic Letz Assignee: Dominic Letz Labels: tombstone Attachments: cassandra-2.0.11-8603.txt, cassandra-2.1-8603.txt, cassandra-2.1-8603_v2.txt, system.log As CQL does not yet support range deletes every delete from CQL results in a Semi-RangeTombstone which actually has the same start and end values - but until today they are copies. Effectively doubling the required heap memory to store the RangeTombstone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8897) Remove FileCacheService, instead pooling the buffers
[ https://issues.apache.org/jira/browse/CASSANDRA-8897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505056#comment-14505056 ] Benedict commented on CASSANDRA-8897: - bq. Are the logging gards something we always use for trace and debug or is there really a big benefit when logging only with parametrized strings and no function calls? Well, honestly I never investigated if the optimisation was worth it. I assumed that isXEnabled would be a final variable lookup, inlinable, and hence extremely cheap. It looks like I was wrong, and that not calling it first in the no-args instance is perhaps more efficient. This is disappointing, though, and we should perhaps write our own logback logger (as well as our own non-blocking appender). For this patch I don't mind which way we go, since both are widespread in the codebase. bq. How is this possible since Allocator.allocateAligned() limits the byte buffer to the exact capacity requested, Oops, yes, my mistake. I misread the method you were using to align. Looking at the code again, it looks to me like we could eliminate the ( == 0) branch though. bq. Did you mean to have one thread unallocate the buffer from another thread pool or take over the entire pool or other? No, I mean in the scenario in which one thread _does_ deallocate another thread's buffer (by mistake we pass ownership) we could have a second count variable that is updated only by threads that detect this scenario (and updated via CAS), so that we can still maintain a safe count. However it is challenging in this scenario to ensure we always detect when the count reaches full, so is perhaps something to leave for another ticket (and only detect that we have deallocated on another thread, log a panic, and accept we've leaked some memory). If you'd like to address it in this ticket, do feel free though, since it does give us some greater safety. We should log errors either way, since this is not a target use case. Remove FileCacheService, instead pooling the buffers Key: CASSANDRA-8897 URL: https://issues.apache.org/jira/browse/CASSANDRA-8897 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Benedict Assignee: Stefania Fix For: 3.0 After CASSANDRA-8893, a RAR will be a very lightweight object and will not need caching, so we can eliminate this cache entirely. Instead we should have a pool of buffers that are page-aligned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9220) Hostname verification for node-to-node encryption
[
https://issues.apache.org/jira/browse/CASSANDRA-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Thompson updated CASSANDRA-9220:
---
Fix Version/s: 3.0
Hostname verification for node-to-node encryption
-
Key: CASSANDRA-9220
URL: https://issues.apache.org/jira/browse/CASSANDRA-9220
Project: Cassandra
Issue Type: New Feature
Components: Core
Reporter: Stefan Podkowinski
Fix For: 3.0
Attachments: sslhostverification-2.0.patch
This patch will will introduce a new ssl server option:
{{require_endpoint_verification}}.
Setting it will enable hostname verification for inter-node SSL
communication. This is necessary to prevent man-in-the-middle attacks when
building a trust chain against a common CA. See
[here|https://tersesystems.com/2014/03/23/fixing-hostname-verification/] for
background details.
Clusters that solely rely on importing all node certificates into each trust
store (as described
[here|http://docs.datastax.com/en/cassandra/2.0/cassandra/security/secureSSLCertificates_t.html])
are not effected.
Clusters that use the same common CA to sign node certificates are
potentially affected. In case the CA signing process will allow other parties
to generate certs for different purposes, those certificates could in turn be
used for MITM attacks. The provided patch will allow to enable hostname
verification to make sure not only to check if the cert is valid but also if
it has been created for the host that we're about to connect.
Corresponding dtest: [Test for
CASSANDRA-9220|https://github.com/riptano/cassandra-dtest/pull/237]
Github:
2.0 -
[diff|https://github.com/apache/cassandra/compare/cassandra-2.0...spodkowinski:feat/sslhostverification],
[patch|https://github.com/apache/cassandra/compare/cassandra-2.0...spodkowinski:feat/sslhostverification.patch],
Trunk -
[diff|https://github.com/apache/cassandra/compare/trunk...spodkowinski:feat/sslhostverification],
[patch|https://github.com/apache/cassandra/compare/trunk...spodkowinski:feat/sslhostverification.patch]
Related patches from the client perspective:
[Java|https://datastax-oss.atlassian.net/browse/JAVA-716],
[Python|https://datastax-oss.atlassian.net/browse/PYTHON-296]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9217) Problems with cqlsh copy command
[
https://issues.apache.org/jira/browse/CASSANDRA-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505084#comment-14505084
]
Tyler Hobbs commented on CASSANDRA-9217:
Issue #3 is CASSANDRA-8093.
Problems with cqlsh copy command
Key: CASSANDRA-9217
URL: https://issues.apache.org/jira/browse/CASSANDRA-9217
Project: Cassandra
Issue Type: Bug
Reporter: Brian Cantoni
Assignee: Tyler Hobbs
Labels: cqlsh
Fix For: 2.1.5
On the current 2.1 branch I notice a few (possibly related) problems with
cqlsh copy commands. I'm writing them here together but we can separate if
there are different causes.
*1. Cannot import from CSV if column name is 'date'*
Test file monthly.csv contents:
{noformat}
stationid,metric,date
LAE,barometricpressure,2014-01-01 00:00:00+
LAE,barometricpressure,2014-02-01 00:00:00+
LAE,barometricpressure,2014-03-01 00:00:00+
{noformat}
Steps:
{noformat}
CREATE KEYSPACE IF NOT EXISTS weathercql WITH replication = { 'class':
'SimpleStrategy', 'replication_factor': '1' };
CREATE TABLE IF NOT EXISTS weathercql.monthly (
stationid text,
metric text,
date timestamp,
primary key (stationid, metric, date)
);
COPY weathercql.monthly (stationid, metric, date) FROM 'monthly.csv' WITH
HEADER='true';
{noformat}
Result: the copy command fails unless date is enclosed in double quotes:
{noformat}
cqlsh COPY weathercql.monthly (stationid, metric, date) FROM 'monthly.csv'
WITH HEADER='true';
Improper COPY command.
cqlsh COPY weathercql.monthly (stationid, metric, date) FROM 'monthly.csv'
WITH HEADER='true';
3 rows imported in 0.096 seconds.
{noformat}
If I instead name the 'date' column as 'datex', it works without quotes.
The same steps work on Cassandra 2.1.4 (release build).
*2. Cannot copy to CSV*
Sample data:
{noformat}
create keyspace if not exists test with replication =
{'class':'SimpleStrategy', 'replication_factor':1};
create table if not exists test.kv (key int primary key, value text);
insert into test.kv (key,value) values (1,'alpha');
insert into test.kv (key,value) values (2,'beta');
insert into test.kv (key,value) values (3,'charlie');
{noformat}
When you try to export to CSV, it throws what appears to be a Python error,
and the file is not created correctly:
{noformat}
cqlsh copy test.kv (key,value) to 'test.csv';
global name 'meter' is not defined
{noformat}
The same steps work on Cassandra 2.1.4 (release build).
*3. Copy from CSV inside CQL command file doesn't work*
File kv.csv:
{noformat}
key,value
1,'a'
2,'b'
3,'c'
{noformat}
File kv.cql:
{noformat}
create keyspace if not exists test with replication = {'class':
'SimpleStrategy', 'replication_factor':1};
create table if not exists test.kv (key int primary key, value text);
truncate test.kv;
copy test.kv (key, value) from 'kv.csv' with header='true';
select * from test.kv;
{noformat}
When command file passed to cqlsh, an error is reported on the `copy` command
and it doesn't work:
{noformat}
$ bin/cqlsh -f kv.cql
kv.cql:5:descriptor 'lower' requires a 'str' object but received a 'unicode'
key | value
-+---
(0 rows)
{noformat}
The same commands work correctly when run directly inside cqlsh or when
executed with -e option like: {{bin/cqlsh -e copy test.kv (key, value) from
'kv.csv' with header='true';}}.
This third issue appears to also be broken in 2.1.4 and 2.1.3 release builds,
but works in 2.1.2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9138) BlacklistingCompactionsTest failing on test-compression target
[
https://issues.apache.org/jira/browse/CASSANDRA-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504959#comment-14504959
]
Aleksey Yeschenko commented on CASSANDRA-9138:
--
+1
BlacklistingCompactionsTest failing on test-compression target
--
Key: CASSANDRA-9138
URL: https://issues.apache.org/jira/browse/CASSANDRA-9138
Project: Cassandra
Issue Type: Bug
Components: Tests
Reporter: Philip Thompson
Assignee: Marcus Eriksson
Fix For: 2.0.15
Attachments: 0001-9138.patch
{{org.apache.cassandra.db.compaction.BlacklistingCompactionsTest.testBlacklistingWithSizeTieredCompactionStrategy}}
and
{{org.apache.cassandra.db.compaction.BlacklistingCompactionsTest.testBlacklistingWithLeveledCompactionStrategy}}
are failing on 2.0-head with the test-compression ant target. I've tried
bisecting, but I can't find a time they passed within 500 commits.
The stack trace is
{code}
junit.framework.AssertionFailedError: expected:25 but was:8
at
org.apache.cassandra.db.compaction.BlacklistingCompactionsTest.testBlacklisting(BlacklistingCompactionsTest.java:158)
at
org.apache.cassandra.db.compaction.BlacklistingCompactionsTest.testBlacklistingWithLeveledCompactionStrategy(BlacklistingCompactionsTest.java:69
{code}
To reproduce run {code}ant test-compression
-Dtest.name=BlacklistingCompactionsTest{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9220) Hostname verification for node-to-node encryption
[
https://issues.apache.org/jira/browse/CASSANDRA-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-9220:
--
Description:
This patch will will introduce a new ssl server option:
{{require_endpoint_verification}}.
Setting it will enable hostname verification for inter-node SSL communication.
This is necessary to prevent man-in-the-middle attacks when building a trust
chain against a common CA. See
[here|https://tersesystems.com/2014/03/23/fixing-hostname-verification/] for
background details.
Clusters that solely rely on importing all node certificates into each trust
store (as described
[here|http://docs.datastax.com/en/cassandra/2.0/cassandra/security/secureSSLCertificates_t.html])
are not effected.
Clusters that use the same common CA to sign node certificates are potentially
affected. In case the CA signing process will allow other parties to generate
certs for different purposes, those certificates could in turn be used for MITM
attacks. The provided patch will allow to enable hostname verification to make
sure not only to check if the cert is valid but also if it has been created for
the host that we're about to connect.
Corresponding dtest: [Test for
CASSANDRA-9220|https://github.com/riptano/cassandra-dtest/pull/237]
Github:
2.0 -
[diff|https://github.com/apache/cassandra/compare/cassandra-2.0...spodkowinski:feat/sslhostverification],
[patch|https://github.com/apache/cassandra/compare/cassandra-2.0...spodkowinski:feat/sslhostverification.patch],
Trunk -
[diff|https://github.com/apache/cassandra/compare/trunk...spodkowinski:feat/sslhostverification],
[patch|https://github.com/apache/cassandra/compare/trunk...spodkowinski:feat/sslhostverification.patch]
Related patches from the client perspective:
[Java|https://datastax-oss.atlassian.net/browse/JAVA-716],
[Python|https://datastax-oss.atlassian.net/browse/PYTHON-296]
was:
This patch will will introduce a new ssl server option:
{{require_endpoint_verification}}.
Setting it will enable hostname verification for inter-node SSL communication.
This is necessary to prevent man-in-the-middle attacks when building a trust
chain against a common CA. See
[here|https://tersesystems.com/2014/03/23/fixing-hostname-verification/] for
background details.
Clusters that solely rely on importing all node certificates into each trust
store (as described
[here|http://docs.datastax.com/en/cassandra/2.0/cassandra/security/secureSSLCertificates_t.html])
are not effected.
Clusters that use the same common CA to sign node certificates are potentially
affected. In case the CA signing process will allow other parties to generate
certs for different purposes, those certificates could in turn be used for MITM
attacks. The provided patch will allow to enable hostname verification to make
sure not only to check if the cert is valid but also if it has been created for
the host that we're about to connect.
Github:
2.0 -
[diff|https://github.com/apache/cassandra/compare/cassandra-2.0...spodkowinski:feat/sslhostverification],
[patch|https://github.com/apache/cassandra/compare/cassandra-2.0...spodkowinski:feat/sslhostverification.patch],
Trunk -
[diff|https://github.com/apache/cassandra/compare/trunk...spodkowinski:feat/sslhostverification],
[patch|https://github.com/apache/cassandra/compare/trunk...spodkowinski:feat/sslhostverification.patch]
Related patches from the client perspective:
[Java|https://datastax-oss.atlassian.net/browse/JAVA-716],
[Python|https://datastax-oss.atlassian.net/browse/PYTHON-296]
Hostname verification for node-to-node encryption
-
Key: CASSANDRA-9220
URL: https://issues.apache.org/jira/browse/CASSANDRA-9220
Project: Cassandra
Issue Type: New Feature
Components: Core
Reporter: Stefan Podkowinski
Attachments: sslhostverification-2.0.patch
This patch will will introduce a new ssl server option:
{{require_endpoint_verification}}.
Setting it will enable hostname verification for inter-node SSL
communication. This is necessary to prevent man-in-the-middle attacks when
building a trust chain against a common CA. See
[here|https://tersesystems.com/2014/03/23/fixing-hostname-verification/] for
background details.
Clusters that solely rely on importing all node certificates into each trust
store (as described
[here|http://docs.datastax.com/en/cassandra/2.0/cassandra/security/secureSSLCertificates_t.html])
are not effected.
Clusters that use the same common CA to sign node certificates are
potentially affected. In case the CA signing process will allow other parties
to generate certs for different purposes, those certificates could in turn be
used for MITM attacks. The provided patch will allow to enable hostname
verification
[jira] [Comment Edited] (CASSANDRA-9217) Problems with cqlsh copy command
[
https://issues.apache.org/jira/browse/CASSANDRA-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505084#comment-14505084
]
Tyler Hobbs edited comment on CASSANDRA-9217 at 4/21/15 3:12 PM:
-
Issue #3 is CASSANDRA-9083
was (Author: thobbs):
Issue #3 is CASSANDRA-8093.
Problems with cqlsh copy command
Key: CASSANDRA-9217
URL: https://issues.apache.org/jira/browse/CASSANDRA-9217
Project: Cassandra
Issue Type: Bug
Reporter: Brian Cantoni
Assignee: Tyler Hobbs
Labels: cqlsh
Fix For: 2.1.5
On the current 2.1 branch I notice a few (possibly related) problems with
cqlsh copy commands. I'm writing them here together but we can separate if
there are different causes.
*1. Cannot import from CSV if column name is 'date'*
Test file monthly.csv contents:
{noformat}
stationid,metric,date
LAE,barometricpressure,2014-01-01 00:00:00+
LAE,barometricpressure,2014-02-01 00:00:00+
LAE,barometricpressure,2014-03-01 00:00:00+
{noformat}
Steps:
{noformat}
CREATE KEYSPACE IF NOT EXISTS weathercql WITH replication = { 'class':
'SimpleStrategy', 'replication_factor': '1' };
CREATE TABLE IF NOT EXISTS weathercql.monthly (
stationid text,
metric text,
date timestamp,
primary key (stationid, metric, date)
);
COPY weathercql.monthly (stationid, metric, date) FROM 'monthly.csv' WITH
HEADER='true';
{noformat}
Result: the copy command fails unless date is enclosed in double quotes:
{noformat}
cqlsh COPY weathercql.monthly (stationid, metric, date) FROM 'monthly.csv'
WITH HEADER='true';
Improper COPY command.
cqlsh COPY weathercql.monthly (stationid, metric, date) FROM 'monthly.csv'
WITH HEADER='true';
3 rows imported in 0.096 seconds.
{noformat}
If I instead name the 'date' column as 'datex', it works without quotes.
The same steps work on Cassandra 2.1.4 (release build).
*2. Cannot copy to CSV*
Sample data:
{noformat}
create keyspace if not exists test with replication =
{'class':'SimpleStrategy', 'replication_factor':1};
create table if not exists test.kv (key int primary key, value text);
insert into test.kv (key,value) values (1,'alpha');
insert into test.kv (key,value) values (2,'beta');
insert into test.kv (key,value) values (3,'charlie');
{noformat}
When you try to export to CSV, it throws what appears to be a Python error,
and the file is not created correctly:
{noformat}
cqlsh copy test.kv (key,value) to 'test.csv';
global name 'meter' is not defined
{noformat}
The same steps work on Cassandra 2.1.4 (release build).
*3. Copy from CSV inside CQL command file doesn't work*
File kv.csv:
{noformat}
key,value
1,'a'
2,'b'
3,'c'
{noformat}
File kv.cql:
{noformat}
create keyspace if not exists test with replication = {'class':
'SimpleStrategy', 'replication_factor':1};
create table if not exists test.kv (key int primary key, value text);
truncate test.kv;
copy test.kv (key, value) from 'kv.csv' with header='true';
select * from test.kv;
{noformat}
When command file passed to cqlsh, an error is reported on the `copy` command
and it doesn't work:
{noformat}
$ bin/cqlsh -f kv.cql
kv.cql:5:descriptor 'lower' requires a 'str' object but received a 'unicode'
key | value
-+---
(0 rows)
{noformat}
The same commands work correctly when run directly inside cqlsh or when
executed with -e option like: {{bin/cqlsh -e copy test.kv (key, value) from
'kv.csv' with header='true';}}.
This third issue appears to also be broken in 2.1.4 and 2.1.3 release builds,
but works in 2.1.2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[3/3] cassandra git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/727c5c83 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/727c5c83 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/727c5c83 Branch: refs/heads/trunk Commit: 727c5c832faf8e1abde69e96e01b491fcf5e959b Parents: 1e8c5e0 7f29201 Author: Joshua McKenzie jmcken...@apache.org Authored: Tue Apr 21 08:43:34 2015 -0500 Committer: Joshua McKenzie jmcken...@apache.org Committed: Tue Apr 21 08:43:34 2015 -0500 -- --
[1/2] cassandra git commit: Revert Add date and time types on 2.1 branch
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1 f18fb0445 - 7f29201ce
http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f29201c/test/unit/org/apache/cassandra/serializers/SimpleDateSerializerTest.java
--
diff --git
a/test/unit/org/apache/cassandra/serializers/SimpleDateSerializerTest.java
b/test/unit/org/apache/cassandra/serializers/SimpleDateSerializerTest.java
deleted file mode 100644
index 4c0751f..000
--- a/test/unit/org/apache/cassandra/serializers/SimpleDateSerializerTest.java
+++ /dev/null
@@ -1,155 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- *http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.cassandra.serializers;
-
-import org.apache.cassandra.db.marshal.SimpleDateType;
-import org.apache.cassandra.utils.Pair;
-import org.junit.Test;
-
-import java.nio.ByteBuffer;
-import java.sql.Timestamp;
-import java.text.SimpleDateFormat;
-import java.util.*;
-
-public class SimpleDateSerializerTest
-{
-private static final long millisPerDay = 1000 * 60 * 60 * 24;
-
-private String dates[] = new String[]
-{
-1970-01-01,
-1970-01-02,
-1969-12-31,
--0001-01-02,
--5877521-01-02,
-2014-01-01,
-5881580-01-10,
-1920-12-01,
-1582-10-19
-};
-
-private static GregorianCalendar testCalendar = new GregorianCalendar();
-private static SimpleDateFormat dateFormatUTC = new
SimpleDateFormat(-MM-dd);
-
-{
-testCalendar.setGregorianChange(new Date(Long.MIN_VALUE));
-testCalendar.setTimeZone(TimeZone.getTimeZone(UTC));
-dateFormatUTC.setCalendar(testCalendar);
-dateFormatUTC.setLenient(false);
-}
-
-@Test
-public void testDateStringToTimestamp()
-{
-ListString unparsedDates = new ArrayList();
-ListString badParseResults = new ArrayList();
-for (String date : dates)
-{
-try
-{
-Integer days = SimpleDateSerializer.dateStringToDays(date);
-ByteBuffer value =
SimpleDateSerializer.instance.serialize(days);
-Integer deserialized =
SimpleDateSerializer.instance.deserialize(value);
-
-String toStringValue =
SimpleDateSerializer.instance.toString(deserialized);
-if (!date.equals(toStringValue)) {
-badParseResults.add(String.format(Failed to parse date
correctly. Expected %s, got %s\n, date, toStringValue));
-}
-}
-catch (MarshalException e)
-{
-System.err.println(Got an exception: + e);
-unparsedDates.add(date);
-}
-}
-assert unparsedDates.isEmpty() : Unable to parse: + unparsedDates;
-assert badParseResults.isEmpty() : Incorrect parse results: +
badParseResults;
-}
-
-@Test
-public void testDaysStringToInt()
-{
-Integer value = SimpleDateSerializer.dateStringToDays(12345);
-assert value.compareTo(12345) == 0 : String.format(Failed to parse
integer based date. Expected %s, got %s,
-12345,
-value);
-}
-
-@Test
-public void testProlepticRange()
-{
-for (int i = 1; i 31; ++i)
-{
-String date = 1582-10-;
-if (i 10) date += 0;
-date += i;
-
-Integer days = SimpleDateSerializer.dateStringToDays(date);
-
-ByteBuffer value =
SimpleDateType.instance.fromString(days.toString());
-Integer deserialized =
SimpleDateSerializer.instance.deserialize(value);
-
-// Serialized values are unsigned int, unwrap bits w/overflow
-deserialized -= Integer.MIN_VALUE;
-
-Timestamp ts = new Timestamp(deserialized * millisPerDay);
-testCalendar.setTime(ts);
-
-Date newDate = testCalendar.getTime();
-assert (dateFormatUTC.format(newDate)).equals(date) :
-String.format(Expected [%s], got [%s], date,
dateFormatUTC.format(newDate).toString());
-}
-}
-
-@Test
[2/3] cassandra git commit: Revert Add date and time types on 2.1 branch
Revert Add date and time types on 2.1 branch This reverts commit 107545b3929c32c61e7d00c6ca448c10046db792. Conflicts: CHANGES.txt NEWS.txt bin/cqlsh pylib/cqlshlib/test/test_keyspace_init.cql Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7f29201c Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7f29201c Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7f29201c Branch: refs/heads/trunk Commit: 7f29201ce2b003ee67658afaa1a1f9840e8c6d3a Parents: f18fb04 Author: Joshua McKenzie jmcken...@apache.org Authored: Tue Apr 21 08:42:31 2015 -0500 Committer: Joshua McKenzie jmcken...@apache.org Committed: Tue Apr 21 08:42:31 2015 -0500 -- CHANGES.txt | 1 - NEWS.txt| 5 - bin/cqlsh | 45 +--- doc/cql3/CQL.textile| 41 +--- lib/joda-time-2.4.jar | Bin 586201 - 0 bytes lib/licenses/joda-time-2.4.txt | 201 --- pylib/cqlshlib/cql3handling.py | 6 +- pylib/cqlshlib/displaying.py| 2 - pylib/cqlshlib/formatting.py| 65 ++--- pylib/cqlshlib/helptopics.py| 23 -- pylib/cqlshlib/test/test_cqlsh_output.py| 17 +- pylib/cqlshlib/test/test_keyspace_init.cql | 23 +- .../org/apache/cassandra/cql3/CQL3Type.java | 34 ++- .../org/apache/cassandra/cql3/Constants.java| 5 +- src/java/org/apache/cassandra/cql3/Cql.g| 4 - .../cassandra/db/marshal/IntegerType.java | 5 - .../cassandra/db/marshal/SimpleDateType.java| 72 -- .../apache/cassandra/db/marshal/TimeType.java | 72 -- .../serializers/SimpleDateSerializer.java | 117 - .../cassandra/serializers/TimeSerializer.java | 198 --- .../cassandra/serializers/TypeSerializer.java | 1 - .../apache/cassandra/transport/DataType.java| 2 - .../db/marshal/SimpleDateTypeTest.java | 153 .../cassandra/db/marshal/TimeTypeTest.java | 61 - .../serializers/SimpleDateSerializerTest.java | 155 .../serializers/TimeSerializerTest.java | 242 --- 26 files changed, 78 insertions(+), 1472 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f29201c/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index a40530a..e3428a7 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -43,7 +43,6 @@ * Fix cassandra-stress so it respects the CL passed in user mode (CASSANDRA-8948) * Fix rare NPE in ColumnDefinition#hasIndexOption() (CASSANDRA-8786) * cassandra-stress reports per-operation statistics, plus misc (CASSANDRA-8769) - * Add SimpleDate (cql date) and Time (cql time) types (CASSANDRA-7523) * Use long for key count in cfstats (CASSANDRA-8913) * Make SSTableRewriter.abort() more robust to failure (CASSANDRA-8832) * Remove cold_reads_to_omit from STCS (CASSANDRA-8860) http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f29201c/NEWS.txt -- diff --git a/NEWS.txt b/NEWS.txt index 0816602..6610d15 100644 --- a/NEWS.txt +++ b/NEWS.txt @@ -21,16 +21,11 @@ Upgrading removed - it is almost always better to use date tiered compaction for workloads that have cold data. -New features - -- New `SimpleDateType` (cql date). 4-byte unsigned integer w/out timestamp -- New `TimeType` (cql time). 8-byte long nano-second resolution timestamp 2.1.4 = The default JMX config now listens to localhost only. You must enable the other JMX flags in cassandra-env.sh manually. - 2.1.3 = http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f29201c/bin/cqlsh -- diff --git a/bin/cqlsh b/bin/cqlsh index bd72a4a..e1bd312 100755 --- a/bin/cqlsh +++ b/bin/cqlsh @@ -126,10 +126,6 @@ from cqlshlib.displaying import (RED, BLUE, CYAN, ANSI_RESET, COLUMN_NAME_COLORS FormattedValue, colorme) from cqlshlib.formatting import format_by_type, formatter_for, format_value_utype from cqlshlib.util import trim_if_present, get_file_encoding_bomsize -from cqlshlib.formatting import DateTimeFormat -from cqlshlib.formatting import DEFAULT_TIMESTAMP_FORMAT -from cqlshlib.formatting import DEFAULT_DATE_FORMAT -from cqlshlib.formatting import DEFAULT_NANOTIME_FORMAT from cqlshlib.tracing import print_trace_session, print_trace DEFAULT_HOST = '127.0.0.1' @@ -137,6 +133,7 @@ DEFAULT_PORT = 9042 DEFAULT_CQLVER =
[1/3] cassandra git commit: Revert Add date and time types on 2.1 branch
Repository: cassandra
Updated Branches:
refs/heads/trunk 1e8c5e0a1 - 727c5c832
http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f29201c/test/unit/org/apache/cassandra/serializers/SimpleDateSerializerTest.java
--
diff --git
a/test/unit/org/apache/cassandra/serializers/SimpleDateSerializerTest.java
b/test/unit/org/apache/cassandra/serializers/SimpleDateSerializerTest.java
deleted file mode 100644
index 4c0751f..000
--- a/test/unit/org/apache/cassandra/serializers/SimpleDateSerializerTest.java
+++ /dev/null
@@ -1,155 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- *http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.cassandra.serializers;
-
-import org.apache.cassandra.db.marshal.SimpleDateType;
-import org.apache.cassandra.utils.Pair;
-import org.junit.Test;
-
-import java.nio.ByteBuffer;
-import java.sql.Timestamp;
-import java.text.SimpleDateFormat;
-import java.util.*;
-
-public class SimpleDateSerializerTest
-{
-private static final long millisPerDay = 1000 * 60 * 60 * 24;
-
-private String dates[] = new String[]
-{
-1970-01-01,
-1970-01-02,
-1969-12-31,
--0001-01-02,
--5877521-01-02,
-2014-01-01,
-5881580-01-10,
-1920-12-01,
-1582-10-19
-};
-
-private static GregorianCalendar testCalendar = new GregorianCalendar();
-private static SimpleDateFormat dateFormatUTC = new
SimpleDateFormat(-MM-dd);
-
-{
-testCalendar.setGregorianChange(new Date(Long.MIN_VALUE));
-testCalendar.setTimeZone(TimeZone.getTimeZone(UTC));
-dateFormatUTC.setCalendar(testCalendar);
-dateFormatUTC.setLenient(false);
-}
-
-@Test
-public void testDateStringToTimestamp()
-{
-ListString unparsedDates = new ArrayList();
-ListString badParseResults = new ArrayList();
-for (String date : dates)
-{
-try
-{
-Integer days = SimpleDateSerializer.dateStringToDays(date);
-ByteBuffer value =
SimpleDateSerializer.instance.serialize(days);
-Integer deserialized =
SimpleDateSerializer.instance.deserialize(value);
-
-String toStringValue =
SimpleDateSerializer.instance.toString(deserialized);
-if (!date.equals(toStringValue)) {
-badParseResults.add(String.format(Failed to parse date
correctly. Expected %s, got %s\n, date, toStringValue));
-}
-}
-catch (MarshalException e)
-{
-System.err.println(Got an exception: + e);
-unparsedDates.add(date);
-}
-}
-assert unparsedDates.isEmpty() : Unable to parse: + unparsedDates;
-assert badParseResults.isEmpty() : Incorrect parse results: +
badParseResults;
-}
-
-@Test
-public void testDaysStringToInt()
-{
-Integer value = SimpleDateSerializer.dateStringToDays(12345);
-assert value.compareTo(12345) == 0 : String.format(Failed to parse
integer based date. Expected %s, got %s,
-12345,
-value);
-}
-
-@Test
-public void testProlepticRange()
-{
-for (int i = 1; i 31; ++i)
-{
-String date = 1582-10-;
-if (i 10) date += 0;
-date += i;
-
-Integer days = SimpleDateSerializer.dateStringToDays(date);
-
-ByteBuffer value =
SimpleDateType.instance.fromString(days.toString());
-Integer deserialized =
SimpleDateSerializer.instance.deserialize(value);
-
-// Serialized values are unsigned int, unwrap bits w/overflow
-deserialized -= Integer.MIN_VALUE;
-
-Timestamp ts = new Timestamp(deserialized * millisPerDay);
-testCalendar.setTime(ts);
-
-Date newDate = testCalendar.getTime();
-assert (dateFormatUTC.format(newDate)).equals(date) :
-String.format(Expected [%s], got [%s], date,
dateFormatUTC.format(newDate).toString());
-}
-}
-
-@Test (expected=MarshalException.class)
-
[2/2] cassandra git commit: Revert Add date and time types on 2.1 branch
Revert Add date and time types on 2.1 branch This reverts commit 107545b3929c32c61e7d00c6ca448c10046db792. Conflicts: CHANGES.txt NEWS.txt bin/cqlsh pylib/cqlshlib/test/test_keyspace_init.cql Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7f29201c Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7f29201c Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7f29201c Branch: refs/heads/cassandra-2.1 Commit: 7f29201ce2b003ee67658afaa1a1f9840e8c6d3a Parents: f18fb04 Author: Joshua McKenzie jmcken...@apache.org Authored: Tue Apr 21 08:42:31 2015 -0500 Committer: Joshua McKenzie jmcken...@apache.org Committed: Tue Apr 21 08:42:31 2015 -0500 -- CHANGES.txt | 1 - NEWS.txt| 5 - bin/cqlsh | 45 +--- doc/cql3/CQL.textile| 41 +--- lib/joda-time-2.4.jar | Bin 586201 - 0 bytes lib/licenses/joda-time-2.4.txt | 201 --- pylib/cqlshlib/cql3handling.py | 6 +- pylib/cqlshlib/displaying.py| 2 - pylib/cqlshlib/formatting.py| 65 ++--- pylib/cqlshlib/helptopics.py| 23 -- pylib/cqlshlib/test/test_cqlsh_output.py| 17 +- pylib/cqlshlib/test/test_keyspace_init.cql | 23 +- .../org/apache/cassandra/cql3/CQL3Type.java | 34 ++- .../org/apache/cassandra/cql3/Constants.java| 5 +- src/java/org/apache/cassandra/cql3/Cql.g| 4 - .../cassandra/db/marshal/IntegerType.java | 5 - .../cassandra/db/marshal/SimpleDateType.java| 72 -- .../apache/cassandra/db/marshal/TimeType.java | 72 -- .../serializers/SimpleDateSerializer.java | 117 - .../cassandra/serializers/TimeSerializer.java | 198 --- .../cassandra/serializers/TypeSerializer.java | 1 - .../apache/cassandra/transport/DataType.java| 2 - .../db/marshal/SimpleDateTypeTest.java | 153 .../cassandra/db/marshal/TimeTypeTest.java | 61 - .../serializers/SimpleDateSerializerTest.java | 155 .../serializers/TimeSerializerTest.java | 242 --- 26 files changed, 78 insertions(+), 1472 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f29201c/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index a40530a..e3428a7 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -43,7 +43,6 @@ * Fix cassandra-stress so it respects the CL passed in user mode (CASSANDRA-8948) * Fix rare NPE in ColumnDefinition#hasIndexOption() (CASSANDRA-8786) * cassandra-stress reports per-operation statistics, plus misc (CASSANDRA-8769) - * Add SimpleDate (cql date) and Time (cql time) types (CASSANDRA-7523) * Use long for key count in cfstats (CASSANDRA-8913) * Make SSTableRewriter.abort() more robust to failure (CASSANDRA-8832) * Remove cold_reads_to_omit from STCS (CASSANDRA-8860) http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f29201c/NEWS.txt -- diff --git a/NEWS.txt b/NEWS.txt index 0816602..6610d15 100644 --- a/NEWS.txt +++ b/NEWS.txt @@ -21,16 +21,11 @@ Upgrading removed - it is almost always better to use date tiered compaction for workloads that have cold data. -New features - -- New `SimpleDateType` (cql date). 4-byte unsigned integer w/out timestamp -- New `TimeType` (cql time). 8-byte long nano-second resolution timestamp 2.1.4 = The default JMX config now listens to localhost only. You must enable the other JMX flags in cassandra-env.sh manually. - 2.1.3 = http://git-wip-us.apache.org/repos/asf/cassandra/blob/7f29201c/bin/cqlsh -- diff --git a/bin/cqlsh b/bin/cqlsh index bd72a4a..e1bd312 100755 --- a/bin/cqlsh +++ b/bin/cqlsh @@ -126,10 +126,6 @@ from cqlshlib.displaying import (RED, BLUE, CYAN, ANSI_RESET, COLUMN_NAME_COLORS FormattedValue, colorme) from cqlshlib.formatting import format_by_type, formatter_for, format_value_utype from cqlshlib.util import trim_if_present, get_file_encoding_bomsize -from cqlshlib.formatting import DateTimeFormat -from cqlshlib.formatting import DEFAULT_TIMESTAMP_FORMAT -from cqlshlib.formatting import DEFAULT_DATE_FORMAT -from cqlshlib.formatting import DEFAULT_NANOTIME_FORMAT from cqlshlib.tracing import print_trace_session, print_trace DEFAULT_HOST = '127.0.0.1' @@ -137,6 +133,7 @@ DEFAULT_PORT = 9042
[jira] [Commented] (CASSANDRA-7168) Add repair aware consistency levels
[ https://issues.apache.org/jira/browse/CASSANDRA-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504983#comment-14504983 ] Aleksey Yeschenko commented on CASSANDRA-7168: -- The 3.0 comment was a bit premature, I take that bit back. Still, should benchmark it once we get some spare time. Think 3.6-3.6 timeframe. Add repair aware consistency levels --- Key: CASSANDRA-7168 URL: https://issues.apache.org/jira/browse/CASSANDRA-7168 Project: Cassandra Issue Type: Improvement Components: Core Reporter: T Jake Luciani Labels: performance Fix For: 3.1 With CASSANDRA-5351 and CASSANDRA-2424 I think there is an opportunity to avoid a lot of extra disk I/O when running queries with higher consistency levels. Since repaired data is by definition consistent and we know which sstables are repaired, we can optimize the read path by having a REPAIRED_QUORUM which breaks reads into two phases: 1) Read from one replica the result from the repaired sstables. 2) Read from a quorum only the un-repaired data. For the node performing 1) we can pipeline the call so it's a single hop. In the long run (assuming data is repaired regularly) we will end up with much closer to CL.ONE performance while maintaining consistency. Some things to figure out: - If repairs fail on some nodes we can have a situation where we don't have a consistent repaired state across the replicas. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9218) Node thinks other nodes are down after heavy GC
[ https://issues.apache.org/jira/browse/CASSANDRA-9218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505013#comment-14505013 ] Brandon Williams commented on CASSANDRA-9218: - bq. I have also observed while this was happening that other nodes were trying to establish connections (SYN packets sent) but the trouble node (A) were not picking up the line (no accept()). If you mean a TCP SYN (vs a gossip SYN) then that is pretty strange, since it would seem to indicate a network problem. Do you have netstat output from both sides when this happens? Node thinks other nodes are down after heavy GC --- Key: CASSANDRA-9218 URL: https://issues.apache.org/jira/browse/CASSANDRA-9218 Project: Cassandra Issue Type: Bug Reporter: Erik Forsberg I have a few troublesome nodes which often end up doing very long GC pauses. The root cause of this is yet to be found, but it's causing another problem - the affected node(s) mark other nodes as down, and they never recover. Here's how it goes: 1. Node goes into troublesome mode, doing heavy GC with long (10+ seconds) GC pauses. 2. While this happens, node will mark other nodes as down. 3. Once the overload situation resolves, the node still thinks the other nodes are down (they are not). It's also quite common that other nodes think the affected node is down. So we often end up with node A thinking there's some 30 nodes down, then a bunch of other nodes beliving node A is down. This in a cluster with 56 nodes. The only way to get out of the situation is to restart node A, and sometimes a few other nodes. And while node A is in this state, any queries that use node A as coordinator have a high risk of getting errors about not enough replicas being available. I have enabled TRACE level gossip debugging while this happens, and on node A, there will be multiple messages about, has already a pending echo, skipping it - i.e the debug line in Gossiper.java line 882. I have also observed while this was happening that other nodes were trying to establish connections (SYN packets sent) but the trouble node (A) were not picking up the line (no accept()). Not knowing exactly how Gossiper works here but it looks like node A is sending out some gossiper echo messages, but then is too busy to get the replies, and never retries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9218) Node thinks other nodes are down after heavy GC
[ https://issues.apache.org/jira/browse/CASSANDRA-9218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505025#comment-14505025 ] Erik Forsberg commented on CASSANDRA-9218: -- I mean TCP SYN. I was running tcpdump on both sides. Sending node was sending SYN, receving node received it but there was no accept() while the receiving node was busy (veeery long GC, I've seen 45+ seconds). So it does not look like we're having a network problem. Node thinks other nodes are down after heavy GC --- Key: CASSANDRA-9218 URL: https://issues.apache.org/jira/browse/CASSANDRA-9218 Project: Cassandra Issue Type: Bug Reporter: Erik Forsberg I have a few troublesome nodes which often end up doing very long GC pauses. The root cause of this is yet to be found, but it's causing another problem - the affected node(s) mark other nodes as down, and they never recover. Here's how it goes: 1. Node goes into troublesome mode, doing heavy GC with long (10+ seconds) GC pauses. 2. While this happens, node will mark other nodes as down. 3. Once the overload situation resolves, the node still thinks the other nodes are down (they are not). It's also quite common that other nodes think the affected node is down. So we often end up with node A thinking there's some 30 nodes down, then a bunch of other nodes beliving node A is down. This in a cluster with 56 nodes. The only way to get out of the situation is to restart node A, and sometimes a few other nodes. And while node A is in this state, any queries that use node A as coordinator have a high risk of getting errors about not enough replicas being available. I have enabled TRACE level gossip debugging while this happens, and on node A, there will be multiple messages about, has already a pending echo, skipping it - i.e the debug line in Gossiper.java line 882. I have also observed while this was happening that other nodes were trying to establish connections (SYN packets sent) but the trouble node (A) were not picking up the line (no accept()). Not knowing exactly how Gossiper works here but it looks like node A is sending out some gossiper echo messages, but then is too busy to get the replies, and never retries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9220) Hostname verification for node-to-node encryption
[
https://issues.apache.org/jira/browse/CASSANDRA-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-9220:
--
Attachment: sslhostverification-2.0.patch
Hostname verification for node-to-node encryption
-
Key: CASSANDRA-9220
URL: https://issues.apache.org/jira/browse/CASSANDRA-9220
Project: Cassandra
Issue Type: New Feature
Components: Core
Reporter: Stefan Podkowinski
Attachments: sslhostverification-2.0.patch
This patch will will introduce a new ssl server option:
{{require_endpoint_verification}}.
Setting it will enable hostname verification for inter-node SSL
communication. This is necessary to prevent man-in-the-middle attacks when
building a trust chain against a common CA. See
[here|https://tersesystems.com/2014/03/23/fixing-hostname-verification/] for
background details.
Clusters that solely rely on importing all node certificates into each trust
store (as described
[here|http://docs.datastax.com/en/cassandra/2.0/cassandra/security/secureSSLCertificates_t.html])
are not effected.
Clusters that use the same common CA to sign node certificates are
potentially affected. In case the CA signing process will allow other parties
to generate certs for different purposes, those certificates could in turn be
used for MITM attacks. The provided patch will allow to enable hostname
verification to make sure not only to check if the cert is valid but also if
it has been created for the host that we're about to connect.
Related patches from the client perspective:
[Java|https://datastax-oss.atlassian.net/browse/JAVA-716],
[Python|https://datastax-oss.atlassian.net/browse/PYTHON-296]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9194) Delete-only workloads crash Cassandra
[
https://issues.apache.org/jira/browse/CASSANDRA-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505033#comment-14505033
]
Benedict commented on CASSANDRA-9194:
-
[~mambocab]: thanks for being diligent about this. In 2.1 we _do_ account for
its on-heap size (if you check), which is what this bug is indicating we do not
track. The deletion itself is not counted towards the live data size though
(i.e. the proportion of heap that is actually data). This is because the space
used is already used. I'm not sure we care enough to special case this, as we
need to check if the deletiontime _was_ live and is now _not_ live and add 8 in
this case, but even this is kind of arbitrary. This live size in general is
kind of an arbitrary measure, so perhaps we can just add 8 to live bytes when
we first insert a row, since there will always be some overhead.
Since the value we're reporting is kind of a fabrication, I'm sanguine about
leaving it as is, or changing it to some other arbitrary value. [~jbellis] got
any opinion either way, since you +1'd the first patch?
Delete-only workloads crash Cassandra
-
Key: CASSANDRA-9194
URL: https://issues.apache.org/jira/browse/CASSANDRA-9194
Project: Cassandra
Issue Type: Bug
Environment: 2.0.14
Reporter: Robert Wille
Assignee: Benedict
Fix For: 2.0.15
Attachments: 9194.txt
The size of a tombstone is not properly accounted for in the memtable. A
memtable which has only tombstones will never get flushed. It will grow until
the JVM runs out of memory. The following program easily demonstrates the
problem.
{code}
Cluster.Builder builder = Cluster.builder();
Cluster c =
builder.addContactPoints(cas121.devf3.com).build();
Session s = c.connect();
s.execute(CREATE KEYSPACE IF NOT EXISTS test WITH replication
= { 'class' : 'SimpleStrategy', 'replication_factor' : 1 });
s.execute(CREATE TABLE IF NOT EXISTS test.test(id INT PRIMARY
KEY));
PreparedStatement stmt = s.prepare(DELETE FROM test.test WHERE
id = :id);
int id = 0;
while (true)
{
s.execute(stmt.bind(id));
id++;
}{code}
This program should run forever, but eventually Cassandra runs out of heap
and craps out. You needn't wait for Cassandra to crash. If you run nodetool
cfstats test.test while it is running, you'll see Memtable cell count grow,
but Memtable data size will remain 0.
This issue was fixed once before. I received a patch for version 2.0.5 (I
believe), which contained the fix, but the fix has apparently been lost,
because it is clearly broken, and I don't see the fix in the change logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-2848) Make the Client API support passing down timeouts
[
https://issues.apache.org/jira/browse/CASSANDRA-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504970#comment-14504970
]
Sylvain Lebresne commented on CASSANDRA-2848:
-
I see what you did there :)
Unfortunately:
# digests are far from being the only problem. Short reads are another, but
more importantly, any range query is pretty much a free for all as far as
timeouts are concerned since we basically reset the timeout for every range
(modulo the concurrency factor, but that can very well be 1) and there can be
very many range with vnodes. In practice, I suspect its range queries that are
more often the culprit when the timeout is not respected.
# I'm sure we can be smarter than halve the timeout when digests are involved
in case we need to redo it. For instance, we could mark the starting of a
query and pass that down to the {{ReadCallback}}, so that it waits for
{{timeout - (now - queryStart)}}. I don't think it would be that weird (outside
of the fact we'd have to pass {{qeryStart}} down the {{ReadCallback}}).
Make the Client API support passing down timeouts
-
Key: CASSANDRA-2848
URL: https://issues.apache.org/jira/browse/CASSANDRA-2848
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Chris Goffinet
Assignee: Sylvain Lebresne
Priority: Minor
Fix For: 3.1
Having a max server RPC timeout is good for worst case, but many applications
that have middleware in front of Cassandra, might have higher timeout
requirements. In a fail fast environment, if my application starting at say
the front-end, only has 20ms to process a request, and it must connect to X
services down the stack, by the time it hits Cassandra, we might only have
10ms. I propose we provide the ability to specify the timeout on each call we
do optionally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-9219) Follow-ups to 7523: protect old clients from new type values and reject empty ByteBuffer
Joshua McKenzie created CASSANDRA-9219: -- Summary: Follow-ups to 7523: protect old clients from new type values and reject empty ByteBuffer Key: CASSANDRA-9219 URL: https://issues.apache.org/jira/browse/CASSANDRA-9219 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Joshua McKenzie Assignee: Joshua McKenzie Priority: Minor Fix For: 3.0 2 follow-ups from CASSANDRA-7523, which has now been removed from the 2.1 branch: bq. The problem is that if someone use one of those new data types with an existing client, Cassandra will currently happily return the new codes, which clients have no reason to know about and may therefore crash in unexpected ways. That is a problem and that's what I meant by Adding new codes to v3 would confuse drivers. bq. So what I mean is that we should special case those 2 types in DataType (maybe in toType, maybe directly in the serialization) so that when the protocol version is = 3, then it write the types as CUSTOM ones. And bq. can we make the new types non-emptiable by default? See the last two CASSANDRA-8951 comments. bq. Would really like us to reject empty BB as a valid value there though -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7523) add date and time types
[ https://issues.apache.org/jira/browse/CASSANDRA-7523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504973#comment-14504973 ] Joshua McKenzie commented on CASSANDRA-7523: Reverted on 2.1 branch, created CASSANDRA-9219 to track remainder of what came up in comments here. add date and time types --- Key: CASSANDRA-7523 URL: https://issues.apache.org/jira/browse/CASSANDRA-7523 Project: Cassandra Issue Type: New Feature Components: API Reporter: Jonathan Ellis Assignee: Joshua McKenzie Priority: Minor Labels: client-impacting, docs Fix For: 2.1.5 http://www.postgresql.org/docs/9.1/static/datatype-datetime.html (we already have timestamp; interval is out of scope for now, and see CASSANDRA-6350 for discussion on timestamp-with-time-zone. but date/time should be pretty easy to add.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9220) Hostname verification for node-to-node encryption
Stefan Podkowinski created CASSANDRA-9220:
-
Summary: Hostname verification for node-to-node encryption
Key: CASSANDRA-9220
URL: https://issues.apache.org/jira/browse/CASSANDRA-9220
Project: Cassandra
Issue Type: New Feature
Components: Core
Reporter: Stefan Podkowinski
This patch will will introduce a new ssl server option:
{{require_endpoint_verification}}.
Setting it will enable hostname verification for inter-node SSL communication.
This is necessary to prevent man-in-the-middle attacks when building a trust
chain against a common CA. See
[here|https://tersesystems.com/2014/03/23/fixing-hostname-verification/] for
background details.
Clusters that solely rely on importing all node certificates into each trust
store (as described
[here|http://docs.datastax.com/en/cassandra/2.0/cassandra/security/secureSSLCertificates_t.html])
are not effected.
Clusters that use the same common CA to sign node certificates are potentially
affected. In case the CA signing process will allow other parties to generate
certs for different purposes, those certificates could in turn be used for MITM
attacks. The provided patch will allow to enable hostname verification to make
sure not only to check if the cert is valid but also if it has been created for
the host that we're about to connect.
Related patches from the client perspective:
[Java|https://datastax-oss.atlassian.net/browse/JAVA-716],
[Python|https://datastax-oss.atlassian.net/browse/PYTHON-296]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9220) Hostname verification for node-to-node encryption
[
https://issues.apache.org/jira/browse/CASSANDRA-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Podkowinski updated CASSANDRA-9220:
--
Description:
This patch will will introduce a new ssl server option:
{{require_endpoint_verification}}.
Setting it will enable hostname verification for inter-node SSL communication.
This is necessary to prevent man-in-the-middle attacks when building a trust
chain against a common CA. See
[here|https://tersesystems.com/2014/03/23/fixing-hostname-verification/] for
background details.
Clusters that solely rely on importing all node certificates into each trust
store (as described
[here|http://docs.datastax.com/en/cassandra/2.0/cassandra/security/secureSSLCertificates_t.html])
are not effected.
Clusters that use the same common CA to sign node certificates are potentially
affected. In case the CA signing process will allow other parties to generate
certs for different purposes, those certificates could in turn be used for MITM
attacks. The provided patch will allow to enable hostname verification to make
sure not only to check if the cert is valid but also if it has been created for
the host that we're about to connect.
Github:
2.0 -
[diff|https://github.com/apache/cassandra/compare/cassandra-2.0...spodkowinski:feat/sslhostverification],
[patch|https://github.com/apache/cassandra/compare/cassandra-2.0...spodkowinski:feat/sslhostverification.patch],
Trunk -
[diff|https://github.com/apache/cassandra/compare/trunk...spodkowinski:feat/sslhostverification],
[patch|https://github.com/apache/cassandra/compare/trunk...spodkowinski:feat/sslhostverification.patch]
Related patches from the client perspective:
[Java|https://datastax-oss.atlassian.net/browse/JAVA-716],
[Python|https://datastax-oss.atlassian.net/browse/PYTHON-296]
was:
This patch will will introduce a new ssl server option:
{{require_endpoint_verification}}.
Setting it will enable hostname verification for inter-node SSL communication.
This is necessary to prevent man-in-the-middle attacks when building a trust
chain against a common CA. See
[here|https://tersesystems.com/2014/03/23/fixing-hostname-verification/] for
background details.
Clusters that solely rely on importing all node certificates into each trust
store (as described
[here|http://docs.datastax.com/en/cassandra/2.0/cassandra/security/secureSSLCertificates_t.html])
are not effected.
Clusters that use the same common CA to sign node certificates are potentially
affected. In case the CA signing process will allow other parties to generate
certs for different purposes, those certificates could in turn be used for MITM
attacks. The provided patch will allow to enable hostname verification to make
sure not only to check if the cert is valid but also if it has been created for
the host that we're about to connect.
Related patches from the client perspective:
[Java|https://datastax-oss.atlassian.net/browse/JAVA-716],
[Python|https://datastax-oss.atlassian.net/browse/PYTHON-296]
Hostname verification for node-to-node encryption
-
Key: CASSANDRA-9220
URL: https://issues.apache.org/jira/browse/CASSANDRA-9220
Project: Cassandra
Issue Type: New Feature
Components: Core
Reporter: Stefan Podkowinski
Attachments: sslhostverification-2.0.patch
This patch will will introduce a new ssl server option:
{{require_endpoint_verification}}.
Setting it will enable hostname verification for inter-node SSL
communication. This is necessary to prevent man-in-the-middle attacks when
building a trust chain against a common CA. See
[here|https://tersesystems.com/2014/03/23/fixing-hostname-verification/] for
background details.
Clusters that solely rely on importing all node certificates into each trust
store (as described
[here|http://docs.datastax.com/en/cassandra/2.0/cassandra/security/secureSSLCertificates_t.html])
are not effected.
Clusters that use the same common CA to sign node certificates are
potentially affected. In case the CA signing process will allow other parties
to generate certs for different purposes, those certificates could in turn be
used for MITM attacks. The provided patch will allow to enable hostname
verification to make sure not only to check if the cert is valid but also if
it has been created for the host that we're about to connect.
Github:
2.0 -
[diff|https://github.com/apache/cassandra/compare/cassandra-2.0...spodkowinski:feat/sslhostverification],
[patch|https://github.com/apache/cassandra/compare/cassandra-2.0...spodkowinski:feat/sslhostverification.patch],
Trunk -
[diff|https://github.com/apache/cassandra/compare/trunk...spodkowinski:feat/sslhostverification],
[jira] [Commented] (CASSANDRA-7168) Add repair aware consistency levels
[ https://issues.apache.org/jira/browse/CASSANDRA-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504981#comment-14504981 ] Sylvain Lebresne commented on CASSANDRA-7168: - bq. But in the meantime I do think we should drop digest reads. I'm not necessarily against that, though I agree with Aleksey that it's worth doing serious benchmarking before affirming that it really doesn't buy us much since it's been there pretty much forever. bq. Do we have any spare resources to do the testing prior to 3.0 release? I know we're always impatient to remove stuff, but since this particular ticket won't make 3.0, I would suggest leaving digests for now and save whatever benchmarking resources we have for 3.0. Just an opinion though. Add repair aware consistency levels --- Key: CASSANDRA-7168 URL: https://issues.apache.org/jira/browse/CASSANDRA-7168 Project: Cassandra Issue Type: Improvement Components: Core Reporter: T Jake Luciani Labels: performance Fix For: 3.1 With CASSANDRA-5351 and CASSANDRA-2424 I think there is an opportunity to avoid a lot of extra disk I/O when running queries with higher consistency levels. Since repaired data is by definition consistent and we know which sstables are repaired, we can optimize the read path by having a REPAIRED_QUORUM which breaks reads into two phases: 1) Read from one replica the result from the repaired sstables. 2) Read from a quorum only the un-repaired data. For the node performing 1) we can pipeline the call so it's a single hop. In the long run (assuming data is repaired regularly) we will end up with much closer to CL.ONE performance while maintaining consistency. Some things to figure out: - If repairs fail on some nodes we can have a situation where we don't have a consistent repaired state across the replicas. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9104) Unit test failures, trunk + Windows
[
https://issues.apache.org/jira/browse/CASSANDRA-9104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504875#comment-14504875
]
Branimir Lambov commented on CASSANDRA-9104:
Thanks for the updates, a few more nits that should be trivial to address:
ScrubTest.testScrubCorruptedCounterRowNoEarlyOpen: I don't believe the
framework will revert the open interval change between tests, so I'd use a
try/finally block to save and restore the previous value to make sure we are
not testing the wrong thing in the tests that happen to run after this.
CommitLog.resetUnsafe: Good idea, calling recover from reset is the right thing
to do. However, does the RecoveryManagerTest2 test work with this change?
recover() is now called twice, and there should not be anything to recover the
second time. Perhaps return the recover() result from resetUnsafe()? And maybe
not convert the IOException as all callers already list it?
CLSM.waitForAllocation: Calling allocatingFrom() achieves exactly the same side
effect (and it's package-visible so there is no problem to call it from
CommitLog). I'd prefer to avoid the repetition, but feel free not to change
this if you think calling allocatingFrom() is not clean enough.
CLSM.stopUnsafe: Is the sleep() vs yield() change necessary? If it is necessary
it is probably masking a concurrency problem.
There is also a binary change that's sneaked into
9104_KeyCache_ScrubTest_v2.txt making the patch fail to apply.
Unit test failures, trunk + Windows
---
Key: CASSANDRA-9104
URL: https://issues.apache.org/jira/browse/CASSANDRA-9104
Project: Cassandra
Issue Type: Test
Reporter: Joshua McKenzie
Assignee: Joshua McKenzie
Labels: Windows
Fix For: 3.0
Attachments: 9104_CFSTest.txt, 9104_KeyCache.txt,
9104_KeyCache_ScrubTest_v2.txt, 9104_RecoveryManager.txt,
9104_RecoveryManager_v2.txt, 9104_ScrubTest.txt
Variety of different test failures have cropped up over the past 2-3 weeks:
h6. -org.apache.cassandra.cql3.UFTest FAILED (timeout)- // No longer failing
/ timing out
h6.
testLoadNewSSTablesAvoidsOverwrites(org.apache.cassandra.db.ColumnFamilyStoreTest):
FAILED
{noformat}
12 SSTables unexpectedly exist
junit.framework.AssertionFailedError: 12 SSTables unexpectedly exist
at
org.apache.cassandra.db.ColumnFamilyStoreTest.testLoadNewSSTablesAvoidsOverwrites(ColumnFamilyStoreTest.java:1896)
{noformat}
h6. org.apache.cassandra.db.KeyCacheTest FAILED
{noformat}
expected:4 but was:2
junit.framework.AssertionFailedError: expected:4 but was:2
at
org.apache.cassandra.db.KeyCacheTest.assertKeyCacheSize(KeyCacheTest.java:221)
at org.apache.cassandra.db.KeyCacheTest.testKeyCache(KeyCacheTest.java:181)
{noformat}
h6. RecoveryManagerTest:
{noformat}
org.apache.cassandra.db.RecoveryManagerTest FAILED
org.apache.cassandra.db.RecoveryManager2Test FAILED
org.apache.cassandra.db.RecoveryManager3Test FAILED
org.apache.cassandra.db.RecoveryManagerTruncateTest FAILED
All are the following:
java.nio.file.AccessDeniedException:
build\test\cassandra\commitlog;0\CommitLog-5-1427995105229.log
FSWriteError in
build\test\cassandra\commitlog;0\CommitLog-5-1427995105229.log
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:128)
at
org.apache.cassandra.db.commitlog.CommitLogSegmentManager.recycleSegment(CommitLogSegmentManager.java:360)
at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:156)
at
org.apache.cassandra.db.RecoveryManagerTest.testNothingToRecover(RecoveryManagerTest.java:75)
Caused by: java.nio.file.AccessDeniedException:
build\test\cassandra\commitlog;0\CommitLog-5-1427995105229.log
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at
sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
at
sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
at java.nio.file.Files.delete(Files.java:1079)
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:124)
{noformat}
h6. testScrubCorruptedCounterRow(org.apache.cassandra.db.ScrubTest): FAILED
{noformat}
Expecting new size of 1, got 2 while replacing
[BigTableReader(path='C:\src\refCassandra\build\test\cassandra\data;0\Keyspace1\Counter1-deab62b2d95c11e489c6e117fe147c1d\la-1-big-Data.db')]
by
[2/3] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f18fb044 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f18fb044 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f18fb044 Branch: refs/heads/trunk Commit: f18fb044527115d9dfe8586bf1615825463b1156 Parents: c6e4379 b325316 Author: Aleksey Yeschenko alek...@apache.org Authored: Tue Apr 21 14:41:08 2015 +0300 Committer: Aleksey Yeschenko alek...@apache.org Committed: Tue Apr 21 14:41:08 2015 +0300 -- CHANGES.txt | 1 + .../cassandra/db/index/SecondaryIndex.java | 10 +++ .../db/index/SecondaryIndexManager.java | 19 + .../cassandra/db/ColumnFamilyStoreTest.java | 77 +--- .../db/index/PerRowSecondaryIndexTest.java | 11 ++- 5 files changed, 74 insertions(+), 44 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/f18fb044/CHANGES.txt -- diff --cc CHANGES.txt index 2777d79,5e1e62c..a40530a --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,86 -1,5 +1,87 @@@ -2.0.15: +2.1.5 + * Re-add deprecated cold_reads_to_omit param for backwards compat (CASSANDRA-9203) + * Make anticompaction visible in compactionstats (CASSANDRA-9098) + * Improve nodetool getendpoints documentation about the partition + key parameter (CASSANDRA-6458) + * Don't check other keyspaces for schema changes when an user-defined + type is altered (CASSANDRA-9187) + * Allow takeColumnFamilySnapshot to take a list of tables (CASSANDRA-8348) + * Limit major sstable operations to their canonical representation (CASSANDRA-8669) + * cqlsh: Add tests for INSERT and UPDATE tab completion (CASSANDRA-9125) + * cqlsh: quote column names when needed in COPY FROM inserts (CASSANDRA-9080) + * Add generate-idea-files target to build.xml (CASSANDRA-9123) + * Do not load read meter for offline operations (CASSANDRA-9082) + * cqlsh: Make CompositeType data readable (CASSANDRA-8919) + * cqlsh: Fix display of triggers (CASSANDRA-9081) + * Fix NullPointerException when deleting or setting an element by index on + a null list collection (CASSANDRA-9077) + * Buffer bloom filter serialization (CASSANDRA-9066) + * Fix anti-compaction target bloom filter size (CASSANDRA-9060) + * Make FROZEN and TUPLE unreserved keywords in CQL (CASSANDRA-9047) + * Prevent AssertionError from SizeEstimatesRecorder (CASSANDRA-9034) + * Avoid overwriting index summaries for sstables with an older format that + does not support downsampling; rebuild summaries on startup when this + is detected (CASSANDRA-8993) + * Fix potential data loss in CompressedSequentialWriter (CASSANDRA-8949) + * Make PasswordAuthenticator number of hashing rounds configurable (CASSANDRA-8085) + * Fix AssertionError when binding nested collections in DELETE (CASSANDRA-8900) + * Check for overlap with non-early sstables in LCS (CASSANDRA-8739) + * Only calculate max purgable timestamp if we have to (CASSANDRA-8914) + * (cqlsh) Greatly improve performance of COPY FROM (CASSANDRA-8225) + * IndexSummary effectiveIndexInterval is now a guideline, not a rule (CASSANDRA-8993) + * Use correct bounds for page cache eviction of compressed files (CASSANDRA-8746) + * SSTableScanner enforces its bounds (CASSANDRA-8946) + * Cleanup cell equality (CASSANDRA-8947) + * Introduce intra-cluster message coalescing (CASSANDRA-8692) + * DatabaseDescriptor throws NPE when rpc_interface is used (CASSANDRA-8839) + * Don't check if an sstable is live for offline compactions (CASSANDRA-8841) + * Don't set clientMode in SSTableLoader (CASSANDRA-8238) + * Fix SSTableRewriter with disabled early open (CASSANDRA-8535) + * Allow invalidating permissions and cache time (CASSANDRA-8722) + * Log warning when queries that will require ALLOW FILTERING in Cassandra 3.0 + are executed (CASSANDRA-8418) + * Fix cassandra-stress so it respects the CL passed in user mode (CASSANDRA-8948) + * Fix rare NPE in ColumnDefinition#hasIndexOption() (CASSANDRA-8786) + * cassandra-stress reports per-operation statistics, plus misc (CASSANDRA-8769) + * Add SimpleDate (cql date) and Time (cql time) types (CASSANDRA-7523) + * Use long for key count in cfstats (CASSANDRA-8913) + * Make SSTableRewriter.abort() more robust to failure (CASSANDRA-8832) + * Remove cold_reads_to_omit from STCS (CASSANDRA-8860) + * Make EstimatedHistogram#percentile() use ceil instead of floor (CASSANDRA-8883) + * Fix top partitions reporting wrong cardinality (CASSANDRA-8834) + * Fix rare NPE in KeyCacheSerializer (CASSANDRA-8067) + * Pick sstables for validation as late as possible inc repairs (CASSANDRA-8366) + * Fix commitlog
[jira] [Commented] (CASSANDRA-9107) More accurate row count estimates
[
https://issues.apache.org/jira/browse/CASSANDRA-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504840#comment-14504840
]
Aleksey Yeschenko commented on CASSANDRA-9107:
--
Should we start to switch terms from 'row' to 'partition' in new metrics when
we mean 'partition'?
Also, should we include the count in flush-pending memtables?
More accurate row count estimates
-
Key: CASSANDRA-9107
URL: https://issues.apache.org/jira/browse/CASSANDRA-9107
Project: Cassandra
Issue Type: Improvement
Reporter: Chris Lohfink
Assignee: Chris Lohfink
Attachments: 9107-cassandra2-1.patch
Currently the estimated row count from cfstats is the sum of the number of
rows in all the sstables. This becomes very inaccurate with wide rows or
heavily updated datasets since the same partition would exist in many
sstables. In example:
{code}
create KEYSPACE test WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 1};
create TABLE wide (key text PRIMARY KEY , value text) WITH compaction =
{'class': 'SizeTieredCompactionStrategy', 'min_threshold': 30,
'max_threshold': 100} ;
---
insert INTO wide (key, value) VALUES ('key', 'value');
// flush
// cfstats output: Number of keys (estimate): 1 (128 in older version from
index)
insert INTO wide (key, value) VALUES ('key', 'value');
// flush
// cfstats output: Number of keys (estimate): 2 (256 in older version from
index)
... etc
{code}
previously it used the index but it still did it per sstable and summed them
up which became inaccurate as there are more sstables (just by much worse).
With new versions of sstables we can merge the cardinalities to resolve this
with a slight hit to accuracy in the case of every sstable having completely
unique partitions.
Furthermore I think it would be pretty minimal effort to include the number
of rows in the memtables to this count. We wont have the cardinality merging
between memtables and sstables but I would consider that a relatively minor
negative.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8717) Top-k queries with custom secondary indexes
[
https://issues.apache.org/jira/browse/CASSANDRA-8717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Yeschenko updated CASSANDRA-8717:
-
Reviewer: Sam Tunnicliffe (was: Aleksey Yeschenko)
Top-k queries with custom secondary indexes
---
Key: CASSANDRA-8717
URL: https://issues.apache.org/jira/browse/CASSANDRA-8717
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Andrés de la Peña
Assignee: Andrés de la Peña
Priority: Minor
Labels: 2i, secondary_index, sort, sorting, top-k
Fix For: 3.0
Attachments: 0001-Add-support-for-top-k-queries-in-2i.patch,
0002-Add-support-for-top-k-queries-in-2i.patch,
0003-Add-support-for-top-k-queries-in-2i.patch
As presented in [Cassandra Summit Europe
2014|https://www.youtube.com/watch?v=Hg5s-hXy_-M], secondary indexes can be
modified to support general top-k queries with minimum changes in Cassandra
codebase. This way, custom 2i implementations could provide relevance search,
sorting by columns, etc.
Top-k queries retrieve the k best results for a certain query. That implies
querying the k best rows in each token range and then sort them in order to
obtain the k globally best rows.
For doing that, we propose two additional methods in class
SecondaryIndexSearcher:
{code:java}
public boolean requiresFullScan(ListIndexExpression clause)
{
return false;
}
public ListRow sort(ListIndexExpression clause, ListRow rows)
{
return rows;
}
{code}
The first one indicates if a query performed in the index requires querying
all the nodes in the ring. It is necessary in top-k queries because we do not
know which node are the best results. The second method specifies how to sort
all the partial node results according to the query.
Then we add two similar methods to the class AbstractRangeCommand:
{code:java}
this.searcher =
Keyspace.open(keyspace).getColumnFamilyStore(columnFamily).indexManager.searcher(rowFilter);
public boolean requiresFullScan() {
return searcher == null ? false : searcher.requiresFullScan(rowFilter);
}
public ListRow combine(ListRow rows)
{
return searcher == null ? trim(rows) : trim(searcher.sort(rowFilter,
rows));
}
{code}
Finnally, we modify StorageProxy#getRangeSlice to use the previous method, as
shown in the attached patch.
We think that the proposed approach provides very useful functionality with
minimum impact in current codebase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7029) Investigate alternative transport protocols for both client and inter-server communications
[ https://issues.apache.org/jira/browse/CASSANDRA-7029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504846#comment-14504846 ] Robert Stupp commented on CASSANDRA-7029: - Also nice : Quic - https://docs.google.com/document/d/1RNHkx_VvKWyWg6Lr8SZ-saqsQx7rFV-ev2jRFUoVD34 (proposed successor of TCP for SPDY) Investigate alternative transport protocols for both client and inter-server communications --- Key: CASSANDRA-7029 URL: https://issues.apache.org/jira/browse/CASSANDRA-7029 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Benedict Labels: performance Fix For: 3.0 There are a number of reasons to think we can do better than TCP for our communications: 1) We can actually tolerate sporadic small message losses, so guaranteed delivery isn't essential (although for larger messages it probably is) 2) As shown in \[1\] and \[2\], Linux can behave quite suboptimally with regard to TCP message delivery when the system is under load. Judging from the theoretical description, this is likely to apply even when the system-load is not high, but the number of processes to schedule is high. Cassandra generally has a lot of threads to schedule, so this is quite pertinent for us. UDP performs substantially better here. 3) Even when the system is not under load, UDP has a lower CPU burden, and that burden is constant regardless of the number of connections it processes. 4) On a simple benchmark on my local PC, using non-blocking IO for UDP and busy spinning on IO I can actually push 20-40% more throughput through loopback (where TCP should be optimal, as no latency), even for very small messages. Since we can see networking taking multiple CPUs' worth of time during a stress test, using a busy-spin for ~100micros after last message receipt is almost certainly acceptable, especially as we can (ultimately) process inter-server and client communications on the same thread/socket in this model. 5) We can optimise the threading model heavily: since we generally process very small messages (200 bytes not at all implausible), the thread signalling costs on the processing thread can actually dramatically impede throughput. In general it costs ~10micros to signal (and passing the message to another thread for processing in the current model requires signalling). For 200-byte messages this caps our throughput at 20MB/s. I propose to knock up a highly naive UDP-based connection protocol with super-trivial congestion control over the course of a few days, with the only initial goal being maximum possible performance (not fairness, reliability, or anything else), and trial it in Netty (possibly making some changes to Netty to mitigate thread signalling costs). The reason for knocking up our own here is to get a ceiling on what the absolute limit of potential for this approach is. Assuming this pans out with performance gains in C* proper, we then look to contributing to/forking the udt-java project and see how easy it is to bring performance in line with what we can get with our naive approach (I don't suggest starting here, as the project is using blocking old-IO, and modifying it with latency in mind may be challenging, and we won't know for sure what the best case scenario is). \[1\] http://test-docdb.fnal.gov/0016/001648/002/Potential%20Performance%20Bottleneck%20in%20Linux%20TCP.PDF \[2\] http://cd-docdb.fnal.gov/cgi-bin/RetrieveFile?docid=1968;filename=Performance%20Analysis%20of%20Linux%20Networking%20-%20Packet%20Receiving%20(Official).pdf;version=2 Further related reading: http://public.dhe.ibm.com/software/commerce/doc/mft/cdunix/41/UDTWhitepaper.pdf https://mospace.umsystem.edu/xmlui/bitstream/handle/10355/14482/ChoiUndPerTcp.pdf?sequence=1 https://access.redhat.com/site/documentation/en-US/JBoss_Enterprise_Web_Platform/5/html/Administration_And_Configuration_Guide/jgroups-perf-udpbuffer.html http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.153.3762rep=rep1type=pdf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9202) Introduce CQL Error Codes
[
https://issues.apache.org/jira/browse/CASSANDRA-9202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504860#comment-14504860
]
Robert Stupp commented on CASSANDRA-9202:
-
Nice - just don't like the separation of code and documentation especially for
error codes.
Maybe we could use Java enums for that. This would enable us to get error codes
translated.
Could be possible in a (pseudo-code) structure like this:
{code}
class ErrorCodes
{
@ErrorCodeOffset(1000)
enum StorageErrorCodes
{
DISK_FULL ( The disk %1 is full, More error description using some
markup language )
}
@ErrorCodeOffset(2000)
enum SyntaxErrorCodes
{
UNKNOWN_TABLE (No such table %2 in keyspace %1, More error
description using some markup language)
}
}
{code}
Means, we provide the English text in Java code but are able to provide
translations in a different way.
Introduce CQL Error Codes
-
Key: CASSANDRA-9202
URL: https://issues.apache.org/jira/browse/CASSANDRA-9202
Project: Cassandra
Issue Type: Improvement
Reporter: Michaël Figuière
Assignee: Benjamin Lerer
As features have been added or modified over the past years, the constraints
applied on the CQL language itself have evolved rapidly, making it
challenging for users, but also for documentation writers, testers and
developers tools developers to keep up with what's possible or not, but also
to understand the meaning of each error message and the rule behind it.
Besides it, as error messages for any single error may change over time to
fix a typo or to rephrase it, they cannot be used as a stable reference of
errors.
It feels like the right time to make error handling more formal in Cassandra,
by introducing a very classic mechanism in the database world: error codes.
The purpose of this ticket is to introduce these codes, but *does not* cover
the way they should then be exposed. Bellow is my proposition along with the
strategy I used:
*Gathering Cassandra errors*
I've walked through the source of all the past releases of Cassandra its git
repo using a script in order to capture all the CQL related
{{InvalidRequestException}} that are thrown. Considering it represents most
of the CQL errors that may be returned by Cassandra.
Bellow is the list of all errors that have been introduced and modified
between Cassandra 1.2.0 and the current trunk:
https://gist.github.com/mfiguiere/3036c2a54af016bbeb58
The complete list of CQL errors declared in each Cassandra source file, along
with the range of versions in which they appeared is as follow:
https://gist.github.com/mfiguiere/42166586647c34b1a41c
That's a lot of them... Clearly we can only focus on Cassandra 3.0+ here,
that is on the current trunk.
*Categorizing errors*
It's common for database to categorize errors in order to make it simpler for
the user to understand its nature or to walk through a list of them:
* PostgreSQL
(http://www.postgresql.org/docs/9.4/static/errcodes-appendix.html)
* MySQL (http://dev.mysql.com/doc/refman/5.6/en/error-messages-server.html)
* Oracle (http://docs.oracle.com/cd/B28359_01/server.111/b28278/toc.htm)
One issue that can be observed in these 3 examples is that the codes they use
are fairly cryptic and not really readable.
It felt to me that a categorization by feature would be helpful for
Cassandra. And rather than building hexadecimal prefixes, let's use readable
string ones. We then end up with the following list of CQL error codes:
https://gist.github.com/mfiguiere/7a19f8368b3ab4fbef3a
That's about 260 errors overall for the current trunk, but broken into
categories it feels to me that it remains very easy to browse and review.
*Native Protocol Error Codes vs. CQL Error Codes*
We actually already introduced the concept of error codes into the Native
Protocol specification. These codes represent execution exceptions and are
used by the clients to understand the kind of problem that occurred.
I propose to rename these error codes to Execution Error Codes and to
introduce with this ticket CQL Error Codes, as they address two different
kind of issues and audiences.
*Introducing CQL Error Codes*
Once an agreement will be reached on the list of error the strategy to
introduce them into the codebase would be as follow:
1. We have to wait until CASSANDRA-8099 is merged, as it'll significantly
change the way Exceptions are manipulated in the CQL codebase.
2. We introduce a {{cql_errors.spec}} file that defines the list of all CQL
errors that may be thrown by Cassandra.
3. We modify the sources to introduce the appropriate cqlErrorCode along with
any error that is declared.
4. Once merged, any subsequent addition or modification of an error in the
sources in the future should
cassandra git commit: Update collection types when nested UDT is altered
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1 7f29201ce - bdef07243
Update collection types when nested UDT is altered
Patch by Jeff Jirsa; reviewed by Benjamin Lerer for CASSANDRA-9148
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bdef0724
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bdef0724
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bdef0724
Branch: refs/heads/cassandra-2.1
Commit: bdef0724352423191aaad141de5102b886d50ae1
Parents: 7f29201
Author: Jeff Jirsa j...@jeffjirsa.net
Authored: Tue Apr 21 11:05:28 2015 -0500
Committer: Tyler Hobbs tylerlho...@gmail.com
Committed: Tue Apr 21 11:05:28 2015 -0500
--
CHANGES.txt | 2 +
.../cql3/statements/AlterTypeStatement.java | 9 +-
.../cassandra/db/composites/CellNames.java | 1 +
.../org/apache/cassandra/cql3/CQLTester.java| 5 +
.../apache/cassandra/cql3/UserTypesTest.java| 111 +++
5 files changed, 127 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/bdef0724/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index e3428a7..ff25db6 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,6 @@
2.1.5
+ * Update collection types that use a user-defined type when that UDT is
+ modified (CASSANDRA-9148)
* Re-add deprecated cold_reads_to_omit param for backwards compat
(CASSANDRA-9203)
* Make anticompaction visible in compactionstats (CASSANDRA-9098)
* Improve nodetool getendpoints documentation about the partition
http://git-wip-us.apache.org/repos/asf/cassandra/blob/bdef0724/src/java/org/apache/cassandra/cql3/statements/AlterTypeStatement.java
--
diff --git
a/src/java/org/apache/cassandra/cql3/statements/AlterTypeStatement.java
b/src/java/org/apache/cassandra/cql3/statements/AlterTypeStatement.java
index 64e7627..2dcf8dc 100644
--- a/src/java/org/apache/cassandra/cql3/statements/AlterTypeStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/AlterTypeStatement.java
@@ -163,7 +163,14 @@ public abstract class AlterTypeStatement extends
SchemaAlteringStatement
default:
// If it's a collection, we still want to modify the
comparator because the collection is aliased in it
if (def.type instanceof CollectionType)
-cfm.comparator =
CellNames.fromAbstractType(updateWith(cfm.comparator.asAbstractType(),
keyspace, toReplace, updated), cfm.comparator.isDense());
+{
+t = updateWith(cfm.comparator.asAbstractType(), keyspace,
toReplace, updated);
+// If t == null, all relevant comparators were updated via
updateWith, which reaches into types and
+// collections
+if (t != null)
+cfm.comparator = CellNames.fromAbstractType(t,
cfm.comparator.isDense());
+}
+break;
}
return true;
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/bdef0724/src/java/org/apache/cassandra/db/composites/CellNames.java
--
diff --git a/src/java/org/apache/cassandra/db/composites/CellNames.java
b/src/java/org/apache/cassandra/db/composites/CellNames.java
index b941166..f73f7a7 100644
--- a/src/java/org/apache/cassandra/db/composites/CellNames.java
+++ b/src/java/org/apache/cassandra/db/composites/CellNames.java
@@ -63,6 +63,7 @@ public abstract class CellNames
}
else
{
+assert type != null;
return new SimpleSparseCellNameType(type);
}
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/bdef0724/test/unit/org/apache/cassandra/cql3/CQLTester.java
--
diff --git a/test/unit/org/apache/cassandra/cql3/CQLTester.java
b/test/unit/org/apache/cassandra/cql3/CQLTester.java
index e994f01..8b24e69 100644
--- a/test/unit/org/apache/cassandra/cql3/CQLTester.java
+++ b/test/unit/org/apache/cassandra/cql3/CQLTester.java
@@ -675,6 +675,11 @@ public abstract class CQLTester
return new TupleValue(values);
}
+protected Object userType(Object... values)
+{
+return new TupleValue(values).toByteBuffer();
+}
+
protected Object list(Object...values)
{
return Arrays.asList(values);
http://git-wip-us.apache.org/repos/asf/cassandra/blob/bdef0724/test/unit/org/apache/cassandra/cql3/UserTypesTest.java
[1/2] cassandra git commit: Update collection types when nested UDT is altered
Repository: cassandra
Updated Branches:
refs/heads/trunk 727c5c832 - 5f663f540
Update collection types when nested UDT is altered
Patch by Jeff Jirsa; reviewed by Benjamin Lerer for CASSANDRA-9148
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bdef0724
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bdef0724
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bdef0724
Branch: refs/heads/trunk
Commit: bdef0724352423191aaad141de5102b886d50ae1
Parents: 7f29201
Author: Jeff Jirsa j...@jeffjirsa.net
Authored: Tue Apr 21 11:05:28 2015 -0500
Committer: Tyler Hobbs tylerlho...@gmail.com
Committed: Tue Apr 21 11:05:28 2015 -0500
--
CHANGES.txt | 2 +
.../cql3/statements/AlterTypeStatement.java | 9 +-
.../cassandra/db/composites/CellNames.java | 1 +
.../org/apache/cassandra/cql3/CQLTester.java| 5 +
.../apache/cassandra/cql3/UserTypesTest.java| 111 +++
5 files changed, 127 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/bdef0724/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index e3428a7..ff25db6 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,6 @@
2.1.5
+ * Update collection types that use a user-defined type when that UDT is
+ modified (CASSANDRA-9148)
* Re-add deprecated cold_reads_to_omit param for backwards compat
(CASSANDRA-9203)
* Make anticompaction visible in compactionstats (CASSANDRA-9098)
* Improve nodetool getendpoints documentation about the partition
http://git-wip-us.apache.org/repos/asf/cassandra/blob/bdef0724/src/java/org/apache/cassandra/cql3/statements/AlterTypeStatement.java
--
diff --git
a/src/java/org/apache/cassandra/cql3/statements/AlterTypeStatement.java
b/src/java/org/apache/cassandra/cql3/statements/AlterTypeStatement.java
index 64e7627..2dcf8dc 100644
--- a/src/java/org/apache/cassandra/cql3/statements/AlterTypeStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/AlterTypeStatement.java
@@ -163,7 +163,14 @@ public abstract class AlterTypeStatement extends
SchemaAlteringStatement
default:
// If it's a collection, we still want to modify the
comparator because the collection is aliased in it
if (def.type instanceof CollectionType)
-cfm.comparator =
CellNames.fromAbstractType(updateWith(cfm.comparator.asAbstractType(),
keyspace, toReplace, updated), cfm.comparator.isDense());
+{
+t = updateWith(cfm.comparator.asAbstractType(), keyspace,
toReplace, updated);
+// If t == null, all relevant comparators were updated via
updateWith, which reaches into types and
+// collections
+if (t != null)
+cfm.comparator = CellNames.fromAbstractType(t,
cfm.comparator.isDense());
+}
+break;
}
return true;
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/bdef0724/src/java/org/apache/cassandra/db/composites/CellNames.java
--
diff --git a/src/java/org/apache/cassandra/db/composites/CellNames.java
b/src/java/org/apache/cassandra/db/composites/CellNames.java
index b941166..f73f7a7 100644
--- a/src/java/org/apache/cassandra/db/composites/CellNames.java
+++ b/src/java/org/apache/cassandra/db/composites/CellNames.java
@@ -63,6 +63,7 @@ public abstract class CellNames
}
else
{
+assert type != null;
return new SimpleSparseCellNameType(type);
}
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/bdef0724/test/unit/org/apache/cassandra/cql3/CQLTester.java
--
diff --git a/test/unit/org/apache/cassandra/cql3/CQLTester.java
b/test/unit/org/apache/cassandra/cql3/CQLTester.java
index e994f01..8b24e69 100644
--- a/test/unit/org/apache/cassandra/cql3/CQLTester.java
+++ b/test/unit/org/apache/cassandra/cql3/CQLTester.java
@@ -675,6 +675,11 @@ public abstract class CQLTester
return new TupleValue(values);
}
+protected Object userType(Object... values)
+{
+return new TupleValue(values).toByteBuffer();
+}
+
protected Object list(Object...values)
{
return Arrays.asList(values);
http://git-wip-us.apache.org/repos/asf/cassandra/blob/bdef0724/test/unit/org/apache/cassandra/cql3/UserTypesTest.java
[jira] [Commented] (CASSANDRA-7225) cqlsh help for CQL3 is often incorrect and should be modernized
[ https://issues.apache.org/jira/browse/CASSANDRA-7225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505669#comment-14505669 ] Robert Stupp commented on CASSANDRA-7225: - Yes, shipping the HTML files (one for CQL.textile and a new one for cqlsh.textile) with C*. Putting these on a web server might not work for everybody (if they have no public internet access from their servers). Maybe there's some work to do to harmonize the anchors in the textile documents, but that's ”just work”. cqlsh help for CQL3 is often incorrect and should be modernized --- Key: CASSANDRA-7225 URL: https://issues.apache.org/jira/browse/CASSANDRA-7225 Project: Cassandra Issue Type: Bug Components: Documentation website, Tools Reporter: Robert Stupp Assignee: Robert Stupp Priority: Trivial Labels: cql Attachments: 7225-cqlhelp.txt, EXPAND.pdf Just a small line of text in cqlsh help command indicates that is = and is = in CQL. This is confusing to many people (including me :) ) because I did not expect to return the equals portion. Please allow distinct behaviours for , =, and = in CQL queries. Maybe in combination with CASSANDRA-5184 and/or CASSANDRA-4914 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9217) Problems with cqlsh copy command
[
https://issues.apache.org/jira/browse/CASSANDRA-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tyler Hobbs updated CASSANDRA-9217:
---
Attachment: 9217-2.1.txt
The attached patch fixes COPY TO (and fixes a couple of minor unrelated cqlsh
test failures). I've also started some basic dtests for COPY here:
https://github.com/riptano/cassandra-dtest/commit/16c75a6f6c5fa6a1cd765775e185d407af373f82
Problems with cqlsh copy command
Key: CASSANDRA-9217
URL: https://issues.apache.org/jira/browse/CASSANDRA-9217
Project: Cassandra
Issue Type: Bug
Reporter: Brian Cantoni
Assignee: Tyler Hobbs
Labels: cqlsh
Fix For: 2.1.5
Attachments: 9217-2.1.txt
On the current 2.1 branch I notice a few (possibly related) problems with
cqlsh copy commands. I'm writing them here together but we can separate if
there are different causes.
*1. Cannot import from CSV if column name is 'date'*
Test file monthly.csv contents:
{noformat}
stationid,metric,date
LAE,barometricpressure,2014-01-01 00:00:00+
LAE,barometricpressure,2014-02-01 00:00:00+
LAE,barometricpressure,2014-03-01 00:00:00+
{noformat}
Steps:
{noformat}
CREATE KEYSPACE IF NOT EXISTS weathercql WITH replication = { 'class':
'SimpleStrategy', 'replication_factor': '1' };
CREATE TABLE IF NOT EXISTS weathercql.monthly (
stationid text,
metric text,
date timestamp,
primary key (stationid, metric, date)
);
COPY weathercql.monthly (stationid, metric, date) FROM 'monthly.csv' WITH
HEADER='true';
{noformat}
Result: the copy command fails unless date is enclosed in double quotes:
{noformat}
cqlsh COPY weathercql.monthly (stationid, metric, date) FROM 'monthly.csv'
WITH HEADER='true';
Improper COPY command.
cqlsh COPY weathercql.monthly (stationid, metric, date) FROM 'monthly.csv'
WITH HEADER='true';
3 rows imported in 0.096 seconds.
{noformat}
If I instead name the 'date' column as 'datex', it works without quotes.
The same steps work on Cassandra 2.1.4 (release build).
*2. Cannot copy to CSV*
Sample data:
{noformat}
create keyspace if not exists test with replication =
{'class':'SimpleStrategy', 'replication_factor':1};
create table if not exists test.kv (key int primary key, value text);
insert into test.kv (key,value) values (1,'alpha');
insert into test.kv (key,value) values (2,'beta');
insert into test.kv (key,value) values (3,'charlie');
{noformat}
When you try to export to CSV, it throws what appears to be a Python error,
and the file is not created correctly:
{noformat}
cqlsh copy test.kv (key,value) to 'test.csv';
global name 'meter' is not defined
{noformat}
The same steps work on Cassandra 2.1.4 (release build).
*3. Copy from CSV inside CQL command file doesn't work*
File kv.csv:
{noformat}
key,value
1,'a'
2,'b'
3,'c'
{noformat}
File kv.cql:
{noformat}
create keyspace if not exists test with replication = {'class':
'SimpleStrategy', 'replication_factor':1};
create table if not exists test.kv (key int primary key, value text);
truncate test.kv;
copy test.kv (key, value) from 'kv.csv' with header='true';
select * from test.kv;
{noformat}
When command file passed to cqlsh, an error is reported on the `copy` command
and it doesn't work:
{noformat}
$ bin/cqlsh -f kv.cql
kv.cql:5:descriptor 'lower' requires a 'str' object but received a 'unicode'
key | value
-+---
(0 rows)
{noformat}
The same commands work correctly when run directly inside cqlsh or when
executed with -e option like: {{bin/cqlsh -e copy test.kv (key, value) from
'kv.csv' with header='true';}}.
This third issue appears to also be broken in 2.1.4 and 2.1.3 release builds,
but works in 2.1.2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9084) Do not generate line number in logs
[
https://issues.apache.org/jira/browse/CASSANDRA-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505635#comment-14505635
]
Jeremiah Jordan commented on CASSANDRA-9084:
This is a bad idea. We don't log in the read or write paths. And line numbers
can be very useful. We should revert this.
Do not generate line number in logs
---
Key: CASSANDRA-9084
URL: https://issues.apache.org/jira/browse/CASSANDRA-9084
Project: Cassandra
Issue Type: Improvement
Components: Config
Reporter: Andrey
Assignee: Benedict
Priority: Minor
According to logback documentation
(http://logback.qos.ch/manual/layouts.html):
{code}
Generating the line number information is not particularly fast. Thus, its
use should be avoided unless execution speed is not an issue.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9217) Error in cqlsh COPY TO
[
https://issues.apache.org/jira/browse/CASSANDRA-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tyler Hobbs updated CASSANDRA-9217:
---
Summary: Error in cqlsh COPY TO (was: Problems with cqlsh copy command)
Error in cqlsh COPY TO
--
Key: CASSANDRA-9217
URL: https://issues.apache.org/jira/browse/CASSANDRA-9217
Project: Cassandra
Issue Type: Bug
Reporter: Brian Cantoni
Assignee: Tyler Hobbs
Labels: cqlsh
Fix For: 2.1.5
Attachments: 9217-2.1.txt
On the current 2.1 branch I notice a few (possibly related) problems with
cqlsh copy commands. I'm writing them here together but we can separate if
there are different causes.
*1. Cannot import from CSV if column name is 'date'*
Test file monthly.csv contents:
{noformat}
stationid,metric,date
LAE,barometricpressure,2014-01-01 00:00:00+
LAE,barometricpressure,2014-02-01 00:00:00+
LAE,barometricpressure,2014-03-01 00:00:00+
{noformat}
Steps:
{noformat}
CREATE KEYSPACE IF NOT EXISTS weathercql WITH replication = { 'class':
'SimpleStrategy', 'replication_factor': '1' };
CREATE TABLE IF NOT EXISTS weathercql.monthly (
stationid text,
metric text,
date timestamp,
primary key (stationid, metric, date)
);
COPY weathercql.monthly (stationid, metric, date) FROM 'monthly.csv' WITH
HEADER='true';
{noformat}
Result: the copy command fails unless date is enclosed in double quotes:
{noformat}
cqlsh COPY weathercql.monthly (stationid, metric, date) FROM 'monthly.csv'
WITH HEADER='true';
Improper COPY command.
cqlsh COPY weathercql.monthly (stationid, metric, date) FROM 'monthly.csv'
WITH HEADER='true';
3 rows imported in 0.096 seconds.
{noformat}
If I instead name the 'date' column as 'datex', it works without quotes.
The same steps work on Cassandra 2.1.4 (release build).
*2. Cannot copy to CSV*
Sample data:
{noformat}
create keyspace if not exists test with replication =
{'class':'SimpleStrategy', 'replication_factor':1};
create table if not exists test.kv (key int primary key, value text);
insert into test.kv (key,value) values (1,'alpha');
insert into test.kv (key,value) values (2,'beta');
insert into test.kv (key,value) values (3,'charlie');
{noformat}
When you try to export to CSV, it throws what appears to be a Python error,
and the file is not created correctly:
{noformat}
cqlsh copy test.kv (key,value) to 'test.csv';
global name 'meter' is not defined
{noformat}
The same steps work on Cassandra 2.1.4 (release build).
*3. Copy from CSV inside CQL command file doesn't work*
File kv.csv:
{noformat}
key,value
1,'a'
2,'b'
3,'c'
{noformat}
File kv.cql:
{noformat}
create keyspace if not exists test with replication = {'class':
'SimpleStrategy', 'replication_factor':1};
create table if not exists test.kv (key int primary key, value text);
truncate test.kv;
copy test.kv (key, value) from 'kv.csv' with header='true';
select * from test.kv;
{noformat}
When command file passed to cqlsh, an error is reported on the `copy` command
and it doesn't work:
{noformat}
$ bin/cqlsh -f kv.cql
kv.cql:5:descriptor 'lower' requires a 'str' object but received a 'unicode'
key | value
-+---
(0 rows)
{noformat}
The same commands work correctly when run directly inside cqlsh or when
executed with -e option like: {{bin/cqlsh -e copy test.kv (key, value) from
'kv.csv' with header='true';}}.
This third issue appears to also be broken in 2.1.4 and 2.1.3 release builds,
but works in 2.1.2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9084) Do not generate line number in logs
[
https://issues.apache.org/jira/browse/CASSANDRA-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505928#comment-14505928
]
Brandon Williams commented on CASSANDRA-9084:
-
The simplest thing to do here is revert, unless someone can provide empirical
evidence that this optimization is worthwhile. And as Jeremiah mentioned, the
user can always remove the line numbers themselves.
Do not generate line number in logs
---
Key: CASSANDRA-9084
URL: https://issues.apache.org/jira/browse/CASSANDRA-9084
Project: Cassandra
Issue Type: Improvement
Components: Config
Reporter: Andrey
Assignee: Benedict
Priority: Minor
According to logback documentation
(http://logback.qos.ch/manual/layouts.html):
{code}
Generating the line number information is not particularly fast. Thus, its
use should be avoided unless execution speed is not an issue.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9084) Do not generate line number in logs
[
https://issues.apache.org/jira/browse/CASSANDRA-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505660#comment-14505660
]
Brandon Williams commented on CASSANDRA-9084:
-
Having just been bitten by this while debugging something that involves a
somewhat complicated state of the cluster to setup, I too am in favor of
reverting this. Whatever we're saving (if we're even saving anything at all)
at production run time isn't worth what we're losing in development time.
Do not generate line number in logs
---
Key: CASSANDRA-9084
URL: https://issues.apache.org/jira/browse/CASSANDRA-9084
Project: Cassandra
Issue Type: Improvement
Components: Config
Reporter: Andrey
Assignee: Benedict
Priority: Minor
According to logback documentation
(http://logback.qos.ch/manual/layouts.html):
{code}
Generating the line number information is not particularly fast. Thus, its
use should be avoided unless execution speed is not an issue.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-9084) Do not generate line number in logs
[
https://issues.apache.org/jira/browse/CASSANDRA-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505635#comment-14505635
]
Jeremiah Jordan edited comment on CASSANDRA-9084 at 4/21/15 8:04 PM:
-
This is a bad idea. We don't log in the read or write paths. And line numbers
can be very useful. We should revert this.
If someone really cares they can turn it off themselves. If you really care
put a commented out version in the log file with them off, and a comment
telling a user why they might want that. We have enough log messages that are
the same, that line numbers are useful.
was (Author: jjordan):
This is a bad idea. We don't log in the read or write paths. And line numbers
can be very useful. We should revert this.
Do not generate line number in logs
---
Key: CASSANDRA-9084
URL: https://issues.apache.org/jira/browse/CASSANDRA-9084
Project: Cassandra
Issue Type: Improvement
Components: Config
Reporter: Andrey
Assignee: Benedict
Priority: Minor
According to logback documentation
(http://logback.qos.ch/manual/layouts.html):
{code}
Generating the line number information is not particularly fast. Thus, its
use should be avoided unless execution speed is not an issue.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8051) Add SERIAL and LOCAL_SERIAL consistency levels to cqlsh
[ https://issues.apache.org/jira/browse/CASSANDRA-8051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tyler Hobbs updated CASSANDRA-8051: --- Reviewer: Stefania (was: Tyler Hobbs) Add SERIAL and LOCAL_SERIAL consistency levels to cqlsh --- Key: CASSANDRA-8051 URL: https://issues.apache.org/jira/browse/CASSANDRA-8051 Project: Cassandra Issue Type: Bug Components: Tools Reporter: Nicolas Favre-Felix Assignee: Carl Yeksigian Priority: Minor Labels: cqlsh Fix For: 2.0.15 Attachments: 8051-2.0.txt cqlsh does not support setting the serial consistency level. The default CL.SERIAL does not let users safely execute LWT alongside an app that runs at LOCAL_SERIAL, and can prevent any LWT from running when a DC is down (e.g. with 2 DCs, RF=3 in each.) Implementing this well is a bit tricky. A user setting the serial CL will probably not want all of their statements to have a serial CL attached, but only the conditional updates. At the same time it would be useful to support serial reads. WITH CONSISTENCY LEVEL used to provide this flexibility. I believe that it is currently impossible to run a SELECT at SERIAL or LOCAL_SERIAL; the only workaround seems to be to run a conditional update with a predicate that always resolves to False, and to rely on the CAS response to read the data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9084) Do not generate line number in logs
[
https://issues.apache.org/jira/browse/CASSANDRA-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505889#comment-14505889
]
Andrey commented on CASSANDRA-9084:
---
Let's setup different logging configuration for development with proper DEBUG
and TRACE appenders if it is required.
The summarized rationale:
* normally it is very easy to find where log message was generated based on
class name and message itself
* when it comes to deep debugging in production I would assume increasing log
level to DEBUG or TRACE rather than add line numbers.
* it's hard to always control whether new log message fits into critical path.
Critical paths might include: compaction stats/GC logs/repair etc.
* I will never know in advance on what hardware and under what load cassandra
will be running. So I would assume create default configuration as performant
as possible. Even it's not on critical path.
Do not generate line number in logs
---
Key: CASSANDRA-9084
URL: https://issues.apache.org/jira/browse/CASSANDRA-9084
Project: Cassandra
Issue Type: Improvement
Components: Config
Reporter: Andrey
Assignee: Benedict
Priority: Minor
According to logback documentation
(http://logback.qos.ch/manual/layouts.html):
{code}
Generating the line number information is not particularly fast. Thus, its
use should be avoided unless execution speed is not an issue.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9194) Delete-only workloads crash Cassandra
[
https://issues.apache.org/jira/browse/CASSANDRA-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505760#comment-14505760
]
Jonathan Ellis commented on CASSANDRA-9194:
---
I figured that in 2.0 it will self-correct via liveRatio, so it doesn't really
matter what constant we use.
Delete-only workloads crash Cassandra
-
Key: CASSANDRA-9194
URL: https://issues.apache.org/jira/browse/CASSANDRA-9194
Project: Cassandra
Issue Type: Bug
Environment: 2.0.14
Reporter: Robert Wille
Assignee: Benedict
Fix For: 2.0.15
Attachments: 9194.txt
The size of a tombstone is not properly accounted for in the memtable. A
memtable which has only tombstones will never get flushed. It will grow until
the JVM runs out of memory. The following program easily demonstrates the
problem.
{code}
Cluster.Builder builder = Cluster.builder();
Cluster c =
builder.addContactPoints(cas121.devf3.com).build();
Session s = c.connect();
s.execute(CREATE KEYSPACE IF NOT EXISTS test WITH replication
= { 'class' : 'SimpleStrategy', 'replication_factor' : 1 });
s.execute(CREATE TABLE IF NOT EXISTS test.test(id INT PRIMARY
KEY));
PreparedStatement stmt = s.prepare(DELETE FROM test.test WHERE
id = :id);
int id = 0;
while (true)
{
s.execute(stmt.bind(id));
id++;
}{code}
This program should run forever, but eventually Cassandra runs out of heap
and craps out. You needn't wait for Cassandra to crash. If you run nodetool
cfstats test.test while it is running, you'll see Memtable cell count grow,
but Memtable data size will remain 0.
This issue was fixed once before. I received a patch for version 2.0.5 (I
believe), which contained the fix, but the fix has apparently been lost,
because it is clearly broken, and I don't see the fix in the change logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7168) Add repair aware consistency levels
[ https://issues.apache.org/jira/browse/CASSANDRA-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505783#comment-14505783 ] sankalp kohli commented on CASSANDRA-7168: -- cc [~krummas] I agree with [~slebresne] that we first need to make sure last repair time is consistent across replicas CASSANDRA-9143. There is lot of overlap here between this ticket and CASSANDRA-6434 but I chose this ticket to comment since there is lot of discussions here :). CASSANDRA-6434 will only drop tombstones from the repaired data. The problem with this is that if repair time could not be sent to one replica with CASSANDRA-9143, it will not drop tombstone for the data which other replicas will. Now during a normal read or repair consistency read, this replica which did not get the repair time will include some tombstones which other replicas won't. This is due to different view of what is repaired and what is not. This will cause digest mismatch leading to spike in latency. We also cannot use Benedict approach of finding the last common repair time since replicas which are ahead would have compacted there tombstones leading to the same problem of digest mismatch. I think we need to do CASSANDRA-9143 and also only drop tombstones when we are sure all replicas has that repair time. Also during the time when replicas are getting the message that these set of stables are repaired and don't include the tombstones from them in read and start dropping tombstones if eligible, this is not going to be done at same time across replicas. This will cause digest mismatch during this time which is not ideal. I have not yet thought through how this could be avoided. Add repair aware consistency levels --- Key: CASSANDRA-7168 URL: https://issues.apache.org/jira/browse/CASSANDRA-7168 Project: Cassandra Issue Type: Improvement Components: Core Reporter: T Jake Luciani Labels: performance Fix For: 3.1 With CASSANDRA-5351 and CASSANDRA-2424 I think there is an opportunity to avoid a lot of extra disk I/O when running queries with higher consistency levels. Since repaired data is by definition consistent and we know which sstables are repaired, we can optimize the read path by having a REPAIRED_QUORUM which breaks reads into two phases: 1) Read from one replica the result from the repaired sstables. 2) Read from a quorum only the un-repaired data. For the node performing 1) we can pipeline the call so it's a single hop. In the long run (assuming data is repaired regularly) we will end up with much closer to CL.ONE performance while maintaining consistency. Some things to figure out: - If repairs fail on some nodes we can have a situation where we don't have a consistent repaired state across the replicas. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (CASSANDRA-9084) Do not generate line number in logs
[
https://issues.apache.org/jira/browse/CASSANDRA-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams reopened CASSANDRA-9084:
-
Do not generate line number in logs
---
Key: CASSANDRA-9084
URL: https://issues.apache.org/jira/browse/CASSANDRA-9084
Project: Cassandra
Issue Type: Improvement
Components: Config
Reporter: Andrey
Assignee: Benedict
Priority: Minor
According to logback documentation
(http://logback.qos.ch/manual/layouts.html):
{code}
Generating the line number information is not particularly fast. Thus, its
use should be avoided unless execution speed is not an issue.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-9224) Figure out a better default float precision rule for cqlsh
Tyler Hobbs created CASSANDRA-9224:
--
Summary: Figure out a better default float precision rule for cqlsh
Key: CASSANDRA-9224
URL: https://issues.apache.org/jira/browse/CASSANDRA-9224
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Tyler Hobbs
Assignee: Stefania
Fix For: 3.0, 2.1.5
We currently use a {{DEFAULT_FLOAT_PRECISION}} of 5 in cqlsh with formatting
{{'%.*g' % (float_precision, val)}}. In practice, this is way too low. For
example, 12345.5 will show up as 123456. Since the float precision is used for
cqlsh's COPY TO, it's particularly important that we maintain as much precision
as is practical by default.
There are some other tricky considerations, though. If the precision is too
high, python will do something like this:
{noformat}
'%.25g' % (12345.,)
'12345.555474711582'
{noformat}
That's not terrible, but it would be nice to avoid if we can.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7168) Add repair aware consistency levels
[ https://issues.apache.org/jira/browse/CASSANDRA-7168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505750#comment-14505750 ] Jonathan Ellis commented on CASSANDRA-7168: --- It's been there forever in what I'm convinced was a premature optimization in the first place. So sure, let's doublecheck but I don't think the bar for proof should be unreasonably high. Add repair aware consistency levels --- Key: CASSANDRA-7168 URL: https://issues.apache.org/jira/browse/CASSANDRA-7168 Project: Cassandra Issue Type: Improvement Components: Core Reporter: T Jake Luciani Labels: performance Fix For: 3.1 With CASSANDRA-5351 and CASSANDRA-2424 I think there is an opportunity to avoid a lot of extra disk I/O when running queries with higher consistency levels. Since repaired data is by definition consistent and we know which sstables are repaired, we can optimize the read path by having a REPAIRED_QUORUM which breaks reads into two phases: 1) Read from one replica the result from the repaired sstables. 2) Read from a quorum only the un-repaired data. For the node performing 1) we can pipeline the call so it's a single hop. In the long run (assuming data is repaired regularly) we will end up with much closer to CL.ONE performance while maintaining consistency. Some things to figure out: - If repairs fail on some nodes we can have a situation where we don't have a consistent repaired state across the replicas. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7225) cqlsh help for CQL3 is often incorrect and should be modernized
[ https://issues.apache.org/jira/browse/CASSANDRA-7225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505780#comment-14505780 ] Tyler Hobbs commented on CASSANDRA-7225: I think we can reasonably assume that devs and operators can access the CQL3 docs from their own machines, though. (If they can't, then they can download them ahead of time, I suppose.) Do we really need to make the help topics for cqlsh an HTML document? It seems like it would be simplest to keep them where they're at. cqlsh help for CQL3 is often incorrect and should be modernized --- Key: CASSANDRA-7225 URL: https://issues.apache.org/jira/browse/CASSANDRA-7225 Project: Cassandra Issue Type: Bug Components: Documentation website, Tools Reporter: Robert Stupp Assignee: Robert Stupp Priority: Trivial Labels: cql Attachments: 7225-cqlhelp.txt, EXPAND.pdf Just a small line of text in cqlsh help command indicates that is = and is = in CQL. This is confusing to many people (including me :) ) because I did not expect to return the equals portion. Please allow distinct behaviours for , =, and = in CQL queries. Maybe in combination with CASSANDRA-5184 and/or CASSANDRA-4914 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9194) Delete-only workloads crash Cassandra
[
https://issues.apache.org/jira/browse/CASSANDRA-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505980#comment-14505980
]
Jim Witschey commented on CASSANDRA-9194:
-
[~benedict] I think I understand now; thanks for the explanation.
I've [changed the
test|https://github.com/mambocab/cassandra-dtest/commit/25ee5b7050e96a85cd4e33eadc41a21cec7da393]
so that it checks {{MemtableOnHeapSize}} for versions = 2.1 and
{{MemtableDataSize}} for 2.0. As you indicated, it passes on 2.1.4 and trunk.
It currently fails on 2.0.4.
I can't seem to apply your patch to 2.0; it looks like it was written for 2.1?
But I'm +1 on the same logic applied to 2.0 if it passes the dtest.
Delete-only workloads crash Cassandra
-
Key: CASSANDRA-9194
URL: https://issues.apache.org/jira/browse/CASSANDRA-9194
Project: Cassandra
Issue Type: Bug
Environment: 2.0.14
Reporter: Robert Wille
Assignee: Benedict
Fix For: 2.0.15
Attachments: 9194.txt
The size of a tombstone is not properly accounted for in the memtable. A
memtable which has only tombstones will never get flushed. It will grow until
the JVM runs out of memory. The following program easily demonstrates the
problem.
{code}
Cluster.Builder builder = Cluster.builder();
Cluster c =
builder.addContactPoints(cas121.devf3.com).build();
Session s = c.connect();
s.execute(CREATE KEYSPACE IF NOT EXISTS test WITH replication
= { 'class' : 'SimpleStrategy', 'replication_factor' : 1 });
s.execute(CREATE TABLE IF NOT EXISTS test.test(id INT PRIMARY
KEY));
PreparedStatement stmt = s.prepare(DELETE FROM test.test WHERE
id = :id);
int id = 0;
while (true)
{
s.execute(stmt.bind(id));
id++;
}{code}
This program should run forever, but eventually Cassandra runs out of heap
and craps out. You needn't wait for Cassandra to crash. If you run nodetool
cfstats test.test while it is running, you'll see Memtable cell count grow,
but Memtable data size will remain 0.
This issue was fixed once before. I received a patch for version 2.0.5 (I
believe), which contained the fix, but the fix has apparently been lost,
because it is clearly broken, and I don't see the fix in the change logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
cassandra git commit: simplify
Repository: cassandra
Updated Branches:
refs/heads/trunk 10ff8c9f8 - 5e68fcec4
simplify
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5e68fcec
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5e68fcec
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5e68fcec
Branch: refs/heads/trunk
Commit: 5e68fcec4329c4751789290059891c03b99be4a6
Parents: 10ff8c9
Author: Dave Brosius dbros...@mebigfatguy.com
Authored: Tue Apr 21 22:36:31 2015 -0400
Committer: Dave Brosius dbros...@mebigfatguy.com
Committed: Tue Apr 21 22:36:31 2015 -0400
--
.../org/apache/cassandra/cql3/statements/AlterTableStatement.java | 2 +-
src/java/org/apache/cassandra/cql3/statements/BatchStatement.java | 2 +-
src/java/org/apache/cassandra/db/compaction/CompactionManager.java | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5e68fcec/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
--
diff --git
a/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
b/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
index 7fdaf81..db4c8dc 100644
--- a/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/AlterTableStatement.java
@@ -260,7 +260,7 @@ public class AlterTableStatement extends
SchemaAlteringStatement
break;
case OPTS:
if (cfProps == null)
-throw new InvalidRequestException(String.format(ALTER
TABLE WITH invoked, but no parameters found));
+throw new InvalidRequestException(ALTER TABLE WITH
invoked, but no parameters found);
cfProps.validate();
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5e68fcec/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
--
diff --git a/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
b/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
index 3f873e7..27a6ffb 100644
--- a/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/BatchStatement.java
@@ -264,7 +264,7 @@ public class BatchStatement implements CQLStatement
{
Tracing.trace(format, ksCfPairs, size, failThreshold, size -
failThreshold, (see batch_size_fail_threshold_in_kb));
logger.error(format, ksCfPairs, size, failThreshold, size -
failThreshold, (see batch_size_fail_threshold_in_kb));
-throw new InvalidRequestException(String.format(Batch too
large));
+throw new InvalidRequestException(Batch too large);
}
else if (logger.isWarnEnabled())
{
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5e68fcec/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
index e319631..f0b1efa 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionManager.java
@@ -501,7 +501,7 @@ public class CompactionManager implements
CompactionManagerMBean
cfs.getDataTracker().unmarkCompacting(sstables);
}
-logger.info(String.format(Completed anticompaction successfully));
+logger.info(Completed anticompaction successfully);
}
public void performMaximal(final ColumnFamilyStore cfStore, boolean
splitOutput)
cassandra git commit: push down allocations to where they are needed
Repository: cassandra
Updated Branches:
refs/heads/trunk 840d9a663 - 10ff8c9f8
push down allocations to where they are needed
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/10ff8c9f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/10ff8c9f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/10ff8c9f
Branch: refs/heads/trunk
Commit: 10ff8c9f892c978a2b0ebd907c3d688f5d8d
Parents: 840d9a6
Author: Dave Brosius dbros...@mebigfatguy.com
Authored: Tue Apr 21 22:24:49 2015 -0400
Committer: Dave Brosius dbros...@mebigfatguy.com
Committed: Tue Apr 21 22:24:49 2015 -0400
--
src/java/org/apache/cassandra/cql3/Constants.java | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/10ff8c9f/src/java/org/apache/cassandra/cql3/Constants.java
--
diff --git a/src/java/org/apache/cassandra/cql3/Constants.java
b/src/java/org/apache/cassandra/cql3/Constants.java
index 8619b24..a0eff94 100644
--- a/src/java/org/apache/cassandra/cql3/Constants.java
+++ b/src/java/org/apache/cassandra/cql3/Constants.java
@@ -319,10 +319,12 @@ public abstract class Constants
public void execute(ByteBuffer rowKey, ColumnFamily cf, Composite
prefix, UpdateParameters params) throws InvalidRequestException
{
-CellName cname = cf.getComparator().create(prefix, column);
ByteBuffer value = t.bindAndGet(params.options);
if (value != ByteBufferUtil.UNSET_BYTE_BUFFER) // use reference
equality and not object equality
+{
+CellName cname = cf.getComparator().create(prefix, column);
cf.addColumn(value == null ? params.makeTombstone(cname) :
params.makeColumn(cname, value));
+}
}
}
[jira] [Commented] (CASSANDRA-9084) Do not generate line number in logs
[
https://issues.apache.org/jira/browse/CASSANDRA-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506005#comment-14506005
]
Benedict commented on CASSANDRA-9084:
-
The results of a quick hacky script to tell us which duplicate log messages
there are:
{noformat}
Validating {}
RepairJob.java:212
RepairJob.java:225
RepairJob.java:266
RepairJob.java:279
Failed to set keep-alive on Thrift socket.
TCustomNonblockingServerSocket.java:58
TCustomServerSocket.java:117
Could not configure socket.
TCustomSocket.java:75
TCustomSocket.java:125
adding secondary index {} to operation
Keyspace.java:446
Keyspace.java:491
skipping {}
KeysSearcher.java:159
CompositesSearcher.java:206
Write timeout; received {} of {} required replies
StorageProxy.java:574
StorageProxy.java:690
Exception in thread {}
CassandraDaemon.java:234
CassandraDaemon.java:235
CassandraDaemon.java:243
Skipping {}
KeysSearcher.java:143
CompositesSearcher.java:196
Loading {}
Schema.java:472
Schema.java:524
Schema.java:555
Schema.java:583
[repair #{}] {}
RepairSession.java:176
RepairSession.java:230
RepairSession.java:243
RepairSession.java:266
RepairJob.java:187
RemoteSyncTask.java:54
LocalSyncTask.java:67
LocalSyncTask.java:107
Notified {}
Gossiper.java:966
Gossiper.java:980
Read only {} ( {}) last page through, must be done
KeysSearcher.java:110
CompositesSearcher.java:167
Sleeping for {}ms to ensure {} does not change
Gossiper.java:515
Gossiper.java:591
JVM doesn't support Adler32 byte buffer access
FBUtilities.java:640
FBUtilities.java:672
DiskAccessMode is {}, indexAccessMode is {}
DatabaseDescriptor.java:315
DatabaseDescriptor.java:320
Enqueuing response to {}
RangeSliceVerbHandler.java:37
MutationVerbHandler.java:52
ReadVerbHandler.java:43
Drop {}
Schema.java:573
Schema.java:601
Updating {}
Schema.java:536
Schema.java:564
Schema.java:592
Could not initialize SIGAR library {}
SigarLibrary.java:51
SigarLibrary.java:55
Empty output skipped, filter empty tuples to suppress this warning
CassandraStorage.java:511
CqlNativeStorage.java:396
Acquiring sstable references
CollationController.java:77
CollationController.java:204
Index build of {} complete
SecondaryIndexManager.java:170
SecondaryIndex.java:222
Miscellaneous task executor still busy after one minute; proceeding with
shutdown
StorageService.java:661
StorageService.java:3827
Failed to set receive buffer size on Thrift socket.
TCustomNonblockingServerSocket.java:81
TCustomServerSocket.java:140
Snapshot-based repair is not yet supported on Windows. Reverting to parallel
repair.
StorageService.java:2793
StorageService.java:2861
Failed to set send buffer size on Thrift socket.
TCustomNonblockingServerSocket.java:69
TCustomServerSocket.java:128
{} found, but does not look like a plain file. Will not watch it for changes
GossipingPropertyFileSnitch.java:92
PropertyFileSnitch.java:79
Opening {} ({} bytes)
SSTableReader.java:384
SSTableReader.java:430
Some replicas have already promised a higher ballot than ours; aborting
StorageProxy.java:376
StorageProxy.java:410
Corrupt sstable {}; skipped
SSTableReader.java:454
SSTableReader.java:477
Could not set socket timeout.
TCustomSocket.java:139
TCustomServerSocket.java:159
Compaction executor has shut down, not submitting task
CompactionManager.java:533
CompactionManager.java:603
Cannot start multiple repair sessions over the same sstables
RepairMessageVerbHandler.java:100
CompactionManager.java:1033
Scanning index {} starting with {}
KeysSearcher.java:115
CompositesSearcher.java:172
Executor has shut down, not submitting background task
CompactionManager.java:183
CompactionManager.java:1294
Reached end of assigned scan range
KeysSearcher.java:166
CompositesSearcher.java:238
Unavailable
StorageProxy.java:582
StorageProxy.java:684
error registering MBean {}
CassandraDaemon.java:551
BlacklistedDirectories.java:54
{noformat}
I don't have a super strong opinion either way, but I do dislike adding line
numbers when they're mostly redundant. I am mostly certain under current
default scenarios it has zero performance impact, but it does leave open a
window for performance regressions if logging happens on any critical paths
without realisation, and on enabling
[jira] [Comment Edited] (CASSANDRA-9084) Do not generate line number in logs
[
https://issues.apache.org/jira/browse/CASSANDRA-9084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506005#comment-14506005
]
Benedict edited comment on CASSANDRA-9084 at 4/21/15 11:27 PM:
---
The results of a quick hacky script to tell us which duplicate log messages
there are:
{noformat}
removed index entry for cleaned-up value {}:{}
CompositesIndex.java:146
AbstractSimplePerColumnSecondaryIndex.java:105
skipping {}
KeysSearcher.java:159
CompositesSearcher.java:206
Timed out waiting on digest mismatch repair acknowledgements
StorageProxy.java:1489
StorageProxy.java:1491
partition keys: {}
CqlRecordWriter.java:356
CqlNativeStorage.java:288
Exception in thread {}
CassandraDaemon.java:234
CassandraDaemon.java:235
CassandraDaemon.java:243
[repair #{}] {}
RepairSession.java:176
RepairSession.java:230
RepairSession.java:243
RepairSession.java:266
RepairJob.java:187
RemoteSyncTask.java:54
LocalSyncTask.java:67
LocalSyncTask.java:107
Expected bloom filter size : {}
DefaultCompactionWriter.java:50
CompactionManager.java:774
Notified {}
Gossiper.java:966
Gossiper.java:980
error writing to {}
OutboundTcpConnection.java:297
OutboundTcpConnection.java:316
Sleeping for {}ms to ensure {} does not change
Gossiper.java:515
Gossiper.java:591
IOException reading from socket; closing
IncomingStreamingConnection.java:69
IncomingTcpConnection.java:99
DiskAccessMode is {}, indexAccessMode is {}
DatabaseDescriptor.java:315
DatabaseDescriptor.java:320
Drop {}
Schema.java:573
Schema.java:601
Updating {}
Schema.java:536
Schema.java:564
Schema.java:592
Conversion error
CassandraAuthorizer.java:442
CassandraRoleManager.java:415
Could not initialize SIGAR library {}
SigarLibrary.java:51
SigarLibrary.java:55
Empty output skipped, filter empty tuples to suppress this warning
CassandraStorage.java:511
CqlNativeStorage.java:396
Index build of {} complete
SecondaryIndexManager.java:170
SecondaryIndex.java:222
Failed to set receive buffer size on Thrift socket.
TCustomNonblockingServerSocket.java:81
TCustomServerSocket.java:140
Snapshot-based repair is not yet supported on Windows. Reverting to parallel
repair.
StorageService.java:2793
StorageService.java:2861
{} found, but does not look like a plain file. Will not watch it for changes
GossipingPropertyFileSnitch.java:92
PropertyFileSnitch.java:79
Opening {} ({} bytes)
SSTableReader.java:384
SSTableReader.java:430
Unable to register metric bean
CassandraMetricsRegistry.java:142
CassandraMetricsRegistry.java:145
Most-selective indexed predicate is {}
KeysSearcher.java:77
CompositesSearcher.java:102
Executor has shut down, not submitting background task
CompactionManager.java:183
CompactionManager.java:1294
Unavailable
StorageProxy.java:582
StorageProxy.java:684
Reading row at
Scrubber.java:140
Verifier.java:142
Compaction buckets are {}
DateTieredCompactionStrategy.java:121
SizeTieredCompactionStrategy.java:91
Validating {}
RepairJob.java:212
RepairJob.java:225
RepairJob.java:266
RepairJob.java:279
RepairMessageVerbHandler.java:110
Failed to set keep-alive on Thrift socket.
TCustomNonblockingServerSocket.java:58
TCustomServerSocket.java:117
name: {}
CqlNativeStorage.java:305
CqlNativeStorage.java:313
Could not configure socket.
TCustomSocket.java:75
TCustomSocket.java:125
adding secondary index {} to operation
Keyspace.java:446
Keyspace.java:491
Write timeout; received {} of {} required replies
StorageProxy.java:574
StorageProxy.java:690
Finished scanning {} rows (estimate was: {})
CqlRecordReader.java:194
ColumnFamilyRecordReader.java:181
Skipping {}
KeysSearcher.java:143
CompositesSearcher.java:196
Unable to mark {} for compaction; probably a background compaction got to it
first. You can disable background compactions temporarily if this is a problem
DateTieredCompactionStrategy.java:375
SizeTieredCompactionStrategy.java:216
Loading {}
Schema.java:472
Schema.java:524
Schema.java:555
Schema.java:583
Timed out waiting on digest mismatch repair requests
StorageProxy.java:1470
StorageProxy.java:1472
Enqueuing response to snapshot request {} to {}
SnapshotVerbHandler.java:43
RepairMessageVerbHandler.java:104
Read only {} ( {}) last page
[jira] [Updated] (CASSANDRA-9151) Anti-compaction is blocking ANTI_ENTROPY stage
[ https://issues.apache.org/jira/browse/CASSANDRA-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Shuler updated CASSANDRA-9151: -- Fix Version/s: 2.1.5 3.0 Anti-compaction is blocking ANTI_ENTROPY stage --- Key: CASSANDRA-9151 URL: https://issues.apache.org/jira/browse/CASSANDRA-9151 Project: Cassandra Issue Type: Bug Reporter: sankalp kohli Assignee: Yuki Morishita Fix For: 3.0, 2.1.5 In RepairMessageVerbHandler.doVerb case ANTICOMPACTION_REQUEST We block on ANTI_ENTROPY stage. This is not good as no other message could be processed till any anticompaction is running. This can cause snapshots for new repairs to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-9194) Delete-only workloads crash Cassandra
[
https://issues.apache.org/jira/browse/CASSANDRA-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505980#comment-14505980
]
Jim Witschey edited comment on CASSANDRA-9194 at 4/21/15 11:16 PM:
---
[~benedict] I think I understand now; thanks for the explanation.
I've [changed the
test|https://github.com/mambocab/cassandra-dtest/commit/25ee5b7050e96a85cd4e33eadc41a21cec7da393]
so that it checks {{MemtableOnHeapSize}} for versions = 2.1 and
{{MemtableDataSize}} for 2.0. As you indicated, it passes on 2.1.4 and trunk.
It currently fails on 2.0.4. Does that sound right to you?
I can't seem to apply your patch to 2.0; it looks like it was written for 2.1?
But I'm +1 on the same logic applied to 2.0 if it passes the dtest.
was (Author: mambocab):
[~benedict] I think I understand now; thanks for the explanation.
I've [changed the
test|https://github.com/mambocab/cassandra-dtest/commit/25ee5b7050e96a85cd4e33eadc41a21cec7da393]
so that it checks {{MemtableOnHeapSize}} for versions = 2.1 and
{{MemtableDataSize}} for 2.0. As you indicated, it passes on 2.1.4 and trunk.
It currently fails on 2.0.4.
I can't seem to apply your patch to 2.0; it looks like it was written for 2.1?
But I'm +1 on the same logic applied to 2.0 if it passes the dtest.
Delete-only workloads crash Cassandra
-
Key: CASSANDRA-9194
URL: https://issues.apache.org/jira/browse/CASSANDRA-9194
Project: Cassandra
Issue Type: Bug
Environment: 2.0.14
Reporter: Robert Wille
Assignee: Benedict
Fix For: 2.0.15
Attachments: 9194.txt
The size of a tombstone is not properly accounted for in the memtable. A
memtable which has only tombstones will never get flushed. It will grow until
the JVM runs out of memory. The following program easily demonstrates the
problem.
{code}
Cluster.Builder builder = Cluster.builder();
Cluster c =
builder.addContactPoints(cas121.devf3.com).build();
Session s = c.connect();
s.execute(CREATE KEYSPACE IF NOT EXISTS test WITH replication
= { 'class' : 'SimpleStrategy', 'replication_factor' : 1 });
s.execute(CREATE TABLE IF NOT EXISTS test.test(id INT PRIMARY
KEY));
PreparedStatement stmt = s.prepare(DELETE FROM test.test WHERE
id = :id);
int id = 0;
while (true)
{
s.execute(stmt.bind(id));
id++;
}{code}
This program should run forever, but eventually Cassandra runs out of heap
and craps out. You needn't wait for Cassandra to crash. If you run nodetool
cfstats test.test while it is running, you'll see Memtable cell count grow,
but Memtable data size will remain 0.
This issue was fixed once before. I received a patch for version 2.0.5 (I
believe), which contained the fix, but the fix has apparently been lost,
because it is clearly broken, and I don't see the fix in the change logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8984) Introduce Transactional API for behaviours that can corrupt system state
[ https://issues.apache.org/jira/browse/CASSANDRA-8984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506033#comment-14506033 ] Benedict commented on CASSANDRA-8984: - OK, I've pushed a smallish refactor that puts prepareToCommit() in the Transactional inerface, and introduces a finish() method into AbstractTransactional. This requires separating out the parameter provision that previously was passed to prepareToCommit() into a separate stage in each case where it occurred, which isn't actually too onerous. It also means storing the _result_ of any prepareToCommit method, which also isn't too unpleasant. We can now chain calls to finish() with a call to finished(), which returns any result state. Let me know what you think of this draft, and I'll look into putting together some of those test cases. Introduce Transactional API for behaviours that can corrupt system state Key: CASSANDRA-8984 URL: https://issues.apache.org/jira/browse/CASSANDRA-8984 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Benedict Assignee: Benedict Fix For: 2.1.5 Attachments: 8984_windows_timeout.txt As a penultimate (and probably final for 2.1, if we agree to introduce it there) round of changes to the internals managing sstable writing, I've introduced a new API called Transactional that I hope will make it much easier to write correct behaviour. As things stand we conflate a lot of behaviours into methods like close - the recent changes unpicked some of these, but didn't go far enough. My proposal here introduces an interface designed to support four actions (on top of their normal function): * prepareToCommit * commit * abort * cleanup In normal operation, once we have finished constructing a state change we call prepareToCommit; once all such state changes are prepared, we call commit. If at any point everything fails, abort is called. In _either_ case, cleanup is called at the very last. These transactional objects are all AutoCloseable, with the behaviour being to rollback any changes unless commit has completed successfully. The changes are actually less invasive than it might sound, since we did recently introduce abort in some places, as well as have commit like methods. This simply formalises the behaviour, and makes it consistent between all objects that interact in this way. Much of the code change is boilerplate, such as moving an object into a try-declaration, although the change is still non-trivial. What it _does_ do is eliminate a _lot_ of special casing that we have had since 2.1 was released. The data tracker API changes and compaction leftover cleanups should finish the job with making this much easier to reason about, but this change I think is worthwhile considering for 2.1, since we've just overhauled this entire area (and not released these changes), and this change is essentially just the finishing touches, so the risk is minimal and the potential gains reasonably significant. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8897) Remove FileCacheService, instead pooling the buffers
[
https://issues.apache.org/jira/browse/CASSANDRA-8897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504758#comment-14504758
]
Stefania edited comment on CASSANDRA-8897 at 4/21/15 10:49 AM:
---
Hi [~benedict] thank you for your review. I'm almost done with the updated
version, I just have a few questions:
- Are the logging gards something we always use for trace and debug or is there
really a big benefit when logging only with parametrized strings and no
function calls?
- I did not understand your comment here:
{code}
ByteBuffer page = chunk.get(BUFFER_SIZE);
if (page.capacity() = BUFFER_SIZE)
buffers.add(page);
else
// this can occur under normal operation, if the chunk allocation is
aligned, since we will have
// a single page spare at the end, and our microchunks are multiple
pages
logger.error(Discarding page with smaller capacity {}, expected {}
bytes, page.capacity(), BUFFER_SIZE);
{code}
How is this possible since {{Allocator.allocateAligned()}} limits the byte
buffer to the exact capacity requested,
which is an exact multiple of the microchunks size (BUFFER_SIZE). The existing
utests do not hit this code path,
any suggestions on how to hit this code path?
- Can you elaborate a bit more on the CAS integer update part, at the end of
this comment here:
{code}
// instead of any ugliness with getAddress() (which seems brittle, if we
somehow grab the wrong value or slice weirdly)
// we can abuse the ByteBuffer internals; the internal attachment property
in DBBs point to the parent buffer of a slice
// to ensure GC of the parent doesn't occur before the child, and hence free
the native memory. We don't need this,
// but either way we can instead point to our Chunk object (which points to
our Chunk ByteBuffer), so that we can
// directly lookup the Chunk we need to maintain. The main added advantage of
this is that we can also detect if
// the chunk is not assigned to this thread, by assigning the Chunk the
current local pool on adoption, and confirming
// we are operating on that exact pool now. We can then have a separate
integer variable we perform a CAS update on
// only in this scenario, so that we can safely recover from the mistake,
and we log some panic warnings. Or we can just
// log the panic warnings for now.
{code}
Did you mean to have one thread unallocate the buffer from another thread pool
or take over the entire pool or other?
BTW, the '=' in allocateChunk() was definitely a mistake, thanks for spotting
it and getting rid of the rollback, which was ugly.
was (Author: stefania):
Hi [~benedict] thank you for your review. I'm almost done with the updated
version, I just have a few questions:
- Are the logging gards something we always use for trace and debug or is there
really a big benefit when logging only with parametrized strings and no
function calls?
- I did not understand your comment here:
{code}
ByteBuffer page = chunk.get(BUFFER_SIZE);
if (page.capacity() = BUFFER_SIZE)
buffers.add(page);
else
// this can occur under normal operation, if the chunk allocation is
aligned, since we will have
// a single page spare at the end, and our microchunks are multiple
pages
logger.error(Discarding page with smaller capacity {}, expected {}
bytes, page.capacity(), BUFFER_SIZE);
{code}
How is this possible since {Allocator.allocateAligned()} limits the byte buffer
to the exact capacity requested,
which is an exact multiple of the microchunks size (BUFFER_SIZE). The existing
utests do not hit this code path,
any suggestions on how to hit this code path?
- Can you elaborate a bit more on the CAS integer update part, at the end of
this comment here:
{code}
// instead of any ugliness with getAddress() (which seems brittle, if we
somehow grab the wrong value or slice weirdly)
// we can abuse the ByteBuffer internals; the internal attachment property
in DBBs point to the parent buffer of a slice
// to ensure GC of the parent doesn't occur before the child, and hence free
the native memory. We don't need this,
// but either way we can instead point to our Chunk object (which points to
our Chunk ByteBuffer), so that we can
// directly lookup the Chunk we need to maintain. The main added advantage of
this is that we can also detect if
// the chunk is not assigned to this thread, by assigning the Chunk the
current local pool on adoption, and confirming
// we are operating on that exact pool now. We can then have a separate
integer variable we perform a CAS update on
// only in this scenario, so that we can safely recover from the mistake,
and we log some panic warnings. Or we can just
// log the panic warnings for now.
{code}
Did you mean to have one thread unallocate the buffer from another thread pool
or take over the
[jira] [Commented] (CASSANDRA-8897) Remove FileCacheService, instead pooling the buffers
[
https://issues.apache.org/jira/browse/CASSANDRA-8897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504758#comment-14504758
]
Stefania commented on CASSANDRA-8897:
-
Hi [~benedict] thank you for your review. I'm almost done with the updated
version, I just have a few questions:
- Are the logging gards something we always use for trace and debug or is there
really a big benefit when logging only with parametrized strings and no
function calls?
- I did not understand your comment here:
{code}
ByteBuffer page = chunk.get(BUFFER_SIZE);
if (page.capacity() = BUFFER_SIZE)
buffers.add(page);
else
// this can occur under normal operation, if the chunk allocation is
aligned, since we will have
// a single page spare at the end, and our microchunks are multiple
pages
logger.error(Discarding page with smaller capacity {}, expected {}
bytes, page.capacity(), BUFFER_SIZE);
{code}
How is this possible since {Allocator.allocateAligned()} limits the byte buffer
to the exact capacity requested,
which is an exact multiple of the microchunks size (BUFFER_SIZE). The existing
utests do not hit this code path,
any suggestions on how to hit this code path?
- Can you elaborate a bit more on the CAS integer update part, at the end of
this comment here:
{code}
// instead of any ugliness with getAddress() (which seems brittle, if we
somehow grab the wrong value or slice weirdly)
// we can abuse the ByteBuffer internals; the internal attachment property
in DBBs point to the parent buffer of a slice
// to ensure GC of the parent doesn't occur before the child, and hence free
the native memory. We don't need this,
// but either way we can instead point to our Chunk object (which points to
our Chunk ByteBuffer), so that we can
// directly lookup the Chunk we need to maintain. The main added advantage of
this is that we can also detect if
// the chunk is not assigned to this thread, by assigning the Chunk the
current local pool on adoption, and confirming
// we are operating on that exact pool now. We can then have a separate
integer variable we perform a CAS update on
// only in this scenario, so that we can safely recover from the mistake,
and we log some panic warnings. Or we can just
// log the panic warnings for now.
{code}
Did you mean to have one thread unallocate the buffer from another thread pool
or take over the entire pool or other?
BTW, the '=' in allocateChunk() was definitely a mistake, thanks for spotting
it and getting rid of the rollback, which was ugly.
Remove FileCacheService, instead pooling the buffers
Key: CASSANDRA-8897
URL: https://issues.apache.org/jira/browse/CASSANDRA-8897
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Benedict
Assignee: Stefania
Fix For: 3.0
After CASSANDRA-8893, a RAR will be a very lightweight object and will not
need caching, so we can eliminate this cache entirely. Instead we should have
a pool of buffers that are page-aligned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
cassandra git commit: Do not attempt to rebuild indexes if no index accepts any column, and only rebuild those that do accept some.
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.0 738229bd7 - 707fa6e74
Do not attempt to rebuild indexes if no index accepts any column, and only
rebuild those that do accept some.
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/707fa6e7
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/707fa6e7
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/707fa6e7
Branch: refs/heads/cassandra-2.0
Commit: 707fa6e74a217e49af0216df008d61cb3d40e386
Parents: 738229b
Author: Sergio Bossa sergio.bo...@gmail.com
Authored: Wed Apr 15 19:22:21 2015 +0100
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Tue Apr 21 14:05:08 2015 +0300
--
.../db/index/SecondaryIndexManager.java | 19 ++
.../cassandra/db/ColumnFamilyStoreTest.java | 27
.../db/index/PerRowSecondaryIndexTest.java | 10 +++-
3 files changed, 55 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/707fa6e7/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
--
diff --git a/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
b/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
index fda79f8..1db7de6 100644
--- a/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
+++ b/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
@@ -131,6 +131,7 @@ public class SecondaryIndexManager
*/
public void maybeBuildSecondaryIndexes(CollectionSSTableReader sstables,
SetString idxNames)
{
+idxNames = filterByColumn(idxNames);
if (idxNames.isEmpty())
return;
@@ -580,6 +581,24 @@ public class SecondaryIndexManager
return true;
}
+private SetString filterByColumn(SetString idxNames)
+{
+SetSecondaryIndex indexes = getIndexesByNames(idxNames);
+SetString filtered = new HashSet(idxNames.size());
+for (SecondaryIndex candidate : indexes)
+{
+for (ColumnDefinition column : baseCfs.metadata.allColumns())
+{
+if (candidate.indexes(column.name))
+{
+filtered.add(candidate.getIndexName());
+break;
+}
+}
+}
+return filtered;
+}
+
public static interface Updater
{
/** called when constructing the index against pre-existing data */
http://git-wip-us.apache.org/repos/asf/cassandra/blob/707fa6e7/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
b/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
index 292b7f2..8f4a18f 100644
--- a/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
+++ b/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
@@ -45,6 +45,7 @@ import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.config.Schema;
import org.apache.cassandra.db.columniterator.IdentityQueryFilter;
import org.apache.cassandra.db.filter.*;
+import org.apache.cassandra.db.index.PerRowSecondaryIndexTest;
import org.apache.cassandra.db.index.SecondaryIndex;
import org.apache.cassandra.db.marshal.CompositeType;
import org.apache.cassandra.db.marshal.LexicalUUIDType;
@@ -2172,4 +2173,30 @@ public class ColumnFamilyStoreTest extends SchemaLoader
});
System.err.println(Row key: + rowKey + Cols: + transformed);
}
+
+@Test
+public void testRebuildSecondaryIndex() throws IOException
+{
+RowMutation rm;
+
+rm = new RowMutation(PerRowSecondaryIndex,
ByteBufferUtil.bytes(k1));
+rm.add(Indexed1, ByteBufferUtil.bytes(indexed),
ByteBufferUtil.bytes(foo), 1);
+rm.apply();
+assertTrue(Arrays.equals(k1.getBytes(),
PerRowSecondaryIndexTest.TestIndex.LAST_INDEXED_KEY.array()));
+
+
Keyspace.open(PerRowSecondaryIndex).getColumnFamilyStore(Indexed1).forceBlockingFlush();
+
+PerRowSecondaryIndexTest.TestIndex.reset();
+
+ColumnFamilyStore.rebuildSecondaryIndex(PerRowSecondaryIndex,
Indexed1, PerRowSecondaryIndexTest.TestIndex.class.getSimpleName());
+assertTrue(Arrays.equals(k1.getBytes(),
PerRowSecondaryIndexTest.TestIndex.LAST_INDEXED_KEY.array()));
+
+PerRowSecondaryIndexTest.TestIndex.reset();
+
+PerRowSecondaryIndexTest.TestIndex.ACTIVE = false;
+ColumnFamilyStore.rebuildSecondaryIndex(PerRowSecondaryIndex,
Indexed1, PerRowSecondaryIndexTest.TestIndex.class.getSimpleName());
+
[1/3] cassandra git commit: Do not attempt to rebuild indexes if no index accepts any column
Repository: cassandra
Updated Branches:
refs/heads/trunk 1fab7b785 - 1e8c5e0a1
Do not attempt to rebuild indexes if no index accepts any column
patch by Sergio Bossa; reviewed by Sam Tunnicliffe for CASSANDRA-9196
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b325316f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b325316f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b325316f
Branch: refs/heads/trunk
Commit: b325316f58a4df07d8037b179fe53c03f97dfd2e
Parents: 738229b
Author: Sergio Bossa sergio.bo...@gmail.com
Authored: Wed Apr 15 19:22:21 2015 +0100
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Tue Apr 21 14:16:41 2015 +0300
--
CHANGES.txt | 1 +
.../db/index/SecondaryIndexManager.java | 19 ++
.../cassandra/db/ColumnFamilyStoreTest.java | 27
.../db/index/PerRowSecondaryIndexTest.java | 10 +++-
4 files changed, 56 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b325316f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 6c546c4..5e1e62c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.15:
+ * Do not attempt to rebuild indexes if no index accepts any column
(CASSANDRA-9196)
* Don't initiate snitch reconnection for dead states (CASSANDRA-7292)
* Fix ArrayIndexOutOfBoundsException in CQLSSTableWriter (CASSANDRA-8978)
* Add shutdown gossip state to prevent timeouts during rolling restarts
(CASSANDRA-8336)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b325316f/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
--
diff --git a/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
b/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
index fda79f8..1db7de6 100644
--- a/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
+++ b/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
@@ -131,6 +131,7 @@ public class SecondaryIndexManager
*/
public void maybeBuildSecondaryIndexes(CollectionSSTableReader sstables,
SetString idxNames)
{
+idxNames = filterByColumn(idxNames);
if (idxNames.isEmpty())
return;
@@ -580,6 +581,24 @@ public class SecondaryIndexManager
return true;
}
+private SetString filterByColumn(SetString idxNames)
+{
+SetSecondaryIndex indexes = getIndexesByNames(idxNames);
+SetString filtered = new HashSet(idxNames.size());
+for (SecondaryIndex candidate : indexes)
+{
+for (ColumnDefinition column : baseCfs.metadata.allColumns())
+{
+if (candidate.indexes(column.name))
+{
+filtered.add(candidate.getIndexName());
+break;
+}
+}
+}
+return filtered;
+}
+
public static interface Updater
{
/** called when constructing the index against pre-existing data */
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b325316f/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
b/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
index 292b7f2..8f4a18f 100644
--- a/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
+++ b/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
@@ -45,6 +45,7 @@ import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.config.Schema;
import org.apache.cassandra.db.columniterator.IdentityQueryFilter;
import org.apache.cassandra.db.filter.*;
+import org.apache.cassandra.db.index.PerRowSecondaryIndexTest;
import org.apache.cassandra.db.index.SecondaryIndex;
import org.apache.cassandra.db.marshal.CompositeType;
import org.apache.cassandra.db.marshal.LexicalUUIDType;
@@ -2172,4 +2173,30 @@ public class ColumnFamilyStoreTest extends SchemaLoader
});
System.err.println(Row key: + rowKey + Cols: + transformed);
}
+
+@Test
+public void testRebuildSecondaryIndex() throws IOException
+{
+RowMutation rm;
+
+rm = new RowMutation(PerRowSecondaryIndex,
ByteBufferUtil.bytes(k1));
+rm.add(Indexed1, ByteBufferUtil.bytes(indexed),
ByteBufferUtil.bytes(foo), 1);
+rm.apply();
+assertTrue(Arrays.equals(k1.getBytes(),
PerRowSecondaryIndexTest.TestIndex.LAST_INDEXED_KEY.array()));
+
+
[3/3] cassandra git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1e8c5e0a Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1e8c5e0a Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1e8c5e0a Branch: refs/heads/trunk Commit: 1e8c5e0a1ad4c02f49a46e668b100be447e3deba Parents: 1fab7b7 f18fb04 Author: Aleksey Yeschenko alek...@apache.org Authored: Tue Apr 21 14:43:21 2015 +0300 Committer: Aleksey Yeschenko alek...@apache.org Committed: Tue Apr 21 14:43:21 2015 +0300 -- CHANGES.txt | 1 + .../cassandra/db/index/SecondaryIndex.java | 10 ++ .../db/index/SecondaryIndexManager.java | 19 +++ .../db/index/PerRowSecondaryIndexTest.java | 11 ++- 4 files changed, 40 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/1e8c5e0a/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/1e8c5e0a/src/java/org/apache/cassandra/db/index/SecondaryIndex.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/1e8c5e0a/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/1e8c5e0a/test/unit/org/apache/cassandra/db/index/PerRowSecondaryIndexTest.java -- diff --cc test/unit/org/apache/cassandra/db/index/PerRowSecondaryIndexTest.java index 372e09a,9dbd3be..596fd41 --- a/test/unit/org/apache/cassandra/db/index/PerRowSecondaryIndexTest.java +++ b/test/unit/org/apache/cassandra/db/index/PerRowSecondaryIndexTest.java @@@ -29,17 -30,12 +29,18 @@@ import org.junit.Test import org.apache.cassandra.SchemaLoader; import org.apache.cassandra.Util; + import org.apache.cassandra.config.ColumnDefinition; import org.apache.cassandra.config.DatabaseDescriptor; +import org.apache.cassandra.config.KSMetaData; import org.apache.cassandra.cql3.QueryProcessor; import org.apache.cassandra.cql3.UntypedResultSet; -import org.apache.cassandra.db.*; -import org.apache.cassandra.db.columniterator.IdentityQueryFilter; +import org.apache.cassandra.db.Cell; +import org.apache.cassandra.db.ColumnFamily; +import org.apache.cassandra.db.ColumnFamilyStore; +import org.apache.cassandra.db.DecoratedKey; +import org.apache.cassandra.db.IndexExpression; +import org.apache.cassandra.db.Mutation; +import org.apache.cassandra.db.Row; import org.apache.cassandra.db.composites.CellName; import org.apache.cassandra.db.filter.ExtendedFilter; import org.apache.cassandra.db.filter.QueryFilter;
cassandra git commit: Do not attempt to rebuild indexes if no index accepts any column
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.0 707fa6e74 - b325316f5 (forced update)
Do not attempt to rebuild indexes if no index accepts any column
patch by Sergio Bossa; reviewed by Sam Tunnicliffe for CASSANDRA-9196
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b325316f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b325316f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b325316f
Branch: refs/heads/cassandra-2.0
Commit: b325316f58a4df07d8037b179fe53c03f97dfd2e
Parents: 738229b
Author: Sergio Bossa sergio.bo...@gmail.com
Authored: Wed Apr 15 19:22:21 2015 +0100
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Tue Apr 21 14:16:41 2015 +0300
--
CHANGES.txt | 1 +
.../db/index/SecondaryIndexManager.java | 19 ++
.../cassandra/db/ColumnFamilyStoreTest.java | 27
.../db/index/PerRowSecondaryIndexTest.java | 10 +++-
4 files changed, 56 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b325316f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 6c546c4..5e1e62c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.15:
+ * Do not attempt to rebuild indexes if no index accepts any column
(CASSANDRA-9196)
* Don't initiate snitch reconnection for dead states (CASSANDRA-7292)
* Fix ArrayIndexOutOfBoundsException in CQLSSTableWriter (CASSANDRA-8978)
* Add shutdown gossip state to prevent timeouts during rolling restarts
(CASSANDRA-8336)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b325316f/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
--
diff --git a/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
b/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
index fda79f8..1db7de6 100644
--- a/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
+++ b/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
@@ -131,6 +131,7 @@ public class SecondaryIndexManager
*/
public void maybeBuildSecondaryIndexes(CollectionSSTableReader sstables,
SetString idxNames)
{
+idxNames = filterByColumn(idxNames);
if (idxNames.isEmpty())
return;
@@ -580,6 +581,24 @@ public class SecondaryIndexManager
return true;
}
+private SetString filterByColumn(SetString idxNames)
+{
+SetSecondaryIndex indexes = getIndexesByNames(idxNames);
+SetString filtered = new HashSet(idxNames.size());
+for (SecondaryIndex candidate : indexes)
+{
+for (ColumnDefinition column : baseCfs.metadata.allColumns())
+{
+if (candidate.indexes(column.name))
+{
+filtered.add(candidate.getIndexName());
+break;
+}
+}
+}
+return filtered;
+}
+
public static interface Updater
{
/** called when constructing the index against pre-existing data */
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b325316f/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
b/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
index 292b7f2..8f4a18f 100644
--- a/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
+++ b/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
@@ -45,6 +45,7 @@ import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.config.Schema;
import org.apache.cassandra.db.columniterator.IdentityQueryFilter;
import org.apache.cassandra.db.filter.*;
+import org.apache.cassandra.db.index.PerRowSecondaryIndexTest;
import org.apache.cassandra.db.index.SecondaryIndex;
import org.apache.cassandra.db.marshal.CompositeType;
import org.apache.cassandra.db.marshal.LexicalUUIDType;
@@ -2172,4 +2173,30 @@ public class ColumnFamilyStoreTest extends SchemaLoader
});
System.err.println(Row key: + rowKey + Cols: + transformed);
}
+
+@Test
+public void testRebuildSecondaryIndex() throws IOException
+{
+RowMutation rm;
+
+rm = new RowMutation(PerRowSecondaryIndex,
ByteBufferUtil.bytes(k1));
+rm.add(Indexed1, ByteBufferUtil.bytes(indexed),
ByteBufferUtil.bytes(foo), 1);
+rm.apply();
+assertTrue(Arrays.equals(k1.getBytes(),
PerRowSecondaryIndexTest.TestIndex.LAST_INDEXED_KEY.array()));
+
+
[2/2] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f18fb044 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f18fb044 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f18fb044 Branch: refs/heads/cassandra-2.1 Commit: f18fb044527115d9dfe8586bf1615825463b1156 Parents: c6e4379 b325316 Author: Aleksey Yeschenko alek...@apache.org Authored: Tue Apr 21 14:41:08 2015 +0300 Committer: Aleksey Yeschenko alek...@apache.org Committed: Tue Apr 21 14:41:08 2015 +0300 -- CHANGES.txt | 1 + .../cassandra/db/index/SecondaryIndex.java | 10 +++ .../db/index/SecondaryIndexManager.java | 19 + .../cassandra/db/ColumnFamilyStoreTest.java | 77 +--- .../db/index/PerRowSecondaryIndexTest.java | 11 ++- 5 files changed, 74 insertions(+), 44 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/f18fb044/CHANGES.txt -- diff --cc CHANGES.txt index 2777d79,5e1e62c..a40530a --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,86 -1,5 +1,87 @@@ -2.0.15: +2.1.5 + * Re-add deprecated cold_reads_to_omit param for backwards compat (CASSANDRA-9203) + * Make anticompaction visible in compactionstats (CASSANDRA-9098) + * Improve nodetool getendpoints documentation about the partition + key parameter (CASSANDRA-6458) + * Don't check other keyspaces for schema changes when an user-defined + type is altered (CASSANDRA-9187) + * Allow takeColumnFamilySnapshot to take a list of tables (CASSANDRA-8348) + * Limit major sstable operations to their canonical representation (CASSANDRA-8669) + * cqlsh: Add tests for INSERT and UPDATE tab completion (CASSANDRA-9125) + * cqlsh: quote column names when needed in COPY FROM inserts (CASSANDRA-9080) + * Add generate-idea-files target to build.xml (CASSANDRA-9123) + * Do not load read meter for offline operations (CASSANDRA-9082) + * cqlsh: Make CompositeType data readable (CASSANDRA-8919) + * cqlsh: Fix display of triggers (CASSANDRA-9081) + * Fix NullPointerException when deleting or setting an element by index on + a null list collection (CASSANDRA-9077) + * Buffer bloom filter serialization (CASSANDRA-9066) + * Fix anti-compaction target bloom filter size (CASSANDRA-9060) + * Make FROZEN and TUPLE unreserved keywords in CQL (CASSANDRA-9047) + * Prevent AssertionError from SizeEstimatesRecorder (CASSANDRA-9034) + * Avoid overwriting index summaries for sstables with an older format that + does not support downsampling; rebuild summaries on startup when this + is detected (CASSANDRA-8993) + * Fix potential data loss in CompressedSequentialWriter (CASSANDRA-8949) + * Make PasswordAuthenticator number of hashing rounds configurable (CASSANDRA-8085) + * Fix AssertionError when binding nested collections in DELETE (CASSANDRA-8900) + * Check for overlap with non-early sstables in LCS (CASSANDRA-8739) + * Only calculate max purgable timestamp if we have to (CASSANDRA-8914) + * (cqlsh) Greatly improve performance of COPY FROM (CASSANDRA-8225) + * IndexSummary effectiveIndexInterval is now a guideline, not a rule (CASSANDRA-8993) + * Use correct bounds for page cache eviction of compressed files (CASSANDRA-8746) + * SSTableScanner enforces its bounds (CASSANDRA-8946) + * Cleanup cell equality (CASSANDRA-8947) + * Introduce intra-cluster message coalescing (CASSANDRA-8692) + * DatabaseDescriptor throws NPE when rpc_interface is used (CASSANDRA-8839) + * Don't check if an sstable is live for offline compactions (CASSANDRA-8841) + * Don't set clientMode in SSTableLoader (CASSANDRA-8238) + * Fix SSTableRewriter with disabled early open (CASSANDRA-8535) + * Allow invalidating permissions and cache time (CASSANDRA-8722) + * Log warning when queries that will require ALLOW FILTERING in Cassandra 3.0 + are executed (CASSANDRA-8418) + * Fix cassandra-stress so it respects the CL passed in user mode (CASSANDRA-8948) + * Fix rare NPE in ColumnDefinition#hasIndexOption() (CASSANDRA-8786) + * cassandra-stress reports per-operation statistics, plus misc (CASSANDRA-8769) + * Add SimpleDate (cql date) and Time (cql time) types (CASSANDRA-7523) + * Use long for key count in cfstats (CASSANDRA-8913) + * Make SSTableRewriter.abort() more robust to failure (CASSANDRA-8832) + * Remove cold_reads_to_omit from STCS (CASSANDRA-8860) + * Make EstimatedHistogram#percentile() use ceil instead of floor (CASSANDRA-8883) + * Fix top partitions reporting wrong cardinality (CASSANDRA-8834) + * Fix rare NPE in KeyCacheSerializer (CASSANDRA-8067) + * Pick sstables for validation as late as possible inc repairs (CASSANDRA-8366) + * Fix
[1/2] cassandra git commit: Do not attempt to rebuild indexes if no index accepts any column
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1 c6e437983 - f18fb0445
Do not attempt to rebuild indexes if no index accepts any column
patch by Sergio Bossa; reviewed by Sam Tunnicliffe for CASSANDRA-9196
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b325316f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b325316f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b325316f
Branch: refs/heads/cassandra-2.1
Commit: b325316f58a4df07d8037b179fe53c03f97dfd2e
Parents: 738229b
Author: Sergio Bossa sergio.bo...@gmail.com
Authored: Wed Apr 15 19:22:21 2015 +0100
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Tue Apr 21 14:16:41 2015 +0300
--
CHANGES.txt | 1 +
.../db/index/SecondaryIndexManager.java | 19 ++
.../cassandra/db/ColumnFamilyStoreTest.java | 27
.../db/index/PerRowSecondaryIndexTest.java | 10 +++-
4 files changed, 56 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b325316f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 6c546c4..5e1e62c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.15:
+ * Do not attempt to rebuild indexes if no index accepts any column
(CASSANDRA-9196)
* Don't initiate snitch reconnection for dead states (CASSANDRA-7292)
* Fix ArrayIndexOutOfBoundsException in CQLSSTableWriter (CASSANDRA-8978)
* Add shutdown gossip state to prevent timeouts during rolling restarts
(CASSANDRA-8336)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b325316f/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
--
diff --git a/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
b/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
index fda79f8..1db7de6 100644
--- a/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
+++ b/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
@@ -131,6 +131,7 @@ public class SecondaryIndexManager
*/
public void maybeBuildSecondaryIndexes(CollectionSSTableReader sstables,
SetString idxNames)
{
+idxNames = filterByColumn(idxNames);
if (idxNames.isEmpty())
return;
@@ -580,6 +581,24 @@ public class SecondaryIndexManager
return true;
}
+private SetString filterByColumn(SetString idxNames)
+{
+SetSecondaryIndex indexes = getIndexesByNames(idxNames);
+SetString filtered = new HashSet(idxNames.size());
+for (SecondaryIndex candidate : indexes)
+{
+for (ColumnDefinition column : baseCfs.metadata.allColumns())
+{
+if (candidate.indexes(column.name))
+{
+filtered.add(candidate.getIndexName());
+break;
+}
+}
+}
+return filtered;
+}
+
public static interface Updater
{
/** called when constructing the index against pre-existing data */
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b325316f/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
b/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
index 292b7f2..8f4a18f 100644
--- a/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
+++ b/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
@@ -45,6 +45,7 @@ import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.config.Schema;
import org.apache.cassandra.db.columniterator.IdentityQueryFilter;
import org.apache.cassandra.db.filter.*;
+import org.apache.cassandra.db.index.PerRowSecondaryIndexTest;
import org.apache.cassandra.db.index.SecondaryIndex;
import org.apache.cassandra.db.marshal.CompositeType;
import org.apache.cassandra.db.marshal.LexicalUUIDType;
@@ -2172,4 +2173,30 @@ public class ColumnFamilyStoreTest extends SchemaLoader
});
System.err.println(Row key: + rowKey + Cols: + transformed);
}
+
+@Test
+public void testRebuildSecondaryIndex() throws IOException
+{
+RowMutation rm;
+
+rm = new RowMutation(PerRowSecondaryIndex,
ByteBufferUtil.bytes(k1));
+rm.add(Indexed1, ByteBufferUtil.bytes(indexed),
ByteBufferUtil.bytes(foo), 1);
+rm.apply();
+assertTrue(Arrays.equals(k1.getBytes(),
PerRowSecondaryIndexTest.TestIndex.LAST_INDEXED_KEY.array()));
+
+
cassandra git commit: Update tuple types when nested UDT is altered
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1 5ed41ea9c - aeadda5ba
Update tuple types when nested UDT is altered
Patch by Benjamin Lerer; reviewed by Tyler Hobbs for CASSANDRA-9192
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/aeadda5b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/aeadda5b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/aeadda5b
Branch: refs/heads/cassandra-2.1
Commit: aeadda5bada0c32719dd74995a672a1ad7589805
Parents: 5ed41ea
Author: blerer benjamin.le...@datastax.com
Authored: Tue Apr 21 11:46:03 2015 -0500
Committer: Tyler Hobbs tylerlho...@gmail.com
Committed: Tue Apr 21 11:46:03 2015 -0500
--
CHANGES.txt | 4 +-
.../cql3/statements/AlterTypeStatement.java | 6 ++
.../apache/cassandra/cql3/UserTypesTest.java| 94
3 files changed, 102 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/aeadda5b/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index ff25db6..9211270 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,6 +1,6 @@
2.1.5
- * Update collection types that use a user-defined type when that UDT is
- modified (CASSANDRA-9148)
+ * Update tuple and collection types that use a user-defined type when that UDT
+ is modified (CASSANDRA-9148, CASSANDRA-9192)
* Re-add deprecated cold_reads_to_omit param for backwards compat
(CASSANDRA-9203)
* Make anticompaction visible in compactionstats (CASSANDRA-9098)
* Improve nodetool getendpoints documentation about the partition
http://git-wip-us.apache.org/repos/asf/cassandra/blob/aeadda5b/src/java/org/apache/cassandra/cql3/statements/AlterTypeStatement.java
--
diff --git
a/src/java/org/apache/cassandra/cql3/statements/AlterTypeStatement.java
b/src/java/org/apache/cassandra/cql3/statements/AlterTypeStatement.java
index 0b64f94..74fafd6 100644
--- a/src/java/org/apache/cassandra/cql3/statements/AlterTypeStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/AlterTypeStatement.java
@@ -191,6 +191,12 @@ public abstract class AlterTypeStatement extends
SchemaAlteringStatement
ListAbstractType? updatedTypes = updateTypes(ut.fieldTypes(),
keyspace, toReplace, updated);
return updatedTypes == null ? null : new UserType(ut.keyspace,
ut.name, new ArrayList(ut.fieldNames()), updatedTypes);
}
+else if (type instanceof TupleType)
+{
+TupleType tt = (TupleType)type;
+ListAbstractType? updatedTypes = updateTypes(tt.allTypes(),
keyspace, toReplace, updated);
+return updatedTypes == null ? null : new TupleType(updatedTypes);
+}
else if (type instanceof CompositeType)
{
CompositeType ct = (CompositeType)type;
http://git-wip-us.apache.org/repos/asf/cassandra/blob/aeadda5b/test/unit/org/apache/cassandra/cql3/UserTypesTest.java
--
diff --git a/test/unit/org/apache/cassandra/cql3/UserTypesTest.java
b/test/unit/org/apache/cassandra/cql3/UserTypesTest.java
index 1d71fe8..c820ece 100644
--- a/test/unit/org/apache/cassandra/cql3/UserTypesTest.java
+++ b/test/unit/org/apache/cassandra/cql3/UserTypesTest.java
@@ -222,4 +222,98 @@ public class UserTypesTest extends CQLTester
row(4, list(userType(null, 4;
}
}
+
+@Test
+public void testAlteringUserTypeNestedWithinTuple() throws Throwable
+{
+String type = createType(CREATE TYPE %s (a int, b int));
+
+createTable(CREATE TABLE %s (a int PRIMARY KEY, b frozentupleint,
+ KEYSPACE + . + type + ));
+
+execute(INSERT INTO %s (a, b) VALUES(1, (1, {a:1, b:1})));
+assertRows(execute(SELECT * FROM %s), row(1, tuple(1, userType(1,
1;
+flush();
+
+execute(ALTER TYPE + KEYSPACE + . + type + ADD c int);
+execute(INSERT INTO %s (a, b) VALUES(2, (2, {a: 2, b: 2, c: 2})));
+execute(INSERT INTO %s (a, b) VALUES(3, (3, {a: 3, b: 3})));
+execute(INSERT INTO %s (a, b) VALUES(4, (4, {b:4})));
+
+assertRows(execute(SELECT * FROM %s),
+ row(1, tuple(1, userType(1, 1))),
+ row(2, tuple(2, userType(2, 2, 2))),
+ row(3, tuple(3, userType(3, 3, null))),
+ row(4, tuple(4, userType(null, 4, null;
+
+flush();
+
+assertRows(execute(SELECT * FROM %s),
+ row(1, tuple(1, userType(1, 1))),
+ row(2, tuple(2, userType(2, 2, 2))),
+ row(3, tuple(3,
[2/2] cassandra git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b379e326 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b379e326 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b379e326 Branch: refs/heads/trunk Commit: b379e32606717dfd1afe1b4e37398f4dfeb0fb24 Parents: a43e43f aeadda5 Author: Tyler Hobbs tylerlho...@gmail.com Authored: Tue Apr 21 11:46:50 2015 -0500 Committer: Tyler Hobbs tylerlho...@gmail.com Committed: Tue Apr 21 11:46:50 2015 -0500 -- CHANGES.txt | 4 +- .../cql3/statements/AlterTypeStatement.java | 6 ++ .../apache/cassandra/cql3/UserTypesTest.java| 94 3 files changed, 102 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/b379e326/CHANGES.txt -- diff --cc CHANGES.txt index 41ed08f,9211270..e14e6cf --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,104 -1,6 +1,104 @@@ +3.0 + * Distinguish between null and unset in protocol v4 (CASSANDRA-7304) + * Add user/role permissions for user-defined functions (CASSANDRA-7557) + * Allow cassandra config to be updated to restart daemon without unloading classes (CASSANDRA-9046) + * Don't initialize compaction writer before checking if iter is empty (CASSANDRA-9117) + * Remove line number generation from default logback.xml + * Don't execute any functions at prepare-time (CASSANDRA-9037) + * Share file handles between all instances of a SegmentedFile (CASSANDRA-8893) + * Make it possible to major compact LCS (CASSANDRA-7272) + * Make FunctionExecutionException extend RequestExecutionException + (CASSANDRA-9055) + * Add support for SELECT JSON, INSERT JSON syntax and new toJson(), fromJson() + functions (CASSANDRA-7970) + * Optimise max purgeable timestamp calculation in compaction (CASSANDRA-8920) + * Constrain internode message buffer sizes, and improve IO class hierarchy (CASSANDRA-8670) + * New tool added to validate all sstables in a node (CASSANDRA-5791) + * Push notification when tracing completes for an operation (CASSANDRA-7807) + * Delay node up and node added notifications until native protocol server is started (CASSANDRA-8236) + * Compressed Commit Log (CASSANDRA-6809) + * Optimise IntervalTree (CASSANDRA-8988) + * Add a key-value payload for third party usage (CASSANDRA-8553) + * Bump metrics-reporter-config dependency for metrics 3.0 (CASSANDRA-8149) + * Partition intra-cluster message streams by size, not type (CASSANDRA-8789) + * Add WriteFailureException to native protocol, notify coordinator of + write failures (CASSANDRA-8592) + * Convert SequentialWriter to nio (CASSANDRA-8709) + * Add role based access control (CASSANDRA-7653, 8650, 7216, 8760, 8849, 8761, 8850) + * Record client ip address in tracing sessions (CASSANDRA-8162) + * Indicate partition key columns in response metadata for prepared + statements (CASSANDRA-7660) + * Merge UUIDType and TimeUUIDType parse logic (CASSANDRA-8759) + * Avoid memory allocation when searching index summary (CASSANDRA-8793) + * Optimise (Time)?UUIDType Comparisons (CASSANDRA-8730) + * Make CRC32Ex into a separate maven dependency (CASSANDRA-8836) + * Use preloaded jemalloc w/ Unsafe (CASSANDRA-8714) + * Avoid accessing partitioner through StorageProxy (CASSANDRA-8244, 8268) + * Upgrade Metrics library and remove depricated metrics (CASSANDRA-5657) + * Serializing Row cache alternative, fully off heap (CASSANDRA-7438) + * Duplicate rows returned when in clause has repeated values (CASSANDRA-6707) + * Make CassandraException unchecked, extend RuntimeException (CASSANDRA-8560) + * Support direct buffer decompression for reads (CASSANDRA-8464) + * DirectByteBuffer compatible LZ4 methods (CASSANDRA-7039) + * Group sstables for anticompaction correctly (CASSANDRA-8578) + * Add ReadFailureException to native protocol, respond + immediately when replicas encounter errors while handling + a read request (CASSANDRA-7886) + * Switch CommitLogSegment from RandomAccessFile to nio (CASSANDRA-8308) + * Allow mixing token and partition key restrictions (CASSANDRA-7016) + * Support index key/value entries on map collections (CASSANDRA-8473) + * Modernize schema tables (CASSANDRA-8261) + * Support for user-defined aggregation functions (CASSANDRA-8053) + * Fix NPE in SelectStatement with empty IN values (CASSANDRA-8419) + * Refactor SelectStatement, return IN results in natural order instead + of IN value list order and ignore duplicate values in partition key IN restrictions (CASSANDRA-7981) + * Support UDTs, tuples, and collections in user-defined + functions (CASSANDRA-7563) + * Fix aggregate fn
