[jira] [Commented] (CASSANDRA-12519) dtest failure in offline_tools_test.TestOfflineTools.sstableofflinerelevel_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362918#comment-17362918
]
Stefania Alborghetti commented on CASSANDRA-12519:
--
The plan proposed above makes sense to me.
I also looked at the PR for trunk and it LGTM.
Great job [~adelapena] and [~bereng].
> dtest failure in
> offline_tools_test.TestOfflineTools.sstableofflinerelevel_test
> ---
>
> Key: CASSANDRA-12519
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12519
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/dtest/python
>Reporter: Sean McCarthy
>Assignee: Andres de la Peña
>Priority: Normal
> Fix For: 4.0-rc2, 4.0, 3.0.x, 3.11.x, 4.0-rc, 4.x
>
> Attachments: node1.log, node1_debug.log, node1_gc.log
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> example failure:
> http://cassci.datastax.com/job/trunk_offheap_dtest/379/testReport/offline_tools_test/TestOfflineTools/sstableofflinerelevel_test/
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/offline_tools_test.py", line 209, in
> sstableofflinerelevel_test
> self.assertGreater(max(final_levels), 1)
> File "/usr/lib/python2.7/unittest/case.py", line 942, in assertGreater
> self.fail(self._formatMessage(msg, standardMsg))
> File "/usr/lib/python2.7/unittest/case.py", line 410, in fail
> raise self.failureException(msg)
> "1 not greater than 1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12519) dtest failure in offline_tools_test.TestOfflineTools.sstableofflinerelevel_test
[
https://issues.apache.org/jira/browse/CASSANDRA-12519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17362042#comment-17362042
]
Stefania Alborghetti commented on CASSANDRA-12519:
--
{quote}
I'm not familiarized with the lifecycle package so I'm not sure whether
skipping the temporary sstables when resetting the levels is right, or whether
the validation error that happens after changing the metadata is caused by a
deeper problem.
{quote}
I would need to see the full reason why the transaction rejected a record and I
wasn't able to find a full failure, but it must have failed the checksum
verification because the metadata file is changed by the standalone tools,
{{sstablelevelreset}} in our case.
The transaction is checking if anything has tampered with a file guarded by it.
This is done by {{LogFile.verify()}} and would also prevent a main Cassandra
process from starting up. This is because there is some automated cleanup done
on startup when {{LogTransaction.removeUnfinishedLeftovers()}} is called. Since
we don't want to mistakenly delete files restored by users for example, we
check using a checksum which is calculated from the files that existed when the
transaction record was created. There are more checks but this is the main one
and the one that I believe must have failed.
So if anything changes any of these files, temporary or permanent, the
transaction detects it. These two standalone tools change the sstable metadata
and hence probably triggered it.
I think it's reasonable to change {{sstablelevelreset}} to skip temporary
files, because if the transaction did not complete, it's as if these files
never existed. However, I don't think this is sufficient to fix the problem,
because changing the old existing metadata files could also trigger a checksum
error. So I may be wrong, but it seems to me that the real fix is to use the
cleanup utility in the test, before running {{sstablelevelreset}} so that there
are no left over transactions.
If these two tools are likely to be used directly from users when the process
is offline, as they seem to be, I believe that they should cleanup leftover
transactions first, or at least issue a warning if there are any. Otherwise the
main process may refuse to start for the same reason explained above. To
cleanup leftovers we can simply call
{{LifecycleTransaction.removeUnfinishedLeftovers(cfs)}} from the tool itself,
before doing any work. We should consider a follow up to do this, or fix this
directly in this ticket. If we fix this here, then we don't need to do this in
the test.
So you can either merge what you have and open a follow up, or add
{{LifecycleTransaction.removeUnfinishedLeftovers(cfs)}}, as well as kipping the
temporary files (which seems more correct to me), and see if this fixes it
without changing the test.
> dtest failure in
> offline_tools_test.TestOfflineTools.sstableofflinerelevel_test
> ---
>
> Key: CASSANDRA-12519
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12519
> Project: Cassandra
> Issue Type: Improvement
> Components: Test/dtest/python
>Reporter: Sean McCarthy
>Assignee: Andres de la Peña
>Priority: Normal
> Fix For: 4.0-rc2, 4.0, 3.0.x, 3.11.x, 4.0-rc, 4.x
>
> Attachments: node1.log, node1_debug.log, node1_gc.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_offheap_dtest/379/testReport/offline_tools_test/TestOfflineTools/sstableofflinerelevel_test/
> {code}
> Stacktrace
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/automaton/cassandra-dtest/offline_tools_test.py", line 209, in
> sstableofflinerelevel_test
> self.assertGreater(max(final_levels), 1)
> File "/usr/lib/python2.7/unittest/case.py", line 942, in assertGreater
> self.fail(self._formatMessage(msg, standardMsg))
> File "/usr/lib/python2.7/unittest/case.py", line 410, in fail
> raise self.failureException(msg)
> "1 not greater than 1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15702) copyutil.py uses Exception.message which doesn't exist in Python 3
[ https://issues.apache.org/jira/browse/CASSANDRA-15702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania Alborghetti updated CASSANDRA-15702: - Since Version: 4.x Source Control Link: https://gitbox.apache.org/repos/asf?p=cassandra.git;a=commit;h=58015fd681dadbe31068ecfb4f4c0eb506de8efa Resolution: Fixed Status: Resolved (was: Ready to Commit) CI was fine, the only error was a code style compliance issue (missing blank line) introduced by CASSANDRA-15679, which I fixed on commit. > copyutil.py uses Exception.message which doesn't exist in Python 3 > -- > > Key: CASSANDRA-15702 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15702 > Project: Cassandra > Issue Type: Bug > Components: Tool/cqlsh >Reporter: Eduard Tudenhoefner >Assignee: Eduard Tudenhoefner >Priority: Normal > Labels: pull-request-available > Fix For: 4.0-alpha > > Time Spent: 10m > Remaining Estimate: 0h > > Python 3 deprecated and removed the use of Exception.message. The purpose of > this ticket is to convert *Exception.message* to something that is Python 3 > compatible -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15702) copyutil.py uses Exception.message which doesn't exist in Python 3
[ https://issues.apache.org/jira/browse/CASSANDRA-15702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania Alborghetti updated CASSANDRA-15702: - Status: Ready to Commit (was: Review In Progress) > copyutil.py uses Exception.message which doesn't exist in Python 3 > -- > > Key: CASSANDRA-15702 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15702 > Project: Cassandra > Issue Type: Bug > Components: Tool/cqlsh >Reporter: Eduard Tudenhoefner >Assignee: Eduard Tudenhoefner >Priority: Normal > Labels: pull-request-available > Fix For: 4.0-alpha > > Time Spent: 10m > Remaining Estimate: 0h > > Python 3 deprecated and removed the use of Exception.message. The purpose of > this ticket is to convert *Exception.message* to something that is Python 3 > compatible -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: Use Exception.message in copyutil.py only if it exists
This is an automated email from the ASF dual-hosted git repository.
stefania pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new 58015fd Use Exception.message in copyutil.py only if it exists
58015fd is described below
commit 58015fd681dadbe31068ecfb4f4c0eb506de8efa
Author: Eduard Tudenhoefner
AuthorDate: Tue Apr 7 17:16:40 2020 +0200
Use Exception.message in copyutil.py only if it exists
Exception.message was removed in Python 3, so only refer to it when it
exists on an Exception instance.
Also fix code style compliance in formatting.py (missing blank lines)
patch by Eduard Tudenhoefner; reviewed by Stefania Alborghetti for
CASSANDRA-15702
---
pylib/cqlshlib/copyutil.py | 8
pylib/cqlshlib/formatting.py | 2 ++
2 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/pylib/cqlshlib/copyutil.py b/pylib/cqlshlib/copyutil.py
index 853893c..169a6e0 100644
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@ -146,7 +146,7 @@ class SendingChannel(object):
msg = self.pending_messages.get()
self.pipe.send(msg)
except Exception as e:
-printmsg('%s: %s' % (e.__class__.__name__, e.message))
+printmsg('%s: %s' % (e.__class__.__name__, e.message if
hasattr(e, 'message') else str(e)))
feeding_thread = threading.Thread(target=feed)
feeding_thread.setDaemon(True)
@@ -1342,7 +1342,7 @@ class FeedingProcess(mp.Process):
try:
reader.start()
except IOError as exc:
-self.outmsg.send(ImportTaskError(exc.__class__.__name__,
exc.message))
+self.outmsg.send(ImportTaskError(exc.__class__.__name__,
exc.message if hasattr(exc, 'message') else str(exc)))
channels = self.worker_channels
max_pending_chunks = self.max_pending_chunks
@@ -1371,7 +1371,7 @@ class FeedingProcess(mp.Process):
if rows:
sent += self.send_chunk(ch, rows)
except Exception as exc:
-self.outmsg.send(ImportTaskError(exc.__class__.__name__,
exc.message))
+self.outmsg.send(ImportTaskError(exc.__class__.__name__,
exc.message if hasattr(exc, 'message') else str(exc)))
if reader.exhausted:
break
@@ -2632,7 +2632,7 @@ class ImportProcess(ChildProcess):
pk = get_row_partition_key_values(row)
rows_by_ring_pos[get_ring_pos(ring,
pk_to_token_value(pk))].append(row)
except Exception as e:
-errors[e.message].append(row)
+errors[e.message if hasattr(e, 'message') else
str(e)].append(row)
if errors:
for msg, rows in errors.items():
diff --git a/pylib/cqlshlib/formatting.py b/pylib/cqlshlib/formatting.py
index e00d91f..a8ee51d 100644
--- a/pylib/cqlshlib/formatting.py
+++ b/pylib/cqlshlib/formatting.py
@@ -239,6 +239,7 @@ def formatter_for(typname):
return f
return registrator
+
class BlobType(object):
def __init__(self, val):
self.val = val
@@ -246,6 +247,7 @@ class BlobType(object):
def __str__(self):
return str(self.val)
+
@formatter_for('BlobType')
def format_value_blob(val, colormap, **_):
bval = ensure_text('0x') + ensure_text(binascii.hexlify(val))
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15702) copyutil.py uses Exception.message which doesn't exist in Python 3
[ https://issues.apache.org/jira/browse/CASSANDRA-15702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17079420#comment-17079420 ] Stefania Alborghetti commented on CASSANDRA-15702: -- Patch looks good, running cqlsh tests here: [53|https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/53/] > copyutil.py uses Exception.message which doesn't exist in Python 3 > -- > > Key: CASSANDRA-15702 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15702 > Project: Cassandra > Issue Type: Bug > Components: Tool/cqlsh >Reporter: Eduard Tudenhoefner >Assignee: Eduard Tudenhoefner >Priority: Normal > Labels: pull-request-available > Fix For: 4.0-alpha > > Time Spent: 10m > Remaining Estimate: 0h > > Python 3 deprecated and removed the use of Exception.message. The purpose of > this ticket is to convert *Exception.message* to something that is Python 3 > compatible -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15702) copyutil.py uses Exception.message which doesn't exist in Python 3
[ https://issues.apache.org/jira/browse/CASSANDRA-15702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania Alborghetti updated CASSANDRA-15702: - Reviewers: Stefania Alborghetti, Stefania Alborghetti (was: Stefania Alborghetti) Stefania Alborghetti, Stefania Alborghetti Status: Review In Progress (was: Patch Available) > copyutil.py uses Exception.message which doesn't exist in Python 3 > -- > > Key: CASSANDRA-15702 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15702 > Project: Cassandra > Issue Type: Bug > Components: Tool/cqlsh >Reporter: Eduard Tudenhoefner >Assignee: Eduard Tudenhoefner >Priority: Normal > Labels: pull-request-available > Fix For: 4.0-alpha > > Time Spent: 10m > Remaining Estimate: 0h > > Python 3 deprecated and removed the use of Exception.message. The purpose of > this ticket is to convert *Exception.message* to something that is Python 3 > compatible -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15679) cqlsh COPY FROM of map of blobs fails with parse error "unhashable type: 'bytearray'"
[
https://issues.apache.org/jira/browse/CASSANDRA-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17079338#comment-17079338
]
Stefania Alborghetti edited comment on CASSANDRA-15679 at 4/9/20, 1:32 PM:
---
CI runs:
[28|https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/28/]
[30|https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/30/]
[31|https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/31/]
[34|https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/34/]
[35|https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/35/]
Dtest changes:
[88cc70b4d6b598c60d4b567cdceff49a76140312|https://github.com/apache/cassandra-dtest/commit/88cc70b4d6b598c60d4b567cdceff49a76140312]
was (Author: stefania):
CI runs:
[28|https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/28/]
[30|https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/30/]
[31|https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/31/]
[34|https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/34/]
[[35|https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/35/|https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/35/]]
Dtest changes:
[88cc70b4d6b598c60d4b567cdceff49a76140312|https://github.com/apache/cassandra-dtest/commit/88cc70b4d6b598c60d4b567cdceff49a76140312]
> cqlsh COPY FROM of map of blobs fails with parse error "unhashable type:
> 'bytearray'"
> -
>
> Key: CASSANDRA-15679
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15679
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/cqlsh
>Reporter: Erick Ramirez
>Assignee: Aleksandr Sorokoumov
>Priority: Normal
> Labels: pull-request-available
> Fix For: 2.2.17, 3.11.7, 2.1.x, 4.0-alpha
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> h2. Background
> A user was having issues loading CSV data with the {{COPY FROM}} command into
> a {{map}} column with {{blob}} values.
> h2. Replication steps
> I can easily replicate the problem with this simple table:
> {noformat}
> CREATE TABLE community.blobmaptable (
> id text PRIMARY KEY,
> blobmapcol map
> )
> {noformat}
> I have this CSV file that contains just 1 row:
> {noformat}
> $ cat blobmap.csv
> c3,{3: 0x74776f}
> {noformat}
> And here's the error when I try to load it:
> {noformat}
> cqlsh:community> COPY blobmaptable (id, blobmapcol) FROM '~/blobmap.csv' ;
> Using 1 child processes
> Starting copy of community.blobmaptable with columns [id, blobmapcol].
> Failed to import 1 rows: ParseError - Failed to parse {3: 0x74776f} :
> unhashable type: 'bytearray', given up without retries
> Failed to process 1 rows; failed rows written to
> import_community_blobmaptable.err
> Processed: 1 rows; Rate: 2 rows/s; Avg. rate: 3 rows/s
> 1 rows imported from 1 files in 0.389 seconds (0 skipped).
> {noformat}
> I've also logged
> [PYTHON-1234|https://datastax-oss.atlassian.net/browse/PYTHON-1234] because I
> wasn't sure if it was a Python driver issue. Cheers!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15679) cqlsh COPY FROM of map of blobs fails with parse error "unhashable type: 'bytearray'"
[
https://issues.apache.org/jira/browse/CASSANDRA-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania Alborghetti updated CASSANDRA-15679:
-
Fix Version/s: (was: 3.0.21)
(was: 2.1.21)
(was: 4.x)
4.0-alpha
2.1.x
Since Version: 2.1.21
Source Control Link:
https://gitbox.apache.org/repos/asf?p=cassandra.git;a=commit;h=f3568c0d50ac7573b53f0043b2567bde3b39bee8
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
CI runs:
[28|https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/28/]
[30|https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/30/]
[31|https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/31/]
[34|https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/34/]
[[35|https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/35/|https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/35/]]
Dtest changes:
[88cc70b4d6b598c60d4b567cdceff49a76140312|https://github.com/apache/cassandra-dtest/commit/88cc70b4d6b598c60d4b567cdceff49a76140312]
> cqlsh COPY FROM of map of blobs fails with parse error "unhashable type:
> 'bytearray'"
> -
>
> Key: CASSANDRA-15679
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15679
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/cqlsh
>Reporter: Erick Ramirez
>Assignee: Aleksandr Sorokoumov
>Priority: Normal
> Labels: pull-request-available
> Fix For: 2.2.17, 3.11.7, 2.1.x, 4.0-alpha
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> h2. Background
> A user was having issues loading CSV data with the {{COPY FROM}} command into
> a {{map}} column with {{blob}} values.
> h2. Replication steps
> I can easily replicate the problem with this simple table:
> {noformat}
> CREATE TABLE community.blobmaptable (
> id text PRIMARY KEY,
> blobmapcol map
> )
> {noformat}
> I have this CSV file that contains just 1 row:
> {noformat}
> $ cat blobmap.csv
> c3,{3: 0x74776f}
> {noformat}
> And here's the error when I try to load it:
> {noformat}
> cqlsh:community> COPY blobmaptable (id, blobmapcol) FROM '~/blobmap.csv' ;
> Using 1 child processes
> Starting copy of community.blobmaptable with columns [id, blobmapcol].
> Failed to import 1 rows: ParseError - Failed to parse {3: 0x74776f} :
> unhashable type: 'bytearray', given up without retries
> Failed to process 1 rows; failed rows written to
> import_community_blobmaptable.err
> Processed: 1 rows; Rate: 2 rows/s; Avg. rate: 3 rows/s
> 1 rows imported from 1 files in 0.389 seconds (0 skipped).
> {noformat}
> I've also logged
> [PYTHON-1234|https://datastax-oss.atlassian.net/browse/PYTHON-1234] because I
> wasn't sure if it was a Python driver issue. Cheers!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15679) cqlsh COPY FROM of map of blobs fails with parse error "unhashable type: 'bytearray'"
[
https://issues.apache.org/jira/browse/CASSANDRA-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania Alborghetti updated CASSANDRA-15679:
-
Status: Ready to Commit (was: Review In Progress)
> cqlsh COPY FROM of map of blobs fails with parse error "unhashable type:
> 'bytearray'"
> -
>
> Key: CASSANDRA-15679
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15679
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/cqlsh
>Reporter: Erick Ramirez
>Assignee: Aleksandr Sorokoumov
>Priority: Normal
> Labels: pull-request-available
> Fix For: 2.1.21, 2.2.17, 3.0.21, 3.11.7, 4.x
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> h2. Background
> A user was having issues loading CSV data with the {{COPY FROM}} command into
> a {{map}} column with {{blob}} values.
> h2. Replication steps
> I can easily replicate the problem with this simple table:
> {noformat}
> CREATE TABLE community.blobmaptable (
> id text PRIMARY KEY,
> blobmapcol map
> )
> {noformat}
> I have this CSV file that contains just 1 row:
> {noformat}
> $ cat blobmap.csv
> c3,{3: 0x74776f}
> {noformat}
> And here's the error when I try to load it:
> {noformat}
> cqlsh:community> COPY blobmaptable (id, blobmapcol) FROM '~/blobmap.csv' ;
> Using 1 child processes
> Starting copy of community.blobmaptable with columns [id, blobmapcol].
> Failed to import 1 rows: ParseError - Failed to parse {3: 0x74776f} :
> unhashable type: 'bytearray', given up without retries
> Failed to process 1 rows; failed rows written to
> import_community_blobmaptable.err
> Processed: 1 rows; Rate: 2 rows/s; Avg. rate: 3 rows/s
> 1 rows imported from 1 files in 0.389 seconds (0 skipped).
> {noformat}
> I've also logged
> [PYTHON-1234|https://datastax-oss.atlassian.net/browse/PYTHON-1234] because I
> wasn't sure if it was a Python driver issue. Cheers!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-dtest] branch master updated: Add cqlsh copy tests for blobs in collections
This is an automated email from the ASF dual-hosted git repository.
stefania pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-dtest.git
The following commit(s) were added to refs/heads/master by this push:
new 88cc70b Add cqlsh copy tests for blobs in collections
88cc70b is described below
commit 88cc70b4d6b598c60d4b567cdceff49a76140312
Author: Aleksandr Sorokoumov
AuthorDate: Thu Apr 2 11:20:06 2020 +0200
Add cqlsh copy tests for blobs in collections
patch by Aleksandr Sorokoumov; reviewed by Stefania Alborghetti for
CASSANDRA-15679
---
cqlsh_tests/test_cqlsh_copy.py | 28
1 file changed, 20 insertions(+), 8 deletions(-)
diff --git a/cqlsh_tests/test_cqlsh_copy.py b/cqlsh_tests/test_cqlsh_copy.py
index 76805e0..286b488 100644
--- a/cqlsh_tests/test_cqlsh_copy.py
+++ b/cqlsh_tests/test_cqlsh_copy.py
@@ -260,7 +260,10 @@ class TestCqlshCopy(Tester):
u frozen>>,
v frozen,set>>,
w frozen>>,
-x map>>
+x map>>,
+y map,
+z list,
+za set
)''')
self.session.cluster.register_user_type('ks', 'name_type', Name)
@@ -302,7 +305,10 @@ class TestCqlshCopy(Tester):
# and this will cause comparison problems when comparing
with csv strings therefore failing
# some tests
ImmutableSet([ImmutableSet(['127.0.0.1']),
ImmutableSet(['127.0.0.1', '127.0.0.2'])]),
- {'key1': ['value1', 'value2']} # map>>
+ {'key1': ['value1', 'value2']}, # map>>
+ {3: bytes.fromhex('74776f')}, # y
+ [bytes.fromhex('74776f')], # z
+ {bytes.fromhex('74776f')} # za
)
def assertCsvResultEqual(self, csv_filename, results, table_name=None,
@@ -1649,12 +1655,11 @@ class TestCqlshCopy(Tester):
@jira_ticket CASSANDRA-9302
"""
-
self.all_datatypes_prepare()
insert_statement = self.session.prepare(
-"""INSERT INTO testdatatype (a, b, c, d, e, f, g, h, i, j, k, l,
m, n, o, p, q, r, s, t, u, v, w, x)
-VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?)""")
+"""INSERT INTO testdatatype (a, b, c, d, e, f, g, h, i, j, k, l,
m, n, o, p, q, r, s, t, u, v, w, x, y, z, za)
+VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?, ?, ?, ?)""")
self.session.execute(insert_statement, self.data)
def _test(prepared_statements):
@@ -1691,13 +1696,20 @@ class TestCqlshCopy(Tester):
writer = csv.writer(csvfile)
# serializing blob bytearray in friendly format
data_set = list(self.data)
+
data_set[2] = self.format_blob(self.data[2])
+# Here we convert containers of blobs to strings that match
exactly the output of the SELECT *
+# because otherwise the comparison fails due to extra quotes added
by the csv writer around the blobs
+# that were converted to strings. White spaces do matter
+data_set[24] = '{3: ' + self.format_blob(self.data[24][3]) + '}'
+data_set[25] = '[' + ', '.join(self.format_blob(b) for b in
self.data[25]) + ']'
+data_set[26] = '{' + ', '.join(self.format_blob(b) for b in
self.data[26]) + '}'
writer.writerow(data_set)
def _test(prepared_statements):
logger.debug('Importing from csv file:
{name}'.format(name=tempfile.name))
out, err, _ = self.run_cqlsh(cmds="COPY ks.testdatatype FROM '{}'
WITH PREPAREDSTATEMENTS = {}"
- .format(tempfile.name, prepared_statements))
+ .format(tempfile.name, prepared_statements))
out, err, _ = self.run_cqlsh(cmds="SELECT * FROM ks.testdatatype")
results = self.parse_cqlsh_query(out=out, num_cols=len(self.data),
timestamps_to_be_rounded=[10, 17])
@@ -1725,8 +1737,8 @@ class TestCqlshCopy(Tester):
self.all_datatypes_prepare()
insert_statement = self.session.prepare(
-"""INSERT INTO testdatatype (a, b, c, d, e, f, g, h, i, j, k, l,
m, n, o, p, q, r, s, t, u, v, w, x)
-VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?)""")
+"""INSERT INTO testdatatype (a, b, c, d, e, f, g, h, i, j, k, l,
m, n, o, p, q, r, s, t, u, v, w, x, y, z, za)
+VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?, ?, ?, ?)""")
self.session.execute(insert_stat
[cassandra] 01/01: Merge branch 'cassandra-3.11' into trunk
This is an automated email from the ASF dual-hosted git repository.
stefania pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 2f781819b184ba306ed0cc4412441a3c444365d6
Merge: 3db7796 57b0bb9
Author: Stefania Alborghetti
AuthorDate: Thu Apr 9 09:05:14 2020 -0400
Merge branch 'cassandra-3.11' into trunk
CHANGES.txt | 2 ++
pylib/cqlshlib/copyutil.py | 7 +--
pylib/cqlshlib/formatting.py | 9 -
3 files changed, 15 insertions(+), 3 deletions(-)
diff --cc pylib/cqlshlib/copyutil.py
index 3bed2ad,b91bb76..853893c
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@@ -56,12 -50,11 +56,12 @@@ from cassandra.metadata import protect_
from cassandra.policies import RetryPolicy, WhiteListRoundRobinPolicy,
DCAwareRoundRobinPolicy, FallthroughRetryPolicy
from cassandra.query import BatchStatement, BatchType, SimpleStatement,
tuple_factory
from cassandra.util import Date, Time
+from cqlshlib.util import profile_on, profile_off
-from cql3handling import CqlRuleSet
-from displaying import NO_COLOR_MAP
-from formatting import format_value_default, CqlType, DateTimeFormat, EMPTY,
get_formatter, BlobType
-from sslhandling import ssl_settings
+from cqlshlib.cql3handling import CqlRuleSet
+from cqlshlib.displaying import NO_COLOR_MAP
- from cqlshlib.formatting import format_value_default, CqlType,
DateTimeFormat, EMPTY, get_formatter
++from cqlshlib.formatting import format_value_default, CqlType,
DateTimeFormat, EMPTY, get_formatter, BlobType
+from cqlshlib.sslhandling import ssl_settings
PROFILE_ON = False
STRACE_ON = False
@@@ -1933,10 -1868,10 +1933,13 @@@ class ImportConversion(object)
return converters.get(t.typename, convert_unknown)(v, ct=t)
def convert_blob(v, **_):
- return bytearray.fromhex(v[2:])
-return BlobType(v[2:].decode("hex"))
++if sys.version_info.major >= 3:
++return bytes.fromhex(v[2:])
++else:
++return BlobType(v[2:].decode("hex"))
def convert_text(v, **_):
-return v
+return ensure_str(v)
def convert_uuid(v, **_):
return UUID(v)
diff --cc pylib/cqlshlib/formatting.py
index 7ed6735,9927aa1..e00d91f
--- a/pylib/cqlshlib/formatting.py
+++ b/pylib/cqlshlib/formatting.py
@@@ -239,10 -236,16 +239,16 @@@ def formatter_for(typname)
return f
return registrator
+ class BlobType(object):
+ def __init__(self, val):
+ self.val = val
- @formatter_for('bytearray')
+ def __str__(self):
+ return str(self.val)
+
+ @formatter_for('BlobType')
def format_value_blob(val, colormap, **_):
-bval = '0x' + binascii.hexlify(val)
+bval = ensure_text('0x') + ensure_text(binascii.hexlify(val))
return colorme(bval, colormap, 'blob')
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-2.1 updated: Fix parse error in cqlsh COPY FROM and formatting for map of blobs
This is an automated email from the ASF dual-hosted git repository.
stefania pushed a commit to branch cassandra-2.1
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cassandra-2.1 by this push:
new f3568c0 Fix parse error in cqlsh COPY FROM and formatting for map of
blobs
f3568c0 is described below
commit f3568c0d50ac7573b53f0043b2567bde3b39bee8
Author: Aleksandr Sorokoumov
AuthorDate: Wed Apr 1 16:32:52 2020 +0200
Fix parse error in cqlsh COPY FROM and formatting for map of blobs
patch by Aleksandr Sorokoumov; reviewed by Stefania Alborghetti for
CASSANDRA-15679
---
CHANGES.txt | 1 +
pylib/cqlshlib/copyutil.py | 8 ++--
pylib/cqlshlib/formatting.py | 9 -
3 files changed, 11 insertions(+), 7 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 3582d4f..d0d0993 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.1.21
+ * Fix parse error in cqlsh COPY FROM and formatting for map of blobs
(CASSANDRA-15679)
* Paged Range Slice queries with DISTINCT can drop rows from results
(CASSANDRA-14956)
* Update release checksum algorithms to SHA-256, SHA-512 (CASSANDRA-14970)
* Check checksum before decompressing data (CASSANDRA-14284)
diff --git a/pylib/cqlshlib/copyutil.py b/pylib/cqlshlib/copyutil.py
index 85e2678..a8bd0a2 100644
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@ -52,7 +52,7 @@ from cassandra.util import Date, Time
from cql3handling import CqlRuleSet
from displaying import NO_COLOR_MAP
-from formatting import format_value_default, EMPTY, get_formatter
+from formatting import format_value_default, EMPTY, get_formatter, BlobType
from sslhandling import ssl_settings
PROFILE_ON = False
@@ -1726,11 +1726,7 @@ class ImportConversion(object):
return converters.get(t.typename, convert_unknown)(v, ct=t)
def convert_blob(v, **_):
-try:
-return bytearray.fromhex(v[2:])
-except TypeError:
-# Work-around for Python 2.6 bug
-return bytearray.fromhex(unicode(v[2:]))
+return BlobType(unicode(v[2:]).decode("hex"))
def convert_text(v, **_):
return v
diff --git a/pylib/cqlshlib/formatting.py b/pylib/cqlshlib/formatting.py
index eac1810..436404a 100644
--- a/pylib/cqlshlib/formatting.py
+++ b/pylib/cqlshlib/formatting.py
@@ -126,13 +126,20 @@ def formatter_for(typname):
return f
return registrator
+class BlobType(object):
+def __init__(self, val):
+self.val = val
-@formatter_for('bytearray')
+def __str__(self):
+return str(self.val)
+
+@formatter_for('BlobType')
def format_value_blob(val, colormap, **_):
bval = '0x' + binascii.hexlify(str(val))
return colorme(bval, colormap, 'blob')
+formatter_for('bytearray')(format_value_blob)
formatter_for('buffer')(format_value_blob)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-2.2' into cassandra-3.0
This is an automated email from the ASF dual-hosted git repository. stefania pushed a commit to branch cassandra-3.0 in repository https://gitbox.apache.org/repos/asf/cassandra.git commit fe263ad9e98c1d43eef74b88fcb111b697f619ef Merge: e3f54d4 c8081c2 Author: Stefania Alborghetti AuthorDate: Thu Apr 9 08:58:55 2020 -0400 Merge branch 'cassandra-2.2' into cassandra-3.0 CHANGES.txt | 2 ++ pylib/cqlshlib/copyutil.py | 4 ++-- pylib/cqlshlib/formatting.py | 9 - 3 files changed, 12 insertions(+), 3 deletions(-) diff --cc CHANGES.txt index 2d14e34,56dd315..eb0b66e --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,8 -1,9 +1,10 @@@ -2.2.18 +3.0.21 + * Memtable memory allocations may deadlock (CASSANDRA-15367) + * Run evictFromMembership in GossipStage (CASSANDRA-15592) +Merged from 2.2: * Disable JMX rebinding (CASSANDRA-15653) + Merged from 2.1: + * Fix parse error in cqlsh COPY FROM and formatting for map of blobs (CASSANDRA-15679) - -2.2.17 * Fix Commit log replays when static column clustering keys are collections (CASSANDRA-14365) * Fix Red Hat init script on newer systemd versions (CASSANDRA-15273) * Allow EXTRA_CLASSPATH to work on tar/source installations (CASSANDRA-15567) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated (3db7796 -> 2f78181)
This is an automated email from the ASF dual-hosted git repository. stefania pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git. from 3db7796 Fix cqlsh output test new f3568c0 Fix parse error in cqlsh COPY FROM and formatting for map of blobs new c8081c2 Merge branch 'cassandra-2.1' into cassandra-2.2 new fe263ad Merge branch 'cassandra-2.2' into cassandra-3.0 new 57b0bb9 Merge branch 'cassandra-3.0' into cassandra-3.11 new 2f78181 Merge branch 'cassandra-3.11' into trunk The 5 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt | 2 ++ pylib/cqlshlib/copyutil.py | 7 +-- pylib/cqlshlib/formatting.py | 9 - 3 files changed, 15 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-2.2 updated (3940a9e -> c8081c2)
This is an automated email from the ASF dual-hosted git repository. stefania pushed a change to branch cassandra-2.2 in repository https://gitbox.apache.org/repos/asf/cassandra.git. from 3940a9e Jenkins 'Cassandra' label applied to the declarative pipeline new f3568c0 Fix parse error in cqlsh COPY FROM and formatting for map of blobs new c8081c2 Merge branch 'cassandra-2.1' into cassandra-2.2 The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt | 2 ++ pylib/cqlshlib/copyutil.py | 4 ++-- pylib/cqlshlib/formatting.py | 9 - 3 files changed, 12 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-2.1' into cassandra-2.2
This is an automated email from the ASF dual-hosted git repository.
stefania pushed a commit to branch cassandra-2.2
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit c8081c2467cfa34efec5e44c7dd31b7a99db5945
Merge: 3940a9e f3568c0
Author: Stefania Alborghetti
AuthorDate: Thu Apr 9 08:56:38 2020 -0400
Merge branch 'cassandra-2.1' into cassandra-2.2
CHANGES.txt | 2 ++
pylib/cqlshlib/copyutil.py | 4 ++--
pylib/cqlshlib/formatting.py | 9 -
3 files changed, 12 insertions(+), 3 deletions(-)
diff --cc CHANGES.txt
index 885d33e,d0d0993..56dd315
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,55 -1,7 +1,57 @@@
-2.1.21
- * Fix parse error in cqlsh COPY FROM and formatting for map of blobs
(CASSANDRA-15679)
+2.2.18
+ * Disable JMX rebinding (CASSANDRA-15653)
++Merged from 2.1:
++ * Fix parse error in cqlsh COPY FROM and formatting for map of blobs
(CASSANDRA-15679)
+
+2.2.17
+ * Fix Commit log replays when static column clustering keys are collections
(CASSANDRA-14365)
+ * Fix Red Hat init script on newer systemd versions (CASSANDRA-15273)
+ * Allow EXTRA_CLASSPATH to work on tar/source installations (CASSANDRA-15567)
+
+2.2.16
+ * Fix SELECT JSON output for empty blobs (CASSANDRA-15435)
+ * In-JVM DTest: Set correct internode message version for upgrade test
(CASSANDRA-15371)
+ * In-JVM DTest: Support NodeTool in dtest
+
+2.2.15
+ * Catch non-IOException in FileUtils.close to make sure that all resources
are closed (CASSANDRA-15225)
+ * Handle exceptions during authentication/authorization (CASSANDRA-15041)
+ * Fix JDK7 compatibility broken in cassandra-2.2 (CASSANDRA-15050)
+ * Support cross version messaging in in-jvm upgrade dtests (CASSANDRA-15078)
+ * Fix index summary redistribution cancellation (CASSANDRA-15045)
+ * Refactor Circle CI configuration (CASSANDRA-14806)
+ * Fixing invalid CQL in security documentation (CASSANDRA-15020)
+ * Make tools/bin/token-generator py2/3 compatible (CASSANDRA-15012)
+ * Multi-version in-JVM dtests (CASSANDRA-14937)
+ * Allow instance class loaders to be garbage collected for inJVM dtest
(CASSANDRA-15170)
+ * Add support for network topology and query tracing for inJVM dtest
(CASSANDRA-15319)
+
+
+2.2.14
+ * CircleCI docker image should bake in more dependencies (CASSANDRA-14985)
+ * Don't enable client transports when bootstrap is pending (CASSANDRA-14525)
+ * MigrationManager attempts to pull schema from different major version
nodes (CASSANDRA-14928)
+ * Don't skip entire sstables when reading backwards with mixed clustering
column order
+ (CASSANDRA-14910)
+ * Cannot perform slice reads in reverse direction against tables with
clustering columns
+ in mixed order (CASSANDRA-14899)
+ * Fix incorrect cqlsh results when selecting same columns multiple times
(CASSANDRA-13262)
+ * Returns null instead of NaN or Infinity in JSON strings (CASSANDRA-14377)
+Merged from 2.1:
* Paged Range Slice queries with DISTINCT can drop rows from results
(CASSANDRA-14956)
* Update release checksum algorithms to SHA-256, SHA-512 (CASSANDRA-14970)
+
+
+2.2.13
+ * Fix bug that prevented compaction of SSTables after full repairs
(CASSANDRA-14423)
+ * Incorrect counting of pending messages in OutboundTcpConnection
(CASSANDRA-11551)
+ * Fix compaction failure caused by reading un-flushed data (CASSANDRA-12743)
+ * Use Bounds instead of Range for sstables in anticompaction
(CASSANDRA-14411)
+ * Fix JSON queries with IN restrictions and ORDER BY clause (CASSANDRA-14286)
+ * CQL fromJson(null) throws NullPointerException (CASSANDRA-13891)
+ * Fix query pager DEBUG log leak causing hit in paged reads throughput
(CASSANDRA-14318)
+ * Backport circleci yaml (CASSANDRA-14240)
+Merged from 2.1:
* Check checksum before decompressing data (CASSANDRA-14284)
* CVE-2017-5929 Security vulnerability in Logback warning in NEWS.txt
(CASSANDRA-14183)
diff --cc pylib/cqlshlib/copyutil.py
index c9c5829,a8bd0a2..b28d96d
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@@ -52,7 -52,7 +52,7 @@@ from cassandra.util import Date, Tim
from cql3handling import CqlRuleSet
from displaying import NO_COLOR_MAP
- from formatting import format_value_default, DateTimeFormat, EMPTY,
get_formatter
-from formatting import format_value_default, EMPTY, get_formatter, BlobType
++from formatting import format_value_default, DateTimeFormat, EMPTY,
get_formatter, BlobType
from sslhandling import ssl_settings
PROFILE_ON = False
@@@ -1810,7 -1726,7 +1810,7 @@@ class ImportConversion(object)
return converters.get(t.typename, convert_unknown)(v, ct=t)
def convert_blob(v, **_):
- return bytearray.fromhex(v[2:])
-return BlobType(unicode(v[2:]).decode("hex"))
++return BlobType(v[2:].decode("hex"))
def convert_text(v, **_):
return v
diff --cc pylib/cqlshlib
[cassandra] 01/01: Merge branch 'cassandra-3.0' into cassandra-3.11
This is an automated email from the ASF dual-hosted git repository.
stefania pushed a commit to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 57b0bb93ab4811529918300c41fbb08433f4f707
Merge: 912c6c4 fe263ad
Author: Stefania Alborghetti
AuthorDate: Thu Apr 9 09:00:58 2020 -0400
Merge branch 'cassandra-3.0' into cassandra-3.11
CHANGES.txt | 2 ++
pylib/cqlshlib/copyutil.py | 4 ++--
pylib/cqlshlib/formatting.py | 9 -
3 files changed, 12 insertions(+), 3 deletions(-)
diff --cc CHANGES.txt
index 9342825,eb0b66e..ce4775b
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -7,6 -3,8 +7,8 @@@ Merged from 3.0
* Run evictFromMembership in GossipStage (CASSANDRA-15592)
Merged from 2.2:
* Disable JMX rebinding (CASSANDRA-15653)
+ Merged from 2.1:
- * Fix parse error in cqlsh COPY FROM and formatting for map of blobs
(CASSANDRA-15679)
++ * Fix parse error in cqlsh COPY FROM and formatting for map of blobs
(CASSANDRA-15679)
* Fix Commit log replays when static column clustering keys are collections
(CASSANDRA-14365)
* Fix Red Hat init script on newer systemd versions (CASSANDRA-15273)
* Allow EXTRA_CLASSPATH to work on tar/source installations (CASSANDRA-15567)
diff --cc pylib/cqlshlib/copyutil.py
index 7f97b49,6a45153..b91bb76
--- a/pylib/cqlshlib/copyutil.py
+++ b/pylib/cqlshlib/copyutil.py
@@@ -53,7 -53,7 +53,7 @@@ from cassandra.util import Date, Tim
from cql3handling import CqlRuleSet
from displaying import NO_COLOR_MAP
- from formatting import format_value_default, CqlType, DateTimeFormat, EMPTY,
get_formatter
-from formatting import format_value_default, DateTimeFormat, EMPTY,
get_formatter, BlobType
++from formatting import format_value_default, CqlType, DateTimeFormat, EMPTY,
get_formatter, BlobType
from sslhandling import ssl_settings
PROFILE_ON = False
diff --cc pylib/cqlshlib/formatting.py
index 803ea63,53ba478..9927aa1
--- a/pylib/cqlshlib/formatting.py
+++ b/pylib/cqlshlib/formatting.py
@@@ -243,8 -160,8 +249,9 @@@ def format_value_blob(val, colormap, **
return colorme(bval, colormap, 'blob')
+ formatter_for('bytearray')(format_value_blob)
formatter_for('buffer')(format_value_blob)
+formatter_for('blob')(format_value_blob)
def format_python_formatted_type(val, colormap, color, quote=False):
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-3.11 updated (912c6c4 -> 57b0bb9)
This is an automated email from the ASF dual-hosted git repository. stefania pushed a change to branch cassandra-3.11 in repository https://gitbox.apache.org/repos/asf/cassandra.git. from 912c6c4 Merge branch 'cassandra-3.0' into cassandra-3.11 new f3568c0 Fix parse error in cqlsh COPY FROM and formatting for map of blobs new c8081c2 Merge branch 'cassandra-2.1' into cassandra-2.2 new fe263ad Merge branch 'cassandra-2.2' into cassandra-3.0 new 57b0bb9 Merge branch 'cassandra-3.0' into cassandra-3.11 The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt | 2 ++ pylib/cqlshlib/copyutil.py | 4 ++-- pylib/cqlshlib/formatting.py | 9 - 3 files changed, 12 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-3.0 updated (e3f54d4 -> fe263ad)
This is an automated email from the ASF dual-hosted git repository. stefania pushed a change to branch cassandra-3.0 in repository https://gitbox.apache.org/repos/asf/cassandra.git. from e3f54d4 Memtable memory allocations may deadlock new f3568c0 Fix parse error in cqlsh COPY FROM and formatting for map of blobs new c8081c2 Merge branch 'cassandra-2.1' into cassandra-2.2 new fe263ad Merge branch 'cassandra-2.2' into cassandra-3.0 The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt | 2 ++ pylib/cqlshlib/copyutil.py | 4 ++-- pylib/cqlshlib/formatting.py | 9 - 3 files changed, 12 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14050) Many cqlsh_copy_tests are busted
[
https://issues.apache.org/jira/browse/CASSANDRA-14050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania Alborghetti updated CASSANDRA-14050:
-
Fix Version/s: 4.0-alpha
Since Version: 4.x
Source Control Link:
https://github.com/apache/cassandra-dtest/commit/da8abe3cab3fc186a6cfb2e3771f647a0dac120e
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
> Many cqlsh_copy_tests are busted
>
>
> Key: CASSANDRA-14050
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14050
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Testing
>Reporter: Michael Kjellman
>Assignee: Stefania Alborghetti
>Priority: Normal
> Fix For: 4.0-alpha
>
>
> Many cqlsh_copy_tests are busted. We should disable the entire suite until
> this is resolved as these tests are currently nothing but a waste of time.
> test_bulk_round_trip_blogposts - cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest
> test_bulk_round_trip_blogposts_with_max_connections -
> cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest
> test_bulk_round_trip_default - cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest
> Error starting node3.
> >> begin captured logging <<
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-S9NfIH
> dtest: DEBUG: Done setting configuration options:
> { 'initial_token': None,
> 'memtable_allocation_type': 'offheap_objects',
> 'num_tokens': '256',
> 'phi_convict_threshold': 5,
> 'range_request_timeout_in_ms': 1,
> 'read_request_timeout_in_ms': 1,
> 'request_timeout_in_ms': 1,
> 'truncate_request_timeout_in_ms': 1,
> 'write_request_timeout_in_ms': 1}
> - >> end captured logging << -
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/cassandra/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py",
> line 2546, in test_bulk_round_trip_blogposts
> stress_table='stresscql.blogposts')
> File "/home/cassandra/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py",
> line 2451, in _test_bulk_round_trip
> self.prepare(nodes=nodes, partitioner=partitioner,
> configuration_options=configuration_options)
> File "/home/cassandra/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py",
> line 115, in prepare
> self.cluster.populate(nodes,
> tokens=tokens).start(wait_for_binary_proto=True)
> File
> "/home/cassandra/env/local/lib/python2.7/site-packages/ccmlib/cluster.py",
> line 423, in start
> raise NodeError("Error starting {0}.".format(node.name), p)
> "Error starting node3.\n >> begin captured logging <<
> \ndtest: DEBUG: cluster ccm directory:
> /tmp/dtest-S9NfIH\ndtest: DEBUG: Done setting configuration options:\n{
> 'initial_token': None,\n'memtable_allocation_type': 'offheap_objects',\n
> 'num_tokens': '256',\n'phi_convict_threshold': 5,\n
> 'range_request_timeout_in_ms': 1,\n'read_request_timeout_in_ms':
> 1,\n'request_timeout_in_ms': 1,\n
> 'truncate_request_timeout_in_ms': 1,\n'write_request_timeout_in_ms':
> 1}\n- >> end captured logging <<
> -"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14050) Many cqlsh_copy_tests are busted
[
https://issues.apache.org/jira/browse/CASSANDRA-14050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania Alborghetti updated CASSANDRA-14050:
-
Status: Ready to Commit (was: Review In Progress)
> Many cqlsh_copy_tests are busted
>
>
> Key: CASSANDRA-14050
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14050
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Testing
>Reporter: Michael Kjellman
>Assignee: Stefania Alborghetti
>Priority: Normal
>
> Many cqlsh_copy_tests are busted. We should disable the entire suite until
> this is resolved as these tests are currently nothing but a waste of time.
> test_bulk_round_trip_blogposts - cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest
> test_bulk_round_trip_blogposts_with_max_connections -
> cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest
> test_bulk_round_trip_default - cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest
> Error starting node3.
> >> begin captured logging <<
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-S9NfIH
> dtest: DEBUG: Done setting configuration options:
> { 'initial_token': None,
> 'memtable_allocation_type': 'offheap_objects',
> 'num_tokens': '256',
> 'phi_convict_threshold': 5,
> 'range_request_timeout_in_ms': 1,
> 'read_request_timeout_in_ms': 1,
> 'request_timeout_in_ms': 1,
> 'truncate_request_timeout_in_ms': 1,
> 'write_request_timeout_in_ms': 1}
> - >> end captured logging << -
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/cassandra/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py",
> line 2546, in test_bulk_round_trip_blogposts
> stress_table='stresscql.blogposts')
> File "/home/cassandra/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py",
> line 2451, in _test_bulk_round_trip
> self.prepare(nodes=nodes, partitioner=partitioner,
> configuration_options=configuration_options)
> File "/home/cassandra/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py",
> line 115, in prepare
> self.cluster.populate(nodes,
> tokens=tokens).start(wait_for_binary_proto=True)
> File
> "/home/cassandra/env/local/lib/python2.7/site-packages/ccmlib/cluster.py",
> line 423, in start
> raise NodeError("Error starting {0}.".format(node.name), p)
> "Error starting node3.\n >> begin captured logging <<
> \ndtest: DEBUG: cluster ccm directory:
> /tmp/dtest-S9NfIH\ndtest: DEBUG: Done setting configuration options:\n{
> 'initial_token': None,\n'memtable_allocation_type': 'offheap_objects',\n
> 'num_tokens': '256',\n'phi_convict_threshold': 5,\n
> 'range_request_timeout_in_ms': 1,\n'read_request_timeout_in_ms':
> 1,\n'request_timeout_in_ms': 1,\n
> 'truncate_request_timeout_in_ms': 1,\n'write_request_timeout_in_ms':
> 1}\n- >> end captured logging <<
> -"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15229) BufferPool Regression
[
https://issues.apache.org/jira/browse/CASSANDRA-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077539#comment-17077539
]
Stefania Alborghetti commented on CASSANDRA-15229:
--
{quote}My view is that having a significant proportion of memory wasted to
fragmentation is a serious bug, irregardless of the total amount of memory that
is wasted.
{quote}
That's absolutely true. However, it's also true that none of our users reported
any problems when the cache was 512 MB and the default file access mode was
mmap. Perhaps there are users in open source that reported problems, I haven't
done a Jira search. So my point was simply meant to say that we should be
mindful of changing critical code late in a release cycle if the existing code
is performing adequately.
{quote}It's not poorly suited to long lived buffers its it? Only to buffers
with widely divergent lifetimes.
{quote}
I implied the fact that lifetimes are divergent, since we're trying to support
a cache, sorry about the confusion.
{quote}Honestly, given chunks are normally the same size, simply re-using the
evicted buffer if possible, and if not allocating new system memory, seems
probably sufficient to me.
{quote}
I'm not too sure that chunks are normally the same size. For data files, they
depend on the compression parameters or on the partition sizes, both could be
different for different tables. Also, indexes would use different chunk sizes
surely? We observed that the chunk cache gradually tends to shift from buffers
coming from data files to buffers coming from index files, as indexes are
accessed more frequently. We have a different index implementation though.
{quote}{quote}I'll try to share some code so you can have a clearer picture.
{quote}
Thanks, that sounds great. I may not get to it immediately, but look forward to
taking a look hopefully soon.
{quote}
I've dropped some files on this
[branch|https://github.com/stef1927/cassandra/tree/15229-4.0]. The buffer pool
is in org.apache.cassandra.utils.memory.buffers. The starting point is the
[BufferPool|https://github.com/apache/cassandra/compare/trunk...stef1927:15229-4.0#diff-72046b5d367f6e120594b58c973bed71R24]
and its concrete implementations or the
[BufferFactory|https://github.com/apache/cassandra/compare/trunk...stef1927:15229-4.0#diff-4fc5fae1de112fc5eb0bd865af532f0aR31].
I've also dropped some related utility classes but not all of them, so clearly
the code doesn't compile and the unit tests are also missing.
> BufferPool Regression
> -
>
> Key: CASSANDRA-15229
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15229
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Caching
>Reporter: Benedict Elliott Smith
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 4.0, 4.0-beta
>
>
> The BufferPool was never intended to be used for a {{ChunkCache}}, and we
> need to either change our behaviour to handle uncorrelated lifetimes or use
> something else. This is particularly important with the default chunk size
> for compressed sstables being reduced. If we address the problem, we should
> also utilise the BufferPool for native transport connections like we do for
> internode messaging, and reduce the number of pooling solutions we employ.
> Probably the best thing to do is to improve BufferPool’s behaviour when used
> for things with uncorrelated lifetimes, which essentially boils down to
> tracking those chunks that have not been freed and re-circulating them when
> we run out of completely free blocks. We should probably also permit
> instantiating separate {{BufferPool}}, so that we can insulate internode
> messaging from the {{ChunkCache}}, or at least have separate memory bounds
> for each, and only share fully-freed chunks.
> With these improvements we can also safely increase the {{BufferPool}} chunk
> size to 128KiB or 256KiB, to guarantee we can fit compressed pages and reduce
> the amount of global coordination and per-allocation overhead. We don’t need
> 1KiB granularity for allocations, nor 16 byte granularity for tiny
> allocations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14050) Many cqlsh_copy_tests are busted
[
https://issues.apache.org/jira/browse/CASSANDRA-14050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania Alborghetti updated CASSANDRA-14050:
-
Test and Documentation Plan:
[https://github.com/apache/cassandra-dtest/pull/62]
The dependency of the dtests to cqlshlib formatting.py has been removed by
running {{SELECT *}} queries with cqlsh rather than the driver. This avoids
applying the formatting manually. In some cases, where formatting was trivial
and the number of rows was significant, the driver is still used and the
formatting is done manually (normally a conversion to a string was sufficient).
This was done to overcome cqlsh paging. Although we can disable paging, parsing
a large number of rows could cause memory or speed problems.
Status: Patch Available (was: In Progress)
I've submitted a [patch|https://github.com/apache/cassandra-dtest/pull/62]
based on the third approach, see documentation plan.
CI runs:
* [2.1|https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/25]
* [2.2|https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/26]
* [3.0|https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/27]
* [3.11|https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/23]
* [4.0|https://ci-cassandra.apache.org/job/Cassandra-devbranch-dtest/24]
> Many cqlsh_copy_tests are busted
>
>
> Key: CASSANDRA-14050
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14050
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Testing
>Reporter: Michael Kjellman
>Assignee: Stefania Alborghetti
>Priority: Normal
>
> Many cqlsh_copy_tests are busted. We should disable the entire suite until
> this is resolved as these tests are currently nothing but a waste of time.
> test_bulk_round_trip_blogposts - cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest
> test_bulk_round_trip_blogposts_with_max_connections -
> cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest
> test_bulk_round_trip_default - cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest
> Error starting node3.
> >> begin captured logging <<
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-S9NfIH
> dtest: DEBUG: Done setting configuration options:
> { 'initial_token': None,
> 'memtable_allocation_type': 'offheap_objects',
> 'num_tokens': '256',
> 'phi_convict_threshold': 5,
> 'range_request_timeout_in_ms': 1,
> 'read_request_timeout_in_ms': 1,
> 'request_timeout_in_ms': 1,
> 'truncate_request_timeout_in_ms': 1,
> 'write_request_timeout_in_ms': 1}
> - >> end captured logging << -
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/cassandra/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py",
> line 2546, in test_bulk_round_trip_blogposts
> stress_table='stresscql.blogposts')
> File "/home/cassandra/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py",
> line 2451, in _test_bulk_round_trip
> self.prepare(nodes=nodes, partitioner=partitioner,
> configuration_options=configuration_options)
> File "/home/cassandra/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py",
> line 115, in prepare
> self.cluster.populate(nodes,
> tokens=tokens).start(wait_for_binary_proto=True)
> File
> "/home/cassandra/env/local/lib/python2.7/site-packages/ccmlib/cluster.py",
> line 423, in start
> raise NodeError("Error starting {0}.".format(node.name), p)
> "Error starting node3.\n >> begin captured logging <<
> \ndtest: DEBUG: cluster ccm directory:
> /tmp/dtest-S9NfIH\ndtest: DEBUG: Done setting configuration options:\n{
> 'initial_token': None,\n'memtable_allocation_type': 'offheap_objects',\n
> 'num_tokens': '256',\n'phi_convict_threshold': 5,\n
> 'range_request_timeout_in_ms': 1,\n'read_request_timeout_in_ms':
> 1,\n'request_timeout_in_ms': 1,\n
> 'truncate_request_timeout_in_ms': 1,\n'write_request_timeout_in_ms':
> 1}\n- >> end captured logging <<
> -"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-14050) Many cqlsh_copy_tests are busted
[
https://issues.apache.org/jira/browse/CASSANDRA-14050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania Alborghetti reassigned CASSANDRA-14050:
Assignee: Stefania Alborghetti (was: Sam Sriramadhesikan)
> Many cqlsh_copy_tests are busted
>
>
> Key: CASSANDRA-14050
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14050
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Testing
>Reporter: Michael Kjellman
>Assignee: Stefania Alborghetti
>Priority: Normal
>
> Many cqlsh_copy_tests are busted. We should disable the entire suite until
> this is resolved as these tests are currently nothing but a waste of time.
> test_bulk_round_trip_blogposts - cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest
> test_bulk_round_trip_blogposts_with_max_connections -
> cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest
> test_bulk_round_trip_default - cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest
> Error starting node3.
> >> begin captured logging <<
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-S9NfIH
> dtest: DEBUG: Done setting configuration options:
> { 'initial_token': None,
> 'memtable_allocation_type': 'offheap_objects',
> 'num_tokens': '256',
> 'phi_convict_threshold': 5,
> 'range_request_timeout_in_ms': 1,
> 'read_request_timeout_in_ms': 1,
> 'request_timeout_in_ms': 1,
> 'truncate_request_timeout_in_ms': 1,
> 'write_request_timeout_in_ms': 1}
> - >> end captured logging << -
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/cassandra/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py",
> line 2546, in test_bulk_round_trip_blogposts
> stress_table='stresscql.blogposts')
> File "/home/cassandra/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py",
> line 2451, in _test_bulk_round_trip
> self.prepare(nodes=nodes, partitioner=partitioner,
> configuration_options=configuration_options)
> File "/home/cassandra/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py",
> line 115, in prepare
> self.cluster.populate(nodes,
> tokens=tokens).start(wait_for_binary_proto=True)
> File
> "/home/cassandra/env/local/lib/python2.7/site-packages/ccmlib/cluster.py",
> line 423, in start
> raise NodeError("Error starting {0}.".format(node.name), p)
> "Error starting node3.\n >> begin captured logging <<
> \ndtest: DEBUG: cluster ccm directory:
> /tmp/dtest-S9NfIH\ndtest: DEBUG: Done setting configuration options:\n{
> 'initial_token': None,\n'memtable_allocation_type': 'offheap_objects',\n
> 'num_tokens': '256',\n'phi_convict_threshold': 5,\n
> 'range_request_timeout_in_ms': 1,\n'read_request_timeout_in_ms':
> 1,\n'request_timeout_in_ms': 1,\n
> 'truncate_request_timeout_in_ms': 1,\n'write_request_timeout_in_ms':
> 1}\n- >> end captured logging <<
> -"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15229) BufferPool Regression
[
https://issues.apache.org/jira/browse/CASSANDRA-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076347#comment-17076347
]
Stefania Alborghetti commented on CASSANDRA-15229:
--
bq. The current implementation isn't really a bump the pointer allocator? It's
bitmap based, though with a very tiny bitmap.
Sorry it's been a while. Of course the current implementation is also bitmap
based. The point is that it is not suitable for long lived buffers, similarly
to our bump the pointer strategy. The transient case is easy to solve, either
approach would work.
bq. Could you elaborate on how these work, as my intuition is that anything
designed for a thread-per-core architecture probably won't translate so well to
the present state of the world. Though, either way, I suppose this is probably
orthogonal to this ticket as we only need to address the {{ChunkCache}} part.
The thread-per-core architecture makes it easy to identify threads that do most
of the work and cause most of the contention. However, thread identification
can be achieved also with thread pools or we can simply give all threads a
local stash of buffers, provided that we return it when the thread dies. I
don't think there is any other dependency on TPC beyond this.
The design choice was mostly dictated by the size of the cache: with AIO reads
the OS page cache is bypassed, and the chunk cache needs therefore to be very
large, which is not the case if we use Java NIO reads or if we eventually
implement asynchronous reads with the new uring API, bypassing AIO completely
(which I do recommend).
bq. We also optimized the chunk cache to store memory addresses rather than
byte buffers, which significantly reduced heap usage. The byte buffers are
materialized on the fly.
bq. This would be a huge improvement, and a welcome backport if it is easy -
though it might (I would guess) depend on Unsafe, which may be going away soon.
It's orthogonal to this ticket, though, I think
Yes it's based on the Unsafe. The addresses come from the slabs, and then we
use the Unsafe to create hollow buffers and to set the address. This is an
optimization and it clearly belongs to a separate ticket.
{quote}
We changed the chunk cache to always store buffers of the same size.
We have global lists of these slabs, sorted by buffer size where each size
is a power-of-two.
How do these two statements reconcile?
{quote}
So let's assume the current workload is mostly on a table with 4k chunks, which
translate to 4k buffers in the cache. Let's also assume that the workload is
shifting towards another table, with 8k chunks. Alternatively, let's assume
compression is ON, and an ALTER TABLE changes the chunk size. So now the chunk
cache is slowly evicting 4k buffers and retaining 8k buffers. These buffers
come from two different lists: the list of slabs serving 4k and the list
serving 8k. Even if we collect all unused 4k slabs, until each slab has every
single buffer returned, there will be wasted memory and we do not control how
long that will take. To be fair, it's an extreme case, and we were perhaps over
cautions in addressing this possibility by fixing the size of buffers in the
cache. So it's possible that the redesigned buffer pool may work even with the
current chunk cache implementation.
bq. Is it your opinion that your entire ChunkCache implementation can be
dropped wholesale into 4.0? I would assume it is still primarily
multi-threaded. If so, it might be preferable to trying to fix the existing
ChunkCache
The changes to the chunk cache are not trivial and should be left as a follow
up for 4.x or later in my opinion.
The changes to the buffer pool can be dropped in 4.0 if you think that:
- they are safe even in the presence of the case described above.
- they are justified: memory wasted due to fragmentation is perhaps not an
issue with a cache as little as 512 MB
I'll try to share some code so you can have a clearer picture.
> BufferPool Regression
> -
>
> Key: CASSANDRA-15229
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15229
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Caching
>Reporter: Benedict Elliott Smith
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 4.0, 4.0-beta
>
>
> The BufferPool was never intended to be used for a {{ChunkCache}}, and we
> need to either change our behaviour to handle uncorrelated lifetimes or use
> something else. This is particularly important with the default chunk size
> for compressed sstables being reduced. If we address the problem, we should
> also utilise the BufferPool for native transport connections like we do for
> internode messaging, and reduce the number of
[jira] [Updated] (CASSANDRA-15672) Testsuite: org.apache.cassandra.repair.consistent.CoordinatorMessagingTest Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.878 sec
[ https://issues.apache.org/jira/browse/CASSANDRA-15672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania Alborghetti updated CASSANDRA-15672: - Since Version: 4.0-alpha Source Control Link: https://gitbox.apache.org/repos/asf?p=cassandra.git;a=commit;h=b4e640a96e76f8d4a45937b1312b64ddc1aeb8ac Resolution: Fixed Status: Resolved (was: Ready to Commit) CI link: https://jenkins-cm4.apache.org/view/patches/job/Cassandra-devbranch/16/ Committed as [b4e640a96e76f8d4a45937b1312b64ddc1aeb8ac|https://gitbox.apache.org/repos/asf?p=cassandra.git;a=commit;h=b4e640a96e76f8d4a45937b1312b64ddc1aeb8ac] > Testsuite: org.apache.cassandra.repair.consistent.CoordinatorMessagingTest > Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.878 sec > - > > Key: CASSANDRA-15672 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15672 > Project: Cassandra > Issue Type: Bug > Components: Test/unit >Reporter: Ekaterina Dimitrova >Assignee: Aleksandr Sorokoumov >Priority: Normal > Labels: pull-request-available > Fix For: 4.0, 4.0-beta > > Time Spent: 10m > Remaining Estimate: 0h > > The following failure was observed: > Testsuite: org.apache.cassandra.repair.consistent.CoordinatorMessagingTest > Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.878 sec > [junit-timeout] > [junit-timeout] Testcase: > testMockedMessagingPrepareFailureP1(org.apache.cassandra.repair.consistent.CoordinatorMessagingTest): >FAILED > [junit-timeout] null > [junit-timeout] junit.framework.AssertionFailedError > [junit-timeout] at > org.apache.cassandra.repair.consistent.CoordinatorMessagingTest.testMockedMessagingPrepareFailure(CoordinatorMessagingTest.java:206) > [junit-timeout] at > org.apache.cassandra.repair.consistent.CoordinatorMessagingTest.testMockedMessagingPrepareFailureP1(CoordinatorMessagingTest.java:154) > [junit-timeout] > [junit-timeout] > Seen on Java8 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15672) Testsuite: org.apache.cassandra.repair.consistent.CoordinatorMessagingTest Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.878 sec
[ https://issues.apache.org/jira/browse/CASSANDRA-15672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania Alborghetti updated CASSANDRA-15672: - Status: Ready to Commit (was: Review In Progress) > Testsuite: org.apache.cassandra.repair.consistent.CoordinatorMessagingTest > Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.878 sec > - > > Key: CASSANDRA-15672 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15672 > Project: Cassandra > Issue Type: Bug > Components: Test/unit >Reporter: Ekaterina Dimitrova >Assignee: Aleksandr Sorokoumov >Priority: Normal > Labels: pull-request-available > Fix For: 4.0, 4.0-beta > > Time Spent: 10m > Remaining Estimate: 0h > > The following failure was observed: > Testsuite: org.apache.cassandra.repair.consistent.CoordinatorMessagingTest > Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.878 sec > [junit-timeout] > [junit-timeout] Testcase: > testMockedMessagingPrepareFailureP1(org.apache.cassandra.repair.consistent.CoordinatorMessagingTest): >FAILED > [junit-timeout] null > [junit-timeout] junit.framework.AssertionFailedError > [junit-timeout] at > org.apache.cassandra.repair.consistent.CoordinatorMessagingTest.testMockedMessagingPrepareFailure(CoordinatorMessagingTest.java:206) > [junit-timeout] at > org.apache.cassandra.repair.consistent.CoordinatorMessagingTest.testMockedMessagingPrepareFailureP1(CoordinatorMessagingTest.java:154) > [junit-timeout] > [junit-timeout] > Seen on Java8 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: Fix flaky CoordinatorMessagingTest and docstring in OutboundSink and ConsistentSession
This is an automated email from the ASF dual-hosted git repository.
stefania pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new b4e640a Fix flaky CoordinatorMessagingTest and docstring in
OutboundSink and ConsistentSession
b4e640a is described below
commit b4e640a96e76f8d4a45937b1312b64ddc1aeb8ac
Author: Aleksandr Sorokoumov
AuthorDate: Tue Mar 31 15:53:51 2020 +0200
Fix flaky CoordinatorMessagingTest and docstring in OutboundSink and
ConsistentSession
patch by Aleksandr Sorokoumov; reviewed by Stefania Alborghetti for
CASSANDRA-15672
---
CHANGES.txt| 1 +
.../org/apache/cassandra/net/OutboundSink.java | 2 +-
.../repair/consistent/ConsistentSession.java | 8 +--
.../consistent/CoordinatorMessagingTest.java | 70 +++---
4 files changed, 53 insertions(+), 28 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 65111d0..fb881de 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
4.0-alpha4
+ * Fix flaky CoordinatorMessagingTest and docstring in OutboundSink and
ConsistentSession (CASSANDRA-15672)
* Fix force compaction of wrapping ranges (CASSANDRA-15664)
* Expose repair streaming metrics (CASSANDRA-15656)
* Set now in seconds in the future for validation repairs (CASSANDRA-15655)
diff --git a/src/java/org/apache/cassandra/net/OutboundSink.java
b/src/java/org/apache/cassandra/net/OutboundSink.java
index d19b3e2..34c72db 100644
--- a/src/java/org/apache/cassandra/net/OutboundSink.java
+++ b/src/java/org/apache/cassandra/net/OutboundSink.java
@@ -25,7 +25,7 @@ import org.apache.cassandra.locator.InetAddressAndPort;
/**
* A message sink that all outbound messages go through.
*
- * Default sink {@link Sink} used by {@link MessagingService} is
MessagingService#doSend(), which proceeds to
+ * Default sink {@link Sink} used by {@link MessagingService} is {@link
MessagingService#doSend(Message, InetAddressAndPort, ConnectionType)}, which
proceeds to
* send messages over the network, but it can be overridden to filter out
certain messages, record the fact
* of attempted delivery, or delay they delivery.
*
diff --git
a/src/java/org/apache/cassandra/repair/consistent/ConsistentSession.java
b/src/java/org/apache/cassandra/repair/consistent/ConsistentSession.java
index 03de157..d9ac927 100644
--- a/src/java/org/apache/cassandra/repair/consistent/ConsistentSession.java
+++ b/src/java/org/apache/cassandra/repair/consistent/ConsistentSession.java
@@ -56,13 +56,13 @@ import org.apache.cassandra.tools.nodetool.RepairAdmin;
* There are 4 stages to a consistent incremental repair.
*
* Repair prepare
- * First, the normal {@link ActiveRepairService#prepareForRepair(UUID,
InetAddressAndPort, Set, RepairOption, List)} stuff
+ * First, the normal {@link ActiveRepairService#prepareForRepair(UUID,
InetAddressAndPort, Set, RepairOption, boolean, List)} stuff
* happens, which sends out {@link PrepareMessage} and creates a {@link
ActiveRepairService.ParentRepairSession}
* on the coordinator and each of the neighbors.
*
* Consistent prepare
* The consistent prepare step promotes the parent repair session to a
consistent session, and isolates the sstables
- * being repaired other sstables. First, the coordinator sends a {@link
PrepareConsistentRequest} message to each repair
+ * being repaired from other sstables. First, the coordinator sends a {@link
PrepareConsistentRequest} message to each repair
* participant (including itself). When received, the node creates a {@link
LocalSession} instance, sets it's state to
* {@code PREPARING}, persists it, and begins a preparing the tables for
incremental repair, which segregates the data
* being repaired from the rest of the table data. When the preparation
completes, the session state is set to
@@ -74,7 +74,7 @@ import org.apache.cassandra.tools.nodetool.RepairAdmin;
* Once the coordinator recieves positive {@code PrepareConsistentResponse}
messages from all the participants, the
* coordinator begins the normal repair process.
*
- * (see {@link CoordinatorSession#handlePrepareResponse(InetAddress, boolean)}
+ * (see {@link CoordinatorSession#handlePrepareResponse(InetAddressAndPort,
boolean)}
*
* Repair
* The coordinator runs the normal data repair process against the sstables
segregated in the previous step. When a
@@ -96,7 +96,7 @@ import org.apache.cassandra.tools.nodetool.RepairAdmin;
* conflicts with in progress compactions. The sstables will be marked
repaired as part of the normal compaction process.
*
*
- * On the coordinator side, see {@link CoordinatorSession#finalizePropose()},
{@link CoordinatorSession#handleFinalizePromise(InetAddress, boolean)},
+ * On the coordinator side, see {@link CoordinatorSession#finalizePropose()},
{@link
[jira] [Commented] (CASSANDRA-14050) Many cqlsh_copy_tests are busted
[
https://issues.apache.org/jira/browse/CASSANDRA-14050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074829#comment-17074829
]
Stefania Alborghetti commented on CASSANDRA-14050:
--
[~sasrira] are you still working on this? If not, [~Ge] and I would like to
take over. We would like to fix these tests before merging CASSANDRA-15679.
The reason for the failures is that the _cqlsh copy tests.py_ links to
_cqlshlib/formatting.py_. It needs this in order to apply the identical
formatting used by cqlsh and determine if the data obtained via
{{self.session.execute("SELECT * FROM testtuple")}} matches the data in the csv
files.
Since cqlshlib on trunk supports both python 3 and python 2, then the cqlsh
copy tests work for trunk. But for older branches that only support python 2,
the tests no longer work.
So to fix the tests we would need to make cqlshlib support both python 2 and
python 3, at least as far as _formatting.py_. There is a problem with this
approach though: this code is mostly tested via dtests, which only support
python 3 (I assume this is the case because of the dependency on Python 3) and
therefore, how would we know if we broke anything for Python 2? Maybe we could
run the dtests from before the migration to python 3, hoping that they still
work.
Another approach would be to copy formatting.py into the dtests repo for the
older branches but this is quite ugly.
Lastly, there is the option of removing the dependency to _formatting.py_. I
think we could try replacing {{self.session.execute("SELECT * FROM
testtuple")}} with the equivalent cqlsh command and see if that works.
> Many cqlsh_copy_tests are busted
>
>
> Key: CASSANDRA-14050
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14050
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Testing
>Reporter: Michael Kjellman
>Assignee: Sam Sriramadhesikan
>Priority: Normal
>
> Many cqlsh_copy_tests are busted. We should disable the entire suite until
> this is resolved as these tests are currently nothing but a waste of time.
> test_bulk_round_trip_blogposts - cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest
> test_bulk_round_trip_blogposts_with_max_connections -
> cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest
> test_bulk_round_trip_default - cqlsh_tests.cqlsh_copy_tests.CqlshCopyTest
> Error starting node3.
> >> begin captured logging <<
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-S9NfIH
> dtest: DEBUG: Done setting configuration options:
> { 'initial_token': None,
> 'memtable_allocation_type': 'offheap_objects',
> 'num_tokens': '256',
> 'phi_convict_threshold': 5,
> 'range_request_timeout_in_ms': 1,
> 'read_request_timeout_in_ms': 1,
> 'request_timeout_in_ms': 1,
> 'truncate_request_timeout_in_ms': 1,
> 'write_request_timeout_in_ms': 1}
> - >> end captured logging << -
> File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
> File "/home/cassandra/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py",
> line 2546, in test_bulk_round_trip_blogposts
> stress_table='stresscql.blogposts')
> File "/home/cassandra/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py",
> line 2451, in _test_bulk_round_trip
> self.prepare(nodes=nodes, partitioner=partitioner,
> configuration_options=configuration_options)
> File "/home/cassandra/cassandra-dtest/cqlsh_tests/cqlsh_copy_tests.py",
> line 115, in prepare
> self.cluster.populate(nodes,
> tokens=tokens).start(wait_for_binary_proto=True)
> File
> "/home/cassandra/env/local/lib/python2.7/site-packages/ccmlib/cluster.py",
> line 423, in start
> raise NodeError("Error starting {0}.".format(node.name), p)
> "Error starting node3.\n >> begin captured logging <<
> \ndtest: DEBUG: cluster ccm directory:
> /tmp/dtest-S9NfIH\ndtest: DEBUG: Done setting configuration options:\n{
> 'initial_token': None,\n'memtable_allocation_type': 'offheap_objects',\n
> 'num_tokens': '256',\n'phi_convict_threshold': 5,\n
> 'range_request_timeout_in_ms': 1,\n'read_request_timeout_in_ms':
> 1,\n'request_timeout_in_ms': 1,\n
> 'truncate_request_timeout_in_ms': 1,\n'write_request_timeout_in_ms':
> 1}\n- >> end captured logging <<
> -"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15672) Testsuite: org.apache.cassandra.repair.consistent.CoordinatorMessagingTest Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.878 sec
[ https://issues.apache.org/jira/browse/CASSANDRA-15672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania Alborghetti updated CASSANDRA-15672: - Status: Review In Progress (was: Changes Suggested) > Testsuite: org.apache.cassandra.repair.consistent.CoordinatorMessagingTest > Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.878 sec > - > > Key: CASSANDRA-15672 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15672 > Project: Cassandra > Issue Type: Bug > Components: Test/unit >Reporter: Ekaterina Dimitrova >Assignee: Aleksandr Sorokoumov >Priority: Normal > Labels: pull-request-available > Fix For: 4.0, 4.0-beta > > Time Spent: 10m > Remaining Estimate: 0h > > The following failure was observed: > Testsuite: org.apache.cassandra.repair.consistent.CoordinatorMessagingTest > Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.878 sec > [junit-timeout] > [junit-timeout] Testcase: > testMockedMessagingPrepareFailureP1(org.apache.cassandra.repair.consistent.CoordinatorMessagingTest): >FAILED > [junit-timeout] null > [junit-timeout] junit.framework.AssertionFailedError > [junit-timeout] at > org.apache.cassandra.repair.consistent.CoordinatorMessagingTest.testMockedMessagingPrepareFailure(CoordinatorMessagingTest.java:206) > [junit-timeout] at > org.apache.cassandra.repair.consistent.CoordinatorMessagingTest.testMockedMessagingPrepareFailureP1(CoordinatorMessagingTest.java:154) > [junit-timeout] > [junit-timeout] > Seen on Java8 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15679) cqlsh COPY FROM of map of blobs fails with parse error "unhashable type: 'bytearray'"
[
https://issues.apache.org/jira/browse/CASSANDRA-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania Alborghetti updated CASSANDRA-15679:
-
Reviewers: Stefania Alborghetti, Stefania Alborghetti (was: Stefania
Alborghetti)
Stefania Alborghetti, Stefania Alborghetti
Status: Review In Progress (was: Patch Available)
> cqlsh COPY FROM of map of blobs fails with parse error "unhashable type:
> 'bytearray'"
> -
>
> Key: CASSANDRA-15679
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15679
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/cqlsh
>Reporter: Erick Ramirez
>Assignee: Aleksandr Sorokoumov
>Priority: Normal
> Labels: pull-request-available
> Fix For: 2.1.21, 2.2.17, 3.0.21, 3.11.7, 4.x
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> h2. Background
> A user was having issues loading CSV data with the {{COPY FROM}} command into
> a {{map}} column with {{blob}} values.
> h2. Replication steps
> I can easily replicate the problem with this simple table:
> {noformat}
> CREATE TABLE community.blobmaptable (
> id text PRIMARY KEY,
> blobmapcol map
> )
> {noformat}
> I have this CSV file that contains just 1 row:
> {noformat}
> $ cat blobmap.csv
> c3,{3: 0x74776f}
> {noformat}
> And here's the error when I try to load it:
> {noformat}
> cqlsh:community> COPY blobmaptable (id, blobmapcol) FROM '~/blobmap.csv' ;
> Using 1 child processes
> Starting copy of community.blobmaptable with columns [id, blobmapcol].
> Failed to import 1 rows: ParseError - Failed to parse {3: 0x74776f} :
> unhashable type: 'bytearray', given up without retries
> Failed to process 1 rows; failed rows written to
> import_community_blobmaptable.err
> Processed: 1 rows; Rate: 2 rows/s; Avg. rate: 3 rows/s
> 1 rows imported from 1 files in 0.389 seconds (0 skipped).
> {noformat}
> I've also logged
> [PYTHON-1234|https://datastax-oss.atlassian.net/browse/PYTHON-1234] because I
> wasn't sure if it was a Python driver issue. Cheers!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15679) cqlsh COPY FROM of map of blobs fails with parse error "unhashable type: 'bytearray'"
[
https://issues.apache.org/jira/browse/CASSANDRA-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania Alborghetti updated CASSANDRA-15679:
-
Test and Documentation Plan:
This bug is triggered when we try to deserialize a blob inside a container,
such as list, set, or map. copyutil stores blobs as bytearrays. Python
bytearrays are mutable and not hashable data structures.
In Python 3 there is bytes type that is exactly what we need in this situation.
On older versions that do not support Python 3 and in case we run cqlsh with
Python 2 at 4.0, we should fall back to str.
https://github.com/apache/cassandra/pull/506
https://github.com/apache/cassandra/pull/507
https://github.com/apache/cassandra/pull/508
https://github.com/apache/cassandra/pull/509
https://github.com/apache/cassandra/pull/510
Status: Patch Available (was: Open)
> cqlsh COPY FROM of map of blobs fails with parse error "unhashable type:
> 'bytearray'"
> -
>
> Key: CASSANDRA-15679
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15679
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/cqlsh
>Reporter: Erick Ramirez
>Assignee: Aleksandr Sorokoumov
>Priority: Normal
> Labels: pull-request-available
> Fix For: 2.1.21, 2.2.17, 3.0.21, 3.11.7, 4.x
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> h2. Background
> A user was having issues loading CSV data with the {{COPY FROM}} command into
> a {{map}} column with {{blob}} values.
> h2. Replication steps
> I can easily replicate the problem with this simple table:
> {noformat}
> CREATE TABLE community.blobmaptable (
> id text PRIMARY KEY,
> blobmapcol map
> )
> {noformat}
> I have this CSV file that contains just 1 row:
> {noformat}
> $ cat blobmap.csv
> c3,{3: 0x74776f}
> {noformat}
> And here's the error when I try to load it:
> {noformat}
> cqlsh:community> COPY blobmaptable (id, blobmapcol) FROM '~/blobmap.csv' ;
> Using 1 child processes
> Starting copy of community.blobmaptable with columns [id, blobmapcol].
> Failed to import 1 rows: ParseError - Failed to parse {3: 0x74776f} :
> unhashable type: 'bytearray', given up without retries
> Failed to process 1 rows; failed rows written to
> import_community_blobmaptable.err
> Processed: 1 rows; Rate: 2 rows/s; Avg. rate: 3 rows/s
> 1 rows imported from 1 files in 0.389 seconds (0 skipped).
> {noformat}
> I've also logged
> [PYTHON-1234|https://datastax-oss.atlassian.net/browse/PYTHON-1234] because I
> wasn't sure if it was a Python driver issue. Cheers!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15679) cqlsh COPY FROM of map of blobs fails with parse error "unhashable type: 'bytearray'"
[
https://issues.apache.org/jira/browse/CASSANDRA-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania Alborghetti updated CASSANDRA-15679:
-
Bug Category: Parent values: Code(13163)Level 1 values: Bug - Unclear
Impact(13164)
Complexity: Normal
Discovered By: User Report
Severity: Low
Status: Open (was: Triage Needed)
> cqlsh COPY FROM of map of blobs fails with parse error "unhashable type:
> 'bytearray'"
> -
>
> Key: CASSANDRA-15679
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15679
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/cqlsh
>Reporter: Erick Ramirez
>Assignee: Aleksandr Sorokoumov
>Priority: Normal
> Labels: pull-request-available
> Fix For: 2.2.17, 3.0.21, 3.11.7, 4.x, 2.1.21
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> h2. Background
> A user was having issues loading CSV data with the {{COPY FROM}} command into
> a {{map}} column with {{blob}} values.
> h2. Replication steps
> I can easily replicate the problem with this simple table:
> {noformat}
> CREATE TABLE community.blobmaptable (
> id text PRIMARY KEY,
> blobmapcol map
> )
> {noformat}
> I have this CSV file that contains just 1 row:
> {noformat}
> $ cat blobmap.csv
> c3,{3: 0x74776f}
> {noformat}
> And here's the error when I try to load it:
> {noformat}
> cqlsh:community> COPY blobmaptable (id, blobmapcol) FROM '~/blobmap.csv' ;
> Using 1 child processes
> Starting copy of community.blobmaptable with columns [id, blobmapcol].
> Failed to import 1 rows: ParseError - Failed to parse {3: 0x74776f} :
> unhashable type: 'bytearray', given up without retries
> Failed to process 1 rows; failed rows written to
> import_community_blobmaptable.err
> Processed: 1 rows; Rate: 2 rows/s; Avg. rate: 3 rows/s
> 1 rows imported from 1 files in 0.389 seconds (0 skipped).
> {noformat}
> I've also logged
> [PYTHON-1234|https://datastax-oss.atlassian.net/browse/PYTHON-1234] because I
> wasn't sure if it was a Python driver issue. Cheers!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15672) Testsuite: org.apache.cassandra.repair.consistent.CoordinatorMessagingTest Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.878 sec
[ https://issues.apache.org/jira/browse/CASSANDRA-15672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania Alborghetti updated CASSANDRA-15672: - Status: Changes Suggested (was: Review In Progress) We can try to synchronize the test by using a latch when messages are intercepted. If this approach fails, we can fall back to relaxing the intermediate checks. > Testsuite: org.apache.cassandra.repair.consistent.CoordinatorMessagingTest > Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.878 sec > - > > Key: CASSANDRA-15672 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15672 > Project: Cassandra > Issue Type: Bug > Components: Test/unit >Reporter: Ekaterina Dimitrova >Assignee: Aleksandr Sorokoumov >Priority: Normal > Labels: pull-request-available > Fix For: 4.0, 4.0-beta > > Time Spent: 10m > Remaining Estimate: 0h > > The following failure was observed: > Testsuite: org.apache.cassandra.repair.consistent.CoordinatorMessagingTest > Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.878 sec > [junit-timeout] > [junit-timeout] Testcase: > testMockedMessagingPrepareFailureP1(org.apache.cassandra.repair.consistent.CoordinatorMessagingTest): >FAILED > [junit-timeout] null > [junit-timeout] junit.framework.AssertionFailedError > [junit-timeout] at > org.apache.cassandra.repair.consistent.CoordinatorMessagingTest.testMockedMessagingPrepareFailure(CoordinatorMessagingTest.java:206) > [junit-timeout] at > org.apache.cassandra.repair.consistent.CoordinatorMessagingTest.testMockedMessagingPrepareFailureP1(CoordinatorMessagingTest.java:154) > [junit-timeout] > [junit-timeout] > Seen on Java8 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15672) Testsuite: org.apache.cassandra.repair.consistent.CoordinatorMessagingTest Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.878 sec
[ https://issues.apache.org/jira/browse/CASSANDRA-15672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania Alborghetti updated CASSANDRA-15672: - Reviewers: Stefania Alborghetti, Stefania Alborghetti (was: Stefania Alborghetti) Stefania Alborghetti, Stefania Alborghetti Status: Review In Progress (was: Patch Available) > Testsuite: org.apache.cassandra.repair.consistent.CoordinatorMessagingTest > Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.878 sec > - > > Key: CASSANDRA-15672 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15672 > Project: Cassandra > Issue Type: Bug > Components: Test/unit >Reporter: Ekaterina Dimitrova >Assignee: Aleksandr Sorokoumov >Priority: Normal > Labels: pull-request-available > Fix For: 4.0, 4.0-beta > > Time Spent: 10m > Remaining Estimate: 0h > > The following failure was observed: > Testsuite: org.apache.cassandra.repair.consistent.CoordinatorMessagingTest > Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.878 sec > [junit-timeout] > [junit-timeout] Testcase: > testMockedMessagingPrepareFailureP1(org.apache.cassandra.repair.consistent.CoordinatorMessagingTest): >FAILED > [junit-timeout] null > [junit-timeout] junit.framework.AssertionFailedError > [junit-timeout] at > org.apache.cassandra.repair.consistent.CoordinatorMessagingTest.testMockedMessagingPrepareFailure(CoordinatorMessagingTest.java:206) > [junit-timeout] at > org.apache.cassandra.repair.consistent.CoordinatorMessagingTest.testMockedMessagingPrepareFailureP1(CoordinatorMessagingTest.java:154) > [junit-timeout] > [junit-timeout] > Seen on Java8 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15229) BufferPool Regression
[ https://issues.apache.org/jira/browse/CASSANDRA-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072895#comment-17072895 ] Stefania Alborghetti edited comment on CASSANDRA-15229 at 4/1/20, 4:14 PM: --- We hit this buffer pool regression problem in our DSE fork a while ago. Because our chunk cache became much larger when it replaced the OS page cache, off-heap memory was growing significantly beyond the limits configured. This was partly due to some leaks, but the fragmentation in the current design of the buffer pool was a big part of it. This is how we solved it: - a bump-the-pointer slab approach for the transient pool, not to dissimilar from the current implementation. We then exploit our thread per core architecture: core threads get a dedicated slab each, other threads share a global slab. - a bitmap-based slab approach for the permanent pool, which is only used by the chunk cache. These slabs can only issue buffers of the same size, one bit is flipped in the bitmap for each buffer issued. When multiple buffers are requested, the slab tries to issue consecutive addresses but this is not guaranteed since we want to avoid memory fragmentation. We have global lists of these slabs, sorted by buffer size where each size is a power-of-two. Slabs are taken out of these lists when they are full, and they are put back into circulation when they have space available. The lists are global but core threads get a thread-local stash of buffers, i.e. they request multiple buffers at the same time in order to reduce contention on the global lists. We changed the chunk cache to always store buffers of the same size. If we need to read chunks of a different size, we use an array of buffers in the cache and we request multiple buffers at the same time. If we get consecutive addresses, we optimize for this case by building a single byte buffer over the first address. We also optimized the chunk cache to store memory addresses rather than byte buffers, which significantly reduced heap usage. The byte buffers are materialized on the fly. For the permanent case, we made the choice of constraining the size of the buffers in the cache so that memory in the pool could be fully used. This may or may not be what people prefer. Our choice was due to the large size of the cache, 20+ GB. An approach that allows some memory fragmentation may be sufficient for smaller cache sizes. Please let me know if there is interest in porting this solution to 4.0 or 4.x. I can share the code if needed. was (Author: stefania): We hit this buffer pool regression problem in our DSE fork a while ago. Because our chunk cache became much larger when it replaced the OS page cache, off-heap memory was growing significantly beyond the limits configured. This was partly due to some leaks, but the fragmentation in the current design of the buffer pool was a big part of it. This is how we solved it: - a bump-the-pointer slab approach for the transient pool, not to dissimilar from the current implementation. We then exploit our thread per core architecture: core threads get a dedicated slab each, other threads share a global slab. - a bitmap-based slab approach for the permanent pool, which is only used by the chunk cache. These slabs can only issue buffers of the same size, one bit is flipped in the bitmap for each buffer issued. When multiple buffers are requested, the slab tries to issue consecutive addresses but this is not guaranteed since we want to avoid memory fragmentation. We have global lists of these slabs, sorted by buffer size where each size is a power-of-two. Slabs are taken out of these lists when they are full, and they are put back into circulation when they have space available. The lists are global but core threads get a thread-local stash of buffers, i.e. they request multiple buffers at the same time in order to reduce contention on the global lists. We changed the chunk cache to always store chunks of the same size. If we need to read chunks of a different size, we use an array of buffers in the cache and we request multiple buffers at the same time. If we get consecutive addresses, we optimize for this case by building a single byte buffer over the first address. We also optimized the chunk cache to store memory addresses rather than byte buffers, which significantly reduced heap usage. The byte buffers are materialized on the fly. For the permanent case, we made the choice of constraining the size of the buffers in the cache so that memory in the pool could be fully used. This may or may not be what people prefer. Our choice was due to the large size of the cache, 20+ GB. An approach that allows some memory fragmentation may be sufficient for smaller cache sizes. Please let me know if there is interest in porting this solution to 4.0 or 4.x. I can
[jira] [Commented] (CASSANDRA-15229) BufferPool Regression
[
https://issues.apache.org/jira/browse/CASSANDRA-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072895#comment-17072895
]
Stefania Alborghetti commented on CASSANDRA-15229:
--
We hit this buffer pool regression problem in our DSE fork a while ago. Because
our chunk cache became much larger when it replaced the OS page cache, off-heap
memory was growing significantly beyond the limits configured. This was partly
due to some leaks, but the fragmentation in the current design of the buffer
pool was a big part of it.
This is how we solved it:
- a bump-the-pointer slab approach for the transient pool, not to dissimilar
from the current implementation. We then exploit our thread per core
architecture: core threads get a dedicated slab each, other threads share a
global slab.
- a bitmap-based slab approach for the permanent pool, which is only used by
the chunk cache. These slabs can only issue buffers of the same size, one bit
is flipped in the bitmap for each buffer issued. When multiple buffers are
requested, the slab tries to issue consecutive addresses but this is not
guaranteed since we want to avoid memory fragmentation. We have global lists of
these slabs, sorted by buffer size where each size is a power-of-two. Slabs are
taken out of these lists when they are full, and they are put back into
circulation when they have space available. The lists are global but core
threads get a thread-local stash of buffers, i.e. they request multiple buffers
at the same time in order to reduce contention on the global lists.
We changed the chunk cache to always store chunks of the same size. If we need
to read chunks of a different size, we use an array of buffers in the cache and
we request multiple buffers at the same time. If we get consecutive addresses,
we optimize for this case by building a single byte buffer over the first
address. We also optimized the chunk cache to store memory addresses rather
than byte buffers, which significantly reduced heap usage. The byte buffers are
materialized on the fly.
For the permanent case, we made the choice of constraining the size of the
buffers in the cache so that memory in the pool could be fully used. This may
or may not be what people prefer. Our choice was due to the large size of the
cache, 20+ GB. An approach that allows some memory fragmentation may be
sufficient for smaller cache sizes.
Please let me know if there is interest in porting this solution to 4.0 or 4.x.
I can share the code if needed.
> BufferPool Regression
> -
>
> Key: CASSANDRA-15229
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15229
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Caching
>Reporter: Benedict Elliott Smith
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 4.0, 4.0-beta
>
>
> The BufferPool was never intended to be used for a {{ChunkCache}}, and we
> need to either change our behaviour to handle uncorrelated lifetimes or use
> something else. This is particularly important with the default chunk size
> for compressed sstables being reduced. If we address the problem, we should
> also utilise the BufferPool for native transport connections like we do for
> internode messaging, and reduce the number of pooling solutions we employ.
> Probably the best thing to do is to improve BufferPool’s behaviour when used
> for things with uncorrelated lifetimes, which essentially boils down to
> tracking those chunks that have not been freed and re-circulating them when
> we run out of completely free blocks. We should probably also permit
> instantiating separate {{BufferPool}}, so that we can insulate internode
> messaging from the {{ChunkCache}}, or at least have separate memory bounds
> for each, and only share fully-freed chunks.
> With these improvements we can also safely increase the {{BufferPool}} chunk
> size to 128KiB or 256KiB, to guarantee we can fit compressed pages and reduce
> the amount of global coordination and per-allocation overhead. We don’t need
> 1KiB granularity for allocations, nor 16 byte granularity for tiny
> allocations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15409) nodetool compactionstats showing extra pending task for TWCS
[
https://issues.apache.org/jira/browse/CASSANDRA-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania Alborghetti updated CASSANDRA-15409:
-
Fix Version/s: (was: 4.x)
4.0
Since Version: 3.0.8
Source Control Link:
https://gitbox.apache.org/repos/asf?p=cassandra.git;a=commitdiff;h=122cf57f1134704769cce9daddf882c3ea578905
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
> nodetool compactionstats showing extra pending task for TWCS
>
>
> Key: CASSANDRA-15409
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15409
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Ekaterina Dimitrova
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Fix For: 3.11.6, 4.0
>
>
> Summary: nodetool compactionstats showing extra pending task for TWCS
> -
> The output of {{nodetool compactionstats}}can show "pending tasks: 1" when
> there are actually none. This seems to be a consistent problem in testing C*
> trunk. In my testing, it looks like the {{nodetool compactionstats}} counter
> output is consistently off by 1 as compared to the table output of the tasks
>
> testing with {{concurrent_compactors: 8}}
> In 12 hours it never ended, always showing 1 pending job
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15409) nodetool compactionstats showing extra pending task for TWCS
[
https://issues.apache.org/jira/browse/CASSANDRA-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania Alborghetti updated CASSANDRA-15409:
-
Status: Ready to Commit (was: Review In Progress)
> nodetool compactionstats showing extra pending task for TWCS
>
>
> Key: CASSANDRA-15409
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15409
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Ekaterina Dimitrova
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Fix For: 3.11.6, 4.x
>
>
> Summary: nodetool compactionstats showing extra pending task for TWCS
> -
> The output of {{nodetool compactionstats}}can show "pending tasks: 1" when
> there are actually none. This seems to be a consistent problem in testing C*
> trunk. In my testing, it looks like the {{nodetool compactionstats}} counter
> output is consistently off by 1 as compared to the table output of the tasks
>
> testing with {{concurrent_compactors: 8}}
> In 12 hours it never ended, always showing 1 pending job
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15409) nodetool compactionstats showing extra pending task for TWCS
[
https://issues.apache.org/jira/browse/CASSANDRA-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993665#comment-16993665
]
Stefania Alborghetti commented on CASSANDRA-15409:
--
CI looked good on our infra, so committed to 3.11 as
[122cf57f1134704769cce9daddf882c3ea578905|https://gitbox.apache.org/repos/asf?p=cassandra.git;a=commit;h=122cf57f1134704769cce9daddf882c3ea578905]
and merged into trunk.
> nodetool compactionstats showing extra pending task for TWCS
>
>
> Key: CASSANDRA-15409
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15409
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Ekaterina Dimitrova
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Fix For: 3.11.6, 4.x
>
>
> Summary: nodetool compactionstats showing extra pending task for TWCS
> -
> The output of {{nodetool compactionstats}}can show "pending tasks: 1" when
> there are actually none. This seems to be a consistent problem in testing C*
> trunk. In my testing, it looks like the {{nodetool compactionstats}} counter
> output is consistently off by 1 as compared to the table output of the tasks
>
> testing with {{concurrent_compactors: 8}}
> In 12 hours it never ended, always showing 1 pending job
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15409) nodetool compactionstats showing extra pending task for TWCS
[
https://issues.apache.org/jira/browse/CASSANDRA-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993665#comment-16993665
]
Stefania Alborghetti edited comment on CASSANDRA-15409 at 12/11/19 4:05 PM:
CI looked good on our infra, so committed to 3.11 as
[122cf57f1134704769cce9daddf882c3ea578905|https://gitbox.apache.org/repos/asf?p=cassandra.git;a=commitdiff;h=122cf57f1134704769cce9daddf882c3ea578905]
and merged into trunk.
was (Author: stefania):
CI looked good on our infra, so committed to 3.11 as
[122cf57f1134704769cce9daddf882c3ea578905|https://gitbox.apache.org/repos/asf?p=cassandra.git;a=commit;h=122cf57f1134704769cce9daddf882c3ea578905]
and merged into trunk.
> nodetool compactionstats showing extra pending task for TWCS
>
>
> Key: CASSANDRA-15409
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15409
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Ekaterina Dimitrova
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Fix For: 3.11.6, 4.x
>
>
> Summary: nodetool compactionstats showing extra pending task for TWCS
> -
> The output of {{nodetool compactionstats}}can show "pending tasks: 1" when
> there are actually none. This seems to be a consistent problem in testing C*
> trunk. In my testing, it looks like the {{nodetool compactionstats}} counter
> output is consistently off by 1 as compared to the table output of the tasks
>
> testing with {{concurrent_compactors: 8}}
> In 12 hours it never ended, always showing 1 pending job
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-3.11 updated: Fix nodetool compactionstats showing extra pending task for TWCS
This is an automated email from the ASF dual-hosted git repository.
stefania pushed a commit to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cassandra-3.11 by this push:
new 122cf57 Fix nodetool compactionstats showing extra pending task for
TWCS
122cf57 is described below
commit 122cf57f1134704769cce9daddf882c3ea578905
Author: Ekaterina Dimitrova
AuthorDate: Wed Oct 30 18:47:20 2019 -0400
Fix nodetool compactionstats showing extra pending task for TWCS
patch by Ekaterina Dimitrova; reviewed by Stefania Alborghetti for
CASSANDRA-15409
---
CHANGES.txt| 2 +-
.../SizeTieredCompactionStrategyOptions.java | 7 ++
.../compaction/TimeWindowCompactionStrategy.java | 77 +++---
.../org/apache/cassandra/tools/nodetool/Stop.java | 2 +-
.../TimeWindowCompactionStrategyTest.java | 15 +++--
5 files changed, 73 insertions(+), 30 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index a75c06f..8507656 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
3.11.6
+ * Fix nodetool compactionstats showing extra pending task for TWCS - patch
implemented (CASSANDRA-15409)
* Fix SELECT JSON formatting for the "duration" type (CASSANDRA-15075)
* Fix LegacyLayout to have same behavior as 2.x when handling unknown column
names (CASSANDRA-15081)
Merged from 3.0:
@@ -17,7 +18,6 @@ Merged from 2.2:
* Fix SELECT JSON output for empty blobs (CASSANDRA-15435)
* In-JVM DTest: Set correct internode message version for upgrade test
(CASSANDRA-15371)
-
3.11.5
* Fix SASI non-literal string comparisons (range operators) (CASSANDRA-15169)
* Make sure user defined compaction transactions are always closed
(CASSANDRA-15123)
diff --git
a/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategyOptions.java
b/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategyOptions.java
index 911bb9f..288af2b 100644
---
a/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategyOptions.java
+++
b/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategyOptions.java
@@ -94,4 +94,11 @@ public final class SizeTieredCompactionStrategyOptions
return uncheckedOptions;
}
+
+@Override
+public String toString()
+{
+return String.format("Min sstable size: %d, bucket low: %f, bucket
high: %f", minSSTableSize, bucketLow, bucketHigh);
+}
+
}
\ No newline at end of file
diff --git
a/src/java/org/apache/cassandra/db/compaction/TimeWindowCompactionStrategy.java
b/src/java/org/apache/cassandra/db/compaction/TimeWindowCompactionStrategy.java
index e0485f9..6186826 100644
---
a/src/java/org/apache/cassandra/db/compaction/TimeWindowCompactionStrategy.java
+++
b/src/java/org/apache/cassandra/db/compaction/TimeWindowCompactionStrategy.java
@@ -166,14 +166,15 @@ public class TimeWindowCompactionStrategy extends
AbstractCompactionStrategy
if(buckets.right > this.highestWindowSeen)
this.highestWindowSeen = buckets.right;
-updateEstimatedCompactionsByTasks(buckets.left);
-List mostInteresting = newestBucket(buckets.left,
-
cfs.getMinimumCompactionThreshold(),
-
cfs.getMaximumCompactionThreshold(),
- options.stcsOptions,
-
this.highestWindowSeen);
-if (!mostInteresting.isEmpty())
-return mostInteresting;
+NewestBucket mostInteresting = newestBucket(buckets.left,
+cfs.getMinimumCompactionThreshold(),
+cfs.getMaximumCompactionThreshold(),
+options.stcsOptions,
+this.highestWindowSeen);
+
+this.estimatedRemainingTasks = mostInteresting.estimatedRemainingTasks;
+if (!mostInteresting.sstables.isEmpty())
+return mostInteresting.sstables;
return null;
}
@@ -253,20 +254,25 @@ public class TimeWindowCompactionStrategy extends
AbstractCompactionStrategy
return Pair.create(buckets, maxTimestamp);
}
-private void updateEstimatedCompactionsByTasks(HashMultimap tasks)
+static final class NewestBucket
{
-int n = 0;
-long now = this.highestWindowSeen;
+/** The sstables that should be compacted next */
+final List sstables;
+
+/** The number of tasks estimated */
+final int estimatedRemainingTasks;
-for(Long key : tasks.keySet())
+NewestBucket(List sstables, int estimatedRemainingTasks)
{
-// For current window, make sure it's compactable
-if (key.compareTo(now) >= 0 && tasks.get(key).size() >
[cassandra] 01/01: Merge branch 'cassandra-3.11' into trunk
This is an automated email from the ASF dual-hosted git repository. stefania pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit ce877cbe2b7c11355b07cac6f1996a9c9006d89f Merge: 1ce979d 122cf57 Author: Stefania Alborghetti AuthorDate: Wed Dec 11 11:00:00 2019 -0500 Merge branch 'cassandra-3.11' into trunk CHANGES.txt| 2 +- .../SizeTieredCompactionStrategyOptions.java | 7 ++ .../compaction/TimeWindowCompactionStrategy.java | 77 +++--- .../org/apache/cassandra/tools/nodetool/Stop.java | 2 +- .../TimeWindowCompactionStrategyTest.java | 15 +++-- 5 files changed, 73 insertions(+), 30 deletions(-) diff --cc CHANGES.txt index e486a43,8507656..4c4af72 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,13 -1,7 +1,14 @@@ -3.11.6 +4.0-alpha3 + * Removed obsolete OldNetworkTopologyStrategy (CASSANDRA-13990) + * Align record header of FQL and audit binary log (CASSANDRA-15076) + * Shuffle forwarding replica for messages to non-local DC (CASSANDRA-15318) + * Optimise native protocol ASCII string encoding (CASSANDRA-15410) + * Make sure all exceptions are propagated in DebuggableThreadPoolExecutor (CASSANDRA-15332) + * Make it possible to resize concurrent read / write thread pools at runtime (CASSANDRA-15277) + * Close channels on error (CASSANDRA-15407) +Merged from 3.11: + * Fix nodetool compactionstats showing extra pending task for TWCS - patch implemented (CASSANDRA-15409) * Fix SELECT JSON formatting for the "duration" type (CASSANDRA-15075) - * Fix LegacyLayout to have same behavior as 2.x when handling unknown column names (CASSANDRA-15081) Merged from 3.0: * Avoid over-scanning data directories in LogFile.verify() (CASSANDRA-15364) * Bump generations and document changes to system_distributed and system_traces in 3.0, 3.11 @@@ -18,19 -18,13 +19,18 @@@ Merged from 2.2 * Fix SELECT JSON output for empty blobs (CASSANDRA-15435) * In-JVM DTest: Set correct internode message version for upgrade test (CASSANDRA-15371) - -3.11.5 +4.0-alpha2 * Fix SASI non-literal string comparisons (range operators) (CASSANDRA-15169) - * Make sure user defined compaction transactions are always closed (CASSANDRA-15123) - * Fix cassandra-env.sh to use $CASSANDRA_CONF to find cassandra-jaas.config (CASSANDRA-14305) - * Fixed nodetool cfstats printing index name twice (CASSANDRA-14903) - * Add flag to disable SASI indexes, and warnings on creation (CASSANDRA-14866) + * Upgrade Guava to 27, and to java-driver 3.6.0 (from 3.4.0-SNAPSHOT) (CASSANDRA-14655) + * Extract an AbstractCompactionController to allow for custom implementations (CASSANDRA-15286) + * Move chronicle-core version from snapshot to stable, and include carrotsearch in generated pom.xml (CASSANDRA-15321) + * Untangle RepairMessage sub-hierarchy of messages, use new messaging (more) correctly (CASSANDRA-15163) + * Add `allocate_tokens_for_local_replication_factor` option for token allocation (CASSANDRA-15260) + * Add Alibaba Cloud Platform snitch (CASSANDRA-15092) Merged from 3.0: + * Fix various data directory prefix matching issues (CASSANDRA-13974) + * Minimize clustering values in metadata collector (CASSANDRA-15400) + * Make sure index summary redistribution does not start when compactions are paused (CASSANDRA-15265) * Add ability to cap max negotiable protocol version (CASSANDRA-15193) * Gossip tokens on startup if available (CASSANDRA-15335) * Fix resource leak in CompressedSequentialWriter (CASSANDRA-15340) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated (1ce979d -> ce877cb)
This is an automated email from the ASF dual-hosted git repository. stefania pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git. from 1ce979d Merge branch 'cassandra-3.11' into trunk new 122cf57 Fix nodetool compactionstats showing extra pending task for TWCS new ce877cb Merge branch 'cassandra-3.11' into trunk The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 2 +- .../SizeTieredCompactionStrategyOptions.java | 7 ++ .../compaction/TimeWindowCompactionStrategy.java | 77 +++--- .../org/apache/cassandra/tools/nodetool/Stop.java | 2 +- .../TimeWindowCompactionStrategyTest.java | 15 +++-- 5 files changed, 73 insertions(+), 30 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15409) nodetool compactionstats showing extra pending task for TWCS
[
https://issues.apache.org/jira/browse/CASSANDRA-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975175#comment-16975175
]
Stefania Alborghetti commented on CASSANDRA-15409:
--
Patch LGTM
Moving forward, it's perhaps easier to post the direct link to the branches:
[CASSANDRA-15409-3.11|https://github.com/ekaterinadimitrova2/cassandra/tree/CASSANDRA-15409-3.11]
[CASSANDRA-15409-trunk|https://github.com/ekaterinadimitrova2/cassandra/tree/CASSANDRA-15409-trunk]
Also, before merging, we need to post the CI results.
> nodetool compactionstats showing extra pending task for TWCS
>
>
> Key: CASSANDRA-15409
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15409
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Ekaterina Dimitrova
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Fix For: 3.11.6, 4.x
>
>
> Summary: nodetool compactionstats showing extra pending task for TWCS
> -
> The output of {{nodetool compactionstats}}can show "pending tasks: 1" when
> there are actually none. This seems to be a consistent problem in testing C*
> trunk. In my testing, it looks like the {{nodetool compactionstats}} counter
> output is consistently off by 1 as compared to the table output of the tasks
>
> testing with {{concurrent_compactors: 8}}
> In 12 hours it never ended, always showing 1 pending job
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15409) nodetool compactionstats showing extra pending task for TWCS
[
https://issues.apache.org/jira/browse/CASSANDRA-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania Alborghetti updated CASSANDRA-15409:
-
Reviewers: Stefania Alborghetti, Stefania Alborghetti (was: Stefania
Alborghetti)
Stefania Alborghetti, Stefania Alborghetti
Status: Review In Progress (was: Patch Available)
> nodetool compactionstats showing extra pending task for TWCS
>
>
> Key: CASSANDRA-15409
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15409
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Ekaterina Dimitrova
>Assignee: Ekaterina Dimitrova
>Priority: Normal
> Fix For: 3.11.6, 4.x
>
>
> Summary: nodetool compactionstats showing extra pending task for TWCS
> -
> The output of {{nodetool compactionstats}}can show "pending tasks: 1" when
> there are actually none. This seems to be a consistent problem in testing C*
> trunk. In my testing, it looks like the {{nodetool compactionstats}} counter
> output is consistently off by 1 as compared to the table output of the tasks
>
> testing with {{concurrent_compactors: 8}}
> In 12 hours it never ended, always showing 1 pending job
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-9259) Bulk Reading from Cassandra
[ https://issues.apache.org/jira/browse/CASSANDRA-9259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania Alborghetti reassigned CASSANDRA-9259: --- Assignee: (was: Stefania Alborghetti) > Bulk Reading from Cassandra > --- > > Key: CASSANDRA-9259 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9259 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/CQL, Legacy/Local Write-Read Paths, > Legacy/Streaming and Messaging, Legacy/Testing, Local/Compaction >Reporter: Brian Hess >Priority: Urgent > Fix For: 4.x > > Attachments: 256_vnodes.jpg, before_after.jpg, > bulk-read-benchmark.1.html, bulk-read-jfr-profiles.1.tar.gz, > bulk-read-jfr-profiles.2.tar.gz, no_vnodes.jpg, spark_benchmark_raw_data.zip > > > This ticket is following on from the 2015 NGCC. This ticket is designed to be > a place for discussing and designing an approach to bulk reading. > The goal is to have a bulk reading path for Cassandra. That is, a path > optimized to grab a large portion of the data for a table (potentially all of > it). This is a core element in the Spark integration with Cassandra, and the > speed at which Cassandra can deliver bulk data to Spark is limiting the > performance of Spark-plus-Cassandra operations. This is especially of > importance as Cassandra will (likely) leverage Spark for internal operations > (for example CASSANDRA-8234). > The core CQL to consider is the following: > SELECT a, b, c FROM myKs.myTable WHERE Token(partitionKey) > X AND > Token(partitionKey) <= Y > There are a few approaches that could be considered. First, we consider a new > "Streaming Compaction" approach. The key observation here is that a bulk read > from Cassandra is a lot like a major compaction, though instead of outputting > a new SSTable we would output CQL rows to a stream/socket/etc. This would be > similar to a CompactionTask, but would strip out some unnecessary things in > there (e.g., some of the indexing, etc). Predicates and projections could > also be encapsulated in this new "StreamingCompactionTask", for example. > Here, we choose X and Y to be contained within one token range (perhaps > considering the primary range of a node without vnodes, for example). This > query pushes 50K-100K rows/sec, which is not very fast if we are doing bulk > operations via Spark (or other processing frameworks - ETL, etc). There are a > few causes (e.g., inefficient paging). > > Another approach would be an alternate storage format. For example, we might > employ Parquet (just as an example) to store the same data as in the primary > Cassandra storage (aka SSTables). This is akin to Global Indexes (an > alternate storage of the same data optimized for a particular query). Then, > Cassandra can choose to leverage this alternate storage for particular CQL > queries (e.g., range scans). > These are just 2 suggestions to get the conversation going. > One thing to note is that it will be useful to have this storage segregated > by token range so that when you extract via these mechanisms you do not get > replications-factor numbers of copies of the data. That will certainly be an > issue for some Spark operations (e.g., counting). Thus, we will want > per-token-range storage (even for single disks), so this will likely leverage > CASSANDRA-6696 (though, we'll want to also consider the single disk case). > It is also worth discussing what the success criteria is here. It is unlikely > to be as fast as EDW or HDFS performance (though, that is still a good goal), > but being within some percentage of that performance should be set as > success. For example, 2x as long as doing bulk operations on HDFS with > similar node count/size/etc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-11520) Implement optimized local read path for CL.ONE
[ https://issues.apache.org/jira/browse/CASSANDRA-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania Alborghetti updated CASSANDRA-11520: - Resolution: Won't Fix Status: Resolved (was: Open) > Implement optimized local read path for CL.ONE > -- > > Key: CASSANDRA-11520 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11520 > Project: Cassandra > Issue Type: Sub-task > Components: Legacy/CQL, Legacy/Local Write-Read Paths > Reporter: Stefania Alborghetti >Assignee: Stefania Alborghetti >Priority: Normal > Fix For: 4.x > > > -Add an option to the CQL SELECT statement to- Bypass the coordination layer > when reading locally at CL.ONE. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-11521) Implement streaming for bulk read requests
[ https://issues.apache.org/jira/browse/CASSANDRA-11521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania Alborghetti updated CASSANDRA-11521: - Fix Version/s: (was: 4.x) > Implement streaming for bulk read requests > -- > > Key: CASSANDRA-11521 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11521 > Project: Cassandra > Issue Type: Sub-task > Components: Legacy/Local Write-Read Paths > Reporter: Stefania Alborghetti >Assignee: Stefania Alborghetti >Priority: Normal > Labels: protocolv5 > Attachments: final-patch-jfr-profiles-1.zip > > > Allow clients to stream data from a C* host, bypassing the coordination layer > and eliminating the need to query individual pages one by one. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-11521) Implement streaming for bulk read requests
[ https://issues.apache.org/jira/browse/CASSANDRA-11521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania Alborghetti updated CASSANDRA-11521: - Status: Open (was: Patch Available) > Implement streaming for bulk read requests > -- > > Key: CASSANDRA-11521 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11521 > Project: Cassandra > Issue Type: Sub-task > Components: Legacy/Local Write-Read Paths > Reporter: Stefania Alborghetti >Assignee: Stefania Alborghetti >Priority: Normal > Labels: protocolv5 > Fix For: 4.x > > Attachments: final-patch-jfr-profiles-1.zip > > > Allow clients to stream data from a C* host, bypassing the coordination layer > and eliminating the need to query individual pages one by one. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-11521) Implement streaming for bulk read requests
[ https://issues.apache.org/jira/browse/CASSANDRA-11521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania Alborghetti updated CASSANDRA-11521: - Resolution: Won't Fix Status: Resolved (was: Open) > Implement streaming for bulk read requests > -- > > Key: CASSANDRA-11521 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11521 > Project: Cassandra > Issue Type: Sub-task > Components: Legacy/Local Write-Read Paths > Reporter: Stefania Alborghetti >Assignee: Stefania Alborghetti >Priority: Normal > Labels: protocolv5 > Fix For: 4.x > > Attachments: final-patch-jfr-profiles-1.zip > > > Allow clients to stream data from a C* host, bypassing the coordination layer > and eliminating the need to query individual pages one by one. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14554) LifecycleTransaction encounters ConcurrentModificationException when used in multi-threaded context
[
https://issues.apache.org/jira/browse/CASSANDRA-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714205#comment-16714205
]
Stefania commented on CASSANDRA-14554:
--
Thanks for the pull requests [~benedict], they LGTM. Do put both names as
authors, since you've done quite a fair bit of work too.
> LifecycleTransaction encounters ConcurrentModificationException when used in
> multi-threaded context
> ---
>
> Key: CASSANDRA-14554
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14554
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
>Reporter: Dinesh Joshi
>Assignee: Stefania
>Priority: Major
> Fix For: 4.0, 3.0.x, 3.11.x
>
>
> When LifecycleTransaction is used in a multi-threaded context, we encounter
> this exception -
> {quote}java.util.ConcurrentModificationException: null
> at
> java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
> at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:742)
> at java.lang.Iterable.forEach(Iterable.java:74)
> at
> org.apache.cassandra.db.lifecycle.LogReplicaSet.maybeCreateReplica(LogReplicaSet.java:78)
> at org.apache.cassandra.db.lifecycle.LogFile.makeRecord(LogFile.java:320)
> at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:285)
> at
> org.apache.cassandra.db.lifecycle.LogTransaction.trackNew(LogTransaction.java:136)
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.trackNew(LifecycleTransaction.java:529)
> {quote}
> During streaming we create a reference to a {{LifeCycleTransaction}} and
> share it between threads -
> [https://github.com/apache/cassandra/blob/5cc68a87359dd02412bdb70a52dfcd718d44a5ba/src/java/org/apache/cassandra/db/streaming/CassandraStreamReader.java#L156]
> This is used in a multi-threaded context inside {{CassandraIncomingFile}}
> which is an {{IncomingStreamMessage}}. This is being deserialized in parallel.
> {{LifecycleTransaction}} is not meant to be used in a multi-threaded context
> and this leads to streaming failures due to object sharing. On trunk, this
> object is shared across all threads that transfer sstables in parallel for
> the given {{TableId}} in a {{StreamSession}}. There are two options to solve
> this - make {{LifecycleTransaction}} and the associated objects thread safe,
> scope the transaction to a single {{CassandraIncomingFile}}. The consequences
> of the latter option is that if we experience streaming failure we may have
> redundant SSTables on disk. This is ok as compaction should clean this up. A
> third option is we synchronize access in the streaming infrastructure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14554) LifecycleTransaction encounters ConcurrentModificationException when used in multi-threaded context
[
https://issues.apache.org/jira/browse/CASSANDRA-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710890#comment-16710890
]
Stefania commented on CASSANDRA-14554:
--
[~benedict], I'm fine with renaming the new interface {{LifecycleNewTracker}},
or with any other change that you see fit.
> LifecycleTransaction encounters ConcurrentModificationException when used in
> multi-threaded context
> ---
>
> Key: CASSANDRA-14554
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14554
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
>Reporter: Dinesh Joshi
>Assignee: Dinesh Joshi
>Priority: Major
> Fix For: 4.0, 3.0.x, 3.11.x
>
>
> When LifecycleTransaction is used in a multi-threaded context, we encounter
> this exception -
> {quote}java.util.ConcurrentModificationException: null
> at
> java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
> at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:742)
> at java.lang.Iterable.forEach(Iterable.java:74)
> at
> org.apache.cassandra.db.lifecycle.LogReplicaSet.maybeCreateReplica(LogReplicaSet.java:78)
> at org.apache.cassandra.db.lifecycle.LogFile.makeRecord(LogFile.java:320)
> at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:285)
> at
> org.apache.cassandra.db.lifecycle.LogTransaction.trackNew(LogTransaction.java:136)
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.trackNew(LifecycleTransaction.java:529)
> {quote}
> During streaming we create a reference to a {{LifeCycleTransaction}} and
> share it between threads -
> [https://github.com/apache/cassandra/blob/5cc68a87359dd02412bdb70a52dfcd718d44a5ba/src/java/org/apache/cassandra/db/streaming/CassandraStreamReader.java#L156]
> This is used in a multi-threaded context inside {{CassandraIncomingFile}}
> which is an {{IncomingStreamMessage}}. This is being deserialized in parallel.
> {{LifecycleTransaction}} is not meant to be used in a multi-threaded context
> and this leads to streaming failures due to object sharing. On trunk, this
> object is shared across all threads that transfer sstables in parallel for
> the given {{TableId}} in a {{StreamSession}}. There are two options to solve
> this - make {{LifecycleTransaction}} and the associated objects thread safe,
> scope the transaction to a single {{CassandraIncomingFile}}. The consequences
> of the latter option is that if we experience streaming failure we may have
> redundant SSTables on disk. This is ok as compaction should clean this up. A
> third option is we synchronize access in the streaming infrastructure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14554) LifecycleTransaction encounters ConcurrentModificationException when used in multi-threaded context
[
https://issues.apache.org/jira/browse/CASSANDRA-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692560#comment-16692560
]
Stefania commented on CASSANDRA-14554:
--
Thanks [~djoshi3]
> LifecycleTransaction encounters ConcurrentModificationException when used in
> multi-threaded context
> ---
>
> Key: CASSANDRA-14554
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14554
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
>Reporter: Dinesh Joshi
>Assignee: Dinesh Joshi
>Priority: Major
> Fix For: 4.0, 3.0.x, 3.11.x
>
>
> When LifecycleTransaction is used in a multi-threaded context, we encounter
> this exception -
> {quote}java.util.ConcurrentModificationException: null
> at
> java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
> at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:742)
> at java.lang.Iterable.forEach(Iterable.java:74)
> at
> org.apache.cassandra.db.lifecycle.LogReplicaSet.maybeCreateReplica(LogReplicaSet.java:78)
> at org.apache.cassandra.db.lifecycle.LogFile.makeRecord(LogFile.java:320)
> at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:285)
> at
> org.apache.cassandra.db.lifecycle.LogTransaction.trackNew(LogTransaction.java:136)
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.trackNew(LifecycleTransaction.java:529)
> {quote}
> During streaming we create a reference to a {{LifeCycleTransaction}} and
> share it between threads -
> [https://github.com/apache/cassandra/blob/5cc68a87359dd02412bdb70a52dfcd718d44a5ba/src/java/org/apache/cassandra/db/streaming/CassandraStreamReader.java#L156]
> This is used in a multi-threaded context inside {{CassandraIncomingFile}}
> which is an {{IncomingStreamMessage}}. This is being deserialized in parallel.
> {{LifecycleTransaction}} is not meant to be used in a multi-threaded context
> and this leads to streaming failures due to object sharing. On trunk, this
> object is shared across all threads that transfer sstables in parallel for
> the given {{TableId}} in a {{StreamSession}}. There are two options to solve
> this - make {{LifecycleTransaction}} and the associated objects thread safe,
> scope the transaction to a single {{CassandraIncomingFile}}. The consequences
> of the latter option is that if we experience streaming failure we may have
> redundant SSTables on disk. This is ok as compaction should clean this up. A
> third option is we synchronize access in the streaming infrastructure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14554) LifecycleTransaction encounters ConcurrentModificationException when used in multi-threaded context
[
https://issues.apache.org/jira/browse/CASSANDRA-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692560#comment-16692560
]
Stefania edited comment on CASSANDRA-14554 at 11/20/18 2:25 AM:
Thanks [~djoshi3] !
was (Author: stefania):
Thanks [~djoshi3]
> LifecycleTransaction encounters ConcurrentModificationException when used in
> multi-threaded context
> ---
>
> Key: CASSANDRA-14554
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14554
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
>Reporter: Dinesh Joshi
>Assignee: Dinesh Joshi
>Priority: Major
> Fix For: 4.0, 3.0.x, 3.11.x
>
>
> When LifecycleTransaction is used in a multi-threaded context, we encounter
> this exception -
> {quote}java.util.ConcurrentModificationException: null
> at
> java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
> at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:742)
> at java.lang.Iterable.forEach(Iterable.java:74)
> at
> org.apache.cassandra.db.lifecycle.LogReplicaSet.maybeCreateReplica(LogReplicaSet.java:78)
> at org.apache.cassandra.db.lifecycle.LogFile.makeRecord(LogFile.java:320)
> at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:285)
> at
> org.apache.cassandra.db.lifecycle.LogTransaction.trackNew(LogTransaction.java:136)
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.trackNew(LifecycleTransaction.java:529)
> {quote}
> During streaming we create a reference to a {{LifeCycleTransaction}} and
> share it between threads -
> [https://github.com/apache/cassandra/blob/5cc68a87359dd02412bdb70a52dfcd718d44a5ba/src/java/org/apache/cassandra/db/streaming/CassandraStreamReader.java#L156]
> This is used in a multi-threaded context inside {{CassandraIncomingFile}}
> which is an {{IncomingStreamMessage}}. This is being deserialized in parallel.
> {{LifecycleTransaction}} is not meant to be used in a multi-threaded context
> and this leads to streaming failures due to object sharing. On trunk, this
> object is shared across all threads that transfer sstables in parallel for
> the given {{TableId}} in a {{StreamSession}}. There are two options to solve
> this - make {{LifecycleTransaction}} and the associated objects thread safe,
> scope the transaction to a single {{CassandraIncomingFile}}. The consequences
> of the latter option is that if we experience streaming failure we may have
> redundant SSTables on disk. This is ok as compaction should clean this up. A
> third option is we synchronize access in the streaming infrastructure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14554) LifecycleTransaction encounters ConcurrentModificationException when used in multi-threaded context
[
https://issues.apache.org/jira/browse/CASSANDRA-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691225#comment-16691225
]
Stefania edited comment on CASSANDRA-14554 at 11/19/18 4:00 AM:
For trunk, we should probably either introduce a synchronized transaction and
add references in the writers, or look into child transactions.
The patch I proposed is suitable for released branches. I see this ticket is
now classified for 4.0 only. In my opinion, there is an issue also for 3.0 and
3.11. The explanation is in my original
[comment|https://issues.apache.org/jira/browse/CASSANDRA-14554?focusedCommentId=16677739=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16677739].
Perhaps we should create a different ticket for this problem and commit the
patch only to 3.0 and 3.11?
I also would like to follow up with a more comprehensive approach for trunk but
I don't know when I'll have time to work on this. I'll post another update if I
do start work.
was (Author: stefania):
For trunk we should probably either introduce a synchronized transaction and
add references in the writers, or look into child transactions.
The patch I proposed is suitable for released branches however. I see this
ticket is now classified for 4.0 only. In my opinion there is an issue also for
3.0 and 3.11.The explanation is in my original comment. Perhaps we should
create a different ticket for this problem and commit the patch only to 3.0 and
3.11?
I also would like to follow up with a more comprehensive approach for trunk but
I don't know when I'll have time to work on this. I'll post another update if I
do start work however.
> LifecycleTransaction encounters ConcurrentModificationException when used in
> multi-threaded context
> ---
>
> Key: CASSANDRA-14554
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14554
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
>Reporter: Dinesh Joshi
>Assignee: Dinesh Joshi
>Priority: Major
> Fix For: 4.0
>
>
> When LifecycleTransaction is used in a multi-threaded context, we encounter
> this exception -
> {quote}java.util.ConcurrentModificationException: null
> at
> java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
> at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:742)
> at java.lang.Iterable.forEach(Iterable.java:74)
> at
> org.apache.cassandra.db.lifecycle.LogReplicaSet.maybeCreateReplica(LogReplicaSet.java:78)
> at org.apache.cassandra.db.lifecycle.LogFile.makeRecord(LogFile.java:320)
> at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:285)
> at
> org.apache.cassandra.db.lifecycle.LogTransaction.trackNew(LogTransaction.java:136)
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.trackNew(LifecycleTransaction.java:529)
> {quote}
> During streaming we create a reference to a {{LifeCycleTransaction}} and
> share it between threads -
> [https://github.com/apache/cassandra/blob/5cc68a87359dd02412bdb70a52dfcd718d44a5ba/src/java/org/apache/cassandra/db/streaming/CassandraStreamReader.java#L156]
> This is used in a multi-threaded context inside {{CassandraIncomingFile}}
> which is an {{IncomingStreamMessage}}. This is being deserialized in parallel.
> {{LifecycleTransaction}} is not meant to be used in a multi-threaded context
> and this leads to streaming failures due to object sharing. On trunk, this
> object is shared across all threads that transfer sstables in parallel for
> the given {{TableId}} in a {{StreamSession}}. There are two options to solve
> this - make {{LifecycleTransaction}} and the associated objects thread safe,
> scope the transaction to a single {{CassandraIncomingFile}}. The consequences
> of the latter option is that if we experience streaming failure we may have
> redundant SSTables on disk. This is ok as compaction should clean this up. A
> third option is we synchronize access in the streaming infrastructure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14554) LifecycleTransaction encounters ConcurrentModificationException when used in multi-threaded context
[
https://issues.apache.org/jira/browse/CASSANDRA-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691225#comment-16691225
]
Stefania commented on CASSANDRA-14554:
--
For trunk we should probably either introduce a synchronized transaction and
add references in the writers, or look into child transactions.
The patch I proposed is suitable for released branches however. I see this
ticket is now classified for 4.0 only. In my opinion there is an issue also for
3.0 and 3.11.The explanation is in my original comment. Perhaps we should
create a different ticket for this problem and commit the patch only to 3.0 and
3.11?
I also would like to follow up with a more comprehensive approach for trunk but
I don't know when I'll have time to work on this. I'll post another update if I
do start work however.
> LifecycleTransaction encounters ConcurrentModificationException when used in
> multi-threaded context
> ---
>
> Key: CASSANDRA-14554
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14554
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
>Reporter: Dinesh Joshi
>Assignee: Dinesh Joshi
>Priority: Major
> Fix For: 4.0
>
>
> When LifecycleTransaction is used in a multi-threaded context, we encounter
> this exception -
> {quote}java.util.ConcurrentModificationException: null
> at
> java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
> at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:742)
> at java.lang.Iterable.forEach(Iterable.java:74)
> at
> org.apache.cassandra.db.lifecycle.LogReplicaSet.maybeCreateReplica(LogReplicaSet.java:78)
> at org.apache.cassandra.db.lifecycle.LogFile.makeRecord(LogFile.java:320)
> at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:285)
> at
> org.apache.cassandra.db.lifecycle.LogTransaction.trackNew(LogTransaction.java:136)
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.trackNew(LifecycleTransaction.java:529)
> {quote}
> During streaming we create a reference to a {{LifeCycleTransaction}} and
> share it between threads -
> [https://github.com/apache/cassandra/blob/5cc68a87359dd02412bdb70a52dfcd718d44a5ba/src/java/org/apache/cassandra/db/streaming/CassandraStreamReader.java#L156]
> This is used in a multi-threaded context inside {{CassandraIncomingFile}}
> which is an {{IncomingStreamMessage}}. This is being deserialized in parallel.
> {{LifecycleTransaction}} is not meant to be used in a multi-threaded context
> and this leads to streaming failures due to object sharing. On trunk, this
> object is shared across all threads that transfer sstables in parallel for
> the given {{TableId}} in a {{StreamSession}}. There are two options to solve
> this - make {{LifecycleTransaction}} and the associated objects thread safe,
> scope the transaction to a single {{CassandraIncomingFile}}. The consequences
> of the latter option is that if we experience streaming failure we may have
> redundant SSTables on disk. This is ok as compaction should clean this up. A
> third option is we synchronize access in the streaming infrastructure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14616) cassandra-stress write hangs with default options
[
https://issues.apache.org/jira/browse/CASSANDRA-14616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691164#comment-16691164
]
Stefania commented on CASSANDRA-14616:
--
[~Yarnspinner], [~jay.zhuang] are you OK with committing Jay's approach? I
don't mind too much which approach, they are both OK, it's just a matter of
picking a default value.
Jay you are a committer correct? So if Jeremey is OK committing your patch, I
assume you prefer to merge it yourself?
> cassandra-stress write hangs with default options
> -
>
> Key: CASSANDRA-14616
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14616
> Project: Cassandra
> Issue Type: Bug
> Components: Stress
>Reporter: Chris Lohfink
>Assignee: Jeremy
>Priority: Major
>
> Cassandra stress sits there for incredibly long time after connecting to JMX.
> To reproduce {code}./tools/bin/cassandra-stress write{code}
> If you give it a -n its not as bad which is why dtests etc dont seem to be
> impacted. Does not occur in 3.0 branch but does in 3.11 and trunk
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14616) cassandra-stress write hangs with default options
[
https://issues.apache.org/jira/browse/CASSANDRA-14616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania updated CASSANDRA-14616:
-
Reviewer: Stefania
> cassandra-stress write hangs with default options
> -
>
> Key: CASSANDRA-14616
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14616
> Project: Cassandra
> Issue Type: Bug
>Reporter: Chris Lohfink
>Assignee: Jeremy
>Priority: Major
>
> Cassandra stress sits there for incredibly long time after connecting to JMX.
> To reproduce {code}./tools/bin/cassandra-stress write{code}
> If you give it a -n its not as bad which is why dtests etc dont seem to be
> impacted. Does not occur in 3.0 branch but does in 3.11 and trunk
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14890) cassandra-stress hang for 200 seconds if `n` is not specified
[
https://issues.apache.org/jira/browse/CASSANDRA-14890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania updated CASSANDRA-14890:
-
Reviewer: (was: Stefania)
> cassandra-stress hang for 200 seconds if `n` is not specified
> -
>
> Key: CASSANDRA-14890
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14890
> Project: Cassandra
> Issue Type: Bug
> Components: Stress
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
>
> if parameter {{n}} is not specified, cassandra-stress will hang (wait) for
> 200 seconds between warm-up and sending traffic.
> For example, the following command will hang 200 seconds before sending the
> traffic:
> {noformat}
> $ ./tools/bin/cassandra-stress write
> ...
> Created keyspaces. Sleeping 1s for propagation.
> Sleeping 2s...
> Warming up WRITE with 0 iterations...
> Failed to connect over JMX; not collecting these stats
> {noformat}
> It's waiting for this:
> [https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/tools/stress/src/org/apache/cassandra/stress/util/Uncertainty.java#L72]
> As there's no warm-up traffic (CASSANDRA-13773), it will wait until:
> {noformat}
> (measurements >= waiter.maxMeasurements)
> {noformat}
> {{maxMeasurements}} is 200 by default:
> [https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/tools/stress/src/org/apache/cassandra/stress/settings/SettingsCommand.java#L153]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14616) cassandra-stress write hangs with default options
[
https://issues.apache.org/jira/browse/CASSANDRA-14616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania updated CASSANDRA-14616:
-
Status: Ready to Commit (was: Patch Available)
> cassandra-stress write hangs with default options
> -
>
> Key: CASSANDRA-14616
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14616
> Project: Cassandra
> Issue Type: Bug
>Reporter: Chris Lohfink
>Assignee: Jeremy
>Priority: Major
>
> Cassandra stress sits there for incredibly long time after connecting to JMX.
> To reproduce {code}./tools/bin/cassandra-stress write{code}
> If you give it a -n its not as bad which is why dtests etc dont seem to be
> impacted. Does not occur in 3.0 branch but does in 3.11 and trunk
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14616) cassandra-stress write hangs with default options
[
https://issues.apache.org/jira/browse/CASSANDRA-14616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687400#comment-16687400
]
Stefania commented on CASSANDRA-14616:
--
[~jay.zhuang] , [~Yarnspinner] , thanks for fixing this problem and writing the
test.
I checked both patches, they are both good. Perhap's Jay's approach of assuming
50,000 iterations for duration tests is slightly preferable since that was the
old behavior.
> cassandra-stress write hangs with default options
> -
>
> Key: CASSANDRA-14616
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14616
> Project: Cassandra
> Issue Type: Bug
>Reporter: Chris Lohfink
>Assignee: Jeremy
>Priority: Major
>
> Cassandra stress sits there for incredibly long time after connecting to JMX.
> To reproduce {code}./tools/bin/cassandra-stress write{code}
> If you give it a -n its not as bad which is why dtests etc dont seem to be
> impacted. Does not occur in 3.0 branch but does in 3.11 and trunk
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14890) cassandra-stress hang for 200 seconds if `n` is not specified
[
https://issues.apache.org/jira/browse/CASSANDRA-14890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania updated CASSANDRA-14890:
-
Reviewer: Stefania
> cassandra-stress hang for 200 seconds if `n` is not specified
> -
>
> Key: CASSANDRA-14890
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14890
> Project: Cassandra
> Issue Type: Bug
> Components: Stress
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
>
> if parameter {{n}} is not specified, cassandra-stress will hang (wait) for
> 200 seconds between warm-up and sending traffic.
> For example, the following command will hang 200 seconds before sending the
> traffic:
> {noformat}
> $ ./tools/bin/cassandra-stress write
> ...
> Created keyspaces. Sleeping 1s for propagation.
> Sleeping 2s...
> Warming up WRITE with 0 iterations...
> Failed to connect over JMX; not collecting these stats
> {noformat}
> It's waiting for this:
> [https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/tools/stress/src/org/apache/cassandra/stress/util/Uncertainty.java#L72]
> As there's no warm-up traffic (CASSANDRA-13773), it will wait until:
> {noformat}
> (measurements >= waiter.maxMeasurements)
> {noformat}
> {{maxMeasurements}} is 200 by default:
> [https://github.com/apache/cassandra/blob/06209037ea56b5a2a49615a99f1542d6ea1b2947/tools/stress/src/org/apache/cassandra/stress/settings/SettingsCommand.java#L153]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14554) LifecycleTransaction encounters ConcurrentModificationException when used in multi-threaded context
[
https://issues.apache.org/jira/browse/CASSANDRA-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685995#comment-16685995
]
Stefania commented on CASSANDRA-14554:
--
Looking at the 3.11 code, {{StreamReceiveTask}} does synchronize all access to
the txn: update/abort/finish. With the proposed patch the same lock is used for
track/untrackNew. So this patch takes care of the txn status at least up to
3.11. Looking briefly at the code on trunk, it seems the txn has been moved to
{{CassandraStreamReceiver}}. I think the intention was still to synchronize
access to it, but I haven't checked in detail whether that was achieved
correctly.
Ultimately we also need to fix the problem of writers still not having created
files when the txn is cleared, or still having files open on Windows.
If you are confident that child transactions are the best solution I'm not
opposed but the same can be achieved with a synchronized transaction and a
reference to the txn kept by the writers.
> LifecycleTransaction encounters ConcurrentModificationException when used in
> multi-threaded context
> ---
>
> Key: CASSANDRA-14554
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14554
> Project: Cassandra
> Issue Type: Bug
>Reporter: Dinesh Joshi
>Assignee: Dinesh Joshi
>Priority: Major
>
> When LifecycleTransaction is used in a multi-threaded context, we encounter
> this exception -
> {quote}java.util.ConcurrentModificationException: null
> at
> java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
> at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:742)
> at java.lang.Iterable.forEach(Iterable.java:74)
> at
> org.apache.cassandra.db.lifecycle.LogReplicaSet.maybeCreateReplica(LogReplicaSet.java:78)
> at org.apache.cassandra.db.lifecycle.LogFile.makeRecord(LogFile.java:320)
> at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:285)
> at
> org.apache.cassandra.db.lifecycle.LogTransaction.trackNew(LogTransaction.java:136)
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.trackNew(LifecycleTransaction.java:529)
> {quote}
> During streaming we create a reference to a {{LifeCycleTransaction}} and
> share it between threads -
> [https://github.com/apache/cassandra/blob/5cc68a87359dd02412bdb70a52dfcd718d44a5ba/src/java/org/apache/cassandra/db/streaming/CassandraStreamReader.java#L156]
> This is used in a multi-threaded context inside {{CassandraIncomingFile}}
> which is an {{IncomingStreamMessage}}. This is being deserialized in parallel.
> {{LifecycleTransaction}} is not meant to be used in a multi-threaded context
> and this leads to streaming failures due to object sharing. On trunk, this
> object is shared across all threads that transfer sstables in parallel for
> the given {{TableId}} in a {{StreamSession}}. There are two options to solve
> this - make {{LifecycleTransaction}} and the associated objects thread safe,
> scope the transaction to a single {{CassandraIncomingFile}}. The consequences
> of the latter option is that if we experience streaming failure we may have
> redundant SSTables on disk. This is ok as compaction should clean this up. A
> third option is we synchronize access in the streaming infrastructure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14554) LifecycleTransaction encounters ConcurrentModificationException when used in multi-threaded context
[
https://issues.apache.org/jira/browse/CASSANDRA-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684618#comment-16684618
]
Stefania edited comment on CASSANDRA-14554 at 11/13/18 2:05 AM:
{quote}The only reason would be simplifying analysis of the code's behaviour.
For instance, it's not clear to me how we either would (or should) behave in
the stream writers actively working (and creating sstable files) but for whom
the transaction has already been cancelled. Does such a scenario even arise? Is
it possible it would leave partially written sstables?
{quote}
I'm not sure if this scenario may arise when a streaming transaction is
aborted, it depends on streaming details which I've forgotten, but let's step
through it:
- The new sstables are recorded as new records before the files are created.
If the recording fails, because the transaction was aborted, the streamer will
abort with an exception. Fine.
- So long as the sstables are recorded, the transaction tidier will delete the
files on disk and so the contents will be removed from disk as soon as the
streamer finishes writing. Also fine.
- We may however have a race is if the streamer has added a new record to a
txn that is about to be aborted, and the streamer hasn't created sstable files
when the transaction tidier is running. This could leave files on disk. It's an
extremely small window, but it's not impossible.
We keep a reference to the txn only for obsoleted readers of existing files, we
should also keep a reference to the txn until all new files are at least
created and the directory has been synced. Child transactions would solve this
without the need for this extra reference, but we would need to enforce them
for all multi-threaded code (the presence of synchronized methods may lure
people on sharing transactions). The alternative to child transactions is to
force writers to reference the txn.
{quote}we could even do it with a delegating SynchronizedLifecycleTransaction,
which would seem to be equivalent to your patch
{quote}
This was exactly the starting point of my patch. I did not implement a fully
synchronized transaction because the API is quite large. I thought it may need
some cleanup in order to extract the methods related to the transaction
behavior. I did not have the time to look into this, and also cleaning up the
API is not an option on our released branches, due to the risk of introducing
problems, so I extracted the three methods that are used by the writers and
implemented the easiest and safest approach.
was (Author: stefania):
{quote}The only reason would be simplifying analysis of the code's behaviour.
For instance, it's not clear to me how we either would (or should) behave in
the stream writers actively working (and creating sstable files) but for whom
the transaction has already been cancelled. Does such a scenario even arise? Is
it possible it would leave partially written sstables?
{quote}
I'm not sure if this scenario may arise when a streaming transaction is
aborted, it depends on streaming details which I've forgotten, but let's step
through it:
- The new sstables are recorded as new records before the files are created.
If the recording fails, because the transaction was aborted, the streamer will
abort with an exception. Fine.
- So long as the sstables are recorded, the transaction tidier will delete the
files on disk and so the contents will be removed from disk as soon as the
streamer finishes writing. Also fine.
- We may however have a race is if the streamer has added a new record to a
txn that is about to be aborted, and the streamer hasn't created sstable files
when the transaction tidier is running. This could leave files on disk. It's an
extremely small window, but it's not impossible.
We keep a reference to the txn only for obsoleted readers of existing files, we
should also keep a reference to the txn until all new files are at least
created and the directory has been synced. Child transactions would solve this
without the need for this extra reference, but we would need to enforce them
for all multi-threaded code (the presence of synchronized methods may lure
people on sharing transactions). The alternative to child transactions is to
force writers to reference the txn.
{quote}we could even do it with a delegating SynchronizedLifecycleTransaction,
which would seem to be equivalent to your patch
{quote}
This was exactly the starting point of my patch. I did not implement a fully
synchronized transaction because the API is quite large. I thought it may need
some cleanup in order to extract the methods related to the transaction
behavior. I did not have the time to look into this, and also cleaning up the
API is not an option on out released branches, due to the risk of introducing
problems, so I just extracted the three methods that are used
[jira] [Commented] (CASSANDRA-14554) LifecycleTransaction encounters ConcurrentModificationException when used in multi-threaded context
[
https://issues.apache.org/jira/browse/CASSANDRA-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684618#comment-16684618
]
Stefania commented on CASSANDRA-14554:
--
{quote}The only reason would be simplifying analysis of the code's behaviour.
For instance, it's not clear to me how we either would (or should) behave in
the stream writers actively working (and creating sstable files) but for whom
the transaction has already been cancelled. Does such a scenario even arise? Is
it possible it would leave partially written sstables?
{quote}
I'm not sure if this scenario may arise when a streaming transaction is
aborted, it depends on streaming details which I've forgotten, but let's step
through it:
- The new sstables are recorded as new records before the files are created.
If the recording fails, because the transaction was aborted, the streamer will
abort with an exception. Fine.
- So long as the sstables are recorded, the transaction tidier will delete the
files on disk and so the contents will be removed from disk as soon as the
streamer finishes writing. Also fine.
- We may however have a race is if the streamer has added a new record to a
txn that is about to be aborted, and the streamer hasn't created sstable files
when the transaction tidier is running. This could leave files on disk. It's an
extremely small window, but it's not impossible.
We keep a reference to the txn only for obsoleted readers of existing files, we
should also keep a reference to the txn until all new files are at least
created and the directory has been synced. Child transactions would solve this
without the need for this extra reference, but we would need to enforce them
for all multi-threaded code (the presence of synchronized methods may lure
people on sharing transactions). The alternative to child transactions is to
force writers to reference the txn.
{quote}we could even do it with a delegating SynchronizedLifecycleTransaction,
which would seem to be equivalent to your patch
{quote}
This was exactly the starting point of my patch. I did not implement a fully
synchronized transaction because the API is quite large. I thought it may need
some cleanup in order to extract the methods related to the transaction
behavior. I did not have the time to look into this, and also cleaning up the
API is not an option on out released branches, due to the risk of introducing
problems, so I just extracted the three methods that are used by the writers
and implemented the safest approach.
> LifecycleTransaction encounters ConcurrentModificationException when used in
> multi-threaded context
> ---
>
> Key: CASSANDRA-14554
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14554
> Project: Cassandra
> Issue Type: Bug
>Reporter: Dinesh Joshi
>Assignee: Dinesh Joshi
>Priority: Major
>
> When LifecycleTransaction is used in a multi-threaded context, we encounter
> this exception -
> {quote}java.util.ConcurrentModificationException: null
> at
> java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
> at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:742)
> at java.lang.Iterable.forEach(Iterable.java:74)
> at
> org.apache.cassandra.db.lifecycle.LogReplicaSet.maybeCreateReplica(LogReplicaSet.java:78)
> at org.apache.cassandra.db.lifecycle.LogFile.makeRecord(LogFile.java:320)
> at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:285)
> at
> org.apache.cassandra.db.lifecycle.LogTransaction.trackNew(LogTransaction.java:136)
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.trackNew(LifecycleTransaction.java:529)
> {quote}
> During streaming we create a reference to a {{LifeCycleTransaction}} and
> share it between threads -
> [https://github.com/apache/cassandra/blob/5cc68a87359dd02412bdb70a52dfcd718d44a5ba/src/java/org/apache/cassandra/db/streaming/CassandraStreamReader.java#L156]
> This is used in a multi-threaded context inside {{CassandraIncomingFile}}
> which is an {{IncomingStreamMessage}}. This is being deserialized in parallel.
> {{LifecycleTransaction}} is not meant to be used in a multi-threaded context
> and this leads to streaming failures due to object sharing. On trunk, this
> object is shared across all threads that transfer sstables in parallel for
> the given {{TableId}} in a {{StreamSession}}. There are two options to solve
> this - make {{LifecycleTransaction}} and the associated objects thread safe,
> scope the transaction to a single {{CassandraIncomingFile}}. The consequences
> of the latter option is that if we experience streaming f
[jira] [Commented] (CASSANDRA-14554) LifecycleTransaction encounters ConcurrentModificationException when used in multi-threaded context
[
https://issues.apache.org/jira/browse/CASSANDRA-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680787#comment-16680787
]
Stefania commented on CASSANDRA-14554:
--
bq. The other methods on a LifecycleTransaction could simply be marked
synchronised as well.
If we are synchronizing the LifecycleTransaction methods anyway, I'm not sure I
understand why we need child transactions. Even in 4.0, where Netty threads
call {{trackNew}}, I don't think we're adding sstables so frequently to
introduce contention on a shared, synchronized txn. Considering {{trackNew}}
performs a file sync as you correctly reminded me, surely this blocks Netty
threads more than a synchronized {{trackNew}}. Maybe if many sstables are
created concurrently during streaming, child transactions would make sense. I'm
still not totally sure.
It's fine with me if we prefer to try a different alternative, the patch is
available at any time. This code is not changing much so there is little risk
of the patch getting stale. For info, internally [~snazy] already reviewed the
patch.
> LifecycleTransaction encounters ConcurrentModificationException when used in
> multi-threaded context
> ---
>
> Key: CASSANDRA-14554
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14554
> Project: Cassandra
> Issue Type: Bug
>Reporter: Dinesh Joshi
>Assignee: Dinesh Joshi
>Priority: Major
>
> When LifecycleTransaction is used in a multi-threaded context, we encounter
> this exception -
> {quote}java.util.ConcurrentModificationException: null
> at
> java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
> at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:742)
> at java.lang.Iterable.forEach(Iterable.java:74)
> at
> org.apache.cassandra.db.lifecycle.LogReplicaSet.maybeCreateReplica(LogReplicaSet.java:78)
> at org.apache.cassandra.db.lifecycle.LogFile.makeRecord(LogFile.java:320)
> at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:285)
> at
> org.apache.cassandra.db.lifecycle.LogTransaction.trackNew(LogTransaction.java:136)
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.trackNew(LifecycleTransaction.java:529)
> {quote}
> During streaming we create a reference to a {{LifeCycleTransaction}} and
> share it between threads -
> [https://github.com/apache/cassandra/blob/5cc68a87359dd02412bdb70a52dfcd718d44a5ba/src/java/org/apache/cassandra/db/streaming/CassandraStreamReader.java#L156]
> This is used in a multi-threaded context inside {{CassandraIncomingFile}}
> which is an {{IncomingStreamMessage}}. This is being deserialized in parallel.
> {{LifecycleTransaction}} is not meant to be used in a multi-threaded context
> and this leads to streaming failures due to object sharing. On trunk, this
> object is shared across all threads that transfer sstables in parallel for
> the given {{TableId}} in a {{StreamSession}}. There are two options to solve
> this - make {{LifecycleTransaction}} and the associated objects thread safe,
> scope the transaction to a single {{CassandraIncomingFile}}. The consequences
> of the latter option is that if we experience streaming failure we may have
> redundant SSTables on disk. This is ok as compaction should clean this up. A
> third option is we synchronize access in the streaming infrastructure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14554) LifecycleTransaction encounters ConcurrentModificationException when used in multi-threaded context
[
https://issues.apache.org/jira/browse/CASSANDRA-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679214#comment-16679214
]
Stefania commented on CASSANDRA-14554:
--
You're welcome [~benedict] !
bq. I wonder if you had considered (and potentially discarded) what might be a
slightly simpler approach of allocating a separate LifecycleTransaction for
each operation, and atomically transferring their contents as they "complete"
to the shared LivecycleTransaction?
No I hadn't considered it. It sounds elegant in principle but in order to
atomically transfer child transactions to their parent, we'd have to add some
complexity to transactions that I'm not sure we need. Obviously, the state of
the parent transaction could change at any time (due to an abort), including
whilst a child transaction is trying to transfer its state. So this would
require some form of synchronization or CAS. The same is true for two child
transactions transferring their state simultaneously. The state on disk should
be fine as long as child transactions are never committed but only transferred.
Child transaction should be allowed to abort independently though. So different
rules for child and parent transactions would apply.
I'm not sure we need this additional complexity because the txn state only
changes rarely. {{LifecycleTransaction}} exposes a large API, but many methods
are probably only used during compaction. Extracting a more comprehensive
interface that can be implemented with a synchronized wrapper may be an easier
approach.
I submitted a safe patch that fixes a known problem with streaming and that is
safe for branches that will not undergo a major release testing cycle.
Unfortunately, I do not have the time to work on a more comprehensive solution,
at least not right now. I could however review whichever approach we choose.
> LifecycleTransaction encounters ConcurrentModificationException when used in
> multi-threaded context
> ---
>
> Key: CASSANDRA-14554
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14554
> Project: Cassandra
> Issue Type: Bug
>Reporter: Dinesh Joshi
>Assignee: Dinesh Joshi
>Priority: Major
>
> When LifecycleTransaction is used in a multi-threaded context, we encounter
> this exception -
> {quote}java.util.ConcurrentModificationException: null
> at
> java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
> at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:742)
> at java.lang.Iterable.forEach(Iterable.java:74)
> at
> org.apache.cassandra.db.lifecycle.LogReplicaSet.maybeCreateReplica(LogReplicaSet.java:78)
> at org.apache.cassandra.db.lifecycle.LogFile.makeRecord(LogFile.java:320)
> at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:285)
> at
> org.apache.cassandra.db.lifecycle.LogTransaction.trackNew(LogTransaction.java:136)
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.trackNew(LifecycleTransaction.java:529)
> {quote}
> During streaming we create a reference to a {{LifeCycleTransaction}} and
> share it between threads -
> [https://github.com/apache/cassandra/blob/5cc68a87359dd02412bdb70a52dfcd718d44a5ba/src/java/org/apache/cassandra/db/streaming/CassandraStreamReader.java#L156]
> This is used in a multi-threaded context inside {{CassandraIncomingFile}}
> which is an {{IncomingStreamMessage}}. This is being deserialized in parallel.
> {{LifecycleTransaction}} is not meant to be used in a multi-threaded context
> and this leads to streaming failures due to object sharing. On trunk, this
> object is shared across all threads that transfer sstables in parallel for
> the given {{TableId}} in a {{StreamSession}}. There are two options to solve
> this - make {{LifecycleTransaction}} and the associated objects thread safe,
> scope the transaction to a single {{CassandraIncomingFile}}. The consequences
> of the latter option is that if we experience streaming failure we may have
> redundant SSTables on disk. This is ok as compaction should clean this up. A
> third option is we synchronize access in the streaming infrastructure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14554) LifecycleTransaction encounters ConcurrentModificationException when used in multi-threaded context
[
https://issues.apache.org/jira/browse/CASSANDRA-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677739#comment-16677739
]
Stefania commented on CASSANDRA-14554:
--
We had a related issue where one of our customers ended up with a corrupt txn
log file during streaming, with an ADD record following an ABORT record. We
couldn't look at the logs as they weren't available any longer, since the
customer only noticed the problem when the node would not restart 22 days
later. However, it's pretty obvious in my opinion that one thread aborted the
streaming session whilst the receiving thread was adding a new sstable. So this
seems the same root cause as reported in this ticket, which is that streaming
is using the txn in a thread unsafe way. In my opinion, the problem exists
since 3.0. However it becomes significanlty more likely with the Netty
streaming refactoring. Our customer was on a branch based on 3.11.
We took a very conservative approach with the fix, in that we didn't want to
fully synchronize abstract transactional and the lifecycle transaction on
released branches. We could consider synchronizing these classes for 4.0
however, or reworking streaming.
Here are the 3.11 changes, if there is interest in this approach I can create
patches for 3.0 and trunk as well:
[https://github.com/apache/cassandra/compare/cassandra-3.11...stef1927:db-2633-3.11]
We simply extracted a new interface, the [sstable
tracker|https://github.com/apache/cassandra/compare/cassandra-3.11...stef1927:db-2633-3.11#diff-9d71c7ad9ad16368bd0429d3b34e2b21R15],
which is also
[implemented|https://github.com/apache/cassandra/compare/cassandra-3.11...stef1927:db-2633-3.11#diff-1a464da4a62ac4a734c725059cbc918bR144]
by {{StreamReceiveTask}} by synchronizing the access to the txn, just like it
does for all its other accesses to the txn. Whilst it's not ideal to have an
additional interface, the change should be quite safe for released branches.
> LifecycleTransaction encounters ConcurrentModificationException when used in
> multi-threaded context
> ---
>
> Key: CASSANDRA-14554
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14554
> Project: Cassandra
> Issue Type: Bug
>Reporter: Dinesh Joshi
>Assignee: Dinesh Joshi
>Priority: Major
>
> When LifecycleTransaction is used in a multi-threaded context, we encounter
> this exception -
> {quote}java.util.ConcurrentModificationException: null
> at
> java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
> at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:742)
> at java.lang.Iterable.forEach(Iterable.java:74)
> at
> org.apache.cassandra.db.lifecycle.LogReplicaSet.maybeCreateReplica(LogReplicaSet.java:78)
> at org.apache.cassandra.db.lifecycle.LogFile.makeRecord(LogFile.java:320)
> at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:285)
> at
> org.apache.cassandra.db.lifecycle.LogTransaction.trackNew(LogTransaction.java:136)
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.trackNew(LifecycleTransaction.java:529)
> {quote}
> During streaming we create a reference to a {{LifeCycleTransaction}} and
> share it between threads -
> [https://github.com/apache/cassandra/blob/5cc68a87359dd02412bdb70a52dfcd718d44a5ba/src/java/org/apache/cassandra/db/streaming/CassandraStreamReader.java#L156]
> This is used in a multi-threaded context inside {{CassandraIncomingFile}}
> which is an {{IncomingStreamMessage}}. This is being deserialized in parallel.
> {{LifecycleTransaction}} is not meant to be used in a multi-threaded context
> and this leads to streaming failures due to object sharing. On trunk, this
> object is shared across all threads that transfer sstables in parallel for
> the given {{TableId}} in a {{StreamSession}}. There are two options to solve
> this - make {{LifecycleTransaction}} and the associated objects thread safe,
> scope the transaction to a single {{CassandraIncomingFile}}. The consequences
> of the latter option is that if we experience streaming failure we may have
> redundant SSTables on disk. This is ok as compaction should clean this up. A
> third option is we synchronize access in the streaming infrastructure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14391) COPY FROM should read columns from file header
[
https://issues.apache.org/jira/browse/CASSANDRA-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania updated CASSANDRA-14391:
-
Summary: COPY FROM should read columns from file header (was: COPY FROM
ignores headers)
> COPY FROM should read columns from file header
> --
>
> Key: CASSANDRA-14391
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14391
> Project: Cassandra
> Issue Type: Improvement
> Components: CQL
> Environment: cqlsh 5.0.1 and Cassandra 3.11.2 on macOS 10.13.2.
>Reporter: M. Justin
>Priority: Major
>
> COPY FROM appears to ignore the headers value, even when "headers = true" is
> specified. This means that if the columns are reordered, the import process
> will save values in the wrong columns. Additionally, if there are missing
> columns, an error occurs, even if those columns are not primary key columns.
> This behavior contradicts the behavior [specified in the
> docs|https://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlshCopy.html]
> (emphasis mine).
> {quote}COPY FROM imports data from a CSV file into an existing table. Each
> line in the source file is imported as a row. All rows in the dataset must
> contain the same number of fields and have values in the PRIMARY KEY fields.
> The process verifies the PRIMARY KEY and updates existing records. If
> {{HEADER = false}} and no column names are specified, the fields are imported
> in deterministic order. *When column names are specified, fields are imported
> in that order. Missing and empty fields are set to null. The source cannot
> have more fields than the target table, however it can have fewer fields.*
> {quote}
> h2. Example
> {noformat:title=temp.csv}
> col2,col1,col3
> column value 1,key2,3
> column value 2,key4,3
> column value 3,key3,3
> column value 4,key1,3
> {noformat}
> {code:sql}
> create keyspace copy_to_from_test WITH replication = { 'class' :
> 'SimpleStrategy', 'replication_factor' : 1 };
> use copy_to_from_test;
> create table test_table (col1 text primary key, col2 text, col3 bigint);
> copy test_table from 'temp.csv' with header = true;
> {code}
> The above code will incorrectly swap the "col2" and "col1" values, since it
> expects the first column to be "col1". If I had instead swapped the order of
> "col3", I would have received an error on input, as it would have attempted
> to store text in a numerical column.
> h2. Expected Behavior
> I would expect specifying "with header = true" on a COPY FROM statement to
> use the headers as column names for insertion, rather than merely skipping
> the header row. Missing non-primary key columns should be set to null.
> h2. Other
> I ran across this issue when copying between two of my environments. One of
> the environments had changed the columns in the primary key, but the other
> had not yet. This caused the order of the columns to vary between the
> environments.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14391) COPY FROM ignores headers
[
https://issues.apache.org/jira/browse/CASSANDRA-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania updated CASSANDRA-14391:
-
Issue Type: Improvement (was: Bug)
> COPY FROM ignores headers
> -
>
> Key: CASSANDRA-14391
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14391
> Project: Cassandra
> Issue Type: Improvement
> Components: CQL
> Environment: cqlsh 5.0.1 and Cassandra 3.11.2 on macOS 10.13.2.
>Reporter: M. Justin
>Priority: Major
>
> COPY FROM appears to ignore the headers value, even when "headers = true" is
> specified. This means that if the columns are reordered, the import process
> will save values in the wrong columns. Additionally, if there are missing
> columns, an error occurs, even if those columns are not primary key columns.
> This behavior contradicts the behavior [specified in the
> docs|https://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlshCopy.html]
> (emphasis mine).
> {quote}COPY FROM imports data from a CSV file into an existing table. Each
> line in the source file is imported as a row. All rows in the dataset must
> contain the same number of fields and have values in the PRIMARY KEY fields.
> The process verifies the PRIMARY KEY and updates existing records. If
> {{HEADER = false}} and no column names are specified, the fields are imported
> in deterministic order. *When column names are specified, fields are imported
> in that order. Missing and empty fields are set to null. The source cannot
> have more fields than the target table, however it can have fewer fields.*
> {quote}
> h2. Example
> {noformat:title=temp.csv}
> col2,col1,col3
> column value 1,key2,3
> column value 2,key4,3
> column value 3,key3,3
> column value 4,key1,3
> {noformat}
> {code:sql}
> create keyspace copy_to_from_test WITH replication = { 'class' :
> 'SimpleStrategy', 'replication_factor' : 1 };
> use copy_to_from_test;
> create table test_table (col1 text primary key, col2 text, col3 bigint);
> copy test_table from 'temp.csv' with header = true;
> {code}
> The above code will incorrectly swap the "col2" and "col1" values, since it
> expects the first column to be "col1". If I had instead swapped the order of
> "col3", I would have received an error on input, as it would have attempted
> to store text in a numerical column.
> h2. Expected Behavior
> I would expect specifying "with header = true" on a COPY FROM statement to
> use the headers as column names for insertion, rather than merely skipping
> the header row. Missing non-primary key columns should be set to null.
> h2. Other
> I ran across this issue when copying between two of my environments. One of
> the environments had changed the columns in the primary key, but the other
> had not yet. This caused the order of the columns to vary between the
> environments.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14391) COPY FROM ignores headers
[
https://issues.apache.org/jira/browse/CASSANDRA-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453359#comment-16453359
]
Stefania commented on CASSANDRA-14391:
--
The columns can be specified in the command line. From typing {{HELP COPY}} in
cqlsh:
{code:java}
COPY [ ( column [, ...] ) ]
FROM ( '<file_pattern_1, file_pattern_2, ... file_pattern_n>' | STDIN )
[ WITH ='value' [AND ...] ];{code}
The {{HEADER}} option simply indicates whether the first line of the file
should be ignored:
{code:java}
HEADER=false - whether to ignore the first line{code}
There is no present way to load column names from the file, this is not a bug
it's the expected behavior. It can be improved but I don't think this ticket
should be a bug.
> COPY FROM ignores headers
> -
>
> Key: CASSANDRA-14391
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14391
> Project: Cassandra
> Issue Type: Bug
> Components: CQL
> Environment: cqlsh 5.0.1 and Cassandra 3.11.2 on macOS 10.13.2.
>Reporter: M. Justin
>Priority: Major
>
> COPY FROM appears to ignore the headers value, even when "headers = true" is
> specified. This means that if the columns are reordered, the import process
> will save values in the wrong columns. Additionally, if there are missing
> columns, an error occurs, even if those columns are not primary key columns.
> This behavior contradicts the behavior [specified in the
> docs|https://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlshCopy.html]
> (emphasis mine).
> {quote}COPY FROM imports data from a CSV file into an existing table. Each
> line in the source file is imported as a row. All rows in the dataset must
> contain the same number of fields and have values in the PRIMARY KEY fields.
> The process verifies the PRIMARY KEY and updates existing records. If
> {{HEADER = false}} and no column names are specified, the fields are imported
> in deterministic order. *When column names are specified, fields are imported
> in that order. Missing and empty fields are set to null. The source cannot
> have more fields than the target table, however it can have fewer fields.*
> {quote}
> h2. Example
> {noformat:title=temp.csv}
> col2,col1,col3
> column value 1,key2,3
> column value 2,key4,3
> column value 3,key3,3
> column value 4,key1,3
> {noformat}
> {code:sql}
> create keyspace copy_to_from_test WITH replication = { 'class' :
> 'SimpleStrategy', 'replication_factor' : 1 };
> use copy_to_from_test;
> create table test_table (col1 text primary key, col2 text, col3 bigint);
> copy test_table from 'temp.csv' with header = true;
> {code}
> The above code will incorrectly swap the "col2" and "col1" values, since it
> expects the first column to be "col1". If I had instead swapped the order of
> "col3", I would have received an error on input, as it would have attempted
> to store text in a numerical column.
> h2. Expected Behavior
> I would expect specifying "with header = true" on a COPY FROM statement to
> use the headers as column names for insertion, rather than merely skipping
> the header row. Missing non-primary key columns should be set to null.
> h2. Other
> I ran across this issue when copying between two of my environments. One of
> the environments had changed the columns in the primary key, but the other
> had not yet. This caused the order of the columns to vary between the
> environments.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14308) Remove invalid SSTables from interrupted compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-14308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408938#comment-16408938
]
Stefania edited comment on CASSANDRA-14308 at 3/22/18 2:23 AM:
---
I'm not sure about deleting them, after all the data is still there. I'd say
maybe move them to a separate folder and issue an error so that the operator
can then delete them manually. I also wonder if it's possible to recreate a
stats component with a valid serialization header but dummy stats, for example
by using
[SerializationHeader.makeWithoutStats()|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/SerializationHeader.java#L69].
This would only be safe if the schema hasn't changed.
was (Author: stefania):
I'm not sure about deleting them, after all the data is still there. I'd say
maybe move them to a separate folder and issue an error so that the operator
can then delete them manually. I also wonder if it's possible to recreate a
stats component with a valid serialization header but dummy stats, for example
by using
[SerializationHeader.makeWithoutStats()|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/SerializationHeader.java#L69].
> Remove invalid SSTables from interrupted compaction
> ---
>
> Key: CASSANDRA-14308
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14308
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
>
> If the JVM crash while compaction is in progress, the incompleted SSTable
> won't be cleaned up, which causes "{{Stats component is missing for
> sstable}}" error in the startup log:
> {noformat}
> ERROR [SSTableBatchOpen:3] 2018-03-11 00:17:35,597 CassandraDaemon.java:207 -
> Exception in thread Thread[SSTableBatchOpen:3,5,main]
> java.lang.AssertionError: Stats component is missing for sstable
> /cassandra/data/keyspace/table-id/mc-12345-big
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:458)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:374)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:533)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_121]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_121]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> [apache-cassandra-3.0.14.jar:3.0.14]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
> {noformat}
> The accumulated incompleted SSTables could take lots of space, especially for
> STCS which could have very large SSTables.
> Here is the script we use to delete the SSTables after node is restarted:
> {noformat}
> grep 'Stats component is missing for sstable' $SYSTEM_LOG | awk '{print $8}'
> > ~/invalid_sstables ; for ss in `cat ~/invalid_sstables`; do echo == $ss; ll
> $ss*; sudo rm $ss* ; done
> {noformat}
> I would suggest to remove these incompleted SSTables while startup.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14308) Remove invalid SSTables from interrupted compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-14308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408938#comment-16408938
]
Stefania commented on CASSANDRA-14308:
--
I'm not sure about deleting them, after all the data is still there. I'd say
maybe move them to a separate folder and issue an error so that the operator
can then delete them manually. I also wonder if it's possible to recreate a
stats component with a valid serialization header but dummy stats, for example
by using
[SerializationHeader.makeWithoutStats()|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/SerializationHeader.java#L69].
> Remove invalid SSTables from interrupted compaction
> ---
>
> Key: CASSANDRA-14308
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14308
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
>
> If the JVM crash while compaction is in progress, the incompleted SSTable
> won't be cleaned up, which causes "{{Stats component is missing for
> sstable}}" error in the startup log:
> {noformat}
> ERROR [SSTableBatchOpen:3] 2018-03-11 00:17:35,597 CassandraDaemon.java:207 -
> Exception in thread Thread[SSTableBatchOpen:3,5,main]
> java.lang.AssertionError: Stats component is missing for sstable
> /cassandra/data/keyspace/table-id/mc-12345-big
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:458)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:374)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:533)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_121]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_121]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> [apache-cassandra-3.0.14.jar:3.0.14]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
> {noformat}
> The accumulated incompleted SSTables could take lots of space, especially for
> STCS which could have very large SSTables.
> Here is the script we use to delete the SSTables after node is restarted:
> {noformat}
> grep 'Stats component is missing for sstable' $SYSTEM_LOG | awk '{print $8}'
> > ~/invalid_sstables ; for ss in `cat ~/invalid_sstables`; do echo == $ss; ll
> $ss*; sudo rm $ss* ; done
> {noformat}
> I would suggest to remove these incompleted SSTables while startup.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14308) Remove invalid SSTables from interrupted compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-14308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407426#comment-16407426
]
Stefania commented on CASSANDRA-14308:
--
I had a good look at the code, the stats seemed to be sync-ed to disk correctly.
Could you maybe, once the problem reproduce, start with logs at trace level and
verify if the compactions leftovers are removed before that exception is shown?
You should see in the logs {{Removing temporary or obsoleted files from
unfinished operations for table...}}. Also, you could check whether there are
any txn log files and if so check what the content is.
> Remove invalid SSTables from interrupted compaction
> ---
>
> Key: CASSANDRA-14308
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14308
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
>
> If the JVM crash while compaction is in progress, the incompleted SSTable
> won't be cleaned up, which causes "{{Stats component is missing for
> sstable}}" error in the startup log:
> {noformat}
> ERROR [SSTableBatchOpen:3] 2018-03-11 00:17:35,597 CassandraDaemon.java:207 -
> Exception in thread Thread[SSTableBatchOpen:3,5,main]
> java.lang.AssertionError: Stats component is missing for sstable
> /cassandra/data/keyspace/table-id/mc-12345-big
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:458)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:374)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:533)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_121]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_121]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> [apache-cassandra-3.0.14.jar:3.0.14]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
> {noformat}
> The accumulated incompleted SSTables could take lots of space, especially for
> STCS which could have very large SSTables.
> Here is the script we use to delete the SSTables after node is restarted:
> {noformat}
> grep 'Stats component is missing for sstable' $SYSTEM_LOG | awk '{print $8}'
> > ~/invalid_sstables ; for ss in `cat ~/invalid_sstables`; do echo == $ss; ll
> $ss*; sudo rm $ss* ; done
> {noformat}
> I would suggest to remove these incompleted SSTables while startup.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14308) Remove invalid SSTables from interrupted compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-14308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407376#comment-16407376
]
Stefania commented on CASSANDRA-14308:
--
That's correct, whilst a compaction is in progress a txn log file keep tracks
of the new and old sstables. The txn log is only committed once the compaction
is finished. Upon startup, if txn log files are found, they are examined and if
they were not committed then the compaction leftovers will be cleared.
May I suggest checking if anything prevents the stats component from being
written, or sync-ed to disk, before the compaction txn is committed?
> Remove invalid SSTables from interrupted compaction
> ---
>
> Key: CASSANDRA-14308
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14308
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
>
> If the JVM crash while compaction is in progress, the incompleted SSTable
> won't be cleaned up, which causes "{{Stats component is missing for
> sstable}}" error in the startup log:
> {noformat}
> ERROR [SSTableBatchOpen:3] 2018-03-11 00:17:35,597 CassandraDaemon.java:207 -
> Exception in thread Thread[SSTableBatchOpen:3,5,main]
> java.lang.AssertionError: Stats component is missing for sstable
> /cassandra/data/keyspace/table-id/mc-12345-big
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:458)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:374)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:533)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_121]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_121]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_121]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> [apache-cassandra-3.0.14.jar:3.0.14]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
> {noformat}
> The accumulated incompleted SSTables could take lots of space, especially for
> STCS which could have very large SSTables.
> Here is the script we use to delete the SSTables after node is restarted:
> {noformat}
> grep 'Stats component is missing for sstable' $SYSTEM_LOG | awk '{print $8}'
> > ~/invalid_sstables ; for ss in `cat ~/invalid_sstables`; do echo == $ss; ll
> $ss*; sudo rm $ss* ; done
> {noformat}
> I would suggest to remove these incompleted SSTables while startup.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-12763) Compaction performance issues when a table has a lot of sstables
[
https://issues.apache.org/jira/browse/CASSANDRA-12763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania updated CASSANDRA-12763:
-
Status: Ready to Commit (was: Patch Available)
> Compaction performance issues when a table has a lot of sstables
>
>
> Key: CASSANDRA-12763
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12763
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Tom van der Woerdt
>Assignee: Marcus Eriksson
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> An issue with a script flooded my cluster with sstables. There is now a table
> with 100k sstables, all on the order of KBytes, and it's taking a long time
> (ETA 20 days) to compact, even though the table is only ~30GB.
> Stack trace :
> {noformat}
> "CompactionExecutor:269" #7536 daemon prio=1 os_prio=4 tid=0x7f4acd40fc00
> nid=0x14f8 runnable [0x7f4798436000]
>java.lang.Thread.State: RUNNABLE
> at java.io.UnixFileSystem.list(Native Method)
> at java.io.File.list(File.java:1122)
> at java.io.File.listFiles(File.java:1248)
> at
> org.apache.cassandra.db.lifecycle.LogRecord.getExistingFiles(LogRecord.java:268)
> at org.apache.cassandra.db.lifecycle.LogRecord.make(LogRecord.java:150)
> at
> org.apache.cassandra.db.lifecycle.LogFile.makeRecord(LogFile.java:293)
> at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:283)
> at
> org.apache.cassandra.db.lifecycle.LogTransaction.obsoleted(LogTransaction.java:158)
> at
> org.apache.cassandra.db.lifecycle.Helpers.prepareForObsoletion(Helpers.java:134)
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.doPrepare(LifecycleTransaction.java:193)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:376)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:84)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:94)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:194)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> listFiles is being called over and over, apparently scaling with the number
> of files in the compaction.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12763) Compaction performance issues when a table has a lot of sstables
[
https://issues.apache.org/jira/browse/CASSANDRA-12763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361803#comment-16361803
]
Stefania commented on CASSANDRA-12763:
--
LGTM - great job [~krummas].
Nits: there are some unused imports and redundant blank lines in
{{HelpersTest}}.
Note: the CI failures look unrelated, but there are just too many failing
dtests to check them all in detail.
> Compaction performance issues when a table has a lot of sstables
>
>
> Key: CASSANDRA-12763
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12763
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Tom van der Woerdt
>Assignee: Marcus Eriksson
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> An issue with a script flooded my cluster with sstables. There is now a table
> with 100k sstables, all on the order of KBytes, and it's taking a long time
> (ETA 20 days) to compact, even though the table is only ~30GB.
> Stack trace :
> {noformat}
> "CompactionExecutor:269" #7536 daemon prio=1 os_prio=4 tid=0x7f4acd40fc00
> nid=0x14f8 runnable [0x7f4798436000]
>java.lang.Thread.State: RUNNABLE
> at java.io.UnixFileSystem.list(Native Method)
> at java.io.File.list(File.java:1122)
> at java.io.File.listFiles(File.java:1248)
> at
> org.apache.cassandra.db.lifecycle.LogRecord.getExistingFiles(LogRecord.java:268)
> at org.apache.cassandra.db.lifecycle.LogRecord.make(LogRecord.java:150)
> at
> org.apache.cassandra.db.lifecycle.LogFile.makeRecord(LogFile.java:293)
> at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:283)
> at
> org.apache.cassandra.db.lifecycle.LogTransaction.obsoleted(LogTransaction.java:158)
> at
> org.apache.cassandra.db.lifecycle.Helpers.prepareForObsoletion(Helpers.java:134)
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.doPrepare(LifecycleTransaction.java:193)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:376)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:84)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:94)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:194)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> listFiles is being called over and over, apparently scaling with the number
> of files in the compaction.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12763) Compaction performance issues when a table has a lot of sstables
[
https://issues.apache.org/jira/browse/CASSANDRA-12763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360205#comment-16360205
]
Stefania commented on CASSANDRA-12763:
--
Sure, I'll try to review it in the next few days.
> Compaction performance issues when a table has a lot of sstables
>
>
> Key: CASSANDRA-12763
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12763
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Tom van der Woerdt
>Assignee: Marcus Eriksson
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> An issue with a script flooded my cluster with sstables. There is now a table
> with 100k sstables, all on the order of KBytes, and it's taking a long time
> (ETA 20 days) to compact, even though the table is only ~30GB.
> Stack trace :
> {noformat}
> "CompactionExecutor:269" #7536 daemon prio=1 os_prio=4 tid=0x7f4acd40fc00
> nid=0x14f8 runnable [0x7f4798436000]
>java.lang.Thread.State: RUNNABLE
> at java.io.UnixFileSystem.list(Native Method)
> at java.io.File.list(File.java:1122)
> at java.io.File.listFiles(File.java:1248)
> at
> org.apache.cassandra.db.lifecycle.LogRecord.getExistingFiles(LogRecord.java:268)
> at org.apache.cassandra.db.lifecycle.LogRecord.make(LogRecord.java:150)
> at
> org.apache.cassandra.db.lifecycle.LogFile.makeRecord(LogFile.java:293)
> at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:283)
> at
> org.apache.cassandra.db.lifecycle.LogTransaction.obsoleted(LogTransaction.java:158)
> at
> org.apache.cassandra.db.lifecycle.Helpers.prepareForObsoletion(Helpers.java:134)
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.doPrepare(LifecycleTransaction.java:193)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:376)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:84)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:94)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:194)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> listFiles is being called over and over, apparently scaling with the number
> of files in the compaction.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-12763) Compaction performance issues when a table has a lot of sstables
[
https://issues.apache.org/jira/browse/CASSANDRA-12763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania updated CASSANDRA-12763:
-
Reviewer: Stefania
> Compaction performance issues when a table has a lot of sstables
>
>
> Key: CASSANDRA-12763
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12763
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Tom van der Woerdt
>Assignee: Marcus Eriksson
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> An issue with a script flooded my cluster with sstables. There is now a table
> with 100k sstables, all on the order of KBytes, and it's taking a long time
> (ETA 20 days) to compact, even though the table is only ~30GB.
> Stack trace :
> {noformat}
> "CompactionExecutor:269" #7536 daemon prio=1 os_prio=4 tid=0x7f4acd40fc00
> nid=0x14f8 runnable [0x7f4798436000]
>java.lang.Thread.State: RUNNABLE
> at java.io.UnixFileSystem.list(Native Method)
> at java.io.File.list(File.java:1122)
> at java.io.File.listFiles(File.java:1248)
> at
> org.apache.cassandra.db.lifecycle.LogRecord.getExistingFiles(LogRecord.java:268)
> at org.apache.cassandra.db.lifecycle.LogRecord.make(LogRecord.java:150)
> at
> org.apache.cassandra.db.lifecycle.LogFile.makeRecord(LogFile.java:293)
> at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:283)
> at
> org.apache.cassandra.db.lifecycle.LogTransaction.obsoleted(LogTransaction.java:158)
> at
> org.apache.cassandra.db.lifecycle.Helpers.prepareForObsoletion(Helpers.java:134)
> at
> org.apache.cassandra.db.lifecycle.LifecycleTransaction.doPrepare(LifecycleTransaction.java:193)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:376)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:84)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:94)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:194)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> listFiles is being called over and over, apparently scaling with the number
> of files in the compaction.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-10112) Refuse to start and print txn log information in case of disk corruption
[
https://issues.apache.org/jira/browse/CASSANDRA-10112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296532#comment-16296532
]
Stefania commented on CASSANDRA-10112:
--
I cannot recall any specific reason, so I am guessing this was considered an
improvement.
> Refuse to start and print txn log information in case of disk corruption
>
>
> Key: CASSANDRA-10112
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10112
> Project: Cassandra
> Issue Type: Improvement
> Components: Local Write-Read Paths
> Reporter: Stefania
>Assignee: Stefania
> Labels: doc-impacting
> Fix For: 3.6
>
>
> Transaction logs were introduced by CASSANDRA-7066 and are read during
> start-up. In case of file system errors, such as disk corruption, we
> currently log a panic error and leave the sstable files and transaction logs
> as they are; this is to avoid rolling back a transaction (i.e. deleting
> files) by mistake.
> We should instead look at the {{disk_failure_policy}} and refuse to start
> unless the failure policy is {{ignore}}.
> We should also consider stashing files that cannot be read during startup,
> either transaction logs or sstables, by moving them to a dedicated
> sub-folder.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13897) nodetool compact and flush fail with "error: null"
[
https://issues.apache.org/jira/browse/CASSANDRA-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania updated CASSANDRA-13897:
-
Component/s: (was: Compaction)
Local Write-Read Paths
> nodetool compact and flush fail with "error: null"
> --
>
> Key: CASSANDRA-13897
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13897
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: * Apache Cassandra 3.11.0
> * Linux 4.4.0-92-generic #115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017 x86_64
> x86_64 x86_64 GNU/Linux
> * jdk1.8.0_144
> Reporter: Jacob Rhoden
>Assignee: Stefania
>Priority: Minor
> Fix For: 3.11.2
>
>
> {{nodetool flush}} and {{nodetool compact}} return an error message that is
> not clear. This could probably be improved. Both of my two nodes return this
> error.
> {{nodetool flush}} Will return this error the first 2-3 times you invoke it,
> then the error temporarily disappears. {{nodetool compress}} always returns
> this error message no matter how many times you invoke it.
> I have tried deleting saved_caches, commit logs, doing nodetool
> compact/rebuild/scrub, and nothing seems to remove the error.
> {noformat}
> cass@s5:~/apache-cassandra-3.11.0$ nodetool compact
> error: null
> -- StackTrace --
> java.lang.AssertionError
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.(ChunkCache.java:222)
> at org.apache.cassandra.cache.ChunkCache.wrap(ChunkCache.java:175)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.maybeCached(FileHandle.java:412)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:381)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:331)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:333)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinalEarly(BigTableWriter.java:318)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.switchWriter(SSTableRewriter.java:322)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:370)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:111)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:121)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:220)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$10.runMayThrow(CompactionManager.java:733)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13897) nodetool compact and flush fail with "error: null"
[
https://issues.apache.org/jira/browse/CASSANDRA-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214523#comment-16214523
]
Stefania commented on CASSANDRA-13897:
--
Thanks for the review!
Committed to 3.11 as {{5b23054f10f4d6553e8dacbf53bd59e552f2a031}} and merged
into trunk.
bq. make the new Config field a primitive instead of a boxed one.
It cannot be a primitive because we need {{== null}} to determine if the user
set this value or not.
> nodetool compact and flush fail with "error: null"
> --
>
> Key: CASSANDRA-13897
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13897
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: * Apache Cassandra 3.11.0
> * Linux 4.4.0-92-generic #115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017 x86_64
> x86_64 x86_64 GNU/Linux
> * jdk1.8.0_144
> Reporter: Jacob Rhoden
>Assignee: Stefania
>Priority: Minor
> Fix For: 3.11.2
>
>
> {{nodetool flush}} and {{nodetool compact}} return an error message that is
> not clear. This could probably be improved. Both of my two nodes return this
> error.
> {{nodetool flush}} Will return this error the first 2-3 times you invoke it,
> then the error temporarily disappears. {{nodetool compress}} always returns
> this error message no matter how many times you invoke it.
> I have tried deleting saved_caches, commit logs, doing nodetool
> compact/rebuild/scrub, and nothing seems to remove the error.
> {noformat}
> cass@s5:~/apache-cassandra-3.11.0$ nodetool compact
> error: null
> -- StackTrace --
> java.lang.AssertionError
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.(ChunkCache.java:222)
> at org.apache.cassandra.cache.ChunkCache.wrap(ChunkCache.java:175)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.maybeCached(FileHandle.java:412)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:381)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:331)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:333)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinalEarly(BigTableWriter.java:318)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.switchWriter(SSTableRewriter.java:322)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:370)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:111)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:121)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:220)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$10.runMayThrow(CompactionManager.java:733)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13897) nodetool compact and flush fail with "error: null"
[
https://issues.apache.org/jira/browse/CASSANDRA-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania updated CASSANDRA-13897:
-
Resolution: Fixed
Fix Version/s: 3.11.2
Reproduced In: 3.11.1, 3.11.0 (was: 3.11.0, 3.11.1)
Status: Resolved (was: Ready to Commit)
> nodetool compact and flush fail with "error: null"
> --
>
> Key: CASSANDRA-13897
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13897
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: * Apache Cassandra 3.11.0
> * Linux 4.4.0-92-generic #115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017 x86_64
> x86_64 x86_64 GNU/Linux
> * jdk1.8.0_144
> Reporter: Jacob Rhoden
>Assignee: Stefania
>Priority: Minor
> Fix For: 3.11.2
>
>
> {{nodetool flush}} and {{nodetool compact}} return an error message that is
> not clear. This could probably be improved. Both of my two nodes return this
> error.
> {{nodetool flush}} Will return this error the first 2-3 times you invoke it,
> then the error temporarily disappears. {{nodetool compress}} always returns
> this error message no matter how many times you invoke it.
> I have tried deleting saved_caches, commit logs, doing nodetool
> compact/rebuild/scrub, and nothing seems to remove the error.
> {noformat}
> cass@s5:~/apache-cassandra-3.11.0$ nodetool compact
> error: null
> -- StackTrace --
> java.lang.AssertionError
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.(ChunkCache.java:222)
> at org.apache.cassandra.cache.ChunkCache.wrap(ChunkCache.java:175)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.maybeCached(FileHandle.java:412)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:381)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:331)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:333)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinalEarly(BigTableWriter.java:318)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.switchWriter(SSTableRewriter.java:322)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:370)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:111)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:121)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:220)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$10.runMayThrow(CompactionManager.java:733)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[2/3] cassandra git commit: Round buffer size to powers of 2 for the chunk cache
Round buffer size to powers of 2 for the chunk cache
patch by Stefania Alborghetti; reviewed by Robert Stupp for CASSANDRA-13897
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5b23054f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5b23054f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5b23054f
Branch: refs/heads/trunk
Commit: 5b23054f10f4d6553e8dacbf53bd59e552f2a031
Parents: bbda201
Author: Stefania Alborghetti <stefania.alborghe...@datastax.com>
Authored: Wed Oct 18 15:36:42 2017 +0800
Committer: Stefania Alborghetti <stefania.alborghe...@datastax.com>
Committed: Mon Oct 23 09:55:04 2017 +0800
--
CHANGES.txt | 1 +
.../org/apache/cassandra/cache/ChunkCache.java | 3 +-
.../org/apache/cassandra/config/Config.java | 11
.../cassandra/config/DatabaseDescriptor.java| 16 ++
.../io/util/DiskOptimizationStrategy.java | 30 ++-
.../apache/cassandra/io/util/FileHandle.java| 3 +-
.../io/util/DiskOptimizationStrategyTest.java | 54
7 files changed, 114 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5b23054f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index a729d35..532c958 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
3.11.2
+ * Round buffer size to powers of 2 for the chunk cache (CASSANDRA-13897)
* Update jackson JSON jars (CASSANDRA-13949)
* Avoid locks when checking LCS fanout and if we should defrag
(CASSANDRA-13930)
Merged from 3.0:
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5b23054f/src/java/org/apache/cassandra/cache/ChunkCache.java
--
diff --git a/src/java/org/apache/cassandra/cache/ChunkCache.java
b/src/java/org/apache/cassandra/cache/ChunkCache.java
index 15cf2ce..4e7f848 100644
--- a/src/java/org/apache/cassandra/cache/ChunkCache.java
+++ b/src/java/org/apache/cassandra/cache/ChunkCache.java
@@ -41,6 +41,7 @@ public class ChunkCache
{
public static final int RESERVED_POOL_SPACE_IN_MB = 32;
public static final long cacheSize = 1024L * 1024L * Math.max(0,
DatabaseDescriptor.getFileCacheSizeInMB() - RESERVED_POOL_SPACE_IN_MB);
+public static final boolean roundUp =
DatabaseDescriptor.getFileCacheRoundUp();
private static boolean enabled = cacheSize > 0;
public static final ChunkCache instance = enabled ? new ChunkCache() :
null;
@@ -219,7 +220,7 @@ public class ChunkCache
{
source = file;
int chunkSize = file.chunkSize();
-assert Integer.bitCount(chunkSize) == 1;// Must be power of two
+assert Integer.bitCount(chunkSize) == 1 : String.format("%d must
be a power of two", chunkSize);
alignmentMask = -chunkSize;
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5b23054f/src/java/org/apache/cassandra/config/Config.java
--
diff --git a/src/java/org/apache/cassandra/config/Config.java
b/src/java/org/apache/cassandra/config/Config.java
index 8fde816..9489af9 100644
--- a/src/java/org/apache/cassandra/config/Config.java
+++ b/src/java/org/apache/cassandra/config/Config.java
@@ -265,6 +265,17 @@ public class Config
public Integer file_cache_size_in_mb;
+/**
+ * Because of the current {@link
org.apache.cassandra.utils.memory.BufferPool} slab sizes of 64 kb, we
+ * store in the file cache buffers that divide 64 kb, so we need to round
the buffer sizes to powers of two.
+ * This boolean controls weather they are rounded up or down. Set it to
true to round up to the
+ * next power of two, set it to false to round down to the previous power
of two. Note that buffer sizes are
+ * already rounded to 4 kb and capped between 4 kb minimum and 64 kb
maximum by the {@link DiskOptimizationStrategy}.
+ * By default, this boolean is set to round down when {@link
#disk_optimization_strategy} is {@code ssd},
+ * and to round up when it is {@code spinning}.
+ */
+public Boolean file_cache_round_up;
+
public boolean buffer_pool_use_heap_if_exhausted = true;
public DiskOptimizationStrategy disk_optimization_strategy =
DiskOptimizationStrategy.ssd;
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5b23054f/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--
diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index 4ce2728.
[1/3] cassandra git commit: Round buffer size to powers of 2 for the chunk cache
Repository: cassandra
Updated Branches:
refs/heads/cassandra-3.11 bbda20155 -> 5b23054f1
refs/heads/trunk 1c3a2d247 -> ec7d5f9da
Round buffer size to powers of 2 for the chunk cache
patch by Stefania Alborghetti; reviewed by Robert Stupp for CASSANDRA-13897
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/5b23054f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/5b23054f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/5b23054f
Branch: refs/heads/cassandra-3.11
Commit: 5b23054f10f4d6553e8dacbf53bd59e552f2a031
Parents: bbda201
Author: Stefania Alborghetti <stefania.alborghe...@datastax.com>
Authored: Wed Oct 18 15:36:42 2017 +0800
Committer: Stefania Alborghetti <stefania.alborghe...@datastax.com>
Committed: Mon Oct 23 09:55:04 2017 +0800
--
CHANGES.txt | 1 +
.../org/apache/cassandra/cache/ChunkCache.java | 3 +-
.../org/apache/cassandra/config/Config.java | 11
.../cassandra/config/DatabaseDescriptor.java| 16 ++
.../io/util/DiskOptimizationStrategy.java | 30 ++-
.../apache/cassandra/io/util/FileHandle.java| 3 +-
.../io/util/DiskOptimizationStrategyTest.java | 54
7 files changed, 114 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5b23054f/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index a729d35..532c958 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
3.11.2
+ * Round buffer size to powers of 2 for the chunk cache (CASSANDRA-13897)
* Update jackson JSON jars (CASSANDRA-13949)
* Avoid locks when checking LCS fanout and if we should defrag
(CASSANDRA-13930)
Merged from 3.0:
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5b23054f/src/java/org/apache/cassandra/cache/ChunkCache.java
--
diff --git a/src/java/org/apache/cassandra/cache/ChunkCache.java
b/src/java/org/apache/cassandra/cache/ChunkCache.java
index 15cf2ce..4e7f848 100644
--- a/src/java/org/apache/cassandra/cache/ChunkCache.java
+++ b/src/java/org/apache/cassandra/cache/ChunkCache.java
@@ -41,6 +41,7 @@ public class ChunkCache
{
public static final int RESERVED_POOL_SPACE_IN_MB = 32;
public static final long cacheSize = 1024L * 1024L * Math.max(0,
DatabaseDescriptor.getFileCacheSizeInMB() - RESERVED_POOL_SPACE_IN_MB);
+public static final boolean roundUp =
DatabaseDescriptor.getFileCacheRoundUp();
private static boolean enabled = cacheSize > 0;
public static final ChunkCache instance = enabled ? new ChunkCache() :
null;
@@ -219,7 +220,7 @@ public class ChunkCache
{
source = file;
int chunkSize = file.chunkSize();
-assert Integer.bitCount(chunkSize) == 1;// Must be power of two
+assert Integer.bitCount(chunkSize) == 1 : String.format("%d must
be a power of two", chunkSize);
alignmentMask = -chunkSize;
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5b23054f/src/java/org/apache/cassandra/config/Config.java
--
diff --git a/src/java/org/apache/cassandra/config/Config.java
b/src/java/org/apache/cassandra/config/Config.java
index 8fde816..9489af9 100644
--- a/src/java/org/apache/cassandra/config/Config.java
+++ b/src/java/org/apache/cassandra/config/Config.java
@@ -265,6 +265,17 @@ public class Config
public Integer file_cache_size_in_mb;
+/**
+ * Because of the current {@link
org.apache.cassandra.utils.memory.BufferPool} slab sizes of 64 kb, we
+ * store in the file cache buffers that divide 64 kb, so we need to round
the buffer sizes to powers of two.
+ * This boolean controls weather they are rounded up or down. Set it to
true to round up to the
+ * next power of two, set it to false to round down to the previous power
of two. Note that buffer sizes are
+ * already rounded to 4 kb and capped between 4 kb minimum and 64 kb
maximum by the {@link DiskOptimizationStrategy}.
+ * By default, this boolean is set to round down when {@link
#disk_optimization_strategy} is {@code ssd},
+ * and to round up when it is {@code spinning}.
+ */
+public Boolean file_cache_round_up;
+
public boolean buffer_pool_use_heap_if_exhausted = true;
public DiskOptimizationStrategy disk_optimization_strategy =
DiskOptimizationStrategy.ssd;
http://git-wip-us.apache.org/repos/asf/cassandra/blob/5b23054f/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--
d
[3/3] cassandra git commit: Merge branch 'cassandra-3.11' into trunk
Merge branch 'cassandra-3.11' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ec7d5f9d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ec7d5f9d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ec7d5f9d Branch: refs/heads/trunk Commit: ec7d5f9da63b60475e9ed76987d632bd7410ed11 Parents: 1c3a2d2 5b23054 Author: Stefania Alborghetti <stefania.alborghe...@datastax.com> Authored: Mon Oct 23 10:00:08 2017 +0800 Committer: Stefania Alborghetti <stefania.alborghe...@datastax.com> Committed: Mon Oct 23 10:00:08 2017 +0800 -- CHANGES.txt | 1 + .../org/apache/cassandra/cache/ChunkCache.java | 3 +- .../org/apache/cassandra/config/Config.java | 11 .../cassandra/config/DatabaseDescriptor.java| 16 ++ .../io/util/DiskOptimizationStrategy.java | 30 ++- .../apache/cassandra/io/util/FileHandle.java| 3 +- .../io/util/DiskOptimizationStrategyTest.java | 54 7 files changed, 114 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/ec7d5f9d/CHANGES.txt -- diff --cc CHANGES.txt index b2d6085,532c958..46c7614 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,158 -1,5 +1,159 @@@ +4.0 + * Add nodetool cmd to print hinted handoff window (CASSANDRA-13728) + * Fix some alerts raised by static analysis (CASSANDRA-13799) + * Checksum sstable metadata (CASSANDRA-13321, CASSANDRA-13593) + * Add result set metadata to prepared statement MD5 hash calculation (CASSANDRA-10786) + * Refactor GcCompactionTest to avoid boxing (CASSANDRA-13941) + * Expose recent histograms in JmxHistograms (CASSANDRA-13642) + * Fix buffer length comparison when decompressing in netty-based streaming (CASSANDRA-13899) + * Properly close StreamCompressionInputStream to release any ByteBuf (CASSANDRA-13906) + * Add SERIAL and LOCAL_SERIAL support for cassandra-stress (CASSANDRA-13925) + * LCS needlessly checks for L0 STCS candidates multiple times (CASSANDRA-12961) + * Correctly close netty channels when a stream session ends (CASSANDRA-13905) + * Update lz4 to 1.4.0 (CASSANDRA-13741) + * Optimize Paxos prepare and propose stage for local requests (CASSANDRA-13862) + * Throttle base partitions during MV repair streaming to prevent OOM (CASSANDRA-13299) + * Use compaction threshold for STCS in L0 (CASSANDRA-13861) + * Fix problem with min_compress_ratio: 1 and disallow ratio < 1 (CASSANDRA-13703) + * Add extra information to SASI timeout exception (CASSANDRA-13677) + * Add incremental repair support for --hosts, --force, and subrange repair (CASSANDRA-13818) + * Rework CompactionStrategyManager.getScanners synchronization (CASSANDRA-13786) + * Add additional unit tests for batch behavior, TTLs, Timestamps (CASSANDRA-13846) + * Add keyspace and table name in schema validation exception (CASSANDRA-13845) + * Emit metrics whenever we hit tombstone failures and warn thresholds (CASSANDRA-13771) + * Make netty EventLoopGroups daemon threads (CASSANDRA-13837) + * Race condition when closing stream sessions (CASSANDRA-13852) + * NettyFactoryTest is failing in trunk on macOS (CASSANDRA-13831) + * Allow changing log levels via nodetool for related classes (CASSANDRA-12696) + * Add stress profile yaml with LWT (CASSANDRA-7960) + * Reduce memory copies and object creations when acting on ByteBufs (CASSANDRA-13789) + * Simplify mx4j configuration (Cassandra-13578) + * Fix trigger example on 4.0 (CASSANDRA-13796) + * Force minumum timeout value (CASSANDRA-9375) + * Use netty for streaming (CASSANDRA-12229) + * Use netty for internode messaging (CASSANDRA-8457) + * Add bytes repaired/unrepaired to nodetool tablestats (CASSANDRA-13774) + * Don't delete incremental repair sessions if they still have sstables (CASSANDRA-13758) + * Fix pending repair manager index out of bounds check (CASSANDRA-13769) + * Don't use RangeFetchMapCalculator when RF=1 (CASSANDRA-13576) + * Don't optimise trivial ranges in RangeFetchMapCalculator (CASSANDRA-13664) + * Use an ExecutorService for repair commands instead of new Thread(..).start() (CASSANDRA-13594) + * Fix race / ref leak in anticompaction (CASSANDRA-13688) + * Expose tasks queue length via JMX (CASSANDRA-12758) + * Fix race / ref leak in PendingRepairManager (CASSANDRA-13751) + * Enable ppc64le runtime as unsupported architecture (CASSANDRA-13615) + * Improve sstablemetadata output (CASSANDRA-11483) + * Support for migrating legacy users to roles has been dropped (CASSANDRA-13371) + * Introduce error metrics for repair (CASSANDRA-13387) + * Refactoring to primitive functional interfaces in AuthCache (CASSANDRA-137
[jira] [Commented] (CASSANDRA-13897) nodetool compact and flush fail with "error: null"
[
https://issues.apache.org/jira/browse/CASSANDRA-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210463#comment-16210463
]
Stefania commented on CASSANDRA-13897:
--
Thanks for the explanation [~blambov], it looks like we should continue
rounding to powers of two for the chunk cache.
I've extended the [patch|https://github.com/stef1927/cassandra/tree/13897-3.11]
to either round up or down [~snazy]. This can be configured via a yaml property
that it only documented in config, not in the actual yaml file. This property
is called {{file_cache_round_up}} and by default it will be false when the disk
optimization strategy is ssd, therefore rounding down, and it will be true when
it is set to spinning, therefore rounding up only for spinning disks.
> nodetool compact and flush fail with "error: null"
> --
>
> Key: CASSANDRA-13897
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13897
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: * Apache Cassandra 3.11.0
> * Linux 4.4.0-92-generic #115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017 x86_64
> x86_64 x86_64 GNU/Linux
> * jdk1.8.0_144
> Reporter: Jacob Rhoden
>Assignee: Stefania
>Priority: Minor
>
> {{nodetool flush}} and {{nodetool compact}} return an error message that is
> not clear. This could probably be improved. Both of my two nodes return this
> error.
> {{nodetool flush}} Will return this error the first 2-3 times you invoke it,
> then the error temporarily disappears. {{nodetool compress}} always returns
> this error message no matter how many times you invoke it.
> I have tried deleting saved_caches, commit logs, doing nodetool
> compact/rebuild/scrub, and nothing seems to remove the error.
> {noformat}
> cass@s5:~/apache-cassandra-3.11.0$ nodetool compact
> error: null
> -- StackTrace --
> java.lang.AssertionError
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.(ChunkCache.java:222)
> at org.apache.cassandra.cache.ChunkCache.wrap(ChunkCache.java:175)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.maybeCached(FileHandle.java:412)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:381)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:331)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:333)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinalEarly(BigTableWriter.java:318)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.switchWriter(SSTableRewriter.java:322)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:370)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:111)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:121)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:220)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$10.runMayThrow(CompactionManager.java:733)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocat
[jira] [Commented] (CASSANDRA-13897) nodetool compact and flush fail with "error: null"
[
https://issues.apache.org/jira/browse/CASSANDRA-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208984#comment-16208984
]
Stefania commented on CASSANDRA-13897:
--
[~blambov], can the {{CachingRebufferer}} be changed to only require a multiple
of 4096 rather than a power of two? [~snazy] and myself are worried that
neither rounding up nor rounding down to a power of two feels too right as in
both cases we may end up being quite far from the initial value derived from
the data and the disk optimization strategy. As far as I have understood, the
rebufferer requires a power of two to calculate the page position, can it not
align to 4096 multiples instead?
> nodetool compact and flush fail with "error: null"
> --
>
> Key: CASSANDRA-13897
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13897
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: * Apache Cassandra 3.11.0
> * Linux 4.4.0-92-generic #115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017 x86_64
> x86_64 x86_64 GNU/Linux
> * jdk1.8.0_144
> Reporter: Jacob Rhoden
>Assignee: Stefania
>Priority: Minor
>
> {{nodetool flush}} and {{nodetool compact}} return an error message that is
> not clear. This could probably be improved. Both of my two nodes return this
> error.
> {{nodetool flush}} Will return this error the first 2-3 times you invoke it,
> then the error temporarily disappears. {{nodetool compress}} always returns
> this error message no matter how many times you invoke it.
> I have tried deleting saved_caches, commit logs, doing nodetool
> compact/rebuild/scrub, and nothing seems to remove the error.
> {noformat}
> cass@s5:~/apache-cassandra-3.11.0$ nodetool compact
> error: null
> -- StackTrace --
> java.lang.AssertionError
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.(ChunkCache.java:222)
> at org.apache.cassandra.cache.ChunkCache.wrap(ChunkCache.java:175)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.maybeCached(FileHandle.java:412)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:381)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:331)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:333)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinalEarly(BigTableWriter.java:318)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.switchWriter(SSTableRewriter.java:322)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:370)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:111)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:121)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:220)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$10.runMayThrow(CompactionManager.java:733)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> at java.lang.Thread.run(Thread.java:748)
[jira] [Commented] (CASSANDRA-13897) nodetool compact and flush fail with "error: null"
[
https://issues.apache.org/jira/browse/CASSANDRA-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208940#comment-16208940
]
Stefania commented on CASSANDRA-13897:
--
Patch for 3.11 is available
[here|https://github.com/stef1927/cassandra/tree/13897-3.11]. It rounds up to
the next power of two since we cap at 64 KB.
CircleCI was having github access problems, so I've launched the CI jobs on our
internal servers.
> nodetool compact and flush fail with "error: null"
> --
>
> Key: CASSANDRA-13897
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13897
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: * Apache Cassandra 3.11.0
> * Linux 4.4.0-92-generic #115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017 x86_64
> x86_64 x86_64 GNU/Linux
> * jdk1.8.0_144
> Reporter: Jacob Rhoden
>Assignee: Stefania
>Priority: Minor
>
> {{nodetool flush}} and {{nodetool compact}} return an error message that is
> not clear. This could probably be improved. Both of my two nodes return this
> error.
> {{nodetool flush}} Will return this error the first 2-3 times you invoke it,
> then the error temporarily disappears. {{nodetool compress}} always returns
> this error message no matter how many times you invoke it.
> I have tried deleting saved_caches, commit logs, doing nodetool
> compact/rebuild/scrub, and nothing seems to remove the error.
> {noformat}
> cass@s5:~/apache-cassandra-3.11.0$ nodetool compact
> error: null
> -- StackTrace --
> java.lang.AssertionError
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.(ChunkCache.java:222)
> at org.apache.cassandra.cache.ChunkCache.wrap(ChunkCache.java:175)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.maybeCached(FileHandle.java:412)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:381)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:331)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:333)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinalEarly(BigTableWriter.java:318)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.switchWriter(SSTableRewriter.java:322)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:370)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:111)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:121)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:220)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$10.runMayThrow(CompactionManager.java:733)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13897) nodetool compact and flush fail with "error: null"
[
https://issues.apache.org/jira/browse/CASSANDRA-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania updated CASSANDRA-13897:
-
Reproduced In: 3.11.1, 3.11.0 (was: 3.11.0, 3.11.1)
Status: Patch Available (was: Open)
> nodetool compact and flush fail with "error: null"
> --
>
> Key: CASSANDRA-13897
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13897
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: * Apache Cassandra 3.11.0
> * Linux 4.4.0-92-generic #115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017 x86_64
> x86_64 x86_64 GNU/Linux
> * jdk1.8.0_144
> Reporter: Jacob Rhoden
>Assignee: Stefania
>Priority: Minor
>
> {{nodetool flush}} and {{nodetool compact}} return an error message that is
> not clear. This could probably be improved. Both of my two nodes return this
> error.
> {{nodetool flush}} Will return this error the first 2-3 times you invoke it,
> then the error temporarily disappears. {{nodetool compress}} always returns
> this error message no matter how many times you invoke it.
> I have tried deleting saved_caches, commit logs, doing nodetool
> compact/rebuild/scrub, and nothing seems to remove the error.
> {noformat}
> cass@s5:~/apache-cassandra-3.11.0$ nodetool compact
> error: null
> -- StackTrace --
> java.lang.AssertionError
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.(ChunkCache.java:222)
> at org.apache.cassandra.cache.ChunkCache.wrap(ChunkCache.java:175)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.maybeCached(FileHandle.java:412)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:381)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:331)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:333)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinalEarly(BigTableWriter.java:318)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.switchWriter(SSTableRewriter.java:322)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:370)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:111)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:121)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:220)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$10.runMayThrow(CompactionManager.java:733)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13897) nodetool compact and flush fail with "error: null"
[
https://issues.apache.org/jira/browse/CASSANDRA-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208743#comment-16208743
]
Stefania commented on CASSANDRA-13897:
--
bq. so switching to a 64 bit version apparently enabled mmap.
That's correct. {{disk_access_mode: auto}} defaults to standard on 32-bit and
mmap on 64-bit, from {{DatabaseDescriptor}}:
{code}
if (conf.disk_access_mode == Config.DiskAccessMode.auto)
{
conf.disk_access_mode = hasLargeAddressSpace() ?
Config.DiskAccessMode.mmap : Config.DiskAccessMode.standard;
indexAccessMode = conf.disk_access_mode;
logger.info("DiskAccessMode 'auto' determined to be {},
indexAccessMode is {}", conf.disk_access_mode, indexAccessMode);
}
{code}
> nodetool compact and flush fail with "error: null"
> --
>
> Key: CASSANDRA-13897
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13897
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: * Apache Cassandra 3.11.0
> * Linux 4.4.0-92-generic #115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017 x86_64
> x86_64 x86_64 GNU/Linux
> * jdk1.8.0_144
> Reporter: Jacob Rhoden
>Assignee: Stefania
>Priority: Minor
>
> {{nodetool flush}} and {{nodetool compact}} return an error message that is
> not clear. This could probably be improved. Both of my two nodes return this
> error.
> {{nodetool flush}} Will return this error the first 2-3 times you invoke it,
> then the error temporarily disappears. {{nodetool compress}} always returns
> this error message no matter how many times you invoke it.
> I have tried deleting saved_caches, commit logs, doing nodetool
> compact/rebuild/scrub, and nothing seems to remove the error.
> {noformat}
> cass@s5:~/apache-cassandra-3.11.0$ nodetool compact
> error: null
> -- StackTrace --
> java.lang.AssertionError
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.(ChunkCache.java:222)
> at org.apache.cassandra.cache.ChunkCache.wrap(ChunkCache.java:175)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.maybeCached(FileHandle.java:412)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:381)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:331)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:333)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinalEarly(BigTableWriter.java:318)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.switchWriter(SSTableRewriter.java:322)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:370)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:111)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:121)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:220)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$10.runMayThrow(CompactionManager.java:733)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLoca
[jira] [Comment Edited] (CASSANDRA-13897) nodetool compact and flush fail with "error: null"
[
https://issues.apache.org/jira/browse/CASSANDRA-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208702#comment-16208702
]
Stefania edited comment on CASSANDRA-13897 at 10/18/17 2:48 AM:
The assertion that fails is {{assert Integer.bitCount(chunkSize) == 1;}}, the
caching rebufferer is requiring a power of two for its chunks. However chunk
sizes derive from buffer sizes, which are multiples of page cache sizes (4096)
and, for example, 4096 * 3 is not a power of two. We should prepare a patch.
Meanwhile, as a workaround, you can use mmap to read data, which will bypass
the chunk cache, by setting {{disk_access_mode: mmap}} in the yaml.
was (Author: stefania):
The assertion that fails is {{assert Integer.bitCount(chunkSize) == 1;}}, the
caching rebufferer is requiring a power of two for its chunks. However chunk
sizes derive from buffer sizes, which are multiples of page cache sizes (4096)
and 4096 * 3 is not a power of two. We should prepare a patch. Meanwhile, as a
workaround, you can use mmap to read data, which will bypass the chunk cache,
by setting {{disk_access_mode: mmap}} in the yaml.
> nodetool compact and flush fail with "error: null"
> --
>
> Key: CASSANDRA-13897
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13897
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: * Apache Cassandra 3.11.0
> * Linux 4.4.0-92-generic #115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017 x86_64
> x86_64 x86_64 GNU/Linux
> * jdk1.8.0_144
> Reporter: Jacob Rhoden
>Assignee: Stefania
>Priority: Minor
>
> {{nodetool flush}} and {{nodetool compact}} return an error message that is
> not clear. This could probably be improved. Both of my two nodes return this
> error.
> {{nodetool flush}} Will return this error the first 2-3 times you invoke it,
> then the error temporarily disappears. {{nodetool compress}} always returns
> this error message no matter how many times you invoke it.
> I have tried deleting saved_caches, commit logs, doing nodetool
> compact/rebuild/scrub, and nothing seems to remove the error.
> {noformat}
> cass@s5:~/apache-cassandra-3.11.0$ nodetool compact
> error: null
> -- StackTrace --
> java.lang.AssertionError
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.(ChunkCache.java:222)
> at org.apache.cassandra.cache.ChunkCache.wrap(ChunkCache.java:175)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.maybeCached(FileHandle.java:412)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:381)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:331)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:333)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinalEarly(BigTableWriter.java:318)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.switchWriter(SSTableRewriter.java:322)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:370)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:111)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:121)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:220)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$10.runMayThrow(CompactionManager.java:733)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Execut
[jira] [Commented] (CASSANDRA-13897) nodetool compact and flush fail with "error: null"
[
https://issues.apache.org/jira/browse/CASSANDRA-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208702#comment-16208702
]
Stefania commented on CASSANDRA-13897:
--
The assertion that fails is {{assert Integer.bitCount(chunkSize) == 1;}}, the
caching rebufferer is requiring a power of two for its chunks. However chunk
sizes derive from buffer sizes, which are multiples of page cache sizes (4096)
and 4096 * 3 is not a power of two. We should prepare a patch. Meanwhile, as a
workaround, you can use mmap to read data, which will bypass the chunk cache,
by setting {{disk_access_mode: mmap}} in the yaml.
> nodetool compact and flush fail with "error: null"
> --
>
> Key: CASSANDRA-13897
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13897
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: * Apache Cassandra 3.11.0
> * Linux 4.4.0-92-generic #115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017 x86_64
> x86_64 x86_64 GNU/Linux
> * jdk1.8.0_144
> Reporter: Jacob Rhoden
>Assignee: Stefania
>Priority: Minor
>
> {{nodetool flush}} and {{nodetool compact}} return an error message that is
> not clear. This could probably be improved. Both of my two nodes return this
> error.
> {{nodetool flush}} Will return this error the first 2-3 times you invoke it,
> then the error temporarily disappears. {{nodetool compress}} always returns
> this error message no matter how many times you invoke it.
> I have tried deleting saved_caches, commit logs, doing nodetool
> compact/rebuild/scrub, and nothing seems to remove the error.
> {noformat}
> cass@s5:~/apache-cassandra-3.11.0$ nodetool compact
> error: null
> -- StackTrace --
> java.lang.AssertionError
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.(ChunkCache.java:222)
> at org.apache.cassandra.cache.ChunkCache.wrap(ChunkCache.java:175)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.maybeCached(FileHandle.java:412)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:381)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:331)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:333)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinalEarly(BigTableWriter.java:318)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.switchWriter(SSTableRewriter.java:322)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:370)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:111)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:121)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:220)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$10.runMayThrow(CompactionManager.java:733)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-13897) nodetool compact and flush fail with "error: null"
[
https://issues.apache.org/jira/browse/CASSANDRA-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania reassigned CASSANDRA-13897:
Assignee: Stefania
> nodetool compact and flush fail with "error: null"
> --
>
> Key: CASSANDRA-13897
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13897
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: * Apache Cassandra 3.11.0
> * Linux 4.4.0-92-generic #115-Ubuntu SMP Thu Aug 10 09:04:33 UTC 2017 x86_64
> x86_64 x86_64 GNU/Linux
> * jdk1.8.0_144
> Reporter: Jacob Rhoden
>Assignee: Stefania
>Priority: Minor
>
> {{nodetool flush}} and {{nodetool compact}} return an error message that is
> not clear. This could probably be improved. Both of my two nodes return this
> error.
> {{nodetool flush}} Will return this error the first 2-3 times you invoke it,
> then the error temporarily disappears. {{nodetool compress}} always returns
> this error message no matter how many times you invoke it.
> I have tried deleting saved_caches, commit logs, doing nodetool
> compact/rebuild/scrub, and nothing seems to remove the error.
> {noformat}
> cass@s5:~/apache-cassandra-3.11.0$ nodetool compact
> error: null
> -- StackTrace --
> java.lang.AssertionError
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.(ChunkCache.java:222)
> at org.apache.cassandra.cache.ChunkCache.wrap(ChunkCache.java:175)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.maybeCached(FileHandle.java:412)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:381)
> at
> org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:331)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:333)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinalEarly(BigTableWriter.java:318)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.switchWriter(SSTableRewriter.java:322)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:370)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:111)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
> at
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:121)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:220)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$10.runMayThrow(CompactionManager.java:733)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13909) Improve TRUNCATE performance with many sstables
[
https://issues.apache.org/jira/browse/CASSANDRA-13909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185214#comment-16185214
]
Stefania commented on CASSANDRA-13909:
--
14 failures in the dtests, but they all failed 14+ times in older builds too,
mostly are due to {{Could not do a git clone}}.
I think we can merge this.
> Improve TRUNCATE performance with many sstables
> ---
>
> Key: CASSANDRA-13909
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13909
> Project: Cassandra
> Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> Truncate is very slow in 3.0, mostly due to {{LogRecord.make}} listing all
> files in the directory for every sstable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13909) Improve TRUNCATE performance with many sstables
[
https://issues.apache.org/jira/browse/CASSANDRA-13909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefania updated CASSANDRA-13909:
-
Status: Ready to Commit (was: Patch Available)
> Improve TRUNCATE performance with many sstables
> ---
>
> Key: CASSANDRA-13909
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13909
> Project: Cassandra
> Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> Truncate is very slow in 3.0, mostly due to {{LogRecord.make}} listing all
> files in the directory for every sstable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13909) Improve TRUNCATE performance with many sstables
[
https://issues.apache.org/jira/browse/CASSANDRA-13909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183746#comment-16183746
]
Stefania commented on CASSANDRA-13909:
--
LGTM - I only found 3 nits, see comments
[here|https://github.com/krummas/cassandra/commit/9ea54dc47f75d0296dde4f74fd6f4d3de392f991].
There seem to be timeouts in both the dtests (they timed out on
manual_join_test) and the unit tests. I doubt very much that they are related
to this patch but the main
[builds|https://builds.apache.org/view/A-D/view/Cassandra/] are not affected by
timeouts (although they did not run in almost a month). I've verified that the
unit test that timed out passes locally.
+1 to commit if you are sure that these timeouts are happening on other
branches too.
> Improve TRUNCATE performance with many sstables
> ---
>
> Key: CASSANDRA-13909
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13909
> Project: Cassandra
> Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> Truncate is very slow in 3.0, mostly due to {{LogRecord.make}} listing all
> files in the directory for every sstable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13789) Reduce memory copies and object creations when acting on ByteBufs
[ https://issues.apache.org/jira/browse/CASSANDRA-13789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183528#comment-16183528 ] Stefania commented on CASSANDRA-13789: -- Sure, thank you so much! > Reduce memory copies and object creations when acting on ByteBufs > -- > > Key: CASSANDRA-13789 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13789 > Project: Cassandra > Issue Type: Improvement >Reporter: Norman Maurer >Assignee: Norman Maurer > Fix For: 4.0 > > Attachments: > 0001-CBUtil.sizeOfLongString-encodes-String-to-byte-to-ca.patch, > 0001-Reduce-memory-copies-and-object-creations-when-actin.patch > > > There are multiple "low-hanging-fruits" when it comes to reduce memory copies > and object allocations when acting on ByteBufs. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
