[jira] [Commented] (CASSANDRA-11574) COPY FROM command in cqlsh throws error
[
https://issues.apache.org/jira/browse/CASSANDRA-11574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251419#comment-15251419
]
Nandakishore Arvapaly commented on CASSANDRA-11574:
---
I'm not sure the exact reason but still facing the same problem.
Traceback (most recent call last):
File "/usr/bin/cqlsh.py", line 1191, in onecmd
self.handle_statement(st, statementtext)
File "/usr/bin/cqlsh.py", line 1228, in handle_statement
return custom_handler(parsed)
File "/usr/bin/cqlsh.py", line 1937, in do_copy
task = ImportTask(self, ks, table, columns, fname, opts,
DEFAULT_PROTOCOL_VERSION, CONFIG_FILE)
File "cqlshlib/copyutil.py", line 1052, in
cqlshlib.copyutil.ImportTask.__init__ (cqlshlib/copyutil.c:27710)
def __init__(self, shell, ks, table, columns, fname, opts,
protocol_version, config_file):
File "cqlshlib/copyutil.py", line 219, in cqlshlib.copyutil.CopyTask.__init__
(cqlshlib/copyutil.c:9708)
self.options = self.parse_options(opts, direction)
File "cqlshlib/copyutil.py", line 320, in
cqlshlib.copyutil.CopyTask.parse_options (cqlshlib/copyutil.c:11850)
anum_processes = 1
TypeError: get_num_processes() takes no keyword arguments
I made the changes as below
[root@nanda03 cassandra_data]# grep "anum_processes"
/usr/lib/python2.7/site-packages/cqlshlib/copyutil.py
anum_processes = 1
copy_options['numprocesses'] = int(opts.pop('numprocesses',
anum_processes))
> COPY FROM command in cqlsh throws error
> ---
>
> Key: CASSANDRA-11574
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11574
> Project: Cassandra
> Issue Type: Bug
> Components: CQL
> Environment: Operating System: Ubuntu Server 14.04
> JDK: Oracle JDK 8 update 77
> Python: 2.7.6
>Reporter: Mahafuzur Rahman
>Assignee: Stefania
> Fix For: 3.0.6
>
>
> Any COPY FROM command in cqlsh is throwing the following error:
> "get_num_processes() takes no keyword arguments"
> Example command:
> COPY inboxdata
> (to_user_id,to_user_network,created_time,attachments,from_user_id,from_user_name,from_user_network,id,message,to_user_name,updated_time)
> FROM 'inbox.csv';
> Similar commands worked parfectly in the previous versions such as 3.0.4
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11621) Stack Overflow inserting value with many columns
[
https://issues.apache.org/jira/browse/CASSANDRA-11621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Petrov updated CASSANDRA-11621:
Description:
I am using CQL to insert into a table that has ~4000 columns
{code}
TABLE_DEFINITION = "
id uuid,
"dimension_n" for n in _.range(N_DIMENSIONS)
ETAG timeuuid,
PRIMARY KEY (id)
"
{code}
I am using the node.js library from Datastax to execute CQL. This creates a
prepared statement and then uses it to perform an insert. It works fine on C*
2.1 but after upgrading to C* 2.2.5 I get the stack overflow below.
I know enough Java to think that recursing an iterator is bad form and should
be easy to fix.
{code}
ERROR 14:59:01 Unexpected exception during request; channel = [id: 0xaac42a5d,
/10.0.7.182:58736 => /10.0.0.87:9042]
java.lang.StackOverflowError: null
at
com.google.common.base.Preconditions.checkPositionIndex(Preconditions.java:339)
~[guava-16.0.jar:na]
at
com.google.common.collect.AbstractIndexedListIterator.(AbstractIndexedListIterator.java:69)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterators$11.(Iterators.java:1048)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterators.forArray(Iterators.java:1048)
~[guava-16.0.jar:na]
at
com.google.common.collect.RegularImmutableList.listIterator(RegularImmutableList.java:106)
~[guava-16.0.jar:na]
at
com.google.common.collect.ImmutableList.listIterator(ImmutableList.java:344)
~[guava-16.0.jar:na]
at
com.google.common.collect.ImmutableList.iterator(ImmutableList.java:340)
~[guava-16.0.jar:na]
at
com.google.common.collect.ImmutableList.iterator(ImmutableList.java:61)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterables.iterators(Iterables.java:504)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterables.access$100(Iterables.java:60)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterables$2.iterator(Iterables.java:494)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterables$3.transform(Iterables.java:508)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterables$3.transform(Iterables.java:505)
~[guava-16.0.jar:na]
at
com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
~[guava-16.0.jar:na]
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
~[guava-16.0.jar:na]
{code}
was:
I am using CQL to insert into a table that has ~4000 columns
TABLE_DEFINITION = "
id uuid,
"dimension_n" for n in _.range(N_DIMENSIONS)
ETAG timeuuid,
PRIMARY KEY (id)
"
I am using the node.js library from Datastax to execute CQL. This creates a
prepared statement and then uses it to perform an insert. It works fine on C*
2.1 but after upgrading to C* 2.2.5 I get the stack overflow below.
I know enough Java to think that recursing an iterator is bad form and should
be easy to fix.
ERROR 14:59:01 Unexpected exception during request; channel
[jira] [Comment Edited] (CASSANDRA-11624) Scrub does not seem to work on previously marked corrupted SSTables
[
https://issues.apache.org/jira/browse/CASSANDRA-11624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251392#comment-15251392
]
Michael Fong edited comment on CASSANDRA-11624 at 4/21/16 6:57 AM:
---
Looking at the source code of Cassandra-2.0.17, it seems the cause might come
from the following logic:
>From org.apache.cassandra.db.compaction.CompactionManager
…
public void performScrub(ColumnFamilyStore cfStore, final boolean
skipCorrupted, final boolean checkData) throws InterruptedException,
ExecutionException
{
performAllSSTableOperation(cfStore, new AllSSTablesOperation()
{
…
private void performAllSSTableOperation(final ColumnFamilyStore cfs, final
AllSSTablesOperation operation) throws InterruptedException, ExecutionException
{
final Iterable sstables = cfs.markAllCompacting();
…
org.apache.cassandra.db. ColumnFamilyStore
…
public Iterable markAllCompacting()
{
Callable callable = new
Callable()
{
public Iterable call() throws Exception
{
assert data.getCompacting().isEmpty() : data.getCompacting();
Iterable sstables =
Lists.newArrayList(AbstractCompactionStrategy.filterSuspectSSTables(getSSTables()));
<-filter out all previously marked suspected SSTables
if (Iterables.isEmpty(sstables))
return null;
…
--
It seems scrub will not perform if SSTable was marked as corrupted (in
blacklist) previously; however, would this defeat the original purpose of scrub
operation?
was (Author: michael.fong):
Looking at the source code of Cassandra-2.0.17, it seems the cause might come
from the following logic:
>From org.apache.cassandra.db.compaction.CompactionManager
…
public void performScrub(ColumnFamilyStore cfStore, final boolean
skipCorrupted, final boolean checkData) throws InterruptedException,
ExecutionException
{
performAllSSTableOperation(cfStore, new AllSSTablesOperation()
{
…
private void performAllSSTableOperation(final ColumnFamilyStore cfs, final
AllSSTablesOperation operation) throws InterruptedException, ExecutionException
{
final Iterable sstables = cfs.markAllCompacting();
…
org.apache.cassandra.db. ColumnFamilyStore
…
public Iterable markAllCompacting()
{
Callable callable = new
Callable()
{
public Iterable call() throws Exception
{
assert data.getCompacting().isEmpty() : data.getCompacting();
Iterable sstables =
Lists.newArrayList(AbstractCompactionStrategy.filterSuspectSSTables(getSSTables()));
<-filter out all previously marked suspected SSTables
if (Iterables.isEmpty(sstables))
return null;
…
--
It seems scrub will not perform if SSTable was marked as corrupted previously;
however, would this defeat the original purpose of scrub operation?
> Scrub does not seem to work on previously marked corrupted SSTables
> ---
>
> Key: CASSANDRA-11624
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11624
> Project: Cassandra
> Issue Type: Bug
>Reporter: Michael Fong
>
> We ran into a scenario that scrub does not seem to work on a previously
> marked-as-corrupted SSTable.
> Here is the log snippet related to the corrupted SSTable and scrub-attempt :
> ERROR [ReadStage:174] 2016-03-17 04:14:39,658 CassandraDaemon.java (line 258)
> Exception in thread Thread[ReadStage:174,5,main]
> java.lang.RuntimeException:
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException:
> mmap segment underflow; remaining is 10197 but 30062 requested for
> /data/ng/db/data/wsg/dpStatusRealTime/wsg-dpStatusRealTime-ic-2-Data.db
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2022)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException:
> java.io.IOException: mmap segment underflow; remaining is 10197 but 30062
> requested for
> /data/ng/db/data/wsg/dpStatusRealTime/wsg-dpStatusRealTime-ic-2-Data.db
> at
> org.apache.cassandra.db.columniterator.IndexedSliceReader.(IndexedSliceReader.java:97)
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65)
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.(SSTableSliceIterator.java:42)
> at
>
[jira] [Comment Edited] (CASSANDRA-11624) Scrub does not seem to work on previously marked corrupted SSTables
[
https://issues.apache.org/jira/browse/CASSANDRA-11624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251392#comment-15251392
]
Michael Fong edited comment on CASSANDRA-11624 at 4/21/16 6:48 AM:
---
Looking at the source code of Cassandra-2.0.17, it seems the cause might come
from the following logic:
>From org.apache.cassandra.db.compaction.CompactionManager
…
public void performScrub(ColumnFamilyStore cfStore, final boolean
skipCorrupted, final boolean checkData) throws InterruptedException,
ExecutionException
{
performAllSSTableOperation(cfStore, new AllSSTablesOperation()
{
…
private void performAllSSTableOperation(final ColumnFamilyStore cfs, final
AllSSTablesOperation operation) throws InterruptedException, ExecutionException
{
final Iterable sstables = cfs.markAllCompacting();
…
org.apache.cassandra.db. ColumnFamilyStore
…
public Iterable markAllCompacting()
{
Callable callable = new
Callable()
{
public Iterable call() throws Exception
{
assert data.getCompacting().isEmpty() : data.getCompacting();
Iterable sstables =
Lists.newArrayList(AbstractCompactionStrategy.filterSuspectSSTables(getSSTables()));
<-filter out all previously marked suspected SSTables
if (Iterables.isEmpty(sstables))
return null;
…
--
It seems scrub will not perform if SSTable was marked as corrupted previously;
however, would this defeat the original purpose of scrub operation?
was (Author: michael.fong):
Looking at the source code of Cassandra-2.0.17, it seems the cause might come
from the following logic:
>From org.apache.cassandra.db.compaction.CompactionManager
…
public void performScrub(ColumnFamilyStore cfStore, final boolean
skipCorrupted, final boolean checkData) throws InterruptedException,
ExecutionException
{
performAllSSTableOperation(cfStore, new AllSSTablesOperation()
{
…
private void performAllSSTableOperation(final ColumnFamilyStore cfs, final
AllSSTablesOperation operation) throws InterruptedException, ExecutionException
{
final Iterable sstables = cfs.markAllCompacting();
…
org.apache.cassandra.db. ColumnFamilyStore
…
public Iterable markAllCompacting()
{
Callable callable = new
Callable()
{
public Iterable call() throws Exception
{
assert data.getCompacting().isEmpty() : data.getCompacting();
Iterable sstables =
Lists.newArrayList(AbstractCompactionStrategy.filterSuspectSSTables(getSSTables()));
<-filter out all previously marked suspected SSTables
if (Iterables.isEmpty(sstables))
return null;
…
--
It seems scrub will not perform if SSTable was marked as corrupted previously;
however, would this defeat the original perform of scrub operation?
> Scrub does not seem to work on previously marked corrupted SSTables
> ---
>
> Key: CASSANDRA-11624
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11624
> Project: Cassandra
> Issue Type: Bug
>Reporter: Michael Fong
>
> We ran into a scenario that scrub does not seem to work on a previously
> marked-as-corrupted SSTable.
> Here is the log snippet related to the corrupted SSTable and scrub-attempt :
> ERROR [ReadStage:174] 2016-03-17 04:14:39,658 CassandraDaemon.java (line 258)
> Exception in thread Thread[ReadStage:174,5,main]
> java.lang.RuntimeException:
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException:
> mmap segment underflow; remaining is 10197 but 30062 requested for
> /data/ng/db/data/wsg/dpStatusRealTime/wsg-dpStatusRealTime-ic-2-Data.db
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2022)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException:
> java.io.IOException: mmap segment underflow; remaining is 10197 but 30062
> requested for
> /data/ng/db/data/wsg/dpStatusRealTime/wsg-dpStatusRealTime-ic-2-Data.db
> at
> org.apache.cassandra.db.columniterator.IndexedSliceReader.(IndexedSliceReader.java:97)
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65)
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.(SSTableSliceIterator.java:42)
> at
>
[jira] [Comment Edited] (CASSANDRA-11624) Scrub does not seem to work on previously marked corrupted SSTables
[
https://issues.apache.org/jira/browse/CASSANDRA-11624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251392#comment-15251392
]
Michael Fong edited comment on CASSANDRA-11624 at 4/21/16 6:48 AM:
---
Looking at the source code of Cassandra-2.0.17, it seems the cause might come
from the following logic:
>From org.apache.cassandra.db.compaction.CompactionManager
…
public void performScrub(ColumnFamilyStore cfStore, final boolean
skipCorrupted, final boolean checkData) throws InterruptedException,
ExecutionException
{
performAllSSTableOperation(cfStore, new AllSSTablesOperation()
{
…
private void performAllSSTableOperation(final ColumnFamilyStore cfs, final
AllSSTablesOperation operation) throws InterruptedException, ExecutionException
{
final Iterable sstables = cfs.markAllCompacting();
…
org.apache.cassandra.db. ColumnFamilyStore
…
public Iterable markAllCompacting()
{
Callable callable = new
Callable()
{
public Iterable call() throws Exception
{
assert data.getCompacting().isEmpty() : data.getCompacting();
Iterable sstables =
Lists.newArrayList(AbstractCompactionStrategy.filterSuspectSSTables(getSSTables()));
<-filter out all previously marked suspected SSTables
if (Iterables.isEmpty(sstables))
return null;
…
--
It seems scrub will not perform if SSTable was marked as corrupted previously;
however, would this defeat the original perform of scrub operation?
was (Author: michael.fong):
Looking at the source code of Cassandra-2.0.17, it seems the cause might come
from the following logic:
>From org.apache.cassandra.db.compaction.CompactionManager
…
public void performScrub(ColumnFamilyStore cfStore, final boolean
skipCorrupted, final boolean checkData) throws InterruptedException,
ExecutionException
{
performAllSSTableOperation(cfStore, new AllSSTablesOperation()
{
…
private void performAllSSTableOperation(final ColumnFamilyStore cfs, final
AllSSTablesOperation operation) throws InterruptedException, ExecutionException
{
final Iterable sstables = cfs.markAllCompacting();
…
org.apache.cassandra.db. ColumnFamilyStore
…
public Iterable markAllCompacting()
{
Callable callable = new
Callable()
{
public Iterable call() throws Exception
{
assert data.getCompacting().isEmpty() : data.getCompacting();
Iterable sstables =
Lists.newArrayList(AbstractCompactionStrategy.filterSuspectSSTables(getSSTables()));
<-filter out all previously marked suspected SSTables
if (Iterables.isEmpty(sstables))
return null;
…
> Scrub does not seem to work on previously marked corrupted SSTables
> ---
>
> Key: CASSANDRA-11624
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11624
> Project: Cassandra
> Issue Type: Bug
>Reporter: Michael Fong
>
> We ran into a scenario that scrub does not seem to work on a previously
> marked-as-corrupted SSTable.
> Here is the log snippet related to the corrupted SSTable and scrub-attempt :
> ERROR [ReadStage:174] 2016-03-17 04:14:39,658 CassandraDaemon.java (line 258)
> Exception in thread Thread[ReadStage:174,5,main]
> java.lang.RuntimeException:
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException:
> mmap segment underflow; remaining is 10197 but 30062 requested for
> /data/ng/db/data/wsg/dpStatusRealTime/wsg-dpStatusRealTime-ic-2-Data.db
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2022)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException:
> java.io.IOException: mmap segment underflow; remaining is 10197 but 30062
> requested for
> /data/ng/db/data/wsg/dpStatusRealTime/wsg-dpStatusRealTime-ic-2-Data.db
> at
> org.apache.cassandra.db.columniterator.IndexedSliceReader.(IndexedSliceReader.java:97)
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65)
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.(SSTableSliceIterator.java:42)
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:238)
> at
>
[jira] [Commented] (CASSANDRA-11624) Scrub does not seem to work on previously marked corrupted SSTables
[
https://issues.apache.org/jira/browse/CASSANDRA-11624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251392#comment-15251392
]
Michael Fong commented on CASSANDRA-11624:
--
Looking at the source code of Cassandra-2.0.17, it seems the cause might come
from the following logic:
>From org.apache.cassandra.db.compaction.CompactionManager
…
public void performScrub(ColumnFamilyStore cfStore, final boolean
skipCorrupted, final boolean checkData) throws InterruptedException,
ExecutionException
{
performAllSSTableOperation(cfStore, new AllSSTablesOperation()
{
…
private void performAllSSTableOperation(final ColumnFamilyStore cfs, final
AllSSTablesOperation operation) throws InterruptedException, ExecutionException
{
final Iterable sstables = cfs.markAllCompacting();
…
org.apache.cassandra.db. ColumnFamilyStore
…
public Iterable markAllCompacting()
{
Callable callable = new
Callable()
{
public Iterable call() throws Exception
{
assert data.getCompacting().isEmpty() : data.getCompacting();
Iterable sstables =
Lists.newArrayList(AbstractCompactionStrategy.filterSuspectSSTables(getSSTables()));
<-filter out all previously marked suspected SSTables
if (Iterables.isEmpty(sstables))
return null;
…
> Scrub does not seem to work on previously marked corrupted SSTables
> ---
>
> Key: CASSANDRA-11624
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11624
> Project: Cassandra
> Issue Type: Bug
>Reporter: Michael Fong
>
> We ran into a scenario that scrub does not seem to work on a previously
> marked-as-corrupted SSTable.
> Here is the log snippet related to the corrupted SSTable and scrub-attempt :
> ERROR [ReadStage:174] 2016-03-17 04:14:39,658 CassandraDaemon.java (line 258)
> Exception in thread Thread[ReadStage:174,5,main]
> java.lang.RuntimeException:
> org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException:
> mmap segment underflow; remaining is 10197 but 30062 requested for
> /data/ng/db/data/wsg/dpStatusRealTime/wsg-dpStatusRealTime-ic-2-Data.db
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2022)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException:
> java.io.IOException: mmap segment underflow; remaining is 10197 but 30062
> requested for
> /data/ng/db/data/wsg/dpStatusRealTime/wsg-dpStatusRealTime-ic-2-Data.db
> at
> org.apache.cassandra.db.columniterator.IndexedSliceReader.(IndexedSliceReader.java:97)
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65)
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.(SSTableSliceIterator.java:42)
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:238)
> at
> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:250)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1642)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1461)
> at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:340)
> at
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:89)
> at
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1445)
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2010)
> ... 3 more
> INFO [CompactionExecutor:98] 2016-03-17 04:14:39,693 OutputHandler.java
> (line 42) Scrubbing
> SSTableReader(path='/data/ng/db/data/wsg/dpStatusRealTime/wsg-dpStatusRealTime-jb-11-Data.db')
> (2230223 bytes)
> INFO [CompactionExecutor:98] 2016-03-17 04:14:39,751 OutputHandler.java
> (line 42) Scrub of
> SSTableReader(path='/data/ng/db/data/wsg/dpStatusRealTime/wsg-dpStatusRealTime-jb-11-Data.db')
> complete: 2 rows in new sstable and 0 empty (tombstoned) rows dropped
> --
> Below is the file information around that time
> --
> -bash-4.1$ ls -alF
[jira] [Created] (CASSANDRA-11624) Scrub does not seem to work on previously marked corrupted SSTables
Michael Fong created CASSANDRA-11624: Summary: Scrub does not seem to work on previously marked corrupted SSTables Key: CASSANDRA-11624 URL: https://issues.apache.org/jira/browse/CASSANDRA-11624 Project: Cassandra Issue Type: Bug Reporter: Michael Fong We ran into a scenario that scrub does not seem to work on a previously marked-as-corrupted SSTable. Here is the log snippet related to the corrupted SSTable and scrub-attempt : ERROR [ReadStage:174] 2016-03-17 04:14:39,658 CassandraDaemon.java (line 258) Exception in thread Thread[ReadStage:174,5,main] java.lang.RuntimeException: org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: mmap segment underflow; remaining is 10197 but 30062 requested for /data/ng/db/data/wsg/dpStatusRealTime/wsg-dpStatusRealTime-ic-2-Data.db at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2022) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: mmap segment underflow; remaining is 10197 but 30062 requested for /data/ng/db/data/wsg/dpStatusRealTime/wsg-dpStatusRealTime-ic-2-Data.db at org.apache.cassandra.db.columniterator.IndexedSliceReader.(IndexedSliceReader.java:97) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.(SSTableSliceIterator.java:42) at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:238) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62) at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:250) at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1642) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1461) at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:340) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:89) at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1445) at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2010) ... 3 more INFO [CompactionExecutor:98] 2016-03-17 04:14:39,693 OutputHandler.java (line 42) Scrubbing SSTableReader(path='/data/ng/db/data/wsg/dpStatusRealTime/wsg-dpStatusRealTime-jb-11-Data.db') (2230223 bytes) INFO [CompactionExecutor:98] 2016-03-17 04:14:39,751 OutputHandler.java (line 42) Scrub of SSTableReader(path='/data/ng/db/data/wsg/dpStatusRealTime/wsg-dpStatusRealTime-jb-11-Data.db') complete: 2 rows in new sstable and 0 empty (tombstoned) rows dropped -- Below is the file information around that time -- -bash-4.1$ ls -alF /data/ng/db/data/wsg/dpStatusRealTime/ total 2328 drwxr-xr-x 2 root root4096 Mar 17 04:14 ./ drwxr-xr-x 264 root root 12288 Mar 16 06:48 ../ -rw-r--r-- 1 root root 72995 Mar 16 07:08 wsg-dpStatusRealTime-ic-2-Data.db -rw-r--r-- 1 root root 75 Mar 16 07:08 wsg-dpStatusRealTime-ic-2-Digest.sha1 -rw-r--r-- 1 root root 16 Mar 16 07:08 wsg-dpStatusRealTime-ic-2-Filter.db -rw-r--r-- 1 root root 132 Mar 16 07:08 wsg-dpStatusRealTime-ic-2-Index.db -rw-r--r-- 1 root root5956 Mar 16 07:08 wsg-dpStatusRealTime-ic-2-Statistics.db -rw-r--r-- 1 root root 244 Mar 16 07:20 wsg-dpStatusRealTime-ic-2-Summary.db -rw-r--r-- 1 root root 72 Mar 16 07:08 wsg-dpStatusRealTime-ic-2-TOC.txt -rw-r--r-- 1 root root 144 Mar 17 04:14 wsg-dpStatusRealTime-jb-12-CRC.db -rw-r--r-- 1 root root 2230223 Mar 17 04:14 wsg-dpStatusRealTime-jb-12-Data.db -rw-r--r-- 1 root root 76 Mar 17 04:14 wsg-dpStatusRealTime-jb-12-Digest.sha1 -rw-r--r-- 1 root root 336 Mar 17 04:14 wsg-dpStatusRealTime-jb-12-Filter.db -rw-r--r-- 1 root root1424 Mar 17 04:14 wsg-dpStatusRealTime-jb-12-Index.db -rw-r--r-- 1 root root6004 Mar 17 04:14 wsg-dpStatusRealTime-jb-12-Statistics.db -rw-r--r-- 1 root root 244 Mar 17 04:14 wsg-dpStatusRealTime-jb-12-Summary.db -rw-r--r-- 1 root root 79 Mar 17 04:14 wsg-dpStatusRealTime-jb-12-TOC.txt -- 1. Please note that the corrupted file is in (ic) version, which is 1.2.19. This test bed was upgraded and attempted to upgradesstable a day ago. There has been some
[jira] [Commented] (CASSANDRA-11381) Node running with join_ring=false and authentication can not serve requests
[
https://issues.apache.org/jira/browse/CASSANDRA-11381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251385#comment-15251385
]
Joel Knighton commented on CASSANDRA-11381:
---
Thanks for the patches [~michaelsembwever] and apologies for the delay again.
This is at the top of my queue now, so I can work on getting this in quickly.
I can confirm that this is a genuine issue - a few thoughts on the patches:
* Is there a reason that we only setup auth in {{StorageService.initServer}} if
the saved tokens aren't empty? This doesn't handle the case when a node starts
with join_ring false on its first boot (a coordinator-only node). This can be
reproduced with a small variant of your dtest in which we do not start node3 in
the initial preparation. Some experimentation on my end suggests this would be
fine.
* On a similar note, is there a reason we can't just move auth setup slightly
earlier in {{StorageService.initServer()}}? If we moved it before the check of
the join_ring property, we could do without the idempotency parts of the
patches. Again, this looks workable in my experimentation.
* Was there any method by which you arrived at the 300 second timeouts for the
cql connection on the dtests? This really drags the test on in failing cases,
and I was able to reduce it to 30 seconds without any false negatives.
* Tests handling the coordinator-only case and using JMX to join the node to
the ring would be great. I think a minimal parameterized test could handle
these all these permutations.
> Node running with join_ring=false and authentication can not serve requests
> ---
>
> Key: CASSANDRA-11381
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11381
> Project: Cassandra
> Issue Type: Bug
>Reporter: mck
>Assignee: mck
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
> Attachments: 11381-2.0.txt, 11381-2.1.txt, 11381-2.2.txt,
> 11381-3.0.txt, 11381-trunk.txt, dtest-11381-trunk.txt
>
>
> Starting up a node with {{-Dcassandra.join_ring=false}} in a cluster that has
> authentication configured, eg PasswordAuthenticator, won't be able to serve
> requests. This is because {{Auth.setup()}} never gets called during the
> startup.
> Without {{Auth.setup()}} having been called in {{StorageService}} clients
> connecting to the node fail with the node throwing
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.cassandra.auth.PasswordAuthenticator.authenticate(PasswordAuthenticator.java:119)
> at
> org.apache.cassandra.thrift.CassandraServer.login(CassandraServer.java:1471)
> at
> org.apache.cassandra.thrift.Cassandra$Processor$login.getResult(Cassandra.java:3505)
> at
> org.apache.cassandra.thrift.Cassandra$Processor$login.getResult(Cassandra.java:3489)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at com.thinkaurelius.thrift.Message.invoke(Message.java:314)
> at
> com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90)
> at
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695)
> at
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689)
> at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The exception thrown from the
> [code|https://github.com/apache/cassandra/blob/cassandra-2.0.16/src/java/org/apache/cassandra/auth/PasswordAuthenticator.java#L119]
> {code}
> ResultMessage.Rows rows =
> authenticateStatement.execute(QueryState.forInternalCalls(), new
> QueryOptions(consistencyForUser(username),
>
>Lists.newArrayList(ByteBufferUtil.bytes(username;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11381) Node running with join_ring=false and authentication can not serve requests
[
https://issues.apache.org/jira/browse/CASSANDRA-11381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joel Knighton updated CASSANDRA-11381:
--
Fix Version/s: (was: 2.2.7)
(was: 3.0.6)
(was: 3.6)
(was: 2.1.14)
3.x
3.0.x
2.2.x
2.1.x
> Node running with join_ring=false and authentication can not serve requests
> ---
>
> Key: CASSANDRA-11381
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11381
> Project: Cassandra
> Issue Type: Bug
>Reporter: mck
>Assignee: mck
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
> Attachments: 11381-2.0.txt, 11381-2.1.txt, 11381-2.2.txt,
> 11381-3.0.txt, 11381-trunk.txt, dtest-11381-trunk.txt
>
>
> Starting up a node with {{-Dcassandra.join_ring=false}} in a cluster that has
> authentication configured, eg PasswordAuthenticator, won't be able to serve
> requests. This is because {{Auth.setup()}} never gets called during the
> startup.
> Without {{Auth.setup()}} having been called in {{StorageService}} clients
> connecting to the node fail with the node throwing
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.cassandra.auth.PasswordAuthenticator.authenticate(PasswordAuthenticator.java:119)
> at
> org.apache.cassandra.thrift.CassandraServer.login(CassandraServer.java:1471)
> at
> org.apache.cassandra.thrift.Cassandra$Processor$login.getResult(Cassandra.java:3505)
> at
> org.apache.cassandra.thrift.Cassandra$Processor$login.getResult(Cassandra.java:3489)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at com.thinkaurelius.thrift.Message.invoke(Message.java:314)
> at
> com.thinkaurelius.thrift.Message$Invocation.execute(Message.java:90)
> at
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:695)
> at
> com.thinkaurelius.thrift.TDisruptorServer$InvocationHandler.onEvent(TDisruptorServer.java:689)
> at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:112)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The exception thrown from the
> [code|https://github.com/apache/cassandra/blob/cassandra-2.0.16/src/java/org/apache/cassandra/auth/PasswordAuthenticator.java#L119]
> {code}
> ResultMessage.Rows rows =
> authenticateStatement.execute(QueryState.forInternalCalls(), new
> QueryOptions(consistencyForUser(username),
>
>Lists.newArrayList(ByteBufferUtil.bytes(username;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11574) COPY FROM command in cqlsh throws error
[
https://issues.apache.org/jira/browse/CASSANDRA-11574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251361#comment-15251361
]
Stefania commented on CASSANDRA-11574:
--
bq. spark already took the available 1 core in that machine which Cassandra is
getting zero for that value. This is the main problem I guess. please let me
know if this is issue.
No I don't think so. {{get_num_processes()}} will never return zero and it uses
{{get_num_cores()}}, which relies on {{mp.cpu_count()}}, doc
[here|https://docs.python.org/2/library/multiprocessing.html]. This returns the
number of cores available on the system, I don't think it would know that Spark
has taken one. Besides, even if the system only has one core, it should still
work with 1 process. We have an environment variable that we set to simulate
1-core machines in our tests and Datastax tested COPY code on single core VMs
as well. Also, it is the cores of the machine that runs cqlsh that matter, not
the machine that runs the Cassandra server (just in case if this wasn't clear
before).
Are you still getting the exact same error with the two lines above? What about
if you don't call {{get_num_processes}} at all and fix {{num_processes}} to 1,
does that work?
Full code here:
{code}
@staticmethod
def get_num_processes(cap):
"""
Pick a reasonable number of child processes. We need to leave at
least one core for the parent or feeder process.
"""
return max(1, min(cap, CopyTask.get_num_cores() - 1))
@staticmethod
def get_num_cores():
"""
Return the number of cores if available. If the test environment
variable
is set, then return the number carried by this variable. This is to
test single-core
machine more easily.
"""
try:
num_cores_for_testing = os.environ.get('CQLSH_COPY_TEST_NUM_CORES',
'')
ret = int(num_cores_for_testing) if num_cores_for_testing else
mp.cpu_count()
printdebugmsg("Detected %d core(s)" % (ret,))
return ret
except NotImplementedError:
printdebugmsg("Failed to detect number of cores, returning 1")
return 1
{code}
> COPY FROM command in cqlsh throws error
> ---
>
> Key: CASSANDRA-11574
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11574
> Project: Cassandra
> Issue Type: Bug
> Components: CQL
> Environment: Operating System: Ubuntu Server 14.04
> JDK: Oracle JDK 8 update 77
> Python: 2.7.6
>Reporter: Mahafuzur Rahman
>Assignee: Stefania
> Fix For: 3.0.6
>
>
> Any COPY FROM command in cqlsh is throwing the following error:
> "get_num_processes() takes no keyword arguments"
> Example command:
> COPY inboxdata
> (to_user_id,to_user_network,created_time,attachments,from_user_id,from_user_name,from_user_network,id,message,to_user_name,updated_time)
> FROM 'inbox.csv';
> Similar commands worked parfectly in the previous versions such as 3.0.4
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11574) COPY FROM command in cqlsh throws error
[ https://issues.apache.org/jira/browse/CASSANDRA-11574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251334#comment-15251334 ] Nandakishore Arvapaly commented on CASSANDRA-11574: --- Thanks for explaining about get_num_processes. I'm running Spark and Cassandra on 2 nodes where spark already took the available 1 core in that machine which Cassandra is getting zero for that value. This is the main problem I guess. please let me know if this is issue. I have 2 nodes where I only 1 core each. When I start spark with these 2 as slaves and one of them as master, its taking the 1 core for each node making it unavailable for get_num_processes. So Can't I run the COPY command with 1 core machine? > COPY FROM command in cqlsh throws error > --- > > Key: CASSANDRA-11574 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11574 > Project: Cassandra > Issue Type: Bug > Components: CQL > Environment: Operating System: Ubuntu Server 14.04 > JDK: Oracle JDK 8 update 77 > Python: 2.7.6 >Reporter: Mahafuzur Rahman >Assignee: Stefania > Fix For: 3.0.6 > > > Any COPY FROM command in cqlsh is throwing the following error: > "get_num_processes() takes no keyword arguments" > Example command: > COPY inboxdata > (to_user_id,to_user_network,created_time,attachments,from_user_id,from_user_name,from_user_network,id,message,to_user_name,updated_time) > FROM 'inbox.csv'; > Similar commands worked parfectly in the previous versions such as 3.0.4 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
