[jira] [Commented] (CASSANDRA-12008) Make decommission operations resumable
[
https://issues.apache.org/jira/browse/CASSANDRA-12008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412708#comment-15412708
]
Kaide Mu commented on CASSANDRA-12008:
--
I've opened a pr: https://github.com/riptano/cassandra-dtest/pull/1192
> Make decommission operations resumable
> --
>
> Key: CASSANDRA-12008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12008
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
>Reporter: Tom van der Woerdt
>Assignee: Kaide Mu
>Priority: Minor
>
> We're dealing with large data sets (multiple terabytes per node) and
> sometimes we need to add or remove nodes. These operations are very dependent
> on the entire cluster being up, so while we're joining a new node (which
> sometimes takes 6 hours or longer) a lot can go wrong and in a lot of cases
> something does.
> It would be great if the ability to retry streams was implemented.
> Example to illustrate the problem :
> {code}
> 03:18 PM ~ $ nodetool decommission
> error: Stream failed
> -- StackTrace --
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186)
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:430)
> at
> org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:622)
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:486)
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
> at java.lang.Thread.run(Thread.java:745)
> 08:04 PM ~ $ nodetool decommission
> nodetool: Unsupported operation: Node in LEAVING state; wait for status to
> become normal or restart
> See 'nodetool help' or 'nodetool help '.
> {code}
> Streaming failed, probably due to load :
> {code}
> ERROR [STREAM-IN-/] 2016-06-14 18:05:47,275 StreamSession.java:520 -
> [Stream #] Streaming error occurred
> java.net.SocketTimeoutException: null
> at
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211)
> ~[na:1.8.0_77]
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> ~[na:1.8.0_77]
> at
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> ~[na:1.8.0_77]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54)
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:268)
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
> {code}
> If implementing retries is not possible, can we have a 'nodetool decommission
> resume'?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12008) Make decommission operations resumable
[
https://issues.apache.org/jira/browse/CASSANDRA-12008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15409938#comment-15409938
]
Kaide Mu commented on CASSANDRA-12008:
--
Thanks for the review!
Implementation patch:
https://github.com/apache/cassandra/compare/trunk...kdmu:trunk-12008?expand=1
Dtests patch:
https://github.com/riptano/cassandra-dtest/compare/cassandra-12008...kdmu:cassandra-12008?expand=1
Brief summary:
* Move tests inside of topology_test
* Use insert_c1c2 and query_c1c2 instead of stress w/r
* simple_decommission_test -> concurrent_decommission_not_allowed_test
* Byteman injection instead of log look-up
* Fixed nits
If everything looks good, I'll open a PR.
bq. It seems the CI dtests failed I think due to some recent changes in dtest
that probably require a update/rebase in your branch.
You mean that I have to rebase my remote branch?
> Make decommission operations resumable
> --
>
> Key: CASSANDRA-12008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12008
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
>Reporter: Tom van der Woerdt
>Assignee: Kaide Mu
>Priority: Minor
>
> We're dealing with large data sets (multiple terabytes per node) and
> sometimes we need to add or remove nodes. These operations are very dependent
> on the entire cluster being up, so while we're joining a new node (which
> sometimes takes 6 hours or longer) a lot can go wrong and in a lot of cases
> something does.
> It would be great if the ability to retry streams was implemented.
> Example to illustrate the problem :
> {code}
> 03:18 PM ~ $ nodetool decommission
> error: Stream failed
> -- StackTrace --
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186)
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:430)
> at
> org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:622)
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:486)
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
> at java.lang.Thread.run(Thread.java:745)
> 08:04 PM ~ $ nodetool decommission
> nodetool: Unsupported operation: Node in LEAVING state; wait for status to
> become normal or restart
> See 'nodetool help' or 'nodetool help '.
> {code}
> Streaming failed, probably due to load :
> {code}
> ERROR [STREAM-IN-/] 2016-06-14 18:05:47,275 StreamSession.java:520 -
> [Stream #] Streaming error occurred
> java.net.SocketTimeoutException: null
> at
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211)
> ~[na:1.8.0_77]
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> ~[na:1.8.0_77]
> at
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> ~[na:1.8.0_77]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54)
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:268)
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
> {code}
> If implementing retries is not possible, can we have a 'nodetool decommission
> resume'?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12008) Make decommission operations resumable
[
https://issues.apache.org/jira/browse/CASSANDRA-12008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406203#comment-15406203
]
Kaide Mu commented on CASSANDRA-12008:
--
New dtests patch:
https://github.com/riptano/cassandra-dtest/compare/master...kdmu:cassandra-12008?expand=1
New implementation patch:
https://github.com/apache/cassandra/compare/trunk...kdmu:trunk-12008?expand=1
bq. Error while decommissioning node is never printed because the
ExecutionException is being wrapped in a RuntimeException on unbootstrap, so
perhaps you can modify unbootstrap to throw ExecutionException |
InterruptedException and catch that on decomission to wrap in RuntimeException.
Now ExecutionException ｜ InterruptedException is handled directly by
unbootstrap.
bq. When verifying if the retrieved data is correct on
resumable_decommission_test, you need to stop either node1 or node3 when
querying the other otherwise the data may be in only one of these nodes (while
it must be in both nodes, since RF=2 and N=2).
Instead of stopping and starting nodes, I changed stress read with a CL=TWO, in
this way I guess we can ensure that node1 and node3 is "replying" to the
request. Also if we do stop and restart node, it seems the restarted node will
raise an error due to it is looking on node2 log that restarting node is alive
which such operation is not possible since node2 is decommissioned.
bq. Instead of counting for decommission_error you can add a self.fail("second
rebuild should fail") after node2.nodetool('decommission') and on the except
part perhaps check that the following message is being print on logs Error
while decommissioning node
I guess I'll use insetead assertRaises which seems more suitable to ensure
NodetoolError is raised. WDYT [~pauloricardomg] [~yukim]?
> Make decommission operations resumable
> --
>
> Key: CASSANDRA-12008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12008
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
>Reporter: Tom van der Woerdt
>Assignee: Kaide Mu
>Priority: Minor
>
> We're dealing with large data sets (multiple terabytes per node) and
> sometimes we need to add or remove nodes. These operations are very dependent
> on the entire cluster being up, so while we're joining a new node (which
> sometimes takes 6 hours or longer) a lot can go wrong and in a lot of cases
> something does.
> It would be great if the ability to retry streams was implemented.
> Example to illustrate the problem :
> {code}
> 03:18 PM ~ $ nodetool decommission
> error: Stream failed
> -- StackTrace --
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186)
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:430)
> at
> org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:622)
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:486)
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
> at java.lang.Thread.run(Thread.java:745)
> 08:04 PM ~ $ nodetool decommission
> nodetool: Unsupported operation: Node in LEAVING state; wait for status to
> become normal or restart
> See 'nodetool help' or 'nodetool help '.
> {code}
> Streaming failed, probably due to load :
> {code}
> ERROR [STREAM-IN-/] 2016-06-14 18:05:47,275 StreamSession.java:520 -
> [Stream #] Streaming error occurred
> java.net.SocketTimeoutException: null
> at
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211)
> ~[na:1.8.0_77]
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> ~[na:1.8.0_77]
> at
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> ~[na:1.8.0_77]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54)
>
[jira] [Commented] (CASSANDRA-12008) Make decommission operations resumable
[
https://issues.apache.org/jira/browse/CASSANDRA-12008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401923#comment-15401923
]
Kaide Mu commented on CASSANDRA-12008:
--
Decommission dtests is available:
https://github.com/kdmu/cassandra-dtest/commit/fd5c95f1b582155f969d4d6e52fc4a4bf46046b1
Both tests are working properly, but I found that streamed range skipping
behaviour log check-up is not working due log capacity (?), i.e. part of log is
overwritten. I tried to test it by running another thread checking skipping
behaviour alongside the main thread and it can find correctly the keyword, but
it doesn't work if I do a watch_log_for as
[L61|https://github.com/kdmu/cassandra-dtest/commit/fd5c95f1b582155f969d4d6e52fc4a4bf46046b1#diff-e726e64e1cbe236026689bfb926669cfR61]
> Make decommission operations resumable
> --
>
> Key: CASSANDRA-12008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12008
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
>Reporter: Tom van der Woerdt
>Assignee: Kaide Mu
>Priority: Minor
>
> We're dealing with large data sets (multiple terabytes per node) and
> sometimes we need to add or remove nodes. These operations are very dependent
> on the entire cluster being up, so while we're joining a new node (which
> sometimes takes 6 hours or longer) a lot can go wrong and in a lot of cases
> something does.
> It would be great if the ability to retry streams was implemented.
> Example to illustrate the problem :
> {code}
> 03:18 PM ~ $ nodetool decommission
> error: Stream failed
> -- StackTrace --
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186)
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:430)
> at
> org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:622)
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:486)
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
> at java.lang.Thread.run(Thread.java:745)
> 08:04 PM ~ $ nodetool decommission
> nodetool: Unsupported operation: Node in LEAVING state; wait for status to
> become normal or restart
> See 'nodetool help' or 'nodetool help '.
> {code}
> Streaming failed, probably due to load :
> {code}
> ERROR [STREAM-IN-/] 2016-06-14 18:05:47,275 StreamSession.java:520 -
> [Stream #] Streaming error occurred
> java.net.SocketTimeoutException: null
> at
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211)
> ~[na:1.8.0_77]
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> ~[na:1.8.0_77]
> at
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> ~[na:1.8.0_77]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54)
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:268)
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
> {code}
> If implementing retries is not possible, can we have a 'nodetool decommission
> resume'?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12008) Make decommission operations resumable
[
https://issues.apache.org/jira/browse/CASSANDRA-12008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401433#comment-15401433
]
Kaide Mu commented on CASSANDRA-12008:
--
Thanks for the review! A new patch is available:
https://github.com/apache/cassandra/compare/trunk...kdmu:trunk-12008?expand=1
Now everything is inside StreamCompleteEvent and keyspace is reflected in the
log.
> Make decommission operations resumable
> --
>
> Key: CASSANDRA-12008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12008
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
>Reporter: Tom van der Woerdt
>Assignee: Kaide Mu
>Priority: Minor
>
> We're dealing with large data sets (multiple terabytes per node) and
> sometimes we need to add or remove nodes. These operations are very dependent
> on the entire cluster being up, so while we're joining a new node (which
> sometimes takes 6 hours or longer) a lot can go wrong and in a lot of cases
> something does.
> It would be great if the ability to retry streams was implemented.
> Example to illustrate the problem :
> {code}
> 03:18 PM ~ $ nodetool decommission
> error: Stream failed
> -- StackTrace --
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186)
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:430)
> at
> org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:622)
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:486)
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
> at java.lang.Thread.run(Thread.java:745)
> 08:04 PM ~ $ nodetool decommission
> nodetool: Unsupported operation: Node in LEAVING state; wait for status to
> become normal or restart
> See 'nodetool help' or 'nodetool help '.
> {code}
> Streaming failed, probably due to load :
> {code}
> ERROR [STREAM-IN-/] 2016-06-14 18:05:47,275 StreamSession.java:520 -
> [Stream #] Streaming error occurred
> java.net.SocketTimeoutException: null
> at
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211)
> ~[na:1.8.0_77]
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> ~[na:1.8.0_77]
> at
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> ~[na:1.8.0_77]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54)
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:268)
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
> {code}
> If implementing retries is not possible, can we have a 'nodetool decommission
> resume'?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12008) Make decommission operations resumable
[
https://issues.apache.org/jira/browse/CASSANDRA-12008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399170#comment-15399170
]
Kaide Mu commented on CASSANDRA-12008:
--
bq. It seems getStreamedRanges is querying the AVAILABLE_RANGES table instead
of STREAMED_RANGES, that's why is generating the Undefined column name
operation error.
Unbelievable, but yes it was the error, already fixed it, thanks!
bq. Maybe it's not working because of the previous error? Perhaps it would help
to add a unit test on StreamStateStoreTest to verify that updateStreamedRanges
and getStreamedRanges is being populated correctly and working as expected. You
can also add debug logs to troubleshoot.
Another stupid error, wasn't adding {{StreamTransferTask}} to
{{SessionCompleteEvent}}, fixed.
bq. SystemKeyspace.getStreamedRanges is being called from inside a for-loop
what may be inefficient, it's maybe better to retrieve it before and re-use it
inside the loop.
I've added a new strategy, please let me know what do you think about it.
Some additional modifications, we are not going to pass description to
{{StreamTransferTask}} constructor, if we do so it will raise an error because
when task is created {{StreamResultFuture}} is not initialized yet, thus
{{StreamSession.description()}} will return a null value at creation time. So
instead we will obtain {{StreamSession}} from
{{StreamTransferTask.getSession()}} when each {{StreamTransferTask}} is
complete i.e when {{StreamStateStore.handleStreamEvent}} is invoked. All these
means that we are going to only pass its responsible keyspace.
Some minor details：
Don't know if there's some problem with current implementation or there's
something weird in the set-up, but it skips twice the same range:
{quote}
DEBUG [RMI TCP Connection(9)-127.0.0.1] 2016-07-29 12:48:36,301
StorageService.java:4556 - Range (3074457345618258602,-9223372036854775808]
already in /127.0.0.3, skipping
DEBUG [RMI TCP Connection(9)-127.0.0.1] 2016-07-29 12:48:36,301
StorageService.java:4556 - Range (3074457345618258602,-9223372036854775808]
already in /127.0.0.3, skipping
{quote}
I think it's the set-up itself since
{{StorageService.getChangedRangesForLeaving}} is also returning the same range
twice
{quote}
DEBUG [RMI TCP Connection(9)-127.0.0.1] 2016-07-29 12:48:36,289
StorageService.java:2526 - Range (3074457345618258602,-9223372036854775808]
will be responsibility of /127.0.0.3
DEBUG [RMI TCP Connection(9)-127.0.0.1] 2016-07-29 12:48:36,294
StorageService.java:2526 - Range (3074457345618258602,-9223372036854775808]
will be responsibility of /127.0.0.3
{quote}
You can find latest working patch via:
https://github.com/apache/cassandra/compare/trunk...kdmu:trunk-12008?expand=1
> Make decommission operations resumable
> --
>
> Key: CASSANDRA-12008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12008
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
>Reporter: Tom van der Woerdt
>Assignee: Kaide Mu
>Priority: Minor
>
> We're dealing with large data sets (multiple terabytes per node) and
> sometimes we need to add or remove nodes. These operations are very dependent
> on the entire cluster being up, so while we're joining a new node (which
> sometimes takes 6 hours or longer) a lot can go wrong and in a lot of cases
> something does.
> It would be great if the ability to retry streams was implemented.
> Example to illustrate the problem :
> {code}
> 03:18 PM ~ $ nodetool decommission
> error: Stream failed
> -- StackTrace --
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186)
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:430)
> at
> org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:622)
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:486)
> at
>
[jira] [Commented] (CASSANDRA-12008) Make decommission operations resumable
[
https://issues.apache.org/jira/browse/CASSANDRA-12008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398120#comment-15398120
]
Kaide Mu commented on CASSANDRA-12008:
--
I've been testing with current implementation. The set-up is 3 nodes,
decommissioning node2 and interrupting node1 at beginning, i.e decommission
failed, but all ranges streamed to node3 should be updated. It seems there's
two problems:
* If we are running decommission for first time, getStreamedRanges is still
invoked raising a nodetool exception "Undefined column name operation", for
solving that we probably should modify isDecommisioning flag behaviour, add a
new flag indicating we are resuming or running for first time or add a new
resumeDecommission method.
* STREAMED_RANGES table is not updated, maybe is due to {{handleStreamEvent}}
only updates the table if {{event.eventType ==
StreamEvent.Type.STREAM_COMPLETE}} and it seems that our StreamEvent status is
FAILED, we may need to change this to also update the table when event failed.
WDYT [~pauloricardomg] [~yukim]]
> Make decommission operations resumable
> --
>
> Key: CASSANDRA-12008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12008
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
>Reporter: Tom van der Woerdt
>Assignee: Kaide Mu
>Priority: Minor
>
> We're dealing with large data sets (multiple terabytes per node) and
> sometimes we need to add or remove nodes. These operations are very dependent
> on the entire cluster being up, so while we're joining a new node (which
> sometimes takes 6 hours or longer) a lot can go wrong and in a lot of cases
> something does.
> It would be great if the ability to retry streams was implemented.
> Example to illustrate the problem :
> {code}
> 03:18 PM ~ $ nodetool decommission
> error: Stream failed
> -- StackTrace --
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186)
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:430)
> at
> org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:622)
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:486)
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
> at java.lang.Thread.run(Thread.java:745)
> 08:04 PM ~ $ nodetool decommission
> nodetool: Unsupported operation: Node in LEAVING state; wait for status to
> become normal or restart
> See 'nodetool help' or 'nodetool help '.
> {code}
> Streaming failed, probably due to load :
> {code}
> ERROR [STREAM-IN-/] 2016-06-14 18:05:47,275 StreamSession.java:520 -
> [Stream #] Streaming error occurred
> java.net.SocketTimeoutException: null
> at
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211)
> ~[na:1.8.0_77]
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> ~[na:1.8.0_77]
> at
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> ~[na:1.8.0_77]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54)
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:268)
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
> {code}
> If implementing retries is not possible, can we have a 'nodetool decommission
> resume'?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12008) Make decommission operations resumable
[ https://issues.apache.org/jira/browse/CASSANDRA-12008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398061#comment-15398061 ] Kaide Mu commented on CASSANDRA-12008: -- New WIP patch uploaded: https://github.com/apache/cassandra/compare/trunk...kdmu:trunk-12008?expand=1 bq. In order to not change the signature of StreamSession.addTransferFiles we can add a Collectionto SSTableStreamingSections so it will be automatically added to StreamTransferTask. Not sure if I did it correctly, but as I [did|https://github.com/apache/cassandra/compare/trunk...kdmu:trunk-12008?expand=1#diff-b397cc5438b1a4be20f026c9ec9ecd6eR367], all {{StreamTransferTask}} from {{getSSTableSectionsForRanges}} will have same ranges, so maybe it's redundant to add its ranges each time we invoke {{StreamTransferTask.addTransferFile}} bq. You will also need to modify bootstrap to use the streamed_ranges table instead of available_ranges, and since that table will become useless you can probably remove it. If you don't mind I'll leave it for later, I'll create another JIRA ticket for re-factoring existing resumable bootstrap code once this one is done. Thanks! > Make decommission operations resumable > -- > > Key: CASSANDRA-12008 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12008 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging >Reporter: Tom van der Woerdt >Assignee: Kaide Mu >Priority: Minor > > We're dealing with large data sets (multiple terabytes per node) and > sometimes we need to add or remove nodes. These operations are very dependent > on the entire cluster being up, so while we're joining a new node (which > sometimes takes 6 hours or longer) a lot can go wrong and in a lot of cases > something does. > It would be great if the ability to retry streams was implemented. > Example to illustrate the problem : > {code} > 03:18 PM ~ $ nodetool decommission > error: Stream failed > -- StackTrace -- > org.apache.cassandra.streaming.StreamException: Stream failed > at > org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) > at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) > at > com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) > at > com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) > at > com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) > at > com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) > at > org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210) > at > org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186) > at > org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:430) > at > org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:622) > at > org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:486) > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274) > at java.lang.Thread.run(Thread.java:745) > 08:04 PM ~ $ nodetool decommission > nodetool: Unsupported operation: Node in LEAVING state; wait for status to > become normal or restart > See 'nodetool help' or 'nodetool help '. > {code} > Streaming failed, probably due to load : > {code} > ERROR [STREAM-IN-/] 2016-06-14 18:05:47,275 StreamSession.java:520 - > [Stream #] Streaming error occurred > java.net.SocketTimeoutException: null > at > sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211) > ~[na:1.8.0_77] > at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) > ~[na:1.8.0_77] > at > java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) > ~[na:1.8.0_77] > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54) > ~[apache-cassandra-3.0.6.jar:3.0.6] > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:268) > ~[apache-cassandra-3.0.6.jar:3.0.6] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77] > {code} > If implementing retries is not possible, can we have a 'nodetool decommission > resume'? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10810) Make rebuild operations resumable
[ https://issues.apache.org/jira/browse/CASSANDRA-10810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396376#comment-15396376 ] Kaide Mu commented on CASSANDRA-10810: -- Pull request created. https://github.com/riptano/cassandra-dtest/pull/1145/files Thanks. > Make rebuild operations resumable > - > > Key: CASSANDRA-10810 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10810 > Project: Cassandra > Issue Type: Wish > Components: Streaming and Messaging >Reporter: Jeremy Hanna >Assignee: Kaide Mu > Fix For: 3.x > > > Related to CASSANDRA-8942, now that we can resume bootstrap operations, this > could also be possible with rebuild operations, such as when you bootstrap > new nodes in a completely new datacenter in two steps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-12008) Make decommission operations resumable
[ https://issues.apache.org/jira/browse/CASSANDRA-12008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392776#comment-15392776 ] Kaide Mu edited comment on CASSANDRA-12008 at 7/27/16 11:09 AM: Here's 3 operations I'm proposing for STREAMED_RANGES: - updateStreamedRanges(String description, InetAddress endpoint, String keyspace, Setrange) - Set getAvailableRanges(String keyspace, InetAdress newReplica, IPartitioner partitioner) - void resetAvailableRanges() Briefly the flow is: # StreamPlan.transferRanges(InetAddress to, InetAddress connecting, String keyspace, Collection ranges) # StreamSession.addTransferRanges(String keyspace, Collection ranges, Collection columnFamilies, boolean flushTables, long repairedAt) # StreamSession.addTransferFiles(Collection sstableDetails) which will create a new StreamTransferTask and add it to the Set, this set will be used for updating STREAMED_RANGES in StreamStateStore. As you can see {{updateStreamedRanges}} will need description, endpoint, keyspace and range, and when we reached 4, from each StreamTransferTask we can only obtain description and endpoint from {{StreamTransferTask.session}}, therefore some information is "lost" and {{updateStreamedRanges}} cannot be accomplished. Thus additionally I propose modify or add a new {{StreamSession.addTransferFiles}} to receive ranges and keyspace since it's where {{StreamTransferTask}} is created and {{StreamTransferTask}}.recordＴransferInformation(Collection range, Keyspace keyspace) WDYT? [~pauloricardomg] [~yukim] was (Author: kdmu): Here's 3 operations I'm proposing for STREAMED_RANGES: - updateStreamedRanges(String description, InetAddress endpoint, String keyspace, Set range) - Set getAvailableRanges(String keyspace, InetAdress newReplica, IPartitioner partitioner) - void resetAvailableRanges() Briefly the flow is: # StreamPlan.transferRanges(InetAddress to, InetAddress connecting, String keyspace, Collection ranges) # StreamSession.addTransferRanges(String keyspace, Collection ranges, Collection columnFamilies, boolean flushTables, long repairedAt) # StreamSession.addTransferFiles(Collection sstableDetails) which will create a new StreamTransferTask and add it to the Set, this set will be used for updating STREAMED_RANGES in StreamStateStore. As you can see {{updateStreamedRanges}} will need description, endpoint, keyspace and range, and when we reached 4, from each StreamTransferTask we can only obtain description and endpoint from {{StreamTransferTask.session}}, therefore some information is "lost" and {{updateStreamedRanges}} cannot be accomplished. Thus additionally I propose {{StreamTransferTask}}.recordＴransferRange(Range range, Keyspace keyspace) for receiving and storing these informations. WDYT? [~pauloricardomg] [~yukim] > Make decommission operations resumable > -- > > Key: CASSANDRA-12008 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12008 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging >Reporter: Tom van der Woerdt >Assignee: Kaide Mu >Priority: Minor > > We're dealing with large data sets (multiple terabytes per node) and > sometimes we need to add or remove nodes. These operations are very dependent > on the entire cluster being up, so while we're joining a new node (which > sometimes takes 6 hours or longer) a lot can go wrong and in a lot of cases > something does. > It would be great if the ability to retry streams was implemented. > Example to illustrate the problem : > {code} > 03:18 PM ~ $ nodetool decommission > error: Stream failed > -- StackTrace -- > org.apache.cassandra.streaming.StreamException: Stream failed > at > org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) > at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) > at > com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) > at > com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) > at > com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) > at > com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) > at > org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210) > at > org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186) > at > org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:430) > at >
[jira] [Comment Edited] (CASSANDRA-12008) Make decommission operations resumable
[ https://issues.apache.org/jira/browse/CASSANDRA-12008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392776#comment-15392776 ] Kaide Mu edited comment on CASSANDRA-12008 at 7/27/16 11:01 AM: Here's 3 operations I'm proposing for STREAMED_RANGES: - updateStreamedRanges(String description, InetAddress endpoint, String keyspace, Setrange) - Set getAvailableRanges(String keyspace, InetAdress newReplica, IPartitioner partitioner) - void resetAvailableRanges() Briefly the flow is: # StreamPlan.transferRanges(InetAddress to, InetAddress connecting, String keyspace, Collection ranges) # StreamSession.addTransferRanges(String keyspace, Collection ranges, Collection columnFamilies, boolean flushTables, long repairedAt) # StreamSession.addTransferFiles(Collection sstableDetails) which will create a new StreamTransferTask and add it to the Set, this set will be used for updating STREAMED_RANGES in StreamStateStore. As you can see {{updateStreamedRanges}} will need description, endpoint, keyspace and range, and when we reached 4, from each StreamTransferTask we can only obtain description and endpoint from {{StreamTransferTask.session}}, therefore some information is "lost" and {{updateStreamedRanges}} cannot be accomplished. Thus additionally I propose {{StreamTransferTask}}.recordＴransferRange(Range range, Keyspace keyspace) for receiving and storing these informations. WDYT? [~pauloricardomg] [~yukim] was (Author: kdmu): Here's 3 operations I'm proposing for STREAMED_RANGES: - updateStreamedRanges(String description, InetAddress endpoint, String keyspace, Set range) - Set getAvailableRanges(String keyspace, InetAdress newReplica, IPartitioner partitioner) - void resetAvailableRanges() Briefly the flow is: # StreamPlan.transferRanges(InetAddress to, InetAddress connecting, String keyspace, Collection ranges) # StreamSession.addTransferRanges(String keyspace, Collection ranges, Collection columnFamilies, boolean flushTables, long repairedAt) # StreamSession.addTransferFiles(Collection sstableDetails) # And finally StreamTransferTask.complete which will make use of {{updateStreamedRanges}} As you can see {{updateStreamedRanges}} will need description, endpoint, keyspace and range, and when we reached 4, we can only obtain description and endpoint from {{StreamTransferTask.session}}, therefore some information is "lost" and {{updateStreamedRanges}} cannot be accomplished. Thus additionally I propose {{StreamTransferTask}} to receive keyspace and range, for that we may need to create a new {{addTransferFiles}} to keep a record of keyspace and transfer ranges. WDYT? [~pauloricardomg] [~yukim] > Make decommission operations resumable > -- > > Key: CASSANDRA-12008 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12008 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging >Reporter: Tom van der Woerdt >Assignee: Kaide Mu >Priority: Minor > > We're dealing with large data sets (multiple terabytes per node) and > sometimes we need to add or remove nodes. These operations are very dependent > on the entire cluster being up, so while we're joining a new node (which > sometimes takes 6 hours or longer) a lot can go wrong and in a lot of cases > something does. > It would be great if the ability to retry streams was implemented. > Example to illustrate the problem : > {code} > 03:18 PM ~ $ nodetool decommission > error: Stream failed > -- StackTrace -- > org.apache.cassandra.streaming.StreamException: Stream failed > at > org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) > at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) > at > com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) > at > com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) > at > com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) > at > com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) > at > org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210) > at > org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186) > at > org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:430) > at > org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:622) > at > org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:486) > at >
[jira] [Commented] (CASSANDRA-12008) Make decommission operations resumable
[
https://issues.apache.org/jira/browse/CASSANDRA-12008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395374#comment-15395374
]
Kaide Mu commented on CASSANDRA-12008:
--
Here's a WIP patch of what I've proposed in last comment
https://github.com/apache/cassandra/compare/trunk...kdmu:trunk-12008?expand=1
> Make decommission operations resumable
> --
>
> Key: CASSANDRA-12008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12008
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
>Reporter: Tom van der Woerdt
>Assignee: Kaide Mu
>Priority: Minor
>
> We're dealing with large data sets (multiple terabytes per node) and
> sometimes we need to add or remove nodes. These operations are very dependent
> on the entire cluster being up, so while we're joining a new node (which
> sometimes takes 6 hours or longer) a lot can go wrong and in a lot of cases
> something does.
> It would be great if the ability to retry streams was implemented.
> Example to illustrate the problem :
> {code}
> 03:18 PM ~ $ nodetool decommission
> error: Stream failed
> -- StackTrace --
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186)
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:430)
> at
> org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:622)
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:486)
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
> at java.lang.Thread.run(Thread.java:745)
> 08:04 PM ~ $ nodetool decommission
> nodetool: Unsupported operation: Node in LEAVING state; wait for status to
> become normal or restart
> See 'nodetool help' or 'nodetool help '.
> {code}
> Streaming failed, probably due to load :
> {code}
> ERROR [STREAM-IN-/] 2016-06-14 18:05:47,275 StreamSession.java:520 -
> [Stream #] Streaming error occurred
> java.net.SocketTimeoutException: null
> at
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211)
> ~[na:1.8.0_77]
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> ~[na:1.8.0_77]
> at
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> ~[na:1.8.0_77]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54)
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:268)
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
> {code}
> If implementing retries is not possible, can we have a 'nodetool decommission
> resume'?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-12008) Make decommission operations resumable
[ https://issues.apache.org/jira/browse/CASSANDRA-12008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392776#comment-15392776 ] Kaide Mu edited comment on CASSANDRA-12008 at 7/27/16 9:54 AM: --- Here's 3 operations I'm proposing for STREAMED_RANGES: - updateStreamedRanges(String description, InetAddress endpoint, String keyspace, Setrange) - Set getAvailableRanges(String keyspace, InetAdress newReplica, IPartitioner partitioner) - void resetAvailableRanges() Briefly the flow is: # StreamPlan.transferRanges(InetAddress to, InetAddress connecting, String keyspace, Collection ranges) # StreamSession.addTransferRanges(String keyspace, Collection ranges, Collection columnFamilies, boolean flushTables, long repairedAt) # StreamSession.addTransferFiles(Collection sstableDetails) # And finally StreamTransferTask.complete which will make use of {{updateStreamedRanges}} As you can see {{updateStreamedRanges}} will need description, endpoint, keyspace and range, and when we reached 4, we can only obtain description and endpoint from {{StreamTransferTask.session}}, therefore some information is "lost" and {{updateStreamedRanges}} cannot be accomplished. Thus additionally I propose {{StreamTransferTask}} to receive keyspace and range, for that we may need to create a new {{addTransferFiles}} to keep a record of keyspace and transfer ranges. WDYT? [~pauloricardomg] [~yukim] was (Author: kdmu): Here's 3 operations I'm proposing for STREAMED_RANGES: - updateStreamedRanges(String description, InetAddress endpoint, String keyspace, Set range) - MultiMap getAvailableRanges(String keyspace, IPartitioner partitioner) - void resetAvailableRanges() Briefly the flow is: # StreamPlan.transferRanges(InetAddress to, InetAddress connecting, String keyspace, Collection ranges) # StreamSession.addTransferRanges(String keyspace, Collection ranges, Collection columnFamilies, boolean flushTables, long repairedAt) # StreamSession.addTransferFiles(Collection sstableDetails) # And finally StreamTransferTask.complete which will make use of {{updateStreamedRanges}} As you can see {{updateStreamedRanges}} will need description, endpoint, keyspace and range, and when we reached 4, we can only obtain description and endpoint from {{StreamTransferTask.session}}, therefore some information is "lost" and {{updateStreamedRanges}} cannot be accomplished. Thus additionally I propose {{StreamTransferTask}} to receive keyspace and range, for that we may need to create a new {{addTransferFiles}} to keep a record of keyspace and transfer ranges. WDYT? [~pauloricardomg] [~yukim] > Make decommission operations resumable > -- > > Key: CASSANDRA-12008 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12008 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging >Reporter: Tom van der Woerdt >Assignee: Kaide Mu >Priority: Minor > > We're dealing with large data sets (multiple terabytes per node) and > sometimes we need to add or remove nodes. These operations are very dependent > on the entire cluster being up, so while we're joining a new node (which > sometimes takes 6 hours or longer) a lot can go wrong and in a lot of cases > something does. > It would be great if the ability to retry streams was implemented. > Example to illustrate the problem : > {code} > 03:18 PM ~ $ nodetool decommission > error: Stream failed > -- StackTrace -- > org.apache.cassandra.streaming.StreamException: Stream failed > at > org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) > at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) > at > com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) > at > com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) > at > com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) > at > com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) > at > org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210) > at > org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186) > at > org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:430) > at > org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:622) > at > org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:486) > at >
[jira] [Commented] (CASSANDRA-12008) Make decommission operations resumable
[ https://issues.apache.org/jira/browse/CASSANDRA-12008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392776#comment-15392776 ] Kaide Mu commented on CASSANDRA-12008: -- Here's 3 operations I'm proposing for STREAMED_RANGES: - updateStreamedRanges(String description, InetAddress endpoint, String keyspace, Setrange) - MultiMap getAvailableRanges(String keyspace, IPartitioner partitioner) - void resetAvailableRanges() Briefly the flow is: # StreamPlan.transferRanges(InetAddress to, InetAddress connecting, String keyspace, Collection ranges) # StreamSession.addTransferRanges(String keyspace, Collection ranges, Collection columnFamilies, boolean flushTables, long repairedAt) # StreamSession.addTransferFiles(Collection sstableDetails) # And finally StreamTransferTask.complete which will make use of {{updateStreamedRanges}} As you can see {{updateStreamedRanges}} will need description, endpoint, keyspace and range, and when we reached 4, we can only obtain description and endpoint from {{StreamTransferTask.session}}, therefore some information is "lost" and {{updateStreamedRanges}} cannot be accomplished. Thus additionally I propose {{StreamTransferTask}} to receive keyspace and range, for that we may need to create a new {{addTransferFiles}} to keep a record of keyspace and transfer ranges. WDYT? [~pauloricardomg] [~yukim] > Make decommission operations resumable > -- > > Key: CASSANDRA-12008 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12008 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging >Reporter: Tom van der Woerdt >Assignee: Kaide Mu >Priority: Minor > > We're dealing with large data sets (multiple terabytes per node) and > sometimes we need to add or remove nodes. These operations are very dependent > on the entire cluster being up, so while we're joining a new node (which > sometimes takes 6 hours or longer) a lot can go wrong and in a lot of cases > something does. > It would be great if the ability to retry streams was implemented. > Example to illustrate the problem : > {code} > 03:18 PM ~ $ nodetool decommission > error: Stream failed > -- StackTrace -- > org.apache.cassandra.streaming.StreamException: Stream failed > at > org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) > at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) > at > com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) > at > com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) > at > com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) > at > com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) > at > org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210) > at > org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186) > at > org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:430) > at > org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:622) > at > org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:486) > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274) > at java.lang.Thread.run(Thread.java:745) > 08:04 PM ~ $ nodetool decommission > nodetool: Unsupported operation: Node in LEAVING state; wait for status to > become normal or restart > See 'nodetool help' or 'nodetool help '. > {code} > Streaming failed, probably due to load : > {code} > ERROR [STREAM-IN-/] 2016-06-14 18:05:47,275 StreamSession.java:520 - > [Stream #] Streaming error occurred > java.net.SocketTimeoutException: null > at > sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211) > ~[na:1.8.0_77] > at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) > ~[na:1.8.0_77] > at > java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) > ~[na:1.8.0_77] > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54) > ~[apache-cassandra-3.0.6.jar:3.0.6] > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:268) > ~[apache-cassandra-3.0.6.jar:3.0.6] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77] > {code} > If implementing retries is not possible, can we have a 'nodetool decommission > resume'? -- This message was sent by Atlassian JIRA
[jira] [Commented] (CASSANDRA-10810) Make rebuild operations resumable
[ https://issues.apache.org/jira/browse/CASSANDRA-10810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392637#comment-15392637 ] Kaide Mu commented on CASSANDRA-10810: -- A new patch is available: https://github.com/riptano/cassandra-dtest/compare/master...kdmu:jira-10810?expand=1 > Make rebuild operations resumable > - > > Key: CASSANDRA-10810 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10810 > Project: Cassandra > Issue Type: Wish > Components: Streaming and Messaging >Reporter: Jeremy Hanna >Assignee: Kaide Mu > Fix For: 3.x > > > Related to CASSANDRA-8942, now that we can resume bootstrap operations, this > could also be possible with rebuild operations, such as when you bootstrap > new nodes in a completely new datacenter in two steps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12008) Make decommission operations resumable
[
https://issues.apache.org/jira/browse/CASSANDRA-12008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15391154#comment-15391154
]
Kaide Mu commented on CASSANDRA-12008:
--
A working skeleton is now available, you check it via
https://github.com/apache/cassandra/compare/trunk...kdmu:trunk-12008
Now we can relaunch decommission if it failed previously and
{{AvailableRanges}} will be truncated according to {{OperationMode}}, but it
seems {{StreamStateStore}} is not recording properly transferred ranges
(nothing is recorded), I did a test with 3 nodes, decommissioning node2 and
interrupting node1, i.e only node3 stream session will be completed, after
partial decommission I manually checked {{AvailableRanges}} with {{SELECT *
FROM system.available_ranges;}} which returned an empty result. I guess
everything is set-up correctly, would you mind to taking a look?
Thanks.
> Make decommission operations resumable
> --
>
> Key: CASSANDRA-12008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12008
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
>Reporter: Tom van der Woerdt
>Assignee: Kaide Mu
>Priority: Minor
>
> We're dealing with large data sets (multiple terabytes per node) and
> sometimes we need to add or remove nodes. These operations are very dependent
> on the entire cluster being up, so while we're joining a new node (which
> sometimes takes 6 hours or longer) a lot can go wrong and in a lot of cases
> something does.
> It would be great if the ability to retry streams was implemented.
> Example to illustrate the problem :
> {code}
> 03:18 PM ~ $ nodetool decommission
> error: Stream failed
> -- StackTrace --
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186)
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:430)
> at
> org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:622)
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:486)
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
> at java.lang.Thread.run(Thread.java:745)
> 08:04 PM ~ $ nodetool decommission
> nodetool: Unsupported operation: Node in LEAVING state; wait for status to
> become normal or restart
> See 'nodetool help' or 'nodetool help '.
> {code}
> Streaming failed, probably due to load :
> {code}
> ERROR [STREAM-IN-/] 2016-06-14 18:05:47,275 StreamSession.java:520 -
> [Stream #] Streaming error occurred
> java.net.SocketTimeoutException: null
> at
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211)
> ~[na:1.8.0_77]
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> ~[na:1.8.0_77]
> at
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> ~[na:1.8.0_77]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54)
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:268)
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
> {code}
> If implementing retries is not possible, can we have a 'nodetool decommission
> resume'?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10810) Make rebuild operations resumable
[
https://issues.apache.org/jira/browse/CASSANDRA-10810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390354#comment-15390354
]
Kaide Mu commented on CASSANDRA-10810:
--
Working dtest available:
https://github.com/riptano/cassandra-dtest/compare/master...kdmu:jira-10810
[~pauloricardomg] or [~yukim] could you take a look?
I've tested that it behaves as expected, but there're still two problems:
# After execution, node3.log that is supposed to be saved in logs/last/ is not
present or partial content is lost, could you double-check it?
# You can see that I insert 10k data into node1 and another 10k into node2 with
{{insert_c1c2}} which inserts data with a key from 0 to 10k-1, therefore there
will be always two data with same key. When we use {{query_c1c2}} to check if
data is available on node3 after rebuild, we only can ensure that 10k data is
present since {{query_c1c2}} works with previous keys, the remaining 10k could
be there or not. It's simple to solve, I think we only need to create a new
{{insert_c1c2_from(key)}} and {{query_c1c2_from(key)}} would be enough, but I
don't know if it's necessary.
> Make rebuild operations resumable
> -
>
> Key: CASSANDRA-10810
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10810
> Project: Cassandra
> Issue Type: Wish
> Components: Streaming and Messaging
>Reporter: Jeremy Hanna
>Assignee: Kaide Mu
> Fix For: 3.x
>
>
> Related to CASSANDRA-8942, now that we can resume bootstrap operations, this
> could also be possible with rebuild operations, such as when you bootstrap
> new nodes in a completely new datacenter in two steps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12008) Make decommission operations resumable
[
https://issues.apache.org/jira/browse/CASSANDRA-12008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388447#comment-15388447
]
Kaide Mu commented on CASSANDRA-12008:
--
I'm working on this ticket, currently decommission is not resumable after
failure due to:
- Node state is changed to {{LEAVING}} after decommission starts, and current
source code prevents all states different from {{NORMAL}} to restart a
decommission operation.
- Streamed ranges are unknown for decommission node, thus although we could
resume decommission, this operation will stream again all ranges.
For solving them I propose the following initial approach:
# Add a new {{isDecommissionMode}} flag.
# Add a new {{SystemKeyspace.streamedRanges}} for storing transferred ranges.
# Add a new {{SystemKeyspace.updateStreamedRanges}}.
# Modify {{StorageService.streamRanges}}.
2. and 3. may not be necessary because {{StreamStateStore.handleStreamEvent}}
always updates {{SystemKeyspace.availableＲanges}} using
{{SystemKeyspace.updateAvailableＲanges}} no matter if the session transferred
or received ranges (Although received ranges is stored, current source code is
not using this functionality), if we want to keep another keyspace for
transferred ranges, we will still need to use
{{SystemKeyspace.updateStreamedRanges}} which will be identical than
{{SystemKeyspace.updateAvailableＲanges}}. So maybe we should adapt
{{StorageService.streamRanges}} to use RangeStreamer that already has all
implemented. WDYT [~pauloricardomg] and [~yukim]?
Here's the source code of
[StreamStateStore.handleＳtreamEvent|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/dht/StreamStateStore.java#L62]
> Make decommission operations resumable
> --
>
> Key: CASSANDRA-12008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12008
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
>Reporter: Tom van der Woerdt
>Assignee: Kaide Mu
>Priority: Minor
>
> We're dealing with large data sets (multiple terabytes per node) and
> sometimes we need to add or remove nodes. These operations are very dependent
> on the entire cluster being up, so while we're joining a new node (which
> sometimes takes 6 hours or longer) a lot can go wrong and in a lot of cases
> something does.
> It would be great if the ability to retry streams was implemented.
> Example to illustrate the problem :
> {code}
> 03:18 PM ~ $ nodetool decommission
> error: Stream failed
> -- StackTrace --
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186)
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:430)
> at
> org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:622)
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:486)
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
> at java.lang.Thread.run(Thread.java:745)
> 08:04 PM ~ $ nodetool decommission
> nodetool: Unsupported operation: Node in LEAVING state; wait for status to
> become normal or restart
> See 'nodetool help' or 'nodetool help '.
> {code}
> Streaming failed, probably due to load :
> {code}
> ERROR [STREAM-IN-/] 2016-06-14 18:05:47,275 StreamSession.java:520 -
> [Stream #] Streaming error occurred
> java.net.SocketTimeoutException: null
> at
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211)
> ~[na:1.8.0_77]
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> ~[na:1.8.0_77]
> at
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> ~[na:1.8.0_77]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54)
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at
>
[jira] [Assigned] (CASSANDRA-12008) Make decommission operations resumable
[
https://issues.apache.org/jira/browse/CASSANDRA-12008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaide Mu reassigned CASSANDRA-12008:
Assignee: Kaide Mu
> Make decommission operations resumable
> --
>
> Key: CASSANDRA-12008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12008
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
>Reporter: Tom van der Woerdt
>Assignee: Kaide Mu
>Priority: Minor
>
> We're dealing with large data sets (multiple terabytes per node) and
> sometimes we need to add or remove nodes. These operations are very dependent
> on the entire cluster being up, so while we're joining a new node (which
> sometimes takes 6 hours or longer) a lot can go wrong and in a lot of cases
> something does.
> It would be great if the ability to retry streams was implemented.
> Example to illustrate the problem :
> {code}
> 03:18 PM ~ $ nodetool decommission
> error: Stream failed
> -- StackTrace --
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186)
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:430)
> at
> org.apache.cassandra.streaming.StreamSession.complete(StreamSession.java:622)
> at
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:486)
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:274)
> at java.lang.Thread.run(Thread.java:745)
> 08:04 PM ~ $ nodetool decommission
> nodetool: Unsupported operation: Node in LEAVING state; wait for status to
> become normal or restart
> See 'nodetool help' or 'nodetool help '.
> {code}
> Streaming failed, probably due to load :
> {code}
> ERROR [STREAM-IN-/] 2016-06-14 18:05:47,275 StreamSession.java:520 -
> [Stream #] Streaming error occurred
> java.net.SocketTimeoutException: null
> at
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211)
> ~[na:1.8.0_77]
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> ~[na:1.8.0_77]
> at
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> ~[na:1.8.0_77]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54)
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:268)
> ~[apache-cassandra-3.0.6.jar:3.0.6]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]
> {code}
> If implementing retries is not possible, can we have a 'nodetool decommission
> resume'?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10810) Make rebuild operations resumable
[ https://issues.apache.org/jira/browse/CASSANDRA-10810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378175#comment-15378175 ] Kaide Mu commented on CASSANDRA-10810: -- Rebuild operation is now resumable, shortly I'll add a new dtest. > Make rebuild operations resumable > - > > Key: CASSANDRA-10810 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10810 > Project: Cassandra > Issue Type: Wish > Components: Streaming and Messaging >Reporter: Jeremy Hanna >Assignee: Kaide Mu > Fix For: 3.x > > > Related to CASSANDRA-8942, now that we can resume bootstrap operations, this > could also be possible with rebuild operations, such as when you bootstrap > new nodes in a completely new datacenter in two steps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-10810) Make rebuild operations resumable
[ https://issues.apache.org/jira/browse/CASSANDRA-10810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaide Mu reassigned CASSANDRA-10810: Assignee: Kaide Mu > Make rebuild operations resumable > - > > Key: CASSANDRA-10810 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10810 > Project: Cassandra > Issue Type: Wish > Components: Streaming and Messaging >Reporter: Jeremy Hanna >Assignee: Kaide Mu > Fix For: 3.x > > > Related to CASSANDRA-8942, now that we can resume bootstrap operations, this > could also be possible with rebuild operations, such as when you bootstrap > new nodes in a completely new datacenter in two steps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-11875) Create sstableconvert tool with support to ma format
[
https://issues.apache.org/jira/browse/CASSANDRA-11875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324708#comment-15324708
]
Kaide Mu edited comment on CASSANDRA-11875 at 6/10/16 4:36 PM:
---
bq. BigVersion.supportsWritingversion(version)
It is now implemented, now we can check if a version different from Version is
supported for writing, I use BigFormat.latestVersion.supportesWritingversion in
StandaloneConverter to check if a given version is supported.
bq. The idea is to abstract only the identical part, leaving specific parsing
to each class (so you can probably extract the parseArgs code from inside
Options).
By doing so I think we have to create a external Option or ConverterOption
class with some abstract class and extend from them in StandaloneUpgrader or
StandaloneConverter. Another way is making StandaloneConverter.Options public.
[~pauloricardomg] do you think is the right way?
bq. The testUnsupportedVersionShouldFail is failing, you should generally use
this format to assert that exceptions are thrown while making the test pass
This is also done, I'll submit a patch once previous issue is solved, but I'm
not sure if there is any other RuntimeException is thrown, do you think we
should create a UnsupportedWritingExeption to ensure it?
Thanks!
was (Author: kdmu):
bq. BigVersion.supportsWritingversion(version)
It is now implemented, now we can check if a version different from Version is
supported for writing, I use BigFormat.latestVersion.supportesWritingversion in
StandaloneConverter to check if a given version is supported.
bq. The idea is to abstract only the identical part, leaving specific parsing
to each class (so you can probably extract the parseArgs code from inside
Options).
By doing so I think we have to create a external Option or ConverterOption
abstract class and extend from them in StandaloneUpgrader or
StandaloneConverter. Another way is making StandaloneConverter.Options public.
[~pauloricardomg] do you think is the right way?
bq. The testUnsupportedVersionShouldFail is failing, you should generally use
this format to assert that exceptions are thrown while making the test pass
This is also done, I'll submit a patch once previous issue is solved, but I'm
not sure if there is any other RuntimeException is thrown, do you think we
should create a UnsupportedWritingExeption to ensure it?
Thanks!
> Create sstableconvert tool with support to ma format
>
>
> Key: CASSANDRA-11875
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11875
> Project: Cassandra
> Issue Type: Sub-task
> Components: Tools
>Reporter: Paulo Motta
>Assignee: Kaide Mu
>Priority: Minor
> Attachments: trunk-11875-WIP-V1.patch
>
>
> Currently {{Upgrader}} receives an sstable in any readable format, and writes
> into {{BigFormat.getLatestVersion()}}. We should generalize it by making it
> receive a {{target}} version and probably also rename it to
> {{SSTableConverter}}.
> Based on this we can create an {{StandaloneDowngrader}} tool which will
> perform downgrade of specified sstables to a target version. To start with,
> we should support only downgrading to {{ma}} format (from current format
> {{mb}}), downgrade to any other version should be forbidden. Since we already
> support serializing to "ma" we will not need to do any data conversion.
> We should also create a test suite that creates an sstable with data in the
> current format, perform the downgrade, and verify data in the new format is
> correct. This will be the base tests suite for more advanced conversions in
> the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-11875) Create sstableconvert tool with support to ma format
[
https://issues.apache.org/jira/browse/CASSANDRA-11875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324708#comment-15324708
]
Kaide Mu edited comment on CASSANDRA-11875 at 6/10/16 4:10 PM:
---
bq. BigVersion.supportsWritingversion(version)
It is now implemented, now we can check if a version different from Version is
supported for writing, I use BigFormat.latestVersion.supportesWritingversion in
StandaloneConverter to check if a given version is supported.
bq. The idea is to abstract only the identical part, leaving specific parsing
to each class (so you can probably extract the parseArgs code from inside
Options).
By doing so I think we have to create a external Option or ConverterOption
abstract class and extend from them in StandaloneUpgrader or
StandaloneConverter. Another way is making StandaloneConverter.Options public.
[~pauloricardomg] do you think is the right way?
bq. The testUnsupportedVersionShouldFail is failing, you should generally use
this format to assert that exceptions are thrown while making the test pass
This is also done, I'll submit a patch once previous issue is solved, but I'm
not sure if there is any other RuntimeException is thrown, do you think we
should create a UnsupportedWritingExeption to ensure it?
Thanks!
was (Author: kdmu):
bq. BigVersion.supportsWritingversion(version)
It is now implemented, now we can check if a version different from Version is
supported for writing, I use BigFormat.latestVersion.supportesWritingversion in
StandaloneConverter to check if a given version is supported.
bq. The idea is to abstract only the identical part, leaving specific parsing
to each class (so you can probably extract the parseArgs code from inside
Options).
By doing so I think we have to create a external Option or ConverterOption
abstract class andextend from them in StandaloneUpgrader or
StandaloneConverter. Another way is making StandaloneConverter.Options public.
[~pauloricardomg] do you think is the right way?
bq. The testUnsupportedVersionShouldFail is failing, you should generally use
this format to assert that exceptions are thrown while making the test pass
This is also done, I'll submit a patch once previous issue is solved, but I'm
not sure if there is any other RuntimeException is thrown, do you think we
should create a UnsupportedWritingExeption to ensure it?
Thanks!
> Create sstableconvert tool with support to ma format
>
>
> Key: CASSANDRA-11875
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11875
> Project: Cassandra
> Issue Type: Sub-task
> Components: Tools
>Reporter: Paulo Motta
>Assignee: Kaide Mu
>Priority: Minor
> Attachments: trunk-11875-WIP-V1.patch
>
>
> Currently {{Upgrader}} receives an sstable in any readable format, and writes
> into {{BigFormat.getLatestVersion()}}. We should generalize it by making it
> receive a {{target}} version and probably also rename it to
> {{SSTableConverter}}.
> Based on this we can create an {{StandaloneDowngrader}} tool which will
> perform downgrade of specified sstables to a target version. To start with,
> we should support only downgrading to {{ma}} format (from current format
> {{mb}}), downgrade to any other version should be forbidden. Since we already
> support serializing to "ma" we will not need to do any data conversion.
> We should also create a test suite that creates an sstable with data in the
> current format, perform the downgrade, and verify data in the new format is
> correct. This will be the base tests suite for more advanced conversions in
> the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11875) Create sstableconvert tool with support to ma format
[
https://issues.apache.org/jira/browse/CASSANDRA-11875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324708#comment-15324708
]
Kaide Mu commented on CASSANDRA-11875:
--
bq. BigVersion.supportsWritingversion(version)
It is now implemented, now we can check if a version different from Version is
supported for writing, I use BigFormat.latestVersion.supportesWritingversion in
StandaloneConverter to check if a given version is supported.
bq. The idea is to abstract only the identical part, leaving specific parsing
to each class (so you can probably extract the parseArgs code from inside
Options).
By doing so I think we have to create a external Option or ConverterOption
abstract class andextend from them in StandaloneUpgrader or
StandaloneConverter. Another way is making StandaloneConverter.Options public.
[~pauloricardomg] do you think is the right way?
bq. The testUnsupportedVersionShouldFail is failing, you should generally use
this format to assert that exceptions are thrown while making the test pass
This is also done, I'll submit a patch once previous issue is solved, but I'm
not sure if there is any other RuntimeException is thrown, do you think we
should create a UnsupportedWritingExeption to ensure it?
Thanks!
> Create sstableconvert tool with support to ma format
>
>
> Key: CASSANDRA-11875
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11875
> Project: Cassandra
> Issue Type: Sub-task
> Components: Tools
>Reporter: Paulo Motta
>Assignee: Kaide Mu
>Priority: Minor
> Attachments: trunk-11875-WIP-V1.patch
>
>
> Currently {{Upgrader}} receives an sstable in any readable format, and writes
> into {{BigFormat.getLatestVersion()}}. We should generalize it by making it
> receive a {{target}} version and probably also rename it to
> {{SSTableConverter}}.
> Based on this we can create an {{StandaloneDowngrader}} tool which will
> perform downgrade of specified sstables to a target version. To start with,
> we should support only downgrading to {{ma}} format (from current format
> {{mb}}), downgrade to any other version should be forbidden. Since we already
> support serializing to "ma" we will not need to do any data conversion.
> We should also create a test suite that creates an sstable with data in the
> current format, perform the downgrade, and verify data in the new format is
> correct. This will be the base tests suite for more advanced conversions in
> the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaide Mu reassigned CASSANDRA-8928: --- Assignee: Kaide Mu > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Assignee: Kaide Mu >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11875) Create sstableconvert tool with support to ma format
[
https://issues.apache.org/jira/browse/CASSANDRA-11875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313350#comment-15313350
]
Kaide Mu commented on CASSANDRA-11875:
--
Hello Paulo, new
[patch|https://github.com/apache/cassandra/compare/trunk...kdmu:trunk-11875?expand=1]
available. Changed isCompatibleForWriting(String version) ->
isCompatibleForWriting() since we are dealing directly with versions so it
would be redundant to pass it.
bq.We should probably add a Descriptor.isCompatibleForWriting so we can check
that directly from the descriptor, and use that to throw an exception on
SSTableConverter if the version is incompatible
Added Descriptor.isCompatibleWriting(), but in case of {{SSTableConverter}} we
don't have specified version {{Descriptor}} so we have to check it from
{{Version}}
{{DatabaseDescriptor.getSSTableFormat().info.getVersion(options.version).isCompatibleForWriting()}},
or maybe there's some way to obtain it?
bq.The idea of your WIP is correct, but instead of calling the main method of
StandaloneUpgrader you could create a common method that receives an Options
object, which in the case of the upgrader it would have a special flag
indicating upgrade to latest version. The argument parsing and Options creation
could be removed from the Options object to the main class so it's different
between the converter and the upgrader. You can do different printing based on
the options flag.
I modified Options access level to protected and parseArgs now varies on
boolean isLatestVersion, so now {{StandaloneUpgrader}} only contains the
execution code which is identical than {{StandaloneConverter}}, do you think we
should abstract it into a common method SSTableConverter.execute(Option
options)?
Thanks.
> Create sstableconvert tool with support to ma format
>
>
> Key: CASSANDRA-11875
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11875
> Project: Cassandra
> Issue Type: Sub-task
> Components: Tools
>Reporter: Paulo Motta
>Assignee: Kaide Mu
>Priority: Minor
> Attachments: trunk-11875-WIP-V1.patch
>
>
> Currently {{Upgrader}} receives an sstable in any readable format, and writes
> into {{BigFormat.getLatestVersion()}}. We should generalize it by making it
> receive a {{target}} version and probably also rename it to
> {{SSTableConverter}}.
> Based on this we can create an {{StandaloneDowngrader}} tool which will
> perform downgrade of specified sstables to a target version. To start with,
> we should support only downgrading to {{ma}} format (from current format
> {{mb}}), downgrade to any other version should be forbidden. Since we already
> support serializing to "ma" we will not need to do any data conversion.
> We should also create a test suite that creates an sstable with data in the
> current format, perform the downgrade, and verify data in the new format is
> correct. This will be the base tests suite for more advanced conversions in
> the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11875) Create sstableconvert tool with support to ma format
[
https://issues.apache.org/jira/browse/CASSANDRA-11875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15312634#comment-15312634
]
Kaide Mu commented on CASSANDRA-11875:
--
New [patch|https://github.com/kdmu/cassandra/tree/trunk-11875] available.
Working test suit included.
I'm not sure if we have to make {{StandaloneUpgrader}} subclass of
{{StandaloneConverter}}. I did a
[WIP|https://gist.github.com/anonymous/29b850c4abed44702c7490606c09fbdc] for
that, since we have to validate passed args (validate and add latestVersion to
it), we should maintain original parseArgs of {{StandaloneUpgrader}}, also
since printUsage is different from {{StandaloneConverter}} we should also
maintain it.
Looking forward for your feedback,
Thanks!
> Create sstableconvert tool with support to ma format
>
>
> Key: CASSANDRA-11875
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11875
> Project: Cassandra
> Issue Type: Sub-task
> Components: Tools
>Reporter: Paulo Motta
>Assignee: Kaide Mu
>Priority: Minor
> Attachments: trunk-11875-WIP-V1.patch
>
>
> Currently {{Upgrader}} receives an sstable in any readable format, and writes
> into {{BigFormat.getLatestVersion()}}. We should generalize it by making it
> receive a {{target}} version and probably also rename it to
> {{SSTableConverter}}.
> Based on this we can create an {{StandaloneDowngrader}} tool which will
> perform downgrade of specified sstables to a target version. To start with,
> we should support only downgrading to {{ma}} format (from current format
> {{mb}}), downgrade to any other version should be forbidden. Since we already
> support serializing to "ma" we will not need to do any data conversion.
> We should also create a test suite that creates an sstable with data in the
> current format, perform the downgrade, and verify data in the new format is
> correct. This will be the base tests suite for more advanced conversions in
> the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11875) Create sstableconvert tool with support to ma format
[
https://issues.apache.org/jira/browse/CASSANDRA-11875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15305295#comment-15305295
]
Kaide Mu commented on CASSANDRA-11875:
--
Just unified things into SSTableConverter and StandaloneConverter.
Also added a new Test suit SSTableConversionTest, but it has some problem when
performing conversion.
{{ERROR 10:39:13 Attempting to load already loaded table
SSTableRewriterTest.Standard1}}
[Patch|https://github.com/kdmu/cassandra/commit/e8637a531e874310a49bc5dff71c94b8841c2e09]
(pushed to trunk instead of a temporary working branch, will fix it)
> Create sstableconvert tool with support to ma format
>
>
> Key: CASSANDRA-11875
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11875
> Project: Cassandra
> Issue Type: Sub-task
> Components: Tools
>Reporter: Paulo Motta
>Assignee: Kaide Mu
>Priority: Minor
> Attachments: trunk-11875-WIP-V1.patch
>
>
> Currently {{Upgrader}} receives an sstable in any readable format, and writes
> into {{BigFormat.getLatestVersion()}}. We should generalize it by making it
> receive a {{target}} version and probably also rename it to
> {{SSTableConverter}}.
> Based on this we can create an {{StandaloneDowngrader}} tool which will
> perform downgrade of specified sstables to a target version. To start with,
> we should support only downgrading to {{ma}} format (from current format
> {{mb}}), downgrade to any other version should be forbidden. Since we already
> support serializing to "ma" we will not need to do any data conversion.
> We should also create a test suite that creates an sstable with data in the
> current format, perform the downgrade, and verify data in the new format is
> correct. This will be the base tests suite for more advanced conversions in
> the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-11875) Create StandaloneDowngrader with support to ma format
[
https://issues.apache.org/jira/browse/CASSANDRA-11875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302250#comment-15302250
]
Kaide Mu edited comment on CASSANDRA-11875 at 5/26/16 3:36 PM:
---
A WIP Patch has been attached, it includes:
- ColumnFamilyStore.getSSTablePath(File directory, String version) which allows
create a SSTable according version specified for example by Downgrader.
- Downgrader and StandaloneDowngrader based on upgrader and StandaloneDowngrader
- OperationType.DOWNGRADE_SSTABLE
Tested the work-flow with SSTableWriterTest creating a SSTable of version "ma"
by specifying it on getWriter of SSTableWriterTestBase. Need to test the same
work-flow using StandaloneDowngrader.
Digest component is always created according to latest
version[(Reference)|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L808],
should it be also modified?
was (Author: kdmu):
A WIP Patch has been attached, it includes:
- ColumnFamilyStore.getSSTablePath(File directory, String version) which allows
create a SSTable according version specified for example by Downgrader.
- Downgrader and StandaloneDowngrader based on upgrader and StandaloneDowngrader
- OperationType.DOWNGRADE_SSTABLE
Tested the work-flow with SSTableWriterTest creating a SSTable of version "ma"
by specifying it on getWriter of SSTableWriterTestBase. Need to test the same
work-flow using StandaloneDowngrader.
> Create StandaloneDowngrader with support to ma format
> -
>
> Key: CASSANDRA-11875
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11875
> Project: Cassandra
> Issue Type: Sub-task
> Components: Tools
>Reporter: Paulo Motta
>Assignee: Kaide Mu
>Priority: Minor
> Attachments: trunk-11875-WIP-V1.patch
>
>
> Currently {{Upgrader}} receives an sstable in any readable format, and writes
> into {{BigFormat.getLatestVersion()}}. We should generalize it by making it
> receive a {{target}} version and probably also rename it to
> {{SSTableConverter}}.
> Based on this we can create an {{StandaloneDowngrader}} tool which will
> perform downgrade of specified sstables to a target version. To start with,
> we should support only downgrading to {{ma}} format (from current format
> {{mb}}), downgrade to any other version should be forbidden. Since we already
> support serializing to "ma" we will not need to do any data conversion.
> We should also create a test suite that creates an sstable with data in the
> current format, perform the downgrade, and verify data in the new format is
> correct. This will be the base tests suite for more advanced conversions in
> the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11875) Create StandaloneDowngrader with support to ma format
[
https://issues.apache.org/jira/browse/CASSANDRA-11875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302250#comment-15302250
]
Kaide Mu commented on CASSANDRA-11875:
--
A WIP Patch has been attached, it includes:
- ColumnFamilyStore.getSSTablePath(File directory, String version) which allows
create a SSTable according version specified for example by Downgrader.
- Downgrader and StandaloneDowngrader based on upgrader and StandaloneDowngrader
- OperationType.DOWNGRADE_SSTABLE
Tested the work-flow with SSTableWriterTest creating a SSTable of version "ma"
by specifying it on getWriter of SSTableWriterTestBase. Need to test the same
work-flow using StandaloneDowngrader.
> Create StandaloneDowngrader with support to ma format
> -
>
> Key: CASSANDRA-11875
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11875
> Project: Cassandra
> Issue Type: Sub-task
> Components: Tools
>Reporter: Paulo Motta
>Assignee: Kaide Mu
>Priority: Minor
> Attachments: trunk-11875-WIP-V1.patch
>
>
> Currently {{Upgrader}} receives an sstable in any readable format, and writes
> into {{BigFormat.getLatestVersion()}}. We should generalize it by making it
> receive a {{target}} version and probably also rename it to
> {{SSTableConverter}}.
> Based on this we can create an {{StandaloneDowngrader}} tool which will
> perform downgrade of specified sstables to a target version. To start with,
> we should support only downgrading to {{ma}} format (from current format
> {{mb}}), downgrade to any other version should be forbidden. Since we already
> support serializing to "ma" we will not need to do any data conversion.
> We should also create a test suite that creates an sstable with data in the
> current format, perform the downgrade, and verify data in the new format is
> correct. This will be the base tests suite for more advanced conversions in
> the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11875) Create StandaloneDowngrader with support to ma format
[
https://issues.apache.org/jira/browse/CASSANDRA-11875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaide Mu updated CASSANDRA-11875:
-
Attachment: trunk-11875-WIP-V1.patch
> Create StandaloneDowngrader with support to ma format
> -
>
> Key: CASSANDRA-11875
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11875
> Project: Cassandra
> Issue Type: Sub-task
> Components: Tools
>Reporter: Paulo Motta
>Assignee: Kaide Mu
>Priority: Minor
> Attachments: trunk-11875-WIP-V1.patch
>
>
> Currently {{Upgrader}} receives an sstable in any readable format, and writes
> into {{BigFormat.getLatestVersion()}}. We should generalize it by making it
> receive a {{target}} version and probably also rename it to
> {{SSTableConverter}}.
> Based on this we can create an {{StandaloneDowngrader}} tool which will
> perform downgrade of specified sstables to a target version. To start with,
> we should support only downgrading to {{ma}} format (from current format
> {{mb}}), downgrade to any other version should be forbidden. Since we already
> support serializing to "ma" we will not need to do any data conversion.
> We should also create a test suite that creates an sstable with data in the
> current format, perform the downgrade, and verify data in the new format is
> correct. This will be the base tests suite for more advanced conversions in
> the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11875) Create StandaloneDowngrader with support to ma format
[
https://issues.apache.org/jira/browse/CASSANDRA-11875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaide Mu updated CASSANDRA-11875:
-
Attachment: (was: trunk-11875.patch)
> Create StandaloneDowngrader with support to ma format
> -
>
> Key: CASSANDRA-11875
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11875
> Project: Cassandra
> Issue Type: Sub-task
> Components: Tools
>Reporter: Paulo Motta
>Assignee: Kaide Mu
>Priority: Minor
>
> Currently {{Upgrader}} receives an sstable in any readable format, and writes
> into {{BigFormat.getLatestVersion()}}. We should generalize it by making it
> receive a {{target}} version and probably also rename it to
> {{SSTableConverter}}.
> Based on this we can create an {{StandaloneDowngrader}} tool which will
> perform downgrade of specified sstables to a target version. To start with,
> we should support only downgrading to {{ma}} format (from current format
> {{mb}}), downgrade to any other version should be forbidden. Since we already
> support serializing to "ma" we will not need to do any data conversion.
> We should also create a test suite that creates an sstable with data in the
> current format, perform the downgrade, and verify data in the new format is
> correct. This will be the base tests suite for more advanced conversions in
> the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11875) Create StandaloneDowngrader with support to ma format
[
https://issues.apache.org/jira/browse/CASSANDRA-11875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaide Mu updated CASSANDRA-11875:
-
Attachment: trunk-11875.patch
> Create StandaloneDowngrader with support to ma format
> -
>
> Key: CASSANDRA-11875
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11875
> Project: Cassandra
> Issue Type: Sub-task
> Components: Tools
>Reporter: Paulo Motta
>Assignee: Kaide Mu
>Priority: Minor
> Attachments: trunk-11875.patch
>
>
> Currently {{Upgrader}} receives an sstable in any readable format, and writes
> into {{BigFormat.getLatestVersion()}}. We should generalize it by making it
> receive a {{target}} version and probably also rename it to
> {{SSTableConverter}}.
> Based on this we can create an {{StandaloneDowngrader}} tool which will
> perform downgrade of specified sstables to a target version. To start with,
> we should support only downgrading to {{ma}} format (from current format
> {{mb}}), downgrade to any other version should be forbidden. Since we already
> support serializing to "ma" we will not need to do any data conversion.
> We should also create a test suite that creates an sstable with data in the
> current format, perform the downgrade, and verify data in the new format is
> correct. This will be the base tests suite for more advanced conversions in
> the future.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8777) Streaming operations should log both endpoint and port associated with the operation
[ https://issues.apache.org/jira/browse/CASSANDRA-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15215188#comment-15215188 ] Kaide Mu commented on CASSANDRA-8777: - Hi, [~pauloricardomg] a new [patch|https://github.com/apache/cassandra/compare/trunk...kdmu:8777-trunk?expand=1] is available. This time I'm trying to work on the forked repository, please let me know if there's any mistake. Thank you. > Streaming operations should log both endpoint and port associated with the > operation > > > Key: CASSANDRA-8777 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8777 > Project: Cassandra > Issue Type: Improvement >Reporter: Jeremy Hanna > Labels: lhf > Fix For: 2.1.x > > Attachments: 8777-2.2.txt > > > Currently we log the endpoint for a streaming operation. If the port has > been overridden, it would be valuable to know that that setting is getting > picked up. Therefore, when logging the endpoint address, it would be nice to > also log the port it's trying to use. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8777) Streaming operations should log both endpoint and port associated with the operation
[ https://issues.apache.org/jira/browse/CASSANDRA-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15212149#comment-15212149 ] Kaide Mu commented on CASSANDRA-8777: - Patch available for review > Streaming operations should log both endpoint and port associated with the > operation > > > Key: CASSANDRA-8777 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8777 > Project: Cassandra > Issue Type: Improvement >Reporter: Jeremy Hanna > Labels: lhf > Fix For: 2.1.x > > Attachments: 8777-2.2.txt > > > Currently we log the endpoint for a streaming operation. If the port has > been overridden, it would be valuable to know that that setting is getting > picked up. Therefore, when logging the endpoint address, it would be nice to > also log the port it's trying to use. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8777) Streaming operations should log both endpoint and port associated with the operation
[ https://issues.apache.org/jira/browse/CASSANDRA-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaide Mu updated CASSANDRA-8777: Status: Patch Available (was: Open) > Streaming operations should log both endpoint and port associated with the > operation > > > Key: CASSANDRA-8777 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8777 > Project: Cassandra > Issue Type: Improvement >Reporter: Jeremy Hanna > Labels: lhf > Fix For: 2.1.x > > Attachments: 8777-2.2.txt > > > Currently we log the endpoint for a streaming operation. If the port has > been overridden, it would be valuable to know that that setting is getting > picked up. Therefore, when logging the endpoint address, it would be nice to > also log the port it's trying to use. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8777) Streaming operations should log both endpoint and port associated with the operation
[ https://issues.apache.org/jira/browse/CASSANDRA-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaide Mu updated CASSANDRA-8777: Attachment: 8777-2.2.txt > Streaming operations should log both endpoint and port associated with the > operation > > > Key: CASSANDRA-8777 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8777 > Project: Cassandra > Issue Type: Improvement >Reporter: Jeremy Hanna > Labels: lhf > Fix For: 2.1.x > > Attachments: 8777-2.2.txt > > > Currently we log the endpoint for a streaming operation. If the port has > been overridden, it would be valuable to know that that setting is getting > picked up. Therefore, when logging the endpoint address, it would be nice to > also log the port it's trying to use. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8888) Disable internode compression by default
[ https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaide Mu updated CASSANDRA-: Status: Patch Available (was: Open) Ｃhanged internode compression from all to dc > Disable internode compression by default > > > Key: CASSANDRA- > URL: https://issues.apache.org/jira/browse/CASSANDRA- > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging >Reporter: Matt Stump > Labels: lhf > Fix For: 3.x > > Attachments: -3.0.txt > > > Internode compression increases GC load, and can cause high CPU utilization > for high throughput use cases. Very rarely are customers restricted by > intra-DC or cross-DC network bandwidth. I'de rather we optimize for the 75% > of cases where internode compression isn't needed and then selectively enable > it for customers where it would provide a benefit. Currently I'm advising all > field consultants disable compression by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8888) Disable internode compression by default
[ https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaide Mu updated CASSANDRA-: Attachment: -3.0.txt > Disable internode compression by default > > > Key: CASSANDRA- > URL: https://issues.apache.org/jira/browse/CASSANDRA- > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging >Reporter: Matt Stump > Labels: lhf > Fix For: 3.x > > Attachments: -3.0.txt > > > Internode compression increases GC load, and can cause high CPU utilization > for high throughput use cases. Very rarely are customers restricted by > intra-DC or cross-DC network bandwidth. I'de rather we optimize for the 75% > of cases where internode compression isn't needed and then selectively enable > it for customers where it would provide a benefit. Currently I'm advising all > field consultants disable compression by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8888) Disable internode compression by default
[
https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211046#comment-15211046
]
Kaide Mu commented on CASSANDRA-:
-
Hi, I'm currently working on this issue, I have modified
[cassandra/conf/cassandra.yaml|https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L932]
internode_compression option to dc only. Could you tell me if there are other
changes required? Also I'm working on {{trunk}}, is that fine?
> Disable internode compression by default
>
>
> Key: CASSANDRA-
> URL: https://issues.apache.org/jira/browse/CASSANDRA-
> Project: Cassandra
> Issue Type: Improvement
> Components: Streaming and Messaging
>Reporter: Matt Stump
> Labels: lhf
> Fix For: 3.x
>
>
> Internode compression increases GC load, and can cause high CPU utilization
> for high throughput use cases. Very rarely are customers restricted by
> intra-DC or cross-DC network bandwidth. I'de rather we optimize for the 75%
> of cases where internode compression isn't needed and then selectively enable
> it for customers where it would provide a benefit. Currently I'm advising all
> field consultants disable compression by default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[
https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15210247#comment-15210247
]
Kaide Mu commented on CASSANDRA-8928:
-
Hi, [~pauloricardomg].
Thank you so much, just fixed format error and added ccm and StandaloneUpgrader
to the proposal, I didn't realize that dtest runs on ccm. I'll check them out.
Regarding {{OldFormatIterator}} and {{OldFormatDeserializer}}, I am trying to
find them on Git repository but I didn't find anything, maybe after setting up
the development environment I can decompile them. Also, can I use openJDK (7)
instead of Oracle JDK? Because I'm trying to setting up Cassandra (trunk) with
IDEA according to
[RunningCassandraInIDEA|http://wiki.apache.org/cassandra/RunningCassandraInIDEA|http://wiki.apache.org/cassandra/RunningCassandraInIDEA]
but it gives me a build failed error when I execute {{ant
generate-idea-files}}.
{code}
build-project:
[echo] apache-cassandra: /home/km/cassandra/build.xml
[javac] Compiling 45 source files to /home/km/cassandra/build/classes/thrift
[javac] javac: invalid target release: 1.8
[javac] Usage: javac
[javac] use -help for a list of possible options
BUILD FAILED
/home/km/cassandra/build.xml:769: Compile failed; see the compiler error output
for details.
{code}
I'll try it again, once I have it setted up, I'll take a look along with other
materials [CASSANDRA-|https://issues.apache.org/jira/browse/CASSANDRA-]
which seems more suitable to start with.
> Add downgradesstables
> -
>
> Key: CASSANDRA-8928
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8928
> Project: Cassandra
> Issue Type: New Feature
> Components: Tools
>Reporter: Jeremy Hanna
>Priority: Minor
> Labels: gsoc2016, mentor
>
> As mentioned in other places such as CASSANDRA-8047 and in the wild,
> sometimes you need to go back. A downgrade sstables utility would be nice
> for a lot of reasons and I don't know that supporting going back to the
> previous major version format would be too much code since we already support
> reading the previous version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208166#comment-15208166 ] Kaide Mu edited comment on CASSANDRA-8928 at 3/23/16 10:21 AM: --- Hi, [~pauloricardomg] and [~yukim]. Sorry I couldn't reply earlier because I have been with exams during these weeks and thanks for reviewing the proposal. I agree that it's better to have first a basic sstabledowngrader than jumping directly to the framework, I have modified it according to your suggestions. I'll make another change if it's needed. Till now I have been looking for inspection tools, I guess I don't have any problems related with the usage. I'll keep on looking for sstablescrubber tool. In addition, do you mean with double cycle support that our sstabledowngrader should work correctly with a flow like "ma" -> "ka" -> "la"? was (Author: kdmu): Hi, [~pauloricardomg] and [~yukim]. Sorry I couldn't reply earlier because I have been with exams during these weeks and thanks for reviewing the proposal. I agree that it's better to have first a basic sstabledowngrader than jumping directly to the framework, thus I have modified it according to your suggestions. I'll make another change if it's needed. Till now I have been looking for inspection tools, I guess I don't have any problems related with the usage. I'll keep on looking for sstablescrubber tool. In addition, do you mean with double cycle support that our sstabledowngrader should work correctly with a flow like "ma" -> "ka" -> "la"? > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208166#comment-15208166 ] Kaide Mu commented on CASSANDRA-8928: - Hi, [~pauloricardomg] and [~yukim]. Sorry I couldn't reply earlier because I have been with exams during these weeks and thanks for reviewing the proposal. I agree that it's better to have first a basic sstabledowngrader than jumping directly to the framework, thus I have modified it according to your suggestions. I'll make another change if it's needed. Till now I have been looking for inspection tools, I guess I don't have any problems related with the usage. I'll keep on looking for sstablescrubber tool. In addition, do you mean with double cycle support that our sstabledowngrader should work correctly with a flow like "ma" -> "ka" -> "la"? > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198548#comment-15198548 ] Kaide Mu commented on CASSANDRA-8928: - Hi, dear [~pauloricardomg]. Thank you for the reply and all these references! I'll look through all of them, especially StandaloneScrubbler which seems it's the key for our design. If I have any questions I'll leave a message in this ticket or via irc. Also, since the student application deadline is next Friday, I had drawn up a [proposal|https://docs.google.com/document/d/10Xor1LggGxEihZ39ItOJ8QZ-EKTOVqcAXaE_B2TY3BM/edit?usp=sharing] for this issue, if you have any time, would you mind take a look and tell me if I missed anything or anything that I should detailed more. Thank you so much. > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15191399#comment-15191399 ] Kaide Mu commented on CASSANDRA-8928: - Hi, dear Paulo. I'd read such blog, I think now I'm clear with changes of Data.db in 3.x, I tried to find something similar of changes on version 2.2 but I didn't find anything, do you have any similar references or similar things, I'll try to see if I can breakdown it, but it seems quite difficult in the sense that I'm not familiar with Cassandra code base. Also before jumping to draw up the framework, I'd like to know is our purpose downgrade an existing SSTable to certain older version or add the possibility to write older version of SSTable? Regards. > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174611#comment-15174611 ] Kaide Mu edited comment on CASSANDRA-8928 at 3/6/16 5:54 PM: - Hi, [~yukim] Thanks for the quick replay, I had been looking for SSTable formats on branch 2.2(la), 2.1(ka) and 2.0(jb). I'm not sure if I'm in the right way, but I had been looking for following stuffs: - Regarding to changelog on [BIgVersion|https://github.com/apache/cassandra/blob/cassandra-3.3/src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java#L116]: -- There's some change with checksums such as adler32 for compression check sum. -- In version 2.1 introduces Statistics, but I found it as component of SSTable in version 2.0(?) -- Vesion 2.2 only introduces a new file name format, "la". - Respect to "Formats", I had been looking for [SSTableFormat|https://github.com/apache/cassandra/blob/trunk/src%2Fjava%2Forg%2Fapache%2Fcassandra%2Fio%2Fsstable%2Fformat%2FSSTableFormat.java#L35] available on branch 2.2, that contains table format type, which is "big", but I didn't find any information about that on branch 2.1 and 2.0. Also there's no indication of this thing in its [Descriptors|https://github.com/apache/cassandra/blob/cassandra-2.1/src%2Fjava%2Forg%2Fapache%2Fcassandra%2Fio%2Fsstable%2FDescriptor.java#L39]. Also as I mentioned before I had been looking for [Components|https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/io/sstable/Component.java#L32] of each branch and seems the main differences are: -- They use different hash functions for digest 2.2 uses adler32, 2.1 and 2.0 uses sha-1 -- 2.0 version has COMPACTED_MARKER, which is removed in forward versions. I'm not sure if i missed something. - Therefore, if I'm correct, please correct me if not, if we want to write (I have a question about this terminology, when we say write in this context do we mean write, or create? Because I found that SSTable is immutable after creation) an older version of SSTable we should have proper Digest and COMPACTED_MARKER also in case of jb. My sincerest apologies for making this comment this way long, If above considerations are correct, I'll continue looking for code base and SSTable API, also would you mind tell me where can I find implementation of changes proposed in the BigVersion and implementation of SSTable, I had been looking for [SSTable|https://github.com/apache/cassandra/blob/cassandra-2.2/src%2Fjava%2Forg%2Fapache%2Fcassandra%2Fio%2Fsstable%2FSSTable.java#L57] but I don't know if it's the right one. Best regards, Kaide Mu was (Author: kdmu): Hi, dear Yuki. Thanks for the quick replay, I had been looking for SSTable formats on branch 2.2(la), 2.1(ka) and 2.0(jb). I'm not sure if I'm in the right way, but I had been looking for following stuffs: - Regarding to changelog on [BIgVersion|https://github.com/apache/cassandra/blob/cassandra-3.3/src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java#L116]: -- There's some change with checksums such as adler32 for compression check sum. -- In version 2.1 introduces Statistics, but I found it as component of SSTable in version 2.0(?) -- Vesion 2.2 only introduces a new file name format, "la". - Respect to "Formats", I had been looking for [SSTableFormat|https://github.com/apache/cassandra/blob/trunk/src%2Fjava%2Forg%2Fapache%2Fcassandra%2Fio%2Fsstable%2Fformat%2FSSTableFormat.java#L35] available on branch 2.2, that contains table format type, which is "big", but I didn't find any information about that on branch 2.1 and 2.0. Also there's no indication of this thing in its [Descriptors|https://github.com/apache/cassandra/blob/cassandra-2.1/src%2Fjava%2Forg%2Fapache%2Fcassandra%2Fio%2Fsstable%2FDescriptor.java#L39]. Also as I mentioned before I had been looking for [Components|https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/io/sstable/Component.java#L32] of each branch and seems the main differences are: -- They use different hash functions for digest 2.2 uses adler32, 2.1 and 2.0 uses sha-1 -- 2.0 version has COMPACTED_MARKER, which is removed in forward versions. I'm not sure if i missed something. - Therefore, if I'm correct, please correct me if not, if we want to write (I have a question about this terminology, when we say write in this context do we mean write, or create? Because I found that SSTable is immutable after creation) an older version of SSTable we should have proper Digest and COMPACTED_MARKER also in case of jb. My sincerest apologies for making this comment this way long, If above considerations are correct, I'll continue looking for code base and SSTable API, also would you mind tell me where can I find implementation of changes proposed in the BigVersion and implementation of SSTable, I had been looking for
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174611#comment-15174611 ] Kaide Mu commented on CASSANDRA-8928: - Hi, dear Yuki. Thanks for the quick replay, I had been looking for SSTable formats on branch 2.2(la), 2.1(ka) and 2.0(jb). I'm not sure if I'm in the right way, but I had been looking for following stuffs: - Regarding to changelog on [BIgVersion|https://github.com/apache/cassandra/blob/cassandra-3.3/src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java#L116]: -- There's some change with checksums such as adler32 for compression check sum. -- In version 2.1 introduces Statistics, but I found it as component of SSTable in version 2.0(?) -- Vesion 2.2 only introduces a new file name format, "la". - Respect to "Formats", I had been looking for [SSTableFormat|https://github.com/apache/cassandra/blob/trunk/src%2Fjava%2Forg%2Fapache%2Fcassandra%2Fio%2Fsstable%2Fformat%2FSSTableFormat.java#L35] available on branch 2.2, that contains table format type, which is "big", but I didn't find any information about that on branch 2.1 and 2.0. Also there's no indication of this thing in its [Descriptors|https://github.com/apache/cassandra/blob/cassandra-2.1/src%2Fjava%2Forg%2Fapache%2Fcassandra%2Fio%2Fsstable%2FDescriptor.java#L39]. Also as I mentioned before I had been looking for [Components|https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/io/sstable/Component.java#L32] of each branch and seems the main differences are: -- They use different hash functions for digest 2.2 uses adler32, 2.1 and 2.0 uses sha-1 -- 2.0 version has COMPACTED_MARKER, which is removed in forward versions. I'm not sure if i missed something. - Therefore, if I'm correct, please correct me if not, if we want to write (I have a question about this terminology, when we say write in this context do we mean write, or create? Because I found that SSTable is immutable after creation) an older version of SSTable we should have proper Digest and COMPACTED_MARKER also in case of jb. My sincerest apologies for making this comment this way long, If above considerations are correct, I'll continue looking for code base and SSTable API, also would you mind tell me where can I find implementation of changes proposed in the BigVersion and implementation of SSTable, I had been looking for [SSTable|https://github.com/apache/cassandra/blob/cassandra-2.2/src%2Fjava%2Forg%2Fapache%2Fcassandra%2Fio%2Fsstable%2FSSTable.java#L57] but I don't know if it's the right one. Best regards, Kaide Mu > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8928) Add downgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-8928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172917#comment-15172917 ] Kaide Mu commented on CASSANDRA-8928: - Hello Apache Cassandra community, I'm Kaide Mu, currently pursuing BSc of Computer Science at Polytechnic University of Valencia, Spain. I'd like to work in this issue as my project of GSoC 2016. About this project, I had been looking in a minor detail for SSTable architecture and some of different available SSTable source code, as I see, there's some different implementations of SSTable, therefore as the issue description indicates, I guess we aim to add a downgradesstable functionality to all existing SSTables, e.g downgrade from a 3.X version SSTable to 2.X SSTable or 1.X ones, please correct if my understanding is wrong. In addition, would you mind providing me any suggestion or references which I should take a look in order to clarify with this issue, I'd very appreciate it. Thank you so much and best regards, Kaide Mu > Add downgradesstables > - > > Key: CASSANDRA-8928 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8928 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jeremy Hanna >Priority: Minor > Labels: gsoc2016, mentor > > As mentioned in other places such as CASSANDRA-8047 and in the wild, > sometimes you need to go back. A downgrade sstables utility would be nice > for a lot of reasons and I don't know that supporting going back to the > previous major version format would be too much code since we already support > reading the previous version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
