[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[
https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644633#comment-16644633
]
Sam Tunnicliffe commented on CASSANDRA-14373:
-
Couple of tiny things:
* I think the help text for --archive-command should show the command quoted,
not in backticks (it's probably obvious to a user that the command should be
quoted, but it doesn't hurt to be explicit)
* the title param of the archiveRetries annotation is wrong (think this is used
by the help command)
otherwise LGTM
> Allow using custom script for chronicle queue BinLog archival
> -
>
> Key: CASSANDRA-14373
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14373
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Stefan Podkowinski
>Assignee: Pramod K Sivaraju
>Priority: Major
> Labels: lhf, pull-request-available
> Fix For: 4.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It would be nice to allow the user to configure an archival script that will
> be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log,
> just as we do in {{CommitLogArchiver}}. The script should be able to copy the
> released file to an external location or do whatever the author hand in mind.
> Deleting the log file should be delegated to the script as well.
> See CASSANDRA-13983, CASSANDRA-12151 for use cases.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[
https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643394#comment-16643394
]
Marcus Eriksson commented on CASSANDRA-14373:
-
bq. are they sequentially numbered?
yeah, the file names should have date/time:
https://github.com/OpenHFT/Chronicle-Queue#topics-and-queue-files
pushed a change to move back to POSIX {{sh}} instead of {{bash}}, [~beobal]
could you have a quick lock at that?
> Allow using custom script for chronicle queue BinLog archival
> -
>
> Key: CASSANDRA-14373
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14373
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Stefan Podkowinski
>Assignee: Pramod K Sivaraju
>Priority: Major
> Labels: lhf, pull-request-available
> Fix For: 4.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It would be nice to allow the user to configure an archival script that will
> be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log,
> just as we do in {{CommitLogArchiver}}. The script should be able to copy the
> released file to an external location or do whatever the author hand in mind.
> Deleting the log file should be delegated to the script as well.
> See CASSANDRA-13983, CASSANDRA-12151 for use cases.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[
https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642046#comment-16642046
]
Ariel Weisberg commented on CASSANDRA-14373:
+1 to the last commit allowing max retries to be configured.
> Allow using custom script for chronicle queue BinLog archival
> -
>
> Key: CASSANDRA-14373
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14373
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Stefan Podkowinski
>Assignee: Pramod K Sivaraju
>Priority: Major
> Labels: lhf, pull-request-available
> Fix For: 4.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It would be nice to allow the user to configure an archival script that will
> be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log,
> just as we do in {{CommitLogArchiver}}. The script should be able to copy the
> released file to an external location or do whatever the author hand in mind.
> Deleting the log file should be delegated to the script as well.
> See CASSANDRA-13983, CASSANDRA-12151 for use cases.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[
https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639945#comment-16639945
]
Ariel Weisberg commented on CASSANDRA-14373:
+1
One interesting thing to note is that the retry mechanism is going to reorder
the things it is archiving when it supplies them to the archive script. It's
probably fine, but I wonder if people can tell when files are missing? Like are
they sequentially numbered?
> Allow using custom script for chronicle queue BinLog archival
> -
>
> Key: CASSANDRA-14373
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14373
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Stefan Podkowinski
>Assignee: Pramod K Sivaraju
>Priority: Major
> Labels: lhf, pull-request-available
> Fix For: 4.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It would be nice to allow the user to configure an archival script that will
> be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log,
> just as we do in {{CommitLogArchiver}}. The script should be able to copy the
> released file to an external location or do whatever the author hand in mind.
> Deleting the log file should be delegated to the script as well.
> See CASSANDRA-13983, CASSANDRA-12151 for use cases.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[
https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639846#comment-16639846
]
Marcus Eriksson commented on CASSANDRA-14373:
-
thanks for the review, pushed a couple of commits to address the comments
bq. Removing documentation of defaults doesn't seem like a pure win since they
still seem to be there?
I removed the defaults from the nodetool command to be able to override options
that are set in cassandra.yaml, do you have a suggestion how to do it in a
better way? We don't know the actual set defaults in the nodetool command, the
user would have to check out cassandra.yaml which might not be very user
friendly.
bq. This isn't just a path right it's a format specified of sorts with %path?
right, changed to 'command' instead
bq. This is BinLogOptions but the comments reference Audit log and there is a
typo in the first sentence here
fixed
bq. Depending on what the archiving script does and why it failed there could
be unfortunate consequences to retrying repeatedly.
added a new configuration param that allows users to set max retries to 0 to
avoid this (defaults to 10 retries as retrying forever might also be bad)
bq. exec forks, forking can be slow because of page table copying which in the
past was slow under things like Xen.. I'm just mentioning it. I don't think you
need to make it better right now. I don't know offhand how you invoke an
external command more efficiently from Java.
yeah not sure what to do here, a quick search tells me ProcessBuilder seems to
be the way to do this
bq. Is this going to enable it for all tests? Is that a good idea can we only
enable it for just the unit tests that require it?
yeah, removed, should not be there
bq. Should use execute instead of submit unless consuming the result future
fixed
bq. Same here
here we actually wait for the future
bq. The dtests are good tests but could they be unit tests since they are
single node.
In general I agree, but in this case it executes the nodetool command as an end
user would, against a running cassandra cluster (well, node, but anyway). I
suppose we could stand up a real cluster in a unit test and execute the
nodetool script as well, but I assume that would take about as long as doing it
using ccm.
> Allow using custom script for chronicle queue BinLog archival
> -
>
> Key: CASSANDRA-14373
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14373
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Stefan Podkowinski
>Assignee: Pramod K Sivaraju
>Priority: Major
> Labels: lhf, pull-request-available
> Fix For: 4.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It would be nice to allow the user to configure an archival script that will
> be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log,
> just as we do in {{CommitLogArchiver}}. The script should be able to copy the
> released file to an external location or do whatever the author hand in mind.
> Deleting the log file should be delegated to the script as well.
> See CASSANDRA-13983, CASSANDRA-12151 for use cases.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[
https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638997#comment-16638997
]
Ariel Weisberg commented on CASSANDRA-14373:
The dtests are good tests but could they be unit tests since they are single
node. The reason I harp on it is that they are so much slower and not adding
more of them is a win. I'm not sure if we can test nodetool from a dtest, but
if we could it would be amazing.
> Allow using custom script for chronicle queue BinLog archival
> -
>
> Key: CASSANDRA-14373
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14373
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Stefan Podkowinski
>Assignee: Pramod K Sivaraju
>Priority: Major
> Labels: lhf, pull-request-available
> Fix For: 4.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It would be nice to allow the user to configure an archival script that will
> be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log,
> just as we do in {{CommitLogArchiver}}. The script should be able to copy the
> released file to an external location or do whatever the author hand in mind.
> Deleting the log file should be delegated to the script as well.
> See CASSANDRA-13983, CASSANDRA-12151 for use cases.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[
https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638942#comment-16638942
]
Ariel Weisberg commented on CASSANDRA-14373:
Some quick review feedback.
* [Removing documentation of defaults doesn't seem like a pure win since they
still seem to be
there?|https://github.com/apache/cassandra/compare/trunk...krummas:marcuse/14373?expand=1#diff-e517aca4b5a9a4773e1957fa4574862aR26]
* [This isn't just a path right it's a format specified of sorts with
%path?|https://github.com/apache/cassandra/compare/trunk...krummas:marcuse/14373?expand=1#diff-e517aca4b5a9a4773e1957fa4574862aR44]
* [This is BinLogOptions but the comments reference Audit log and there is a
typo in the first sentence
here|https://github.com/apache/cassandra/compare/trunk...krummas:marcuse/14373?expand=1#diff-8d9638dbfb056fb8020e9b32e60e3a61R33]
* Depending on what the archiving script does and why it failed there could be
unfortunate consequences to retrying repeatedly.
* [exec forks, forking can be slow because of page table copying which in the
was slow under things like
Xen.|https://github.com/apache/cassandra/compare/trunk...krummas:marcuse/14373?expand=1#diff-3a9be5c6d02a55bd3efddb1053dc87dbR160].
I'm just mentioning it. I don't think you need to make it better right now. I
don't know offhand how you invoke an external command more efficiently from
Java.
* [Is this going to enable it for all
tests?|https://github.com/apache/cassandra/compare/trunk...krummas:marcuse/14373?expand=1#diff-b1c721618608f703239339811634ef3dR51]
Is that a good idea can we only enable it for just the unit tests that require
it?
* [Should use execute instead of submit unless consuming the result
future|https://github.com/apache/cassandra/compare/trunk...krummas:marcuse/14373?expand=1#diff-3a9be5c6d02a55bd3efddb1053dc87dbR77]
* [Same here |
https://github.com/apache/cassandra/compare/trunk...krummas:marcuse/14373?expand=1#diff-3a9be5c6d02a55bd3efddb1053dc87dbR120]
I'll look at the dtests now.
> Allow using custom script for chronicle queue BinLog archival
> -
>
> Key: CASSANDRA-14373
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14373
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Stefan Podkowinski
>Assignee: Pramod K Sivaraju
>Priority: Major
> Labels: lhf, pull-request-available
> Fix For: 4.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It would be nice to allow the user to configure an archival script that will
> be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log,
> just as we do in {{CommitLogArchiver}}. The script should be able to copy the
> released file to an external location or do whatever the author hand in mind.
> Deleting the log file should be delegated to the script as well.
> See CASSANDRA-13983, CASSANDRA-12151 for use cases.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[
https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632164#comment-16632164
]
Pramod K Sivaraju commented on CASSANDRA-14373:
---
Thanks [~krummas]. Sorry for the trouble. I should have raised the pull request
earlier in the release cycle. Let me know if any other fixes are required.
> Allow using custom script for chronicle queue BinLog archival
> -
>
> Key: CASSANDRA-14373
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14373
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Stefan Podkowinski
>Assignee: Pramod K Sivaraju
>Priority: Major
> Labels: lhf, pull-request-available
> Fix For: 4.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It would be nice to allow the user to configure an archival script that will
> be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log,
> just as we do in {{CommitLogArchiver}}. The script should be able to copy the
> released file to an external location or do whatever the author hand in mind.
> Deleting the log file should be delegated to the script as well.
> See CASSANDRA-13983, CASSANDRA-12151 for use cases.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[
https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630259#comment-16630259
]
Marcus Eriksson commented on CASSANDRA-14373:
-
hey [~psivaraju] thanks for the patch, I rewrote it a bit here:
[https://github.com/krummas/cassandra/commits/marcuse/14373]
* make sure nodetool can take multi-word arguments (ie, {{nodetool
enablefullquerylog --archive-command "/path/to/executable.sh %path"}} should
work) - my solution required me to change the nodetool interpreter from
{{/bin/sh}} to {{/bin/bash}} to get arrays, I'll try to figure out a way to do
in in posix sh though.
* remove the option to set the --archive-command from enableauditlog - it
wasn't working and it looks like it might need some more refactoring of that
code to get it working
* Break out the archiving logic in to two separate classes
({{DeletingArchiver}} and {{ExternalArchiver}}) for easier testing
* Reworked the configuration to add binlog specific configuration to its own
class
Also wrote a bunch of dtests for audit/fullquerylogging here:
[https://github.com/krummas/cassandra-dtest/commits/marcuse/14373] (these also
required a CCM change which has been
[merged|https://github.com/riptano/ccm/commit/a76e45c1fbe7ebb4ef43de686644c90181e80b29])
I also filed CASSANDRA-14772 to further clean up the audit/fullquerylogging code
Note that 4.0 is currently in feature freeze and one could argue that this is a
new feature, but as it simplifies collecting full query logs for replay
testing, I'd like to get it in 4.0.
> Allow using custom script for chronicle queue BinLog archival
> -
>
> Key: CASSANDRA-14373
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14373
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Stefan Podkowinski
>Assignee: Pramod K Sivaraju
>Priority: Major
> Labels: lhf, pull-request-available
> Fix For: 4.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It would be nice to allow the user to configure an archival script that will
> be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log,
> just as we do in {{CommitLogArchiver}}. The script should be able to copy the
> released file to an external location or do whatever the author hand in mind.
> Deleting the log file should be delegated to the script as well.
> See CASSANDRA-13983, CASSANDRA-12151 for use cases.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[
https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457086#comment-16457086
]
Pramod K Sivaraju commented on CASSANDRA-14373:
---
Thanks [~jasobrown].
> Allow using custom script for chronicle queue BinLog archival
> -
>
> Key: CASSANDRA-14373
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14373
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Stefan Podkowinski
>Assignee: Pramod K Sivaraju
>Priority: Major
> Labels: lhf
> Fix For: 4.x
>
>
> It would be nice to allow the user to configure an archival script that will
> be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log,
> just as we do in {{CommitLogArchiver}}. The script should be able to copy the
> released file to an external location or do whatever the author hand in mind.
> Deleting the log file should be delegated to the script as well.
> See CASSANDRA-13983, CASSANDRA-12151 for use cases.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[
https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457075#comment-16457075
]
Jason Brown commented on CASSANDRA-14373:
-
Assigned to you, [~psivaraju]
> Allow using custom script for chronicle queue BinLog archival
> -
>
> Key: CASSANDRA-14373
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14373
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Stefan Podkowinski
>Assignee: Pramod K Sivaraju
>Priority: Major
> Labels: lhf
> Fix For: 4.x
>
>
> It would be nice to allow the user to configure an archival script that will
> be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log,
> just as we do in {{CommitLogArchiver}}. The script should be able to copy the
> released file to an external location or do whatever the author hand in mind.
> Deleting the log file should be delegated to the script as well.
> See CASSANDRA-13983, CASSANDRA-12151 for use cases.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[
https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457055#comment-16457055
]
Pramod K Sivaraju commented on CASSANDRA-14373:
---
I'm interested in picking up this change request. I will update my progress
here.
> Allow using custom script for chronicle queue BinLog archival
> -
>
> Key: CASSANDRA-14373
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14373
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Stefan Podkowinski
>Priority: Major
> Labels: lhf
> Fix For: 4.x
>
>
> It would be nice to allow the user to configure an archival script that will
> be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log,
> just as we do in {{CommitLogArchiver}}. The script should be able to copy the
> released file to an external location or do whatever the author hand in mind.
> Deleting the log file should be delegated to the script as well.
> See CASSANDRA-13983, CASSANDRA-12151 for use cases.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
