[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[ https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644633#comment-16644633 ] Sam Tunnicliffe commented on CASSANDRA-14373: - Couple of tiny things: * I think the help text for --archive-command should show the command quoted, not in backticks (it's probably obvious to a user that the command should be quoted, but it doesn't hurt to be explicit) * the title param of the archiveRetries annotation is wrong (think this is used by the help command) otherwise LGTM > Allow using custom script for chronicle queue BinLog archival > - > > Key: CASSANDRA-14373 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14373 > Project: Cassandra > Issue Type: Improvement >Reporter: Stefan Podkowinski >Assignee: Pramod K Sivaraju >Priority: Major > Labels: lhf, pull-request-available > Fix For: 4.x > > Time Spent: 10m > Remaining Estimate: 0h > > It would be nice to allow the user to configure an archival script that will > be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log, > just as we do in {{CommitLogArchiver}}. The script should be able to copy the > released file to an external location or do whatever the author hand in mind. > Deleting the log file should be delegated to the script as well. > See CASSANDRA-13983, CASSANDRA-12151 for use cases. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[ https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643394#comment-16643394 ] Marcus Eriksson commented on CASSANDRA-14373: - bq. are they sequentially numbered? yeah, the file names should have date/time: https://github.com/OpenHFT/Chronicle-Queue#topics-and-queue-files pushed a change to move back to POSIX {{sh}} instead of {{bash}}, [~beobal] could you have a quick lock at that? > Allow using custom script for chronicle queue BinLog archival > - > > Key: CASSANDRA-14373 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14373 > Project: Cassandra > Issue Type: Improvement >Reporter: Stefan Podkowinski >Assignee: Pramod K Sivaraju >Priority: Major > Labels: lhf, pull-request-available > Fix For: 4.x > > Time Spent: 10m > Remaining Estimate: 0h > > It would be nice to allow the user to configure an archival script that will > be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log, > just as we do in {{CommitLogArchiver}}. The script should be able to copy the > released file to an external location or do whatever the author hand in mind. > Deleting the log file should be delegated to the script as well. > See CASSANDRA-13983, CASSANDRA-12151 for use cases. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[ https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642046#comment-16642046 ] Ariel Weisberg commented on CASSANDRA-14373: +1 to the last commit allowing max retries to be configured. > Allow using custom script for chronicle queue BinLog archival > - > > Key: CASSANDRA-14373 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14373 > Project: Cassandra > Issue Type: Improvement >Reporter: Stefan Podkowinski >Assignee: Pramod K Sivaraju >Priority: Major > Labels: lhf, pull-request-available > Fix For: 4.x > > Time Spent: 10m > Remaining Estimate: 0h > > It would be nice to allow the user to configure an archival script that will > be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log, > just as we do in {{CommitLogArchiver}}. The script should be able to copy the > released file to an external location or do whatever the author hand in mind. > Deleting the log file should be delegated to the script as well. > See CASSANDRA-13983, CASSANDRA-12151 for use cases. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[ https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16639945#comment-16639945 ] Ariel Weisberg commented on CASSANDRA-14373: +1 One interesting thing to note is that the retry mechanism is going to reorder the things it is archiving when it supplies them to the archive script. It's probably fine, but I wonder if people can tell when files are missing? Like are they sequentially numbered? > Allow using custom script for chronicle queue BinLog archival > - > > Key: CASSANDRA-14373 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14373 > Project: Cassandra > Issue Type: Improvement >Reporter: Stefan Podkowinski >Assignee: Pramod K Sivaraju >Priority: Major > Labels: lhf, pull-request-available > Fix For: 4.x > > Time Spent: 10m > Remaining Estimate: 0h > > It would be nice to allow the user to configure an archival script that will > be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log, > just as we do in {{CommitLogArchiver}}. The script should be able to copy the > released file to an external location or do whatever the author hand in mind. > Deleting the log file should be delegated to the script as well. > See CASSANDRA-13983, CASSANDRA-12151 for use cases. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[ https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16639846#comment-16639846 ] Marcus Eriksson commented on CASSANDRA-14373: - thanks for the review, pushed a couple of commits to address the comments bq. Removing documentation of defaults doesn't seem like a pure win since they still seem to be there? I removed the defaults from the nodetool command to be able to override options that are set in cassandra.yaml, do you have a suggestion how to do it in a better way? We don't know the actual set defaults in the nodetool command, the user would have to check out cassandra.yaml which might not be very user friendly. bq. This isn't just a path right it's a format specified of sorts with %path? right, changed to 'command' instead bq. This is BinLogOptions but the comments reference Audit log and there is a typo in the first sentence here fixed bq. Depending on what the archiving script does and why it failed there could be unfortunate consequences to retrying repeatedly. added a new configuration param that allows users to set max retries to 0 to avoid this (defaults to 10 retries as retrying forever might also be bad) bq. exec forks, forking can be slow because of page table copying which in the past was slow under things like Xen.. I'm just mentioning it. I don't think you need to make it better right now. I don't know offhand how you invoke an external command more efficiently from Java. yeah not sure what to do here, a quick search tells me ProcessBuilder seems to be the way to do this bq. Is this going to enable it for all tests? Is that a good idea can we only enable it for just the unit tests that require it? yeah, removed, should not be there bq. Should use execute instead of submit unless consuming the result future fixed bq. Same here here we actually wait for the future bq. The dtests are good tests but could they be unit tests since they are single node. In general I agree, but in this case it executes the nodetool command as an end user would, against a running cassandra cluster (well, node, but anyway). I suppose we could stand up a real cluster in a unit test and execute the nodetool script as well, but I assume that would take about as long as doing it using ccm. > Allow using custom script for chronicle queue BinLog archival > - > > Key: CASSANDRA-14373 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14373 > Project: Cassandra > Issue Type: Improvement >Reporter: Stefan Podkowinski >Assignee: Pramod K Sivaraju >Priority: Major > Labels: lhf, pull-request-available > Fix For: 4.x > > Time Spent: 10m > Remaining Estimate: 0h > > It would be nice to allow the user to configure an archival script that will > be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log, > just as we do in {{CommitLogArchiver}}. The script should be able to copy the > released file to an external location or do whatever the author hand in mind. > Deleting the log file should be delegated to the script as well. > See CASSANDRA-13983, CASSANDRA-12151 for use cases. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[ https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16638997#comment-16638997 ] Ariel Weisberg commented on CASSANDRA-14373: The dtests are good tests but could they be unit tests since they are single node. The reason I harp on it is that they are so much slower and not adding more of them is a win. I'm not sure if we can test nodetool from a dtest, but if we could it would be amazing. > Allow using custom script for chronicle queue BinLog archival > - > > Key: CASSANDRA-14373 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14373 > Project: Cassandra > Issue Type: Improvement >Reporter: Stefan Podkowinski >Assignee: Pramod K Sivaraju >Priority: Major > Labels: lhf, pull-request-available > Fix For: 4.x > > Time Spent: 10m > Remaining Estimate: 0h > > It would be nice to allow the user to configure an archival script that will > be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log, > just as we do in {{CommitLogArchiver}}. The script should be able to copy the > released file to an external location or do whatever the author hand in mind. > Deleting the log file should be delegated to the script as well. > See CASSANDRA-13983, CASSANDRA-12151 for use cases. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[ https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16638942#comment-16638942 ] Ariel Weisberg commented on CASSANDRA-14373: Some quick review feedback. * [Removing documentation of defaults doesn't seem like a pure win since they still seem to be there?|https://github.com/apache/cassandra/compare/trunk...krummas:marcuse/14373?expand=1#diff-e517aca4b5a9a4773e1957fa4574862aR26] * [This isn't just a path right it's a format specified of sorts with %path?|https://github.com/apache/cassandra/compare/trunk...krummas:marcuse/14373?expand=1#diff-e517aca4b5a9a4773e1957fa4574862aR44] * [This is BinLogOptions but the comments reference Audit log and there is a typo in the first sentence here|https://github.com/apache/cassandra/compare/trunk...krummas:marcuse/14373?expand=1#diff-8d9638dbfb056fb8020e9b32e60e3a61R33] * Depending on what the archiving script does and why it failed there could be unfortunate consequences to retrying repeatedly. * [exec forks, forking can be slow because of page table copying which in the was slow under things like Xen.|https://github.com/apache/cassandra/compare/trunk...krummas:marcuse/14373?expand=1#diff-3a9be5c6d02a55bd3efddb1053dc87dbR160]. I'm just mentioning it. I don't think you need to make it better right now. I don't know offhand how you invoke an external command more efficiently from Java. * [Is this going to enable it for all tests?|https://github.com/apache/cassandra/compare/trunk...krummas:marcuse/14373?expand=1#diff-b1c721618608f703239339811634ef3dR51] Is that a good idea can we only enable it for just the unit tests that require it? * [Should use execute instead of submit unless consuming the result future|https://github.com/apache/cassandra/compare/trunk...krummas:marcuse/14373?expand=1#diff-3a9be5c6d02a55bd3efddb1053dc87dbR77] * [Same here | https://github.com/apache/cassandra/compare/trunk...krummas:marcuse/14373?expand=1#diff-3a9be5c6d02a55bd3efddb1053dc87dbR120] I'll look at the dtests now. > Allow using custom script for chronicle queue BinLog archival > - > > Key: CASSANDRA-14373 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14373 > Project: Cassandra > Issue Type: Improvement >Reporter: Stefan Podkowinski >Assignee: Pramod K Sivaraju >Priority: Major > Labels: lhf, pull-request-available > Fix For: 4.x > > Time Spent: 10m > Remaining Estimate: 0h > > It would be nice to allow the user to configure an archival script that will > be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log, > just as we do in {{CommitLogArchiver}}. The script should be able to copy the > released file to an external location or do whatever the author hand in mind. > Deleting the log file should be delegated to the script as well. > See CASSANDRA-13983, CASSANDRA-12151 for use cases. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[ https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16632164#comment-16632164 ] Pramod K Sivaraju commented on CASSANDRA-14373: --- Thanks [~krummas]. Sorry for the trouble. I should have raised the pull request earlier in the release cycle. Let me know if any other fixes are required. > Allow using custom script for chronicle queue BinLog archival > - > > Key: CASSANDRA-14373 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14373 > Project: Cassandra > Issue Type: Improvement >Reporter: Stefan Podkowinski >Assignee: Pramod K Sivaraju >Priority: Major > Labels: lhf, pull-request-available > Fix For: 4.x > > Time Spent: 10m > Remaining Estimate: 0h > > It would be nice to allow the user to configure an archival script that will > be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log, > just as we do in {{CommitLogArchiver}}. The script should be able to copy the > released file to an external location or do whatever the author hand in mind. > Deleting the log file should be delegated to the script as well. > See CASSANDRA-13983, CASSANDRA-12151 for use cases. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[ https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16630259#comment-16630259 ] Marcus Eriksson commented on CASSANDRA-14373: - hey [~psivaraju] thanks for the patch, I rewrote it a bit here: [https://github.com/krummas/cassandra/commits/marcuse/14373] * make sure nodetool can take multi-word arguments (ie, {{nodetool enablefullquerylog --archive-command "/path/to/executable.sh %path"}} should work) - my solution required me to change the nodetool interpreter from {{/bin/sh}} to {{/bin/bash}} to get arrays, I'll try to figure out a way to do in in posix sh though. * remove the option to set the --archive-command from enableauditlog - it wasn't working and it looks like it might need some more refactoring of that code to get it working * Break out the archiving logic in to two separate classes ({{DeletingArchiver}} and {{ExternalArchiver}}) for easier testing * Reworked the configuration to add binlog specific configuration to its own class Also wrote a bunch of dtests for audit/fullquerylogging here: [https://github.com/krummas/cassandra-dtest/commits/marcuse/14373] (these also required a CCM change which has been [merged|https://github.com/riptano/ccm/commit/a76e45c1fbe7ebb4ef43de686644c90181e80b29]) I also filed CASSANDRA-14772 to further clean up the audit/fullquerylogging code Note that 4.0 is currently in feature freeze and one could argue that this is a new feature, but as it simplifies collecting full query logs for replay testing, I'd like to get it in 4.0. > Allow using custom script for chronicle queue BinLog archival > - > > Key: CASSANDRA-14373 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14373 > Project: Cassandra > Issue Type: Improvement >Reporter: Stefan Podkowinski >Assignee: Pramod K Sivaraju >Priority: Major > Labels: lhf, pull-request-available > Fix For: 4.x > > Time Spent: 10m > Remaining Estimate: 0h > > It would be nice to allow the user to configure an archival script that will > be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log, > just as we do in {{CommitLogArchiver}}. The script should be able to copy the > released file to an external location or do whatever the author hand in mind. > Deleting the log file should be delegated to the script as well. > See CASSANDRA-13983, CASSANDRA-12151 for use cases. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[ https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16457086#comment-16457086 ] Pramod K Sivaraju commented on CASSANDRA-14373: --- Thanks [~jasobrown]. > Allow using custom script for chronicle queue BinLog archival > - > > Key: CASSANDRA-14373 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14373 > Project: Cassandra > Issue Type: Improvement >Reporter: Stefan Podkowinski >Assignee: Pramod K Sivaraju >Priority: Major > Labels: lhf > Fix For: 4.x > > > It would be nice to allow the user to configure an archival script that will > be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log, > just as we do in {{CommitLogArchiver}}. The script should be able to copy the > released file to an external location or do whatever the author hand in mind. > Deleting the log file should be delegated to the script as well. > See CASSANDRA-13983, CASSANDRA-12151 for use cases. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[ https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16457075#comment-16457075 ] Jason Brown commented on CASSANDRA-14373: - Assigned to you, [~psivaraju] > Allow using custom script for chronicle queue BinLog archival > - > > Key: CASSANDRA-14373 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14373 > Project: Cassandra > Issue Type: Improvement >Reporter: Stefan Podkowinski >Assignee: Pramod K Sivaraju >Priority: Major > Labels: lhf > Fix For: 4.x > > > It would be nice to allow the user to configure an archival script that will > be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log, > just as we do in {{CommitLogArchiver}}. The script should be able to copy the > released file to an external location or do whatever the author hand in mind. > Deleting the log file should be delegated to the script as well. > See CASSANDRA-13983, CASSANDRA-12151 for use cases. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14373) Allow using custom script for chronicle queue BinLog archival
[ https://issues.apache.org/jira/browse/CASSANDRA-14373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16457055#comment-16457055 ] Pramod K Sivaraju commented on CASSANDRA-14373: --- I'm interested in picking up this change request. I will update my progress here. > Allow using custom script for chronicle queue BinLog archival > - > > Key: CASSANDRA-14373 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14373 > Project: Cassandra > Issue Type: Improvement >Reporter: Stefan Podkowinski >Priority: Major > Labels: lhf > Fix For: 4.x > > > It would be nice to allow the user to configure an archival script that will > be executed in {{BinLog.onReleased(cycle, file)}} for every deleted bin log, > just as we do in {{CommitLogArchiver}}. The script should be able to copy the > released file to an external location or do whatever the author hand in mind. > Deleting the log file should be delegated to the script as well. > See CASSANDRA-13983, CASSANDRA-12151 for use cases. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org