[jira] [Commented] (CASSANDRA-1443) shell scripts are not posix compliant
[
https://issues.apache.org/jira/browse/CASSANDRA-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750863#comment-16750863
]
Murukesh Mohanan commented on CASSANDRA-1443:
-
The syntax itself seems to be POSIX-compliant, but at least one tool used
isn't: {{which}}. {{which}} isn't POSIX; the POSIX way to look up the path of a
command seems to be {{command -v}} (see [this comprehensive post on Unix &
Linux|https://unix.stackexchange.com/a/85250/70524] for details).
Most uses of {{which}} were eliminated by [this patch for making C8 work with
Java 8 and
11|https://github.com/apache/cassandra/commit/6ba2fb9395226491872b41312d978a169f36fcdb];
there are still a few left:
{code}
$ grep 'which.*>' bin tools/bin/ -R
bin/cqlsh.py:which python$pyver > /dev/null 2>&1 && exec python$pyver "$0"
"$@"
bin/cassandra:if which numactl >/dev/null 2>/dev/null && numactl $NUMACTL_ARGS
ls / >/dev/null 2>/dev/null
bin/cassandra:which ldconfig > /dev/null 2>&1
bin/cqlsh:which python$pyver > /dev/null 2>&1 && exec python$pyver
"`python$pyver -c "import
os;print(os.path.dirname(os.path.realpath('$0')))"`/cqlsh.py" "$@"
{code}
It might be worthwhile to eliminate those as well.
---
Before that patch, many C* utilities used this check:
{code}
# Use JAVA_HOME if set, otherwise look for java in PATH
if [ -x "$JAVA_HOME/bin/java" ]; then
JAVA="$JAVA_HOME/bin/java"
else
JAVA="`which java`"
fi
{code}
I noticed this when checking out ScyllaDB; their Docker images don't include
{{which}} and use an older version of various C* tools. Their solution was to
create a {{/bin/java}} symlink since the first check will then succeed (with
{{JAVA_HOME}} unset, that becomes {{/bin/java}}).
> shell scripts are not posix compliant
> -
>
> Key: CASSANDRA-1443
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1443
> Project: Cassandra
> Issue Type: Bug
>Affects Versions: 0.6
>Reporter: Brandon Williams
>Assignee: Brandon Williams
>Priority: Minor
> Fix For: 0.6.6
>
> Attachments: 1443.txt
>
>
> Our shells scripts currently require bash, but invoke /bin/sh which may or
> may not be bash. Instead, we should make these scripts POSIX compliant so
> they work under any shell.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14954) Website documentation for stable and latest, with stable as default linked
[
https://issues.apache.org/jira/browse/CASSANDRA-14954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16735436#comment-16735436
]
Murukesh Mohanan commented on CASSANDRA-14954:
--
Should CASSANDRA-13907 be duped to this, since work is being done here?
> Website documentation for stable and latest, with stable as default linked
> --
>
> Key: CASSANDRA-14954
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14954
> Project: Cassandra
> Issue Type: Task
> Components: Documentation/Website
>Reporter: mck
>Assignee: mck
>Priority: Trivial
> Fix For: 3.11.x, 4.x
>
> Attachments: make-add-stable-doc.patch
>
>
> The website should link Documentation to the docs generated for our most
> recent stable release.
> By providing directory listing (using {{`htaccess Indexes`}}) under /doc/,
> and having two symlinks {{latest}} and {{stable}}, we can by default link to
> {{stable}}.
> The following patch
> - adds to the website Makefile the task {{add-stable-doc}}
> - changes the default documentation link to {{/doc/stable/}}
> - removes the html redirecting from {{doc/ --> doc/latest/}}
> - adds directory listing to {{/doc/}} for a simple view of versioned docs
> available
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-13960) update cassandra.yaml links to point to new docs instead of the wiki
[
https://issues.apache.org/jira/browse/CASSANDRA-13960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Murukesh Mohanan reassigned CASSANDRA-13960:
Assignee: Gabriel Nobrega
> update cassandra.yaml links to point to new docs instead of the wiki
>
>
> Key: CASSANDRA-13960
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13960
> Project: Cassandra
> Issue Type: Improvement
> Components: Documentation and Website
>Reporter: Jon Haddad
>Assignee: Gabriel Nobrega
>Priority: Minor
> Labels: lhf
>
> The wiki is dead, let's clean up the links and be sure we have the right into
> on cassandra.apache.org.
> {code}
> # NOTE:
> # See http://wiki.apache.org/cassandra/StorageConfiguration for
> # full explanations of configuration directives
> # /NOTE
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-10989) Move away from SEDA to TPC
[
https://issues.apache.org/jira/browse/CASSANDRA-10989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Murukesh Mohanan updated CASSANDRA-10989:
-
Description:
Since its inception, Cassandra has been utilising
[SEDA|http://www.mdw.la/papers/seda-sosp01.pdf] at its core.
As originally conceived, it means every request is split into several stages,
and each stage is backed by a thread pool. That imposes certain challenges:
- thread parking/unparking overheads (partially improved by SEPExecutor in
CASSANDRA-4718)
- extensive context switching (i-/d- caches thrashing)
- less than optimal multiple writer/multiple reader data structures for
memtables, partitions, metrics, more
- hard to grok concurrent code
- large number of GC roots, longer TTSP
- increased complexity for moving data structures off java heap
- inability to easily balance writes/reads/compaction/flushing
Latency implications of SEDA have been acknowledged by the authors themselves -
see 2010 [retrospective on
SEDA|http://matt-welsh.blogspot.co.uk/2010/07/retrospective-on-seda.html].
To fix these issues (and more), two years ago at NGCC [~benedict] suggested
moving Cassandra away from SEDA to the more mechanically sympathetic thread per
core architecture (TPC). See the slides from the original presentation
[here|https://docs.google.com/presentation/d/19_U8I7mq9JKBjgPmmi6Hri3y308QEx1FmXLt-53QqEw/edit?ts=56265eb4#slide=id.g98ad32b25_1_19].
In a nutshell, each core would become a logical shared nothing micro instance
of Cassandra, taking over a portion of the node’s range {{*}}.
Client connections will be assigned randomly to one of the cores (sharing a
single listen socket). A request that cannot be served by the client’s core
will be proxied to the one owning the data, similar to the way we perform
remote coordination today.
Each thread (pinned to an exclusive core) would have a single event loop, and
be responsible for both serving requests and performing maintenance tasks
(flushing, compaction, repair), scheduling them intelligently.
One notable exception from the original proposal is that we cannot,
unfortunately, use linux AIO for file I/O, as it's only properly implemented
for xfs. We might, however, have a specialised implementation for xfs and
Windows (based on IOCP) later. In the meantime, we have no other choice other
than to hand off I/O that cannot be served from cache to a separate threadpool.
Transitioning from SEDA to TPC will be done in stages, incrementally and in
parallel.
This is a high-level overview meta-ticket that will track JIRA issues for each
individual stage.
{{*}} they’ll share certain things still, like schema, gossip, file I/O
threadpool(s), and maybe MessagingService.
was:
Since its inception, Cassandra has been utilising [SEDA
|http://www.eecs.harvard.edu/~mdw/papers/seda-sosp01.pdf] at its core.
As originally conceived, it means every request is split into several stages,
and each stage is backed by a thread pool. That imposes certain challenges:
- thread parking/unparking overheads (partially improved by SEPExecutor in
CASSANDRA-4718)
- extensive context switching (i-/d- caches thrashing)
- less than optimal multiple writer/multiple reader data structures for
memtables, partitions, metrics, more
- hard to grok concurrent code
- large number of GC roots, longer TTSP
- increased complexity for moving data structures off java heap
- inability to easily balance writes/reads/compaction/flushing
Latency implications of SEDA have been acknowledged by the authors themselves -
see 2010 [retrospective on
SEDA|http://matt-welsh.blogspot.co.uk/2010/07/retrospective-on-seda.html].
To fix these issues (and more), two years ago at NGCC [~benedict] suggested
moving Cassandra away from SEDA to the more mechanically sympathetic thread per
core architecture (TPC). See the slides from the original presentation
[here|https://docs.google.com/presentation/d/19_U8I7mq9JKBjgPmmi6Hri3y308QEx1FmXLt-53QqEw/edit?ts=56265eb4#slide=id.g98ad32b25_1_19].
In a nutshell, each core would become a logical shared nothing micro instance
of Cassandra, taking over a portion of the node’s range {{*}}.
Client connections will be assigned randomly to one of the cores (sharing a
single listen socket). A request that cannot be served by the client’s core
will be proxied to the one owning the data, similar to the way we perform
remote coordination today.
Each thread (pinned to an exclusive core) would have a single event loop, and
be responsible for both serving requests and performing maintenance tasks
(flushing, compaction, repair), scheduling them intelligently.
One notable exception from the original proposal is that we cannot,
unfortunately, use linux AIO for file I/O, as it's only properly implemented
for xfs. We might, however, have a specialised implementation for xfs and
Windows (based on IOCP) later. In the meantime,
[jira] [Assigned] (CASSANDRA-13001) pluggable slow query logging / handling
[ https://issues.apache.org/jira/browse/CASSANDRA-13001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Murukesh Mohanan reassigned CASSANDRA-13001: Assignee: Sumanth Pasupuleti (was: Murukesh Mohanan) > pluggable slow query logging / handling > --- > > Key: CASSANDRA-13001 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13001 > Project: Cassandra > Issue Type: New Feature >Reporter: Jon Haddad >Assignee: Sumanth Pasupuleti >Priority: Major > Fix For: 4.0 > > Attachments: > 0001-Add-basic-pluggable-logging-to-debug.log-and-table.patch, > 0001-Add-multiple-logging-methods-for-slow-queries-CASSAN.patch > > > Currently CASSANDRA-12403 logs slow queries as DEBUG to a file. It would be > better to have this as an interface which we can log to alternative > locations, such as to a table on the cluster or to a remote location (statsd, > graphite, etc). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13001) pluggable slow query logging / handling
[ https://issues.apache.org/jira/browse/CASSANDRA-13001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407555#comment-16407555 ] Murukesh Mohanan commented on CASSANDRA-13001: -- Not recently, no. If you want to take it up, please do. I'm willing to help out in anyway I can, though. > pluggable slow query logging / handling > --- > > Key: CASSANDRA-13001 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13001 > Project: Cassandra > Issue Type: New Feature >Reporter: Jon Haddad >Assignee: Murukesh Mohanan >Priority: Major > Fix For: 4.0 > > Attachments: > 0001-Add-basic-pluggable-logging-to-debug.log-and-table.patch, > 0001-Add-multiple-logging-methods-for-slow-queries-CASSAN.patch > > > Currently CASSANDRA-12403 logs slow queries as DEBUG to a file. It would be > better to have this as an interface which we can log to alternative > locations, such as to a table on the cluster or to a remote location (statsd, > graphite, etc). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14312) Create sub directories for version specific documentation
[ https://issues.apache.org/jira/browse/CASSANDRA-14312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16398151#comment-16398151 ] Murukesh Mohanan commented on CASSANDRA-14312: -- Is this related to CASSANDRA-13907? > Create sub directories for version specific documentation > - > > Key: CASSANDRA-14312 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14312 > Project: Cassandra > Issue Type: Improvement >Reporter: Kenneth Brotman >Priority: Major > > I want to create two sub directories "2_x" and "3_x" under "doc". I'll then > copy the files under "doc" to each of the new subdirectories so we can begin > making version specific documentation. No links are changed yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13915) Create a Docker container to build the docs
[
https://issues.apache.org/jira/browse/CASSANDRA-13915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240034#comment-16240034
]
Murukesh Mohanan commented on CASSANDRA-13915:
--
Since the intention here is to ensure dependencies are easily handled, should
we ask for specific versions when doing {{pip install sphinx sphinx_rtd_theme}}
({{requirements.txt}}-style, perhaps)?
I thought of installing both of these using {{apt-get}} - {{apt-get install
python-sphinx python-sphinx-rtd-theme}}, but the {{python:2.7}} Docker image is
based on Debian jessie, where the [sphinx
packages|https://packages.debian.org/jessie/python-sphinx] seem to be pretty
old (1.2.3 vs ~1.6 now).
> Create a Docker container to build the docs
> ---
>
> Key: CASSANDRA-13915
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13915
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Joaquin Casares
>Assignee: Joaquin Casares
>
> As requested by [~rustyrazorblade], I will be adding a Docker container to
> build the docs without any prereqs (other than Docker).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13915) Create a Docker container to build the docs
[
https://issues.apache.org/jira/browse/CASSANDRA-13915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240026#comment-16240026
]
Murukesh Mohanan commented on CASSANDRA-13915:
--
{quote}Is there a linux alternative? Is it even possible to identify the host
OS? I'd be fine with spitting out the open command on mac and a backup "open
build/html/index.html file in your browser to view the docs" message if we
don't know it's mac. Otherwise the backup message all the time works for
me.{quote}
On Linux, [{{xdg-open}}|https://portland.freedesktop.org/doc/xdg-open.html] can
be a safe DE-independent alternative (it wraps around DE-specific
{{gnome-open}}/{{gvfs-open}}, {{kde-open}}, etc.). I've never heard of
{{exe-open}} before. It doesn't seem to be a command [available from any Ubuntu
package|https://packages.ubuntu.com/search?suite=xenial=any=exactfilename=contents=exe-open],
and Googling does not throw anything obvious, so I'm not sure what that is.
> Create a Docker container to build the docs
> ---
>
> Key: CASSANDRA-13915
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13915
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Joaquin Casares
>Assignee: Joaquin Casares
>
> As requested by [~rustyrazorblade], I will be adding a Docker container to
> build the docs without any prereqs (other than Docker).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13907) Versioned documentation on cassandra.apache.org
[ https://issues.apache.org/jira/browse/CASSANDRA-13907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210929#comment-16210929 ] Murukesh Mohanan commented on CASSANDRA-13907: -- Nothing further on labeling the versions from me, then. I'd rather have multiple versions supported than bicker about what to call them. Thanks for the link to the source. Reading the README, I can appreciate why Michael might generate just the trunk and leave it alone. Integrating Jekyll with a non-Jekyll source is a PITA. Having a consistent theme and structure to the docs and the main Cassandra site is important, and that's probably why this devolved to the way it is now. As it always does, it looks like this will take more effort than I originally thought it would. :) In the meantime, I'll try to set it up so that the POC is automatically updated so that versioned docs are available somewhere. > Versioned documentation on cassandra.apache.org > --- > > Key: CASSANDRA-13907 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13907 > Project: Cassandra > Issue Type: Improvement > Components: Documentation and Website >Reporter: Murukesh Mohanan >Priority: Minor > > Services like https://readthedocs.org and http://www.javadoc.io/ make it easy > to browse the documentation for a particular version or commit of various > open source projects. It would be nice to be able to browse the docs for a > particular release on http://cassandra.apache.org/doc/. > Currently it seems only CQL has this at http://cassandra.apache.org/doc/old/. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13907) Versioned documentation on cassandra.apache.org
[
https://issues.apache.org/jira/browse/CASSANDRA-13907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210844#comment-16210844
]
Murukesh Mohanan commented on CASSANDRA-13907:
--
Having all versions would be impractical (I think the services which do
per-commit docs generate them on-demand, but I'm not sure). Since {{make html}}
only worked for 5 tags, I just added them all to my site.
How about something like the Debian system: old-stable (2.X), stable (3.0),
latest (3.11), and trunk (rolling, updated regularly, maybe once a week)? If
links like {{/stable}}, {{/latest}} can be provided, it would be even more
convenient.
Is there something I can do to help with this? Are the scripts for the website
open?
> Versioned documentation on cassandra.apache.org
> ---
>
> Key: CASSANDRA-13907
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13907
> Project: Cassandra
> Issue Type: Improvement
> Components: Documentation and Website
>Reporter: Murukesh Mohanan
>Priority: Minor
>
> Services like https://readthedocs.org and http://www.javadoc.io/ make it easy
> to browse the documentation for a particular version or commit of various
> open source projects. It would be nice to be able to browse the docs for a
> particular release on http://cassandra.apache.org/doc/.
> Currently it seems only CQL has this at http://cassandra.apache.org/doc/old/.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13907) Versioned documentation on cassandra.apache.org
[
https://issues.apache.org/jira/browse/CASSANDRA-13907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209566#comment-16209566
]
Murukesh Mohanan commented on CASSANDRA-13907:
--
As a POC, I created a GitHub Pages site:
https://murukeshm.github.io/cassandra/3.9/ (check the dropdown near the
copyright notice at the bottom of the page). The commands I used to create them
are in https://github.com/murukeshm/cassandra/blob/gh-pages/gen.sh It's a bit
hacky: inserting a {{}} for versions after the HTML docs have been
generated, and I haven't created a central index page yet, but it can serve as
a baseline.
> Versioned documentation on cassandra.apache.org
> ---
>
> Key: CASSANDRA-13907
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13907
> Project: Cassandra
> Issue Type: Improvement
> Components: Documentation and Website
>Reporter: Murukesh Mohanan
>Priority: Minor
>
> Services like https://readthedocs.org and http://www.javadoc.io/ make it easy
> to browse the documentation for a particular version or commit of various
> open source projects. It would be nice to be able to browse the docs for a
> particular release on http://cassandra.apache.org/doc/.
> Currently it seems only CQL has this at http://cassandra.apache.org/doc/old/.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13907) Versioned documentation on cassandra.apache.org
[ https://issues.apache.org/jira/browse/CASSANDRA-13907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209254#comment-16209254 ] Murukesh Mohanan commented on CASSANDRA-13907: -- I do, that's why I think this is possible in the first place. Being able to read the docs without having C* installed, or of a different version than the one you have, is extremely convenient. Think of the versioned manpage browsing sites ([Ubuntu|http://manpages.ubuntu.com], or [Debian|https://manpages.debian.org] or the absolutely fantastic [FreeBSD manpage browser|https://freebsd.org/cgi/man.cgi]), or what's probably the best documentation site of a software project, [the Python docs|https://docs.python.org/] (which are canonical, feature all supported versions, support multiple languages, has excellent search, etc.). Having some documentation online is good ... but given that we have a release every few months or so, the online docs will quickly outpace anything that people will likely have installed (and are, therefore, using). It's not essential, but it's nice to have. > Versioned documentation on cassandra.apache.org > --- > > Key: CASSANDRA-13907 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13907 > Project: Cassandra > Issue Type: Improvement > Components: Documentation and Website >Reporter: Murukesh Mohanan >Priority: Minor > > Services like https://readthedocs.org and http://www.javadoc.io/ make it easy > to browse the documentation for a particular version or commit of various > open source projects. It would be nice to be able to browse the docs for a > particular release on http://cassandra.apache.org/doc/. > Currently it seems only CQL has this at http://cassandra.apache.org/doc/old/. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13863) Speculative retry causes read repair even if read_repair_chance is 0.0.
[
https://issues.apache.org/jira/browse/CASSANDRA-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Murukesh Mohanan updated CASSANDRA-13863:
-
Attachment: 0001-Use-read_repair_chance-when-starting-repairs-due-to-.patch
As a quick fix I tried using {{read_repair_chance}} in the exception handler
for {{DigestMismatchException}}. After running benchmarks with YCSB (default
settings - so {{read_repair_chance}} is 0), {{workloada}}) on 3.0.(8,9,12) and
3.0.12 with the patch, the results averaged across ~50 runs are so:
3.0.8:
{code}
[OVERALL], RunTime(ms), 5287.62
[OVERALL], Throughput(ops/sec), 189.70
[TOTAL_GCS_PS_Scavenge], Count, 1
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 14.47
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.27
[TOTAL_GCS_PS_MarkSweep], Count, 0
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 0
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0
[TOTAL_GCs], Count, 1
[TOTAL_GC_TIME], Time(ms), 14.47
[TOTAL_GC_TIME_%], Time(%), 0.27
[READ], Operations, 502.55
[READ], AverageLatency(us), 2701.96
[READ], MinLatency(us), 1144.75
[READ], MaxLatency(us), 21410.62
[READ], 95thPercentileLatency(us), 4606.09
[READ], 99thPercentileLatency(us), 8593.26
[READ], Return=OK, 502.55
[CLEANUP], Operations, 1
[CLEANUP], AverageLatency(us), 2230368.60
[CLEANUP], MinLatency(us), 2229344.60
[CLEANUP], MaxLatency(us), 2231391.60
[CLEANUP], 95thPercentileLatency(us), 2231391.60
[CLEANUP], 99thPercentileLatency(us), 2231391.60
[UPDATE], Operations, 497.45
[UPDATE], AverageLatency(us), 2118.83
[UPDATE], MinLatency(us), 976.21
[UPDATE], MaxLatency(us), 21953.26
[UPDATE], 95thPercentileLatency(us), 3519.23
[UPDATE], 99thPercentileLatency(us), 7775.53
[UPDATE], Return=OK, 497.45
{code}
3.0.9:
{code}
[OVERALL], RunTime(ms), 5269.64
[OVERALL], Throughput(ops/sec), 190.36
[TOTAL_GCS_PS_Scavenge], Count, 1
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 14.26
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.27
[TOTAL_GCS_PS_MarkSweep], Count, 0
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 0
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0
[TOTAL_GCs], Count, 1
[TOTAL_GC_TIME], Time(ms), 14.26
[TOTAL_GC_TIME_%], Time(%), 0.27
[READ], Operations, 499.26
[READ], AverageLatency(us), 2673.89
[READ], MinLatency(us), 1141.89
[READ], MaxLatency(us), 21053.04
[READ], 95thPercentileLatency(us), 4392.28
[READ], 99thPercentileLatency(us), 8742.70
[READ], Return=OK, 499.26
[CLEANUP], Operations, 1
[CLEANUP], AverageLatency(us), 2230214.04
[CLEANUP], MinLatency(us), 2229190.04
[CLEANUP], MaxLatency(us), 2231237.04
[CLEANUP], 95thPercentileLatency(us), 2231237.04
[CLEANUP], 99thPercentileLatency(us), 2231237.04
[UPDATE], Operations, 500.74
[UPDATE], AverageLatency(us), 2106.96
[UPDATE], MinLatency(us), 967.11
[UPDATE], MaxLatency(us), 21862.40
[UPDATE], 95thPercentileLatency(us), 3477.83
[UPDATE], 99thPercentileLatency(us), 7677.11
[UPDATE], Return=OK, 500.74
{code}
3.0.12:
{code}
[OVERALL], RunTime(ms), 5425.13
[OVERALL], Throughput(ops/sec), 184.86
[TOTAL_GCS_PS_Scavenge], Count, 1
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 17.42
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.32
[TOTAL_GCS_PS_MarkSweep], Count, 0
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 0
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0
[TOTAL_GCs], Count, 1
[TOTAL_GC_TIME], Time(ms), 17.42
[TOTAL_GC_TIME_%], Time(%), 0.32
[READ], Operations, 500.49
[READ], AverageLatency(us), 2805.40
[READ], MinLatency(us), 1158.47
[READ], MaxLatency(us), 24314.62
[READ], 95thPercentileLatency(us), 4903.83
[READ], 99thPercentileLatency(us), 9662.70
[READ], Return=OK, 500.49
[CLEANUP], Operations, 1
[CLEANUP], AverageLatency(us), 2230716.38
[CLEANUP], MinLatency(us), 2229692.38
[CLEANUP], MaxLatency(us), 2231739.38
[CLEANUP], 95thPercentileLatency(us), 2231739.38
[CLEANUP], 99thPercentileLatency(us), 2231739.38
[UPDATE], Operations, 499.51
[UPDATE], AverageLatency(us), 2225.51
[UPDATE], MinLatency(us), 971.92
[UPDATE], MaxLatency(us), 23552.06
[UPDATE], 95thPercentileLatency(us), 3822.02
[UPDATE], 99thPercentileLatency(us), 9153.19
[UPDATE], Return=OK, 499.51
{code}
3.0.12 with patch:
{code}
[OVERALL], RunTime(ms), 5128.40
[OVERALL], Throughput(ops/sec), 195.93
[TOTAL_GCS_PS_Scavenge], Count, 1
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 12.13
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.24
[TOTAL_GCS_PS_MarkSweep], Count, 0
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 0
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0
[TOTAL_GCs], Count, 1
[TOTAL_GC_TIME], Time(ms), 12.13
[TOTAL_GC_TIME_%], Time(%), 0.24
[READ], Operations, 500.79
[READ], AverageLatency(us), 2557.40
[READ], MinLatency(us), 1081.06
[READ], MaxLatency(us), 21607.91
[READ], 95thPercentileLatency(us), 4195.49
[READ], 99thPercentileLatency(us), 7990.74
[READ], Return=OK, 500.79
[CLEANUP], Operations, 1
[CLEANUP], AverageLatency(us), 2229325.28
[CLEANUP], MinLatency(us), 2228301.28
[CLEANUP], MaxLatency(us), 2230348.28
[CLEANUP], 95thPercentileLatency(us), 2230348.28
[CLEANUP], 99thPercentileLatency(us), 2230348.28
[UPDATE],
[jira] [Updated] (CASSANDRA-13000) slow query log analysis tool
[
https://issues.apache.org/jira/browse/CASSANDRA-13000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Murukesh Mohanan updated CASSANDRA-13000:
-
Attachment: 0001-Adds-a-cqldumpslow-tool-which-analyses-the-debug-log.patch
Updated the patch to remove some JSON parsing code - I'd originally included
that to work in tandem with my initial misguided patch for CASSANDRA-13001. It
doesn't make any sense to keep it there. (Also fixed from some code indentation
complaints from flake8.)
> slow query log analysis tool
>
>
> Key: CASSANDRA-13000
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13000
> Project: Cassandra
> Issue Type: New Feature
> Components: Observability
>Reporter: Jon Haddad
>Assignee: Murukesh Mohanan
> Fix For: 4.0
>
> Attachments:
> 0001-Adds-a-cqldumpslow-tool-which-analyses-the-debug-log.patch,
> 0001-Adds-a-cqldumpslow-tool-which-analyses-the-debug-log.patch,
> csqldumpslow.py
>
>
> As a follow up to CASSANDRA-12403, it would be very helpful to have a tool to
> process the slow queries that are logged. In the MySQL world, there's a tool
> called mysqldumpslow, which processes a slow query log, abstracts the
> parameters to prepared statements, and shows the queries which are causing
> problems based on frequency. The {{mysqldumpslow}} utillity shows an
> aggregated count & time statistics spent on slow queries. For instance:
> {code}shell> mysqldumpslow
> Reading mysql slow query log from
> /usr/local/mysql/data/mysqld51-apple-slow.log
> Count: 1 Time=4.32s (4s) Lock=0.00s (0s) Rows=0.0 (0), root[root]@localhost
> insert into t2 select * from t1
> Count: 3 Time=2.53s (7s) Lock=0.00s (0s) Rows=0.0 (0), root[root]@localhost
> insert into t2 select * from t1 limit N
> Count: 3 Time=2.13s (6s) Lock=0.00s (0s) Rows=0.0 (0), root[root]@localhost
> insert into t1 select * from t1{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13002) per table slow query times
[ https://issues.apache.org/jira/browse/CASSANDRA-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192359#comment-16192359 ] Murukesh Mohanan commented on CASSANDRA-13002: -- Just checking to avoid confusion: do the updated patches still cause Cassandra to crash when starting with data from a 3.0 instance? > per table slow query times > -- > > Key: CASSANDRA-13002 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13002 > Project: Cassandra > Issue Type: New Feature > Components: Observability >Reporter: Jon Haddad >Assignee: Murukesh Mohanan > Fix For: 4.x > > Attachments: > 0001-Add-per-table-slow_query_log_timeout_in_ms-property.patch, > 0001-Add-per-table-slow_query_log_timeout_in_ms-property.patch, > 0001-Add-per-table-slow_query_log_timeout_in_ms-property.patch > > > CASSANDRA-12403 made it possible to log slow queries, but the time specified > is a global one. This isn't useful if we know different tables have > different access patterns, as we'll end up with a lot of noise. We should be > able to override the slow query time at a per table level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13002) per table slow query times
[
https://issues.apache.org/jira/browse/CASSANDRA-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16191018#comment-16191018
]
Murukesh Mohanan commented on CASSANDRA-13002:
--
Except for {{CHANGES.txt}}, the patch still applies cleanly on trunk. I've
created an issue and PR for the python driver:
https://datastax-oss.atlassian.net/browse/PYTHON-835
https://github.com/datastax/python-driver/pull/838
> per table slow query times
> --
>
> Key: CASSANDRA-13002
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13002
> Project: Cassandra
> Issue Type: New Feature
> Components: Observability
>Reporter: Jon Haddad
>Assignee: Murukesh Mohanan
> Fix For: 4.x
>
> Attachments:
> 0001-Add-per-table-slow_query_log_timeout_in_ms-property.patch,
> 0001-Add-per-table-slow_query_log_timeout_in_ms-property.patch,
> 0001-Add-per-table-slow_query_log_timeout_in_ms-property.patch
>
>
> CASSANDRA-12403 made it possible to log slow queries, but the time specified
> is a global one. This isn't useful if we know different tables have
> different access patterns, as we'll end up with a lot of noise. We should be
> able to override the slow query time at a per table level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-13907) Versioned documentation on cassandra.apache.org
Murukesh Mohanan created CASSANDRA-13907: Summary: Versioned documentation on cassandra.apache.org Key: CASSANDRA-13907 URL: https://issues.apache.org/jira/browse/CASSANDRA-13907 Project: Cassandra Issue Type: Improvement Components: Documentation and Website Reporter: Murukesh Mohanan Priority: Minor Services like https://readthedocs.org and http://www.javadoc.io/ make it easy to browse the documentation for a particular version or commit of various open source projects. It would be nice to be able to browse the docs for a particular release on http://cassandra.apache.org/doc/. Currently it seems only CQL has this at http://cassandra.apache.org/doc/old/. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13863) Speculative retry causes read repair even if read_repair_chance is 0.0.
[
https://issues.apache.org/jira/browse/CASSANDRA-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181007#comment-16181007
]
Murukesh Mohanan commented on CASSANDRA-13863:
--
How about:
* having the {{DigestMismatchException}} handler get a read repair decision?
While discussing this problem with Wakabayashi-san, I made a quick patch that
does this, but I have the feeling it's not the right way to go about this.
* or, a global flag to disable automatic read repairs? I'm not sure how this
would go - I'm guessing there's quite a few places where read repairs can be
initiated?
> Speculative retry causes read repair even if read_repair_chance is 0.0.
> ---
>
> Key: CASSANDRA-13863
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13863
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Hiro Wakabayashi
>
> {{read_repair_chance = 0.0}} and {{dclocal_read_repair_chance = 0.0}} should
> cause no read repair, but read repair happens with speculative retry. I think
> {{read_repair_chance = 0.0}} and {{dclocal_read_repair_chance = 0.0}} should
> stop read repair completely because the user wants to stop read repair in
> some cases.
> {panel:title=Case 1: TWCS users}
> The
> [documentation|http://cassandra.apache.org/doc/latest/operating/compaction.html?highlight=read_repair_chance]
> states how to disable read repair.
> {quote}While TWCS tries to minimize the impact of comingled data, users
> should attempt to avoid this behavior. Specifically, users should avoid
> queries that explicitly set the timestamp via CQL USING TIMESTAMP.
> Additionally, users should run frequent repairs (which streams data in such a
> way that it does not become comingled), and disable background read repair by
> setting the table’s read_repair_chance and dclocal_read_repair_chance to 0.
> {quote}
> {panel}
> {panel:title=Case 2. Strict SLA for read latency}
> In a peak time, read latency is a key for us but, read repair causes latency
> higher than no read repair. We can use anti entropy repair in off peak time
> for consistency.
> {panel}
>
> Here is my procedure to reproduce the problem.
> h3. 1. Create a cluster and set {{hinted_handoff_enabled}} to false.
> {noformat}
> $ ccm create -v 3.0.14 -n 3 cluster_3.0.14
> $ for h in $(seq 1 3) ; do perl -pi -e 's/hinted_handoff_enabled:
> true/hinted_handoff_enabled: false/'
> ~/.ccm/cluster_3.0.14/node$h/conf/cassandra.yaml ; done
> $ for h in $(seq 1 3) ; do grep "hinted_handoff_enabled:"
> ~/.ccm/cluster_3.0.14/node$h/conf/cassandra.yaml ; done
> hinted_handoff_enabled: false
> hinted_handoff_enabled: false
> hinted_handoff_enabled: false
> $ ccm start{noformat}
> h3. 2. Create a keyspace and a table.
> {noformat}
> $ ccm node1 cqlsh
> DROP KEYSPACE IF EXISTS ks1;
> CREATE KEYSPACE ks1 WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': '3'} AND durable_writes = true;
> CREATE TABLE ks1.t1 (
> key text PRIMARY KEY,
> value blob
> ) WITH bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = 'ALWAYS';
> QUIT;
> {noformat}
> h3. 3. Stop node2 and node3. Insert a row.
> {noformat}
> $ ccm node3 stop && ccm node2 stop && ccm status
> Cluster: 'cluster_3.0.14'
> --
> node1: UP
> node3: DOWN
> node2: DOWN
> $ ccm node1 cqlsh -k ks1 -e "consistency; tracing on; insert into ks1.t1
> (key, value) values ('mmullass', bigintAsBlob(1));"
> Current consistency level is ONE.
> Now Tracing is enabled
> Tracing session: 01d74590-97cb-11e7-8ea7-c1bd4d549501
> activity
>| timestamp | source|
> source_elapsed
> -++---+
>
> Execute CQL3 query | 2017-09-12 23:59:42.316000 | 127.0.0.1 |
> 0
> Parsing insert into ks1.t1 (key, value) values ('mmullass',
[jira] [Assigned] (CASSANDRA-13782) Cassandra RPM has wrong owner for /usr/share directories
[ https://issues.apache.org/jira/browse/CASSANDRA-13782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Murukesh Mohanan reassigned CASSANDRA-13782: Assignee: Sasatani Takenobu > Cassandra RPM has wrong owner for /usr/share directories > > > Key: CASSANDRA-13782 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13782 > Project: Cassandra > Issue Type: Sub-task > Components: Packaging >Reporter: Hannu Kröger >Assignee: Sasatani Takenobu > Labels: lhf > > Some Cassandra RPM directories are owned by cassandra user against the fedora > package guidelines. > Offending lines: > https://github.com/apache/cassandra/blob/trunk/redhat/cassandra.spec#L135-L136 > "Permissions on files MUST be set properly. Inside of /usr, files should be > owned by root:root unless a more specific user or group is needed for > security." > - > https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelines#File_Permissions -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13782) Cassandra RPM has wrong owner for /usr/share directories
[ https://issues.apache.org/jira/browse/CASSANDRA-13782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Murukesh Mohanan updated CASSANDRA-13782: - Issue Type: Sub-task (was: Bug) Parent: CASSANDRA-13433 > Cassandra RPM has wrong owner for /usr/share directories > > > Key: CASSANDRA-13782 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13782 > Project: Cassandra > Issue Type: Sub-task > Components: Packaging >Reporter: Hannu Kröger > Labels: lhf > > Some Cassandra RPM directories are owned by cassandra user against the fedora > package guidelines. > Offending lines: > https://github.com/apache/cassandra/blob/trunk/redhat/cassandra.spec#L135-L136 > "Permissions on files MUST be set properly. Inside of /usr, files should be > owned by root:root unless a more specific user or group is needed for > security." > - > https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelines#File_Permissions -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-9430) Add startup options to cqlshrc
[
https://issues.apache.org/jira/browse/CASSANDRA-9430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143349#comment-16143349
]
Murukesh Mohanan commented on CASSANDRA-9430:
-
(Just a note to anybody interested in implementing this.) If this were to be
implemented with an option like Python's {{-i}}, I still wouldn't expect {{echo
"paging off;" | cqlsh -i}} to give me an interactive shell. That would require
{{cqlsh}} to detect whether it is connected to a TTY (since stdin naturally
would be from the pipe), then provide a prompt on that TTY, and read from that
TTY. That could be done ({{sudo}} is a command which does that), but it would
be quite hacky. I think it would be best if cqlsh didn't try to play guessing
games with stdin, and simply take it as given. Reading from a file given as an
option argument is a different case, of course.
> Add startup options to cqlshrc
> --
>
> Key: CASSANDRA-9430
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9430
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
>Reporter: Jeremy Hanna
>Priority: Minor
> Labels: cqlsh, lhf
>
> There are certain settings that would be nice to set defaults for in the
> cqlshrc file. For example, a user may want to set the paging to off by
> default for their environment. You can't simply do
> {code}
> echo "paging off;" | cqlsh
> {code}
> because this would disable paging and immediately exit cqlsh.
> So it would be nice to have a section of the cqlshrc to include default
> settings on startup.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Resolved] (CASSANDRA-13702) Error on keyspace create/alter if referencing non-existing DC in cluster
[ https://issues.apache.org/jira/browse/CASSANDRA-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Murukesh Mohanan resolved CASSANDRA-13702. -- Resolution: Duplicate Fix Version/s: 4.0 > Error on keyspace create/alter if referencing non-existing DC in cluster > > > Key: CASSANDRA-13702 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13702 > Project: Cassandra > Issue Type: Improvement > Components: CQL, Distributed Metadata >Reporter: Johnny Miller >Priority: Minor > Labels: lhf > Fix For: 4.0 > > > It is possible to create/alter a keyspace using NetworkTopologyStrategy and a > DC that does not exist. It would be great if this was validated to prevent > accidents. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13148) Systemd support for RPM
[
https://issues.apache.org/jira/browse/CASSANDRA-13148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085526#comment-16085526
]
Murukesh Mohanan commented on CASSANDRA-13148:
--
[~hoall] Carrying over the discussion from CASSANDRA-13436, if you want to
start services when Cassandra starts *listening*, you need a slightly more
complex setup. See [this thread on the systemd-devel mailing
list|https://lists.freedesktop.org/archives/systemd-devel/2015-May/032278.html]
or this [Unix & Linux Stack Exchange
post|https://unix.stackexchange.com/q/241962/70524] (both of which
coincidentally happen to be about Cassandra!), for example. The consensus seems
to be essentially:
1. Use an {{ExecStartPost}} command which waits for the relevant port to start
listening, or
2. Use an additional service which waits for the relevant port to start
listening and depend on it
The latter is more flexible, since you can start services dependent on the
various ports (JMX, CQL, etc.) independently.
> Systemd support for RPM
> ---
>
> Key: CASSANDRA-13148
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13148
> Project: Cassandra
> Issue Type: New Feature
> Components: Packaging
>Reporter: Spiros Ioannou
>
> Since CASSANDRA-12967 will provide an RPM file, it would be greatly
> beneficial if this package included systemd startup unit configuration
> instead of the current traditional init-based, which misbehaves on
> RHEL7/CentOS7.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13436) Stopping Cassandra shows status "failed" due to non-zero exit status
[
https://issues.apache.org/jira/browse/CASSANDRA-13436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067859#comment-16067859
]
Murukesh Mohanan commented on CASSANDRA-13436:
--
I had a look at the current init script, and tried writing a systemd
equivalent. This raised some questions:
1. The part that doesn't easily translate to systemd is the JVM search code
(which searches for Java in certain directories and sets the command and
{{JAVA_HOME}} appropriately. Perhaps we should leave it to the system
administrator to set it in {{/etc/default/cassandra}}?
2. The PID file becomes unnecessary since systemd can track the processes
started by a service. But I'm not sure if nothing outside the init script uses
the PID file.
3. The log file could also be considered unnecessary since systemd-journald
will gather up any output. But, again, people might expect the logs to stored
in {{/var/log/cassandra/cassandra.log}}.
So, we could have a minimal systemd unit like so (running cassandra in the
foreground making it a simple service):
{code}
[Unit]
Description=Cassandra is a distributed storage system for structured data.
[Service]
Environment=CASSANDRA_HOME=/usr/share/cassandra
CASSANDRA_CONF=/etc/cassandra/conf
CASSANDRA_INCLUDE=/usr/share/cassandra/cassandra.in.sh
EnvironmentFile=-/etc/default/cassandra
User=cassandra
ExecStart=/usr/sbin/cassandra -f
TimeoutStopSec=25
SuccessExitStatus=0 143
[Install]
WantedBy=multi-user.target
{code}
Or more verbose:
{code}
[Unit]
Description=Cassandra is a distributed storage system for structured data.
[Service]
Environment=CASSANDRA_HOME=/usr/share/cassandra
CASSANDRA_CONF=/etc/cassandra/conf
CASSANDRA_INCLUDE=/usr/share/cassandra/cassandra.in.sh
EnvironmentFile=-/etc/default/cassandra
User=cassandra
PermissionsStartOnly=true
ExecStartPre=/usr/bin/install -m 755 -o cassandra -g cassandra -d
/var/run/cassandra
ExecStart=/bin/sh -c '/usr/sbin/cassandra -p /var/run/cassandra/cassandra.pid >
/var/log/cassandra/cassandra.log 2>&1'
Type=forking
PIDFile=/var/run/cassandra/cassandra.pid
TimeoutStopSec=25
SuccessExitStatus=0 143
[Install]
WantedBy=multi-user.target
{code}
I prefer the simpler unit. Thoughts?
> Stopping Cassandra shows status "failed" due to non-zero exit status
>
>
> Key: CASSANDRA-13436
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13436
> Project: Cassandra
> Issue Type: Sub-task
> Components: Packaging
>Reporter: Stefan Podkowinski
>
> Systemd will monitor the process from the pid file and save the return status
> once if has been stopped. In case the process terminates with a status other
> than zero, it will assume the process terminated abnormaly. Stopping
> Cassandra using the cassandra script will send a kill signal to the JVM
> causing it to terminate. If this happen, the JVM will exit with status 143,
> no matter if shutdown hooks have been executed or not. In order to make
> systemd recognize this as a normal exit code, the following should be added
> to the yet to be created unit file:
> {noformat}
> [Service]
> ...
> SuccessExitStatus=0 143
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13634) Create repository and do initial import for cassandra-dtest
[
https://issues.apache.org/jira/browse/CASSANDRA-13634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Murukesh Mohanan updated CASSANDRA-13634:
-
Issue Type: Sub-task (was: Bug)
Parent: CASSANDRA-13613
> Create repository and do initial import for cassandra-dtest
> ---
>
> Key: CASSANDRA-13634
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13634
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Nate McCall
>Assignee: Nate McCall
>
> Given the completion of CASSANDRA-13584, it's time to create the repository
> and do the initial import of code.
> Note that this repo will be configured similarly our main one in that pull
> requests will go to {{pr@c.a.o}} address.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12748) Cassandra fails to start up if GREP_COLOR is set
[
https://issues.apache.org/jira/browse/CASSANDRA-12748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952955#comment-15952955
]
Murukesh Mohanan commented on CASSANDRA-12748:
--
{{env}} has an {{-i}} option which allows us to run a command with a clean
environment ([this is specified by
POSIX|http://pubs.opengroup.org/onlinepubs/9699919799/utilities/env.html], so
should be portable). Since we don't know what environment variables can affect
commands invoked in our scripts, should we consider re-executing the script
with {{env -i var1="$var1" var2="$var2" ...}} where {{var1}}, {{var2}}, etc.
are explicitly listed out as only those variables of interest to us?
As an example, [Ubuntu's {{service}}
command|http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/trusty/sysvinit/trusty/view/head:/debian/service/service#L102]
calls init scripts with a clean environment, retaining only the {{LANG}},
{{PATH}} and {{TERM}} variables. Of course, we would need to retain more.
> Cassandra fails to start up if GREP_COLOR is set
>
>
> Key: CASSANDRA-12748
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12748
> Project: Cassandra
> Issue Type: Bug
>Reporter: Benjamin Roth
>Priority: Minor
>
> If env var GREP_COLOR is set, CS fails to start as startup command then
> contains ANSI codes.
> See cassandra-env.sh when JVM_OPTS_FILE is parsed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-12622) Snap package of Cassandra
[
https://issues.apache.org/jira/browse/CASSANDRA-12622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935598#comment-15935598
]
Murukesh Mohanan commented on CASSANDRA-12622:
--
I had a go at reviewing this, and there is a major problem: {{snapcraft
cleanbuild}} doesn't build this snap cleanly:
{code}
Copying needed target link from the system /etc/ssl/certs/java/cacerts
Pulling cassandra
Cloning into '/root/parts/cassandra/src'...
done.
Preparing to pull glue
Pulling glue
Preparing to build cassandra
Building cassandra
Buildfile: /root/parts/cassandra/build/build.xml
BUILD FAILED
/root/parts/cassandra/build/build.xml:1051: taskdef A class needed by class
org.krummas.junit.JStackJUnitTask cannot be found:
org/apache/tools/ant/taskdefs/optional/junit/JUnitTask
using the classloader
AntClassLoader[/root/parts/cassandra/build/lib/jstackjunit-0.0.1.jar]
Total time: 1 second
Command '['/bin/sh', '/tmp/tmpmfqzqafe', 'ant', 'artifacts',
'-Ddist.dir=/root/parts/cassandra/install']' returned non-zero exit status 1
ant artifacts -Ddist.dir=/root/parts/cassandra/install
Command '['lxc', 'exec', 'snapcraft-yearly-finer-glider', '--', 'snapcraft',
'snap', '--output', 'cassandra_3.10-SNAPSHOT_amd64.snap']' returned non-zero
exit status 1
Stopping snapcraft-yearly-finer-glider
{code}
Now, a plain {{snapcraft}} works, but I had problems with locale [1], but IMHO
from a package maintenance perspective, if a {{cleanbuild}} fails, then the
packaging configuration has problems.
[1]: I was working with {{LC_ALL=C}}, and that causes {{javadoc}} to fail if it
finds Unicode characters in files. I had to use {{LC_ALL=C.UTF-8}} before the
snap built. But, in any case, while building packages a clean-room environment
should be used, and it shouldn't be affected my personal quirky settings.
> Snap package of Cassandra
> -
>
> Key: CASSANDRA-12622
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12622
> Project: Cassandra
> Issue Type: New Feature
> Components: Packaging
>Reporter: Evan
>Priority: Minor
> Fix For: 3.11.x
>
>
> Picking up the conversation from [1], I'd like to propose that Cassandra
> publish snap packages (http://snapcraft.io).
> I've put together a patch:
> https://github.com/apache/cassandra/compare/trunk...evandandrea:snap
> This could be used to build and publish a snap on every commit to trunk [2,
> 3], or as a quicker way for developers to one-off build something more
> lightweight than a container for testing.
> Alternatively, you could keep snap publication to released versions of
> Cassandra. Dependencies are bundled, so you would get to decide Oracle vs
> OpenJDK and the exact version. For the end user it would mean confidence that
> Cassandra with this bundled set of dependencies had been tested by the
> project. Uploads would instantly reach all of Ubuntu and a fair few other
> distributions without any changes [4], hopefully simplifying install
> instructions.
> I couldn't find where the machinery for driving the Cassandra release process
> lives, but if someone can point me in the right direction I'd be happy to
> submit a patch for that.
> 1: https://www.mail-archive.com/dev@cassandra.apache.org/msg09216.html
> 2: Builds of trunk would be best published to the edge channel:
> http://snapcraft.io/#snapcraft_home_using-snaps_channels
> 3: What automatic building and publishing could look like using Travis:
> https://travis-ci.org/evandandrea/cassandra-snap/builds/158449135#L3937
> 4: http://snapcraft.io/docs/core/install
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (CASSANDRA-13262) Incorrect cqlsh results when selecting same columns multiple times
[
https://issues.apache.org/jira/browse/CASSANDRA-13262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Murukesh Mohanan updated CASSANDRA-13262:
-
Attachment: (was:
0001-Fix-incorrect-cqlsh-results-when-selecting-same-colu.patch)
> Incorrect cqlsh results when selecting same columns multiple times
> --
>
> Key: CASSANDRA-13262
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13262
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefan Podkowinski
>Assignee: Murukesh Mohanan
>Priority: Minor
> Labels: lhf
> Attachments:
> 0001-Fix-incorrect-cqlsh-results-when-selecting-same-colu.patch
>
>
> Just stumbled over this on trunk:
> {quote}
> cqlsh:test1> select a, b, c from table1;
> a | b| c
> ---+--+-
> 1 |b | 2
> 2 | null | 2.2
> (2 rows)
> cqlsh:test1> select a, a, b, c from table1;
> a | a| b | c
> ---+--+-+--
> 1 |b | 2 | null
> 2 | null | 2.2 | null
> (2 rows)
> cqlsh:test1> select a, a, a, b, c from table1;
> a | a| a | b| c
> ---+--+---+--+--
> 1 |b | 2.0 | null | null
> 2 | null | 2.2004768 | null | null
> {quote}
> My guess is that his is on the Python side, but haven't really looked into it.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (CASSANDRA-13262) Incorrect cqlsh results when selecting same columns multiple times
[
https://issues.apache.org/jira/browse/CASSANDRA-13262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Murukesh Mohanan updated CASSANDRA-13262:
-
Attachment: 0001-Fix-incorrect-cqlsh-results-when-selecting-same-colu.patch
Sorry for the double upload, there was noise due to some whitespace differences
in the previous one.
> Incorrect cqlsh results when selecting same columns multiple times
> --
>
> Key: CASSANDRA-13262
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13262
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefan Podkowinski
>Assignee: Murukesh Mohanan
>Priority: Minor
> Labels: lhf
> Attachments:
> 0001-Fix-incorrect-cqlsh-results-when-selecting-same-colu.patch
>
>
> Just stumbled over this on trunk:
> {quote}
> cqlsh:test1> select a, b, c from table1;
> a | b| c
> ---+--+-
> 1 |b | 2
> 2 | null | 2.2
> (2 rows)
> cqlsh:test1> select a, a, b, c from table1;
> a | a| b | c
> ---+--+-+--
> 1 |b | 2 | null
> 2 | null | 2.2 | null
> (2 rows)
> cqlsh:test1> select a, a, a, b, c from table1;
> a | a| a | b| c
> ---+--+---+--+--
> 1 |b | 2.0 | null | null
> 2 | null | 2.2004768 | null | null
> {quote}
> My guess is that his is on the Python side, but haven't really looked into it.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (CASSANDRA-13262) Incorrect cqlsh results when selecting same columns multiple times
[
https://issues.apache.org/jira/browse/CASSANDRA-13262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Murukesh Mohanan updated CASSANDRA-13262:
-
Assignee: Murukesh Mohanan
Status: Patch Available (was: Open)
> Incorrect cqlsh results when selecting same columns multiple times
> --
>
> Key: CASSANDRA-13262
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13262
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefan Podkowinski
>Assignee: Murukesh Mohanan
>Priority: Minor
> Labels: lhf
> Attachments:
> 0001-Fix-incorrect-cqlsh-results-when-selecting-same-colu.patch
>
>
> Just stumbled over this on trunk:
> {quote}
> cqlsh:test1> select a, b, c from table1;
> a | b| c
> ---+--+-
> 1 |b | 2
> 2 | null | 2.2
> (2 rows)
> cqlsh:test1> select a, a, b, c from table1;
> a | a| b | c
> ---+--+-+--
> 1 |b | 2 | null
> 2 | null | 2.2 | null
> (2 rows)
> cqlsh:test1> select a, a, a, b, c from table1;
> a | a| a | b| c
> ---+--+---+--+--
> 1 |b | 2.0 | null | null
> 2 | null | 2.2004768 | null | null
> {quote}
> My guess is that his is on the Python side, but haven't really looked into it.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (CASSANDRA-13262) Incorrect cqlsh results when selecting same columns multiple times
[
https://issues.apache.org/jira/browse/CASSANDRA-13262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Murukesh Mohanan updated CASSANDRA-13262:
-
Attachment: 0001-Fix-incorrect-cqlsh-results-when-selecting-same-colu.patch
> Incorrect cqlsh results when selecting same columns multiple times
> --
>
> Key: CASSANDRA-13262
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13262
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefan Podkowinski
>Priority: Minor
> Labels: lhf
> Attachments:
> 0001-Fix-incorrect-cqlsh-results-when-selecting-same-colu.patch
>
>
> Just stumbled over this on trunk:
> {quote}
> cqlsh:test1> select a, b, c from table1;
> a | b| c
> ---+--+-
> 1 |b | 2
> 2 | null | 2.2
> (2 rows)
> cqlsh:test1> select a, a, b, c from table1;
> a | a| b | c
> ---+--+-+--
> 1 |b | 2 | null
> 2 | null | 2.2 | null
> (2 rows)
> cqlsh:test1> select a, a, a, b, c from table1;
> a | a| a | b| c
> ---+--+---+--+--
> 1 |b | 2.0 | null | null
> 2 | null | 2.2004768 | null | null
> {quote}
> My guess is that his is on the Python side, but haven't really looked into it.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13262) Incorrect cqlsh results when selecting same columns multiple times
[
https://issues.apache.org/jira/browse/CASSANDRA-13262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15906894#comment-15906894
]
Murukesh Mohanan commented on CASSANDRA-13262:
--
This is on the Python side, specifically because the results are converted to
an OrderedDict
([bin/cqlsh.py#L500|https://github.com/apache/cassandra/blob/trunk/bin/cqlsh.py#L500]):
{code}
self.session.row_factory = ordered_dict_factory
{code}
Dictionaries of course don't support duplicate keys. The default row_factory is
a named tuple, which also doesn't like duplicate keys, so we have changes to
the key names:
{code}
Row(rack=u'rack1', timeout=5000, rack_=u'rack1')
OrderedDict([(u'rack', u'rack1'), (u'timeout', 5000)])
{code}
The simple fix would be explicitly list the values corresponding to each column
in
[print_static_result()|https://github.com/apache/cassandra/blob/trunk/bin/cqlsh.py#L1115]:
{code}
formatted_values = [map(self.myformat_value, [row[c] for c in column_names],
cql_types) for row in result.current_rows]
{code}
And that sort of negates the point of using an OrderedDict in the first place.
> Incorrect cqlsh results when selecting same columns multiple times
> --
>
> Key: CASSANDRA-13262
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13262
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefan Podkowinski
>Priority: Minor
> Labels: lhf
>
> Just stumbled over this on trunk:
> {quote}
> cqlsh:test1> select a, b, c from table1;
> a | b| c
> ---+--+-
> 1 |b | 2
> 2 | null | 2.2
> (2 rows)
> cqlsh:test1> select a, a, b, c from table1;
> a | a| b | c
> ---+--+-+--
> 1 |b | 2 | null
> 2 | null | 2.2 | null
> (2 rows)
> cqlsh:test1> select a, a, a, b, c from table1;
> a | a| a | b| c
> ---+--+---+--+--
> 1 |b | 2.0 | null | null
> 2 | null | 2.2004768 | null | null
> {quote}
> My guess is that his is on the Python side, but haven't really looked into it.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (CASSANDRA-13275) Cassandra throws an exception during CQL select query filtering on map key
[
https://issues.apache.org/jira/browse/CASSANDRA-13275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Murukesh Mohanan updated CASSANDRA-13275:
-
Description:
Env: cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4
Using this table structure:
{code}CREATE TABLE mytable (
mymap frozen>> PRIMARY KEY
)
{code}
Executing:
{code} select * from mytable where mymap contains key UUID;
{code}
Within cqlsh shows this message:
{code}
ServerError: java.lang.UnsupportedOperationException
system.log:
java.lang.UnsupportedOperationException: null
at
org.apache.cassandra.cql3.restrictions.SingleColumnRestriction$ContainsRestriction.appendTo(SingleColumnRestriction.java:456)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.restrictions.PartitionKeySingleRestrictionSet.values(PartitionKeySingleRestrictionSet.java:86)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.restrictions.StatementRestrictions.getPartitionKeys(StatementRestrictions.java:585)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.statements.SelectStatement.getSliceCommands(SelectStatement.java:474)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.statements.SelectStatement.getQuery(SelectStatement.java:262)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:227)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:76)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:219)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:204)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513)
[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407)
[apache-cassandra-3.9.jar:3.9]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_121]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
[apache-cassandra-3.9.jar:3.9]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
{code}
was:
Env : [cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]
Using this table structure :
CREATE TABLE mytable (
mymap frozen>> PRIMARY KEY
)
Executing :
select * from mytable where mymap contains key UUID;
Within cqlsh shows this message :
ServerError: java.lang.UnsupportedOperationException
system.log:
java.lang.UnsupportedOperationException: null
at
org.apache.cassandra.cql3.restrictions.SingleColumnRestriction$ContainsRestriction.appendTo(SingleColumnRestriction.java:456)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.restrictions.PartitionKeySingleRestrictionSet.values(PartitionKeySingleRestrictionSet.java:86)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.restrictions.StatementRestrictions.getPartitionKeys(StatementRestrictions.java:585)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.statements.SelectStatement.getSliceCommands(SelectStatement.java:474)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.statements.SelectStatement.getQuery(SelectStatement.java:262)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:227)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:76)
~[apache-cassandra-3.9.jar:3.9]
at
[jira] [Updated] (CASSANDRA-13001) pluggable slow query logging / handling
[
https://issues.apache.org/jira/browse/CASSANDRA-13001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Murukesh Mohanan updated CASSANDRA-13001:
-
Attachment: 0001-Add-basic-pluggable-logging-to-debug.log-and-table.patch
Jon, can you go over the new patch and see if I'm headed in the right direction
this time?
Essentially:
- a new interface {{o.a.c.db.monitoring.IQueryLogger}}, with one function
{{void logQueries(MonitoringTask.AggregatedOperations operations,
MonitoringTask.OperationType type, long elapsed)}}, accordingly, various
elements in {{MonitoringTask}} have been made {{public}}
- Two classes implementing it, {{o.a.c.d.m.QueryDebugLogger}} and
o.a.c.d.m.QueryTableLogger}}. The former implements the current
{{debug.log}} behaviour and is default, and the latter saves stats and metadata
to a table (hard-coded to {{perf.slow_log}}, looking for better ideas on this).
- New configuration setting {{slow_query_logger}}, works like {{authenticator}}.
- Rolled up both timed out and slow operation logging in {{MonitoringTask}} to
use the interface. Accordingly cleaned up {{MonitoringTask}}. Perhaps the
configuration setting should be renamed, again looking for ideas.
The patch is far from ready, but ...
> pluggable slow query logging / handling
> ---
>
> Key: CASSANDRA-13001
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13001
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Jon Haddad
>Assignee: Murukesh Mohanan
> Fix For: 4.0
>
> Attachments:
> 0001-Add-basic-pluggable-logging-to-debug.log-and-table.patch,
> 0001-Add-multiple-logging-methods-for-slow-queries-CASSAN.patch
>
>
> Currently CASSANDRA-12403 logs slow queries as DEBUG to a file. It would be
> better to have this as an interface which we can log to alternative
> locations, such as to a table on the cluster or to a remote location (statsd,
> graphite, etc).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13001) pluggable slow query logging / handling
[
https://issues.apache.org/jira/browse/CASSANDRA-13001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15898600#comment-15898600
]
Murukesh Mohanan commented on CASSANDRA-13001:
--
{{MonitoringTask}} also logs timed-out queries using pretty much identical code
as that used for slow queries. Does it make sense to move both of them to the
same (pluggable) logging method? If so, do I pass something like a {{Type:
"slow"}} or {{Type: "time-out"}} to the method doing the logging?
> pluggable slow query logging / handling
> ---
>
> Key: CASSANDRA-13001
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13001
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Jon Haddad
>Assignee: Murukesh Mohanan
> Fix For: 4.0
>
> Attachments:
> 0001-Add-multiple-logging-methods-for-slow-queries-CASSAN.patch
>
>
> Currently CASSANDRA-12403 logs slow queries as DEBUG to a file. It would be
> better to have this as an interface which we can log to alternative
> locations, such as to a table on the cluster or to a remote location (statsd,
> graphite, etc).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13001) pluggable slow query logging / handling
[ https://issues.apache.org/jira/browse/CASSANDRA-13001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15888378#comment-15888378 ] Murukesh Mohanan commented on CASSANDRA-13001: -- Somewhat. Do you mean something like http://www.datastax.com/dev/blog/pluggable-metrics-reporting-in-cassandra-2-0-2? I considered that and it looked like I'd have to move this out of MonitoringTask. I also tried using Timer objects, but then there's the problem that the actual CQL query isn't available here, but a reconstruction, so I wasn't really sure what I could track against. > pluggable slow query logging / handling > --- > > Key: CASSANDRA-13001 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13001 > Project: Cassandra > Issue Type: New Feature >Reporter: Jon Haddad >Assignee: Murukesh Mohanan > Fix For: 4.0 > > Attachments: > 0001-Add-multiple-logging-methods-for-slow-queries-CASSAN.patch > > > Currently CASSANDRA-12403 logs slow queries as DEBUG to a file. It would be > better to have this as an interface which we can log to alternative > locations, such as to a table on the cluster or to a remote location (statsd, > graphite, etc). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CASSANDRA-13001) pluggable slow query logging / handling
[
https://issues.apache.org/jira/browse/CASSANDRA-13001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Murukesh Mohanan updated CASSANDRA-13001:
-
Assignee: Murukesh Mohanan
Fix Version/s: 4.0
Status: Patch Available (was: Open)
In this patch, I add three new configuration options in {{cassandra.yaml}} to
decide how slow queries are logged:
- {{slow_query_log_methods}} - a string of space-separated logging methods
({{log}}, {{command}} and {{table}}).
1. {{log}} is what's already happening,
2. {{command}} will run an external command and send JSON-encoded
slow-query data to its input. The script I wrote for CASSANDRA-13000 can be a
consumer here. A script can parse the JSON and send data to external servers.
3. {{table}} will save the entries to a specified Cassandra table
- {{slow_query_log_command}} - an array of strings, specifying the path to the
command and any arguments
- {{slow_query_log_table}} - the table where the queries should be logged (in
{{keyspace.table}} format)
Multiple methods can be specified.
> pluggable slow query logging / handling
> ---
>
> Key: CASSANDRA-13001
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13001
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Jon Haddad
>Assignee: Murukesh Mohanan
> Fix For: 4.0
>
> Attachments:
> 0001-Add-multiple-logging-methods-for-slow-queries-CASSAN.patch
>
>
> Currently CASSANDRA-12403 logs slow queries as DEBUG to a file. It would be
> better to have this as an interface which we can log to alternative
> locations, such as to a table on the cluster or to a remote location (statsd,
> graphite, etc).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (CASSANDRA-13001) pluggable slow query logging / handling
[ https://issues.apache.org/jira/browse/CASSANDRA-13001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Murukesh Mohanan updated CASSANDRA-13001: - Attachment: 0001-Add-multiple-logging-methods-for-slow-queries-CASSAN.patch > pluggable slow query logging / handling > --- > > Key: CASSANDRA-13001 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13001 > Project: Cassandra > Issue Type: New Feature >Reporter: Jon Haddad > Attachments: > 0001-Add-multiple-logging-methods-for-slow-queries-CASSAN.patch > > > Currently CASSANDRA-12403 logs slow queries as DEBUG to a file. It would be > better to have this as an interface which we can log to alternative > locations, such as to a table on the cluster or to a remote location (statsd, > graphite, etc). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CASSANDRA-13000) slow query log analysis tool
[
https://issues.apache.org/jira/browse/CASSANDRA-13000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Murukesh Mohanan updated CASSANDRA-13000:
-
Assignee: Murukesh Mohanan
Fix Version/s: 4.0
Status: Patch Available (was: Open)
I have wrapped the previously uploaded script in a patch, placing it in
{{tools/bin}} and creating the bat/shell wrappers for it. I also added a
{{-g/--grep}} option for filtering queries with regex.
> slow query log analysis tool
>
>
> Key: CASSANDRA-13000
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13000
> Project: Cassandra
> Issue Type: New Feature
> Components: Observability
>Reporter: Jon Haddad
>Assignee: Murukesh Mohanan
> Fix For: 4.0
>
> Attachments:
> 0001-Adds-a-cqldumpslow-tool-which-analyses-the-debug-log.patch,
> csqldumpslow.py
>
>
> As a follow up to CASSANDRA-12403, it would be very helpful to have a tool to
> process the slow queries that are logged. In the MySQL world, there's a tool
> called mysqldumpslow, which processes a slow query log, abstracts the
> parameters to prepared statements, and shows the queries which are causing
> problems based on frequency. The {{mysqldumpslow}} utillity shows an
> aggregated count & time statistics spent on slow queries. For instance:
> {code}shell> mysqldumpslow
> Reading mysql slow query log from
> /usr/local/mysql/data/mysqld51-apple-slow.log
> Count: 1 Time=4.32s (4s) Lock=0.00s (0s) Rows=0.0 (0), root[root]@localhost
> insert into t2 select * from t1
> Count: 3 Time=2.53s (7s) Lock=0.00s (0s) Rows=0.0 (0), root[root]@localhost
> insert into t2 select * from t1 limit N
> Count: 3 Time=2.13s (6s) Lock=0.00s (0s) Rows=0.0 (0), root[root]@localhost
> insert into t1 select * from t1{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (CASSANDRA-13000) slow query log analysis tool
[
https://issues.apache.org/jira/browse/CASSANDRA-13000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Murukesh Mohanan updated CASSANDRA-13000:
-
Attachment: 0001-Adds-a-cqldumpslow-tool-which-analyses-the-debug-log.patch
> slow query log analysis tool
>
>
> Key: CASSANDRA-13000
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13000
> Project: Cassandra
> Issue Type: New Feature
> Components: Observability
>Reporter: Jon Haddad
> Attachments:
> 0001-Adds-a-cqldumpslow-tool-which-analyses-the-debug-log.patch,
> csqldumpslow.py
>
>
> As a follow up to CASSANDRA-12403, it would be very helpful to have a tool to
> process the slow queries that are logged. In the MySQL world, there's a tool
> called mysqldumpslow, which processes a slow query log, abstracts the
> parameters to prepared statements, and shows the queries which are causing
> problems based on frequency. The {{mysqldumpslow}} utillity shows an
> aggregated count & time statistics spent on slow queries. For instance:
> {code}shell> mysqldumpslow
> Reading mysql slow query log from
> /usr/local/mysql/data/mysqld51-apple-slow.log
> Count: 1 Time=4.32s (4s) Lock=0.00s (0s) Rows=0.0 (0), root[root]@localhost
> insert into t2 select * from t1
> Count: 3 Time=2.53s (7s) Lock=0.00s (0s) Rows=0.0 (0), root[root]@localhost
> insert into t2 select * from t1 limit N
> Count: 3 Time=2.13s (6s) Lock=0.00s (0s) Rows=0.0 (0), root[root]@localhost
> insert into t1 select * from t1{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (CASSANDRA-13002) per table slow query times
[
https://issues.apache.org/jira/browse/CASSANDRA-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Murukesh Mohanan updated CASSANDRA-13002:
-
Attachment: 0001-Add-per-table-slow_query_log_timeout_in_ms-property.patch
I added a description of the new property to the table of options in
{{ddl.rst}}.
> per table slow query times
> --
>
> Key: CASSANDRA-13002
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13002
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Jon Haddad
>Assignee: Murukesh Mohanan
> Fix For: 4.x
>
> Attachments:
> 0001-Add-per-table-slow_query_log_timeout_in_ms-property.patch,
> 0001-Add-per-table-slow_query_log_timeout_in_ms-property.patch,
> 0001-Add-per-table-slow_query_log_timeout_in_ms-property.patch
>
>
> CASSANDRA-12403 made it possible to log slow queries, but the time specified
> is a global one. This isn't useful if we know different tables have
> different access patterns, as we'll end up with a lot of noise. We should be
> able to override the slow query time at a per table level.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (CASSANDRA-13002) per table slow query times
[
https://issues.apache.org/jira/browse/CASSANDRA-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Murukesh Mohanan updated CASSANDRA-13002:
-
Attachment: 0001-Add-per-table-slow_query_log_timeout_in_ms-property.patch
:facepalm: I forgot a check in {{createTableParamsFromRow}}. However, I can't
even get it apply cleanly on {{cassandra-3.0}}, so I tested off {{trunk}}.
Where do I add user documentation? {{doc/source/cql}}, or
{{doc/source/operating}}, or {{doc/cql3/CQL.textile}} (or somewhere else
altogether)?
For {{DESC Table}}, the python cassandra-driver will need to be patched. Since
that is in a separate Github repo, I'm not sure how to coordinate changes here
and there. (Relevant mailing list discussion that led to this discovery:
https://lists.apache.org/thread.html/ed138af46f48033aa7dfed679b52c2001e42b04f55f91214a0bc4c1e@%3Cdev.cassandra.apache.org%3E)
> per table slow query times
> --
>
> Key: CASSANDRA-13002
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13002
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Jon Haddad
>Assignee: Murukesh Mohanan
> Fix For: 4.x
>
> Attachments:
> 0001-Add-per-table-slow_query_log_timeout_in_ms-property.patch,
> 0001-Add-per-table-slow_query_log_timeout_in_ms-property.patch
>
>
> CASSANDRA-12403 made it possible to log slow queries, but the time specified
> is a global one. This isn't useful if we know different tables have
> different access patterns, as we'll end up with a lot of noise. We should be
> able to override the slow query time at a per table level.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (CASSANDRA-13000) slow query log analysis tool
[
https://issues.apache.org/jira/browse/CASSANDRA-13000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Murukesh Mohanan updated CASSANDRA-13000:
-
Attachment: csqldumpslow.py
I have written a Python script that mimics functions of {{mysqldumpslow}}.
Quoting the help text from it:
{code}
usage: csqldumpslow.py [-h] [-s TYPE] [-r] [-t N] [-j] [-o FILE]
[FILE [FILE ...]]
Provide a summary of the slow queries listed in Cassandra debug logs.
Multiple log files can be provided, in which case the logs are combined.
positional arguments:
FILE Input files. Standrad input is -. Default:
logs/debug.log
optional arguments:
-h, --helpshow this help message and exit
-s TYPE, --sort TYPE Sort the input by TYPE
-r, --reverse Reverse the sort order
-t N, --top N Print only the top N queries
-j, --jsonAssume JSON-encoded input
-o FILE, --output FILE
Save output to FILE
Sorting types:
t - total time
at - average time
c - count
{code}
Some of the information available in MySQL's logs are not available (or
applicable) here. Accordingly, I haven't tried to implement options from
{{mysqldumpslow}} which use those.
I thought about implementing the {{-g}} option, but it seems the query string
printed out doesn't always match the actual query, so I don't know how useful
it would be.
With input from [Code Review Stack
Exchange|http://codereview.stackexchange.com/questions/155563/cassandra-slow-query-log-analysis-tool].
> slow query log analysis tool
>
>
> Key: CASSANDRA-13000
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13000
> Project: Cassandra
> Issue Type: New Feature
> Components: Observability
>Reporter: Jon Haddad
> Attachments: csqldumpslow.py
>
>
> As a follow up to CASSANDRA-12403, it would be very helpful to have a tool to
> process the slow queries that are logged. In the MySQL world, there's a tool
> called mysqldumpslow, which processes a slow query log, abstracts the
> parameters to prepared statements, and shows the queries which are causing
> problems based on frequency. The {{mysqldumpslow}} utillity shows an
> aggregated count & time statistics spent on slow queries. For instance:
> {code}shell> mysqldumpslow
> Reading mysql slow query log from
> /usr/local/mysql/data/mysqld51-apple-slow.log
> Count: 1 Time=4.32s (4s) Lock=0.00s (0s) Rows=0.0 (0), root[root]@localhost
> insert into t2 select * from t1
> Count: 3 Time=2.53s (7s) Lock=0.00s (0s) Rows=0.0 (0), root[root]@localhost
> insert into t2 select * from t1 limit N
> Count: 3 Time=2.13s (6s) Lock=0.00s (0s) Rows=0.0 (0), root[root]@localhost
> insert into t1 select * from t1{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13001) pluggable slow query logging / handling
[
https://issues.apache.org/jira/browse/CASSANDRA-13001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865514#comment-15865514
]
Murukesh Mohanan commented on CASSANDRA-13001:
--
I have a hacky WIP patch for this, but, looking at the Datastax
{{node_slow_log}}, I think it falls woefully short of what Datastax already
does. The logging is currently done in the {{MonitorableImpl}} class, which
doesn't, as far as I can tell, have access to that much information. Worse,
even the query isn't really available, as [the comment on function that
provides access to
it|https://github.com/apache/cassandra/blob/cassandra-3.10/src/java/org/apache/cassandra/db/ReadCommand.java#L616]
notes.
Right now my changes are confined to a single file, but doing what
{{node_slow_log}} provides would be more invasive. Thoughts? Should I try for
it?
> pluggable slow query logging / handling
> ---
>
> Key: CASSANDRA-13001
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13001
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Jon Haddad
>
> Currently CASSANDRA-12403 logs slow queries as DEBUG to a file. It would be
> better to have this as an interface which we can log to alternative
> locations, such as to a table on the cluster or to a remote location (statsd,
> graphite, etc).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (CASSANDRA-13002) per table slow query times
[ https://issues.apache.org/jira/browse/CASSANDRA-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Murukesh Mohanan updated CASSANDRA-13002: - Attachment: 0001-Add-per-table-slow_query_log_timeout_in_ms-property.patch > per table slow query times > -- > > Key: CASSANDRA-13002 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13002 > Project: Cassandra > Issue Type: New Feature >Reporter: Jon Haddad >Assignee: Murukesh Mohanan > Fix For: 4.x > > Attachments: > 0001-Add-per-table-slow_query_log_timeout_in_ms-property.patch > > > CASSANDRA-12403 made it possible to log slow queries, but the time specified > is a global one. This isn't useful if we know different tables have > different access patterns, as we'll end up with a lot of noise. We should be > able to override the slow query time at a per table level. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (CASSANDRA-13002) per table slow query times
[
https://issues.apache.org/jira/browse/CASSANDRA-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Murukesh Mohanan updated CASSANDRA-13002:
-
Assignee: Murukesh Mohanan
Fix Version/s: 4.x
Status: Patch Available (was: Open)
I have implemented this by adding a table property with the same name as the
global one {{slow_query_log_timeout_in_ms}}, which overrides the global one if
it is not negative. So, with this, if we wanted to:
- use a different value (say 200ms) for a table foo.bar:
{code}
ALTER TABLE foo.bar with slow_query_log_timeout_in_ms = 200;
{code}
- disable it for a table foo.bar:
{code}
ALTER TABLE foo.bar with slow_query_log_timeout_in_ms = 0;
{code}
- go back to using the global value:
{code}
ALTER TABLE foo.bar with slow_query_log_timeout_in_ms = -1;
{code}
Thoughts?
> per table slow query times
> --
>
> Key: CASSANDRA-13002
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13002
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Jon Haddad
>Assignee: Murukesh Mohanan
> Fix For: 4.x
>
>
> CASSANDRA-12403 made it possible to log slow queries, but the time specified
> is a global one. This isn't useful if we know different tables have
> different access patterns, as we'll end up with a lot of noise. We should be
> able to override the slow query time at a per table level.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (CASSANDRA-12244) progress in compactionstats is reported wrongly for view builds
[
https://issues.apache.org/jira/browse/CASSANDRA-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15656122#comment-15656122
]
Murukesh Mohanan commented on CASSANDRA-12244:
--
[~jasonstack] IMHO the ranges don't need to be sorted. You could do something
like:
{code}
for (Range range :
StorageService.instance.getLocalRanges(baseCfs.keyspace.getName()))
{
rangesTotal++;
// Each range whose right boundary (largest value) is less than the
lastToken can be considered as seen
// by the node.
if ((lastToken != null) && lastToken.compareTo(range.right) > 0)
rangesCompleted++;
}
{code}
Making a collection and sorting it is a waste when you need to iterate over the
ranges only once.
> progress in compactionstats is reported wrongly for view builds
> ---
>
> Key: CASSANDRA-12244
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12244
> Project: Cassandra
> Issue Type: Bug
>Reporter: Tom van der Woerdt
>Assignee: ZhaoYang
>Priority: Minor
> Labels: lhf
> Fix For: 3.0.9
>
>
> In the view build progress given by compactionstats, there are several issues
> :
> {code}
> id compaction type keyspace
> table completed total unit progress
>038d3690-4dbe-11e6-b207-21ec388d48e6View build mykeyspace
> mytable 844 bytes 967 bytes ranges 87.28%
> Active compaction remaining time :n/a
> {code}
> 1) those are ranges, not bytes
> 2) it's not at 87.28%, it's at ~4%. the method for calculating progress in
> Cassandra is wrong: it neglects to sort the tokens it's iterating through
> (ViewBuilder.java) and thus ends up with a random number.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12273) Casandra stress graph: option to create directory for graph if it doesn't exist
[
https://issues.apache.org/jira/browse/CASSANDRA-12273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15652591#comment-15652591
]
Murukesh Mohanan commented on CASSANDRA-12273:
--
[~jkni], I'd be happy to look into the {{hdrfile}} thing. Just to confirm, the
file to be attached would look like:
{code}
>From b98e9b4de506fd0b344d9aaf4cb380651a0a68a0 Mon Sep 17 00:00:00 2001
From: Murukesh Mohanan
Date: Thu, 10 Nov 2016 09:58:34 +0900
Subject: [PATCH] Create log directories as needed, handling symbolic links
(CASSANDRA-12273)
patch by Murukesh Mohanan; reviewed by Joel Knighton
---
CHANGES.txt | 1 +
.../stress/src/org/apache/cassandra/stress/StressGraph.java | 13 +
2 files changed, 14 insertions(+)
diff --git a/CHANGES.txt b/CHANGES.txt
index 69a05c2..ac2e9c3 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
4.0
+ * Create log directories as needed, handling symbolic links (CASSANDRA-12273)
* Add column definition kind to dropped columns in schema (CASSANDRA-12705)
* Add (automate) Nodetool Documentation (CASSANDRA-12672)
* Update bundled cqlsh python driver to 3.7.0 (CASSANDRA-12736)
diff --git a/tools/stress/src/org/apache/cassandra/stress/StressGraph.java
b/tools/stress/src/org/apache/cassandra/stress/StressGraph.java
index 17b718d..a4d9744 100644
...
{code}
> Casandra stress graph: option to create directory for graph if it doesn't
> exist
> ---
>
> Key: CASSANDRA-12273
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12273
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
>Reporter: Christopher Batey
>Assignee: Murukesh Mohanan
>Priority: Minor
> Labels: lhf
> Attachments: 12273.patch
>
>
> I am running it in CI with ephemeral workspace / build dirs. It would be
> nice if CS would create the directory so my build tool doesn't have to
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11888) cassandra-env.sh may not work properly with jvm args containing a space
[
https://issues.apache.org/jira/browse/CASSANDRA-11888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603777#comment-15603777
]
Murukesh Mohanan commented on CASSANDRA-11888:
--
The {{bin/cassandra}} script relies on unquoted expansion of {{$JVM_OPTS}} to
split supplied options to separate arguments. Unfortunately, there's no simple
way to safeguard spaces in an environment variable that gets included in this
unquoted expansion (no combination of quoting or backslash-escaping can help).
The simplest way out of this problem is to use a script whenever the command
you need to run contains whitespace or globs ({{?}}, {{*}}, etc.).
More modern shells support arrays, which can help with keeping arguments
separate, but the first step of dealing with multiple options in
{{$JVM_EXTRA_OPTS}} which are separated by whitespace, and any of which might
contain whitespace themselves remains a problem. This cannot be fixed by simple
changes (options include reading in all options from a file and using something
like {{xargs}} to create the command, or restricting {{$JVM_EXTRA_OPTS}} to a
single option so that whitespace can be preserved for it, or some ugly hack
involving {{eval}}).
{{bin/cassandra}} also has this note at the start, after the boilerplate:
{code}
# NB: Developers should be aware that this script should remain compatible with
# POSIX sh and Solaris sh. This means, in particular, no $(( )) and no $( ).
{code}
This means using a modern shell is out of the question, so something like the
following can't be used, where we take advantage of bash's {{read}}'s ability
to handle escaped input:
{code}
bash-4.4$ JVM_EXTRA_OPTS='-XX:foo\ bar\ \*'
bash-4.4$ JVM_OPTS='-XX:blah'
bash-4.4$ JVM_OPTS="$JVM_OPTS $JVM_EXTRA_OPTS"
bash-4.4$ printf "%s\n" "$JVM_OPTS" | while read -a args; do printf "|%s|\n"
"${args[@]}" ; done
|-XX:blah|
|-XX:foo bar *|
{code}
Workarounds like reserving a character like tab for this could also break
things for others who rely on the default IFS (space, tab, newline).
I suppose this issue should be closed as won't fix.
Related: [Is it possible to “protect” an IFS character from field splitting? -
Unix & Linux|http://unix.stackexchange.com/q/318460/70524]
> cassandra-env.sh may not work properly with jvm args containing a space
> ---
>
> Key: CASSANDRA-11888
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11888
> Project: Cassandra
> Issue Type: Bug
> Environment: linux
>Reporter: Russ Hatch
>Priority: Minor
> Labels: lhf
>
> When using the JVM_EXTRA_OPTS environment variable, it looks like there's
> some cases that may not work if an env var contains a space.
> For example, setting:
> {noformat}
> export JVM_EXTRA_OPTS='-XX:OnOutOfMemoryError="echo oh_no"'
> {noformat}
> Results in the jvm not starting because the resultant startup command looks
> to java like it should load a class called oh_no.
> {noformat}
> Error: Could not find or load main class oh_no
> {noformat}
> I think this results from the last line of cassandra-env.sh, where it does
> this:
> {noformat}
> JVM_OPTS="$JVM_OPTS $JVM_EXTRA_OPTS"
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12273) Casandra stess graph: option to create directory for graph if it doesn't exist
[ https://issues.apache.org/jira/browse/CASSANDRA-12273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Murukesh Mohanan updated CASSANDRA-12273: - Status: Patch Available (was: Open) > Casandra stess graph: option to create directory for graph if it doesn't exist > -- > > Key: CASSANDRA-12273 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12273 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Christopher Batey >Assignee: Christopher Batey >Priority: Minor > Labels: lhf > Attachments: 12273.patch > > > I am running it in CI with ephemeral workspace / build dirs. It would be > nice if CS would create the directory so my build tool doesn't have to -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-12273) Casandra stess graph: option to create directory for graph if it doesn't exist
[
https://issues.apache.org/jira/browse/CASSANDRA-12273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15585145#comment-15585145
]
Murukesh Mohanan edited comment on CASSANDRA-12273 at 10/18/16 10:46 AM:
-
The attached patch uses {{Files.createDirectories}} to create any missing
parent directories. {{Files.createDirectories}} will just return {{false}} if
the parent directory path already exists and is a directory, or {{true}} if it
could create the directory, so we can safely ignore the return value and look
only at the exceptions it throws. If the path was a file, it will throw an
exception. Unfortunately, it will *also* throw an exception if the path was a
symbolic link, so I have added a check for that.
Result when running {{tools/bin/cassandra-stress write n=1900 -graph
file=bar/foo.html -port native=9142 > /dev/null; find bar -ls}}:
h4. Before
Case: {{bar/}} exists
{code}
17013858 drwxr-xr-x1 user group3 Oct 18 19:39 bar
17013868 -rw-r--r--1 user group3 Oct 18 19:39
bar/foo.html
{code}
Case: {{bar/}} doesn't exist
{code}
java.lang.RuntimeException: Couldn't write stats html.
at
org.apache.cassandra.stress.StressGraph.generateGraph(StressGraph.java:95)
at org.apache.cassandra.stress.Stress.run(Stress.java:146)
at org.apache.cassandra.stress.Stress.main(Stress.java:62)
find: {{bar/}}: No such file or directory
{code}
Case: {{bar/}} is a file
{code}
java.lang.RuntimeException: Couldn't write stats html.
at
org.apache.cassandra.stress.StressGraph.generateGraph(StressGraph.java:95)
at org.apache.cassandra.stress.Stress.run(Stress.java:146)
at org.apache.cassandra.stress.Stress.main(Stress.java:62)
17014670 -rw-r--r--1 user group0 Oct 18 19:41 bar
{code}
Case: {{bar/}} is a link to non-existing file
{code}
java.lang.RuntimeException: Couldn't write stats html.
at
org.apache.cassandra.stress.StressGraph.generateGraph(StressGraph.java:95)
at org.apache.cassandra.stress.Stress.run(Stress.java:146)
at org.apache.cassandra.stress.Stress.main(Stress.java:62)
17014728 lrwxr-xr-x1 user group8 Oct 18 19:41 bar
-> /nothing
{code}
Case: {{bar/}} is a link to an existing directory
{code}
17014778 lrwxr-xr-x1 user group3 Oct 18 19:41 bar
-> foo
17014760 drwxr-xr-x3 user group 102 Oct 18 19:41 bar
1701484 520 -rw-r--r--1 user group 265292 Oct 18 19:41
bar/foo.html
{code}
h4. After
Case: {{bar/}} exists
{code}
17010940 drwxr-xr-x3 user group 102 Oct 18 19:30 bar
1701099 520 -rw-r--r--1 user group 265295 Oct 18 19:30
bar/foo.html
{code}
Case: {{bar/}} doesn't exist
{code}
17010940 drwxr-xr-x3 user group 102 Oct 18 19:30 bar
1701095 520 -rw-r--r--1 user group 265381 Oct 18 19:30
bar/foo.html
{code}
Case: {{bar/}} is a file
{code}
java.lang.RuntimeException: Parent directory of stats html is a file.
at
org.apache.cassandra.stress.StressGraph.generateGraph(StressGraph.java:104)
at org.apache.cassandra.stress.Stress.run(Stress.java:146)
at org.apache.cassandra.stress.Stress.main(Stress.java:62)
17011000 -rw-r--r--1 user group0 Oct 18 19:30 bar
{code}
Case: {{bar/}} is a link to non-existing file
{code}
java.lang.RuntimeException: Couldn't write stats html.
at
org.apache.cassandra.stress.StressGraph.generateGraph(StressGraph.java:108)
at org.apache.cassandra.stress.Stress.run(Stress.java:146)
at org.apache.cassandra.stress.Stress.main(Stress.java:62)
17011078 lrwxr-xr-x1 user group8 Oct 18 19:30 bar
-> /nothing
{code}
Case: {{bar/}} is a link to an existing directory
{code}
17011128 lrwxr-xr-x1 user group3 Oct 18 19:30 bar
-> foo
1700 drwxr-xr-x3 user group 102 Oct 18 19:31 bar
1701116 520 -rw-r--r--1 user group 265301 Oct 18 19:31
bar/foo.html
{code}
was (Author: muru):
The attached patch uses {{Files.createDirectories}} to create any missing
parent directories. {{Files.createDirectories}} will just return {{false}} if
the parent directory path already exists and is a directory, or {{true}} if it
could create the directory, so we can safely ignore the return value and look
only at the exceptions it throws. If the path was a file, it will throw an
exception. Unfortunately, it will *also* throw an exception if the path was a
symbolic link, so I have added a check for that.
Result when running {{tools/bin/cassandra-stress write n=1900 -graph
file=bar/foo.html -port native=9142 > /dev/null; find bar -ls}}:
h4. Before
Case: {{bar/}} exists
{code}
1701385
[jira] [Updated] (CASSANDRA-12273) Casandra stess graph: option to create directory for graph if it doesn't exist
[
https://issues.apache.org/jira/browse/CASSANDRA-12273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Murukesh Mohanan updated CASSANDRA-12273:
-
Attachment: 12273.patch
The attached patch uses {{Files.createDirectories}} to create any missing
parent directories. {{Files.createDirectories}} will just return {{false}} if
the parent directory path already exists and is a directory, or {{true}} if it
could create the directory, so we can safely ignore the return value and look
only at the exceptions it throws. If the path was a file, it will throw an
exception. Unfortunately, it will *also* throw an exception if the path was a
symbolic link, so I have added a check for that.
Result when running {{tools/bin/cassandra-stress write n=1900 -graph
file=bar/foo.html -port native=9142 > /dev/null; find bar -ls}}:
h4. Before
Case: {{bar/}} exists
{code}
17013858 lrwxr-xr-x1 user group3 Oct 18 19:39 bar
-> foo
{code}
Case: {{bar/}} doesn't exist
{code}
java.lang.RuntimeException: Couldn't write stats html.
at
org.apache.cassandra.stress.StressGraph.generateGraph(StressGraph.java:95)
at org.apache.cassandra.stress.Stress.run(Stress.java:146)
at org.apache.cassandra.stress.Stress.main(Stress.java:62)
find: {{bar/}}: No such file or directory
{code}
Case: {{bar/}} is a file
{code}
java.lang.RuntimeException: Couldn't write stats html.
at
org.apache.cassandra.stress.StressGraph.generateGraph(StressGraph.java:95)
at org.apache.cassandra.stress.Stress.run(Stress.java:146)
at org.apache.cassandra.stress.Stress.main(Stress.java:62)
17014670 -rw-r--r--1 user group0 Oct 18 19:41 bar
{code}
Case: {{bar/}} is a link to non-existing file
{code}
java.lang.RuntimeException: Couldn't write stats html.
at
org.apache.cassandra.stress.StressGraph.generateGraph(StressGraph.java:95)
at org.apache.cassandra.stress.Stress.run(Stress.java:146)
at org.apache.cassandra.stress.Stress.main(Stress.java:62)
17014728 lrwxr-xr-x1 user group8 Oct 18 19:41 bar
-> /nothing
{code}
Case: {{bar/}} is a link to an existing directory
{code}
17014778 lrwxr-xr-x1 user group3 Oct 18 19:41 bar
-> foo
17014760 drwxr-xr-x3 user group 102 Oct 18 19:41 bar
1701484 520 -rw-r--r--1 user group 265292 Oct 18 19:41
bar/foo.html
{code}
h4. After
Case: {{bar/}} exists
{code}
17010940 drwxr-xr-x3 user group 102 Oct 18 19:30 bar
1701099 520 -rw-r--r--1 user group 265295 Oct 18 19:30
bar/foo.html
{code}
Case: {{bar/}} doesn't exist
{code}
17010940 drwxr-xr-x3 user group 102 Oct 18 19:30 bar
1701095 520 -rw-r--r--1 user group 265381 Oct 18 19:30
bar/foo.html
{code}
Case: {{bar/}} is a file
{code}
java.lang.RuntimeException: Parent directory of stats html is a file.
at
org.apache.cassandra.stress.StressGraph.generateGraph(StressGraph.java:104)
at org.apache.cassandra.stress.Stress.run(Stress.java:146)
at org.apache.cassandra.stress.Stress.main(Stress.java:62)
17011000 -rw-r--r--1 user group0 Oct 18 19:30 bar
{code}
Case: {{bar/}} is a link to non-existing file
{code}
java.lang.RuntimeException: Couldn't write stats html.
at
org.apache.cassandra.stress.StressGraph.generateGraph(StressGraph.java:108)
at org.apache.cassandra.stress.Stress.run(Stress.java:146)
at org.apache.cassandra.stress.Stress.main(Stress.java:62)
17011078 lrwxr-xr-x1 user group8 Oct 18 19:30 bar
-> /nothing
{code}
Case: {{bar/}} is a link to an existing directory
{code}
17011128 lrwxr-xr-x1 user group3 Oct 18 19:30 bar
-> foo
1700 drwxr-xr-x3 user group 102 Oct 18 19:31 bar
1701116 520 -rw-r--r--1 user group 265301 Oct 18 19:31
bar/foo.html
{code}
> Casandra stess graph: option to create directory for graph if it doesn't exist
> --
>
> Key: CASSANDRA-12273
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12273
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
>Reporter: Christopher Batey
>Assignee: Christopher Batey
>Priority: Minor
> Labels: lhf
> Attachments: 12273.patch
>
>
> I am running it in CI with ephemeral workspace / build dirs. It would be
> nice if CS would create the directory so my build tool doesn't have to
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
