[
https://issues.apache.org/jira/browse/CASSANDRA-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14646961#comment-14646961
]
Paulo Motta commented on CASSANDRA-9902:
----------------------------------------
The original error is caused by conflicting JDKs (namely 7 and 8) installed in
the same system, which are normally switched by changing the {{$JAVA_HOME}} and
adding {{$JAVA_HOME\bin}} to the {{$PATH}} environment variable. It may also
necessary to update the registry when switching JDKs by running the following
command on PowerShell:
{noformat}
Set-ItemProperty -Path 'Registry::HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java
Runtime Environment\' CurrentVersion 1.7
{noformat}
When "java" without the parent directory is called via python's
{{subprocess.check_output}} on
[cassandra-dtest/jmxutils.py|https://github.com/riptano/cassandra-dtest/blob/master/jmxutils.py#L80],
the {{$PATH}} env var is not considered due to this
[bug|http://bugs.python.org/issue8557], so instead, the java executable from
{{c:\windows\system32\}} is picked, which can be pointing to another version.
The simple fix is to use the {{$JAVA_HOME}} variable, if present, as done in
this [v1|https://github.com/pauloricardomg/cassandra-dtest/tree/9902-v1] patch.
However, this is not all the fun, after this fix, the following error appears:
{noformat}
Failed to start jolokia agent (command was: C:\Program
Files\Java\jdk1.7.0_76\bin\java -jar lib\jolokia-jvm-1.2.3-agent.jar --host
127.0.0.1 start 1992 --debug --verbose): Command '('C:\\Program
Files\\Java\\jdk1.7.0_76\\bin\\java', '-jar',
'lib\\jolokia-jvm-1.2.3-agent.jar', '--host', '127.0.0.1', 'start', '1992',
'--debug', '--verbose')' returned non-zero exit status 1
Exit status was: 1
Output was: java.util.ServiceConfigurationError:
com.sun.tools.attach.spi.AttachProvider: Provider
sun.tools.attach.WindowsAttachProvider could not be instantiated
InvocationTarget class com.sun.tools.attach.VirtualMachine (command: start) :
null
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.jolokia.jvmagent.client.util.VirtualMachineHandler.attachVirtualMachine(VirtualMachineHandler.java:66)
at org.jolokia.jvmagent.client.AgentLauncher.main(AgentLauncher.java:57)
Caused by: com.sun.tools.attach.AttachNotSupportedException: no providers
installed
at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)
... 6 more
{noformat}
According to this [blog
post|http://wayne-adams.blogspot.com.br/2011/10/jdi-three-ways-to-attach-to-java.html],
this issue happens when the tools.jar is not loaded by the JRE. After
inspecting the jolokia [vm instatiation
code|https://github.com/rhuss/jolokia/blob/aadf5c19c2052bdfa2bad470c55f4c15c1005d26/agent/jvm/src/main/java/org/jolokia/jvmagent/client/util/VirtualMachineHandler.java],
I found that the {{com.sun.tools.attach.VirtualMachine}} class is [dynamically
loaded|https://github.com/rhuss/jolokia/blob/aadf5c19c2052bdfa2bad470c55f4c15c1005d26/agent/jvm/src/main/java/org/jolokia/jvmagent/client/util/ToolsClassFinder.java]
from the tools.jar, however when {{VirtualMachine.attach}} is called, the JVM
is not able to load the {{sun.tools.attach.WindowsAttachProvider}} class
because the tools.jar is not on the classpath (so that's probably a bug with
jolokia, or Java on Windows).
The simple fix is to explicitly add the tools.jar to the jolokia classpath, as
suggested
[here|https://github.com/rhuss/jolokia/blob/aadf5c19c2052bdfa2bad470c55f4c15c1005d26/agent/jvm/src/main/java/org/jolokia/jvmagent/client/util/VirtualMachineHandler.java#L206].
This was done in
[v2|https://github.com/pauloricardomg/cassandra-dtest/tree/9902-v2] patch.
Apparently this seems to make jolokia connect with the testing JVM, however,
this makes the Cassandra JVM crash without java crash logs or heap dumps. The
only thing we have is a windows crash log ([attached
here|https://issues.apache.org/jira/secure/attachment/12747879/win-crashlogs.zip]),
which indicates the JVM crashes because of a [Data Execution
Prevention|https://technet.microsoft.com/en-us/library/cc738483%28v=ws.10%29.aspx]
exception. There is a feature to [whitelist a program from
DEP|http://windows.microsoft.com/en-us/windows7/Change-Data-Execution-Prevention-settings],
but apparently it's not possible to disable DEP for 64-bit programs o_O.
I don't know what might be causing this, if a JVM option that is set by
Cassandra or an actual Windows JVM bug. In order to simplify reproduction, I
made a simple class to reproduce the issue
([VMAttach.java|https://issues.apache.org/jira/secure/attachment/12747880/VMAttach.java]),
which receives the PID of a java process and tries to attach to it using sun
attach tools (the same used by jolokia). In order to compile and run:
{noformat}
javac -cp "$ENV:JAVA_HOME\lib\tools.jar;." VMAttach.java
java -cp "$ENV:JAVA_HOME\lib\tools.jar;." VMAttach.java <pid>
{noformat}
When this program is run against a cassandra process/pid, it makes the
cassandra process crash silently.
Could you please try reproducing this on your box [~JoshuaMcKenzie]? Do you
have any idea of what might be causing this?
> Windows dtest: invalid jolokia agent path
> -----------------------------------------
>
> Key: CASSANDRA-9902
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9902
> Project: Cassandra
> Issue Type: Sub-task
> Reporter: Joshua McKenzie
> Assignee: Paulo Motta
> Priority: Minor
> Labels: Windows
> Fix For: 2.2.x
>
> Attachments: VMAttach.java, win-crashlogs.zip
>
>
> A variety of tests are failing with similar:
> {noformat}
> 05:39:07
> ======================================================================
> 05:39:07 ERROR: change_durable_writes_test
> (configuration_test.TestConfiguration)
> 05:39:07
> ----------------------------------------------------------------------
> 05:39:07 Traceback (most recent call last):
> 05:39:07 File
> "D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\configuration_test.py",
> line 63, in change_durable_writes_test
> 05:39:07 durable_init_size = commitlog_size(durable_node)
> 05:39:07 File
> "D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\configuration_test.py",
> line 127, in commitlog_size
> 05:39:07 with JolokiaAgent(node) as jmx:
> 05:39:07 File
> "D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\jmxutils.py",
> line 188, in __enter__
> 05:39:07 self.start()
> 05:39:07 File
> "D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\jmxutils.py",
> line 80, in start
> 05:39:07 subprocess.check_output(args, stderr=subprocess.STDOUT)
> 05:39:07 File "D:\Python27\lib\subprocess.py", line 573, in check_output
> 05:39:07 raise CalledProcessError(retcode, cmd, output=output)
> 05:39:07 CalledProcessError: Command '('java', '-jar',
> 'lib\\jolokia-jvm-1.2.3-agent.jar', '--host', '127.0.0.1', 'start', '3436')'
> returned non-zero exit status 2
> 05:39:07 -------------------- >> begin captured logging <<
> --------------------
> 05:39:07 dtest: DEBUG: cluster ccm directory: d:\temp\dtest-kjkypw
> 05:39:07 --------------------- >> end captured logging <<
> ---------------------
> {noformat}
> Likely related:
> {noformat}
> 20:54:46 Failed to start jolokia agent (command was: java -jar
> lib\jolokia-jvm-1.2.3-agent.jar --host 127.0.0.1 start 3436): Command
> '('java', '-jar', 'lib\\jolokia-jvm-1.2.3-agent.jar', '--host', '127.0.0.1',
> 'start', '3436')' returned non-zero exit status 2
> 20:54:46 Exit status was: 2
> 20:54:46 Output was: Error: Registry key 'Software\JavaSoft\Java Runtime
> Environment'\CurrentVersion'
> 20:54:46 has value '1.8', but '1.7' is required.
> 20:54:46 Error: could not find java.dll
> 20:54:46 Error: Could not find Java SE Runtime Environment.
> {noformat}
> And at the top of jmxutils.py we see:
> {code}
> JOLOKIA_JAR = os.path.join('lib', 'jolokia-jvm-1.2.3-agent.jar')
> {code}
> So that's not going to work out too well on Windows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
