That‘ what the script should do: it must invoke „cygpath“ to resolve windows paths and convert to Cygwin paths. There are already code paths in it.
I would still recommend to install a real windows python like Jenkins. The main problem here is: windows (ANT) -> Cygwin (python) -> windows (Java) and that is likely to fail and is not really a supported configuration. For me, with native win32 python downloaded as windows installer all works as it should (I did not wait for it to finish). ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen <http://www.thetaphi.de/> http://www.thetaphi.de eMail: [email protected] From: Shai Erera [mailto:[email protected]] Sent: Monday, April 22, 2013 8:48 AM To: [email protected] Subject: Re: nightly-smoke on Cygwin I don't pass any -D arguments. I simply run 'ant nightly-smoke', just like I do 'ant precommit'. Only the latter recognizes JAVA7_HOME properly (set to /d/java/sun-java-70) and e.g. fails if it's not Oracle's 1.7. I don't know if the problem is in Ant or the python script. On one hand, since the rest of the targets work when I invoke them from Cygwin, I think it's the python script. But on the other hand, when I invoke smokeTestRelease to verify an RC, and I give it a relative path to place the downloaded artifacts, it resolves them properly. I think the problem here is that Ant resolves the path to D:\, which breaks python. I.e. perhaps if the script replaced D:\ and C:\ with /cygdrive/d and /cygdrive/c it will work? Do you know how to do that in python? I don't mind giving it a try. Shai On Mon, Apr 22, 2013 at 9:31 AM, Uwe Schindler <[email protected]> wrote: If you pass a relative JAVA7_HOME to the smoke tester, it will fail at a later stage, because when running parts of the ANT build, the CWD is different, so JAVA_HOME passed to ANT is wrong. This is the reason why the sysprop is “transformed” to be absolute. The Windows-style full path in JAVA7_HOME is correct from the java side (it expects a Windows path), so you have to pass a Windows path, not a Cygwin path. The python script should convert JAVA7_HOME to its own filesystem conventions with the “cygpath” tool when doing checks – and that is the bug here. The second error is related to you environment args. Whn using –D properties passed to ANT, you have to use windows-style paths. The double backslash is broken. Mabye use a forward slash if you are invoking from Cygwin shell, to prevent the special meaning of \ in cygwin. Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen <http://www.thetaphi.de/> http://www.thetaphi.de eMail: [email protected] From: Shai Erera [mailto:[email protected]] Sent: Monday, April 22, 2013 8:18 AM To: [email protected] Subject: Re: nightly-smoke on Cygwin Not sure I follow all your arguments. Other targets run fine from Cygwin, even while the paths are defined absolute / relative. The weird error I got the second time seems unrelated to Cygwin at all (nor Ant): /cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp Looks like the python script assumes the argument given to it is already relative? I.e. in Java File it would be the equivalent of new File(dir, arg)? And as for the first replacement of file= by value=, according to the <exec> task documentation, I think it's ok to change to value=, because we're passing an environment variable, not a file location. Why do other targets run fine when they just refer to ${JAVA7_HOME}, but you think we must pass a full path reference to the environment variable given to <exec>? Shai On Mon, Apr 22, 2013 at 9:09 AM, Uwe Schindler <[email protected]> wrote: Hi, Hi this is a general problem with horrible CYGWIN J I strongly disagree with replacing file=”…” by value=”..” because this makes the general build less portable between platforms and not depend on relative directories set correct (especially inside the lucene subfolders, where some build.xml set properties for other build.xmls, so all paths should be absolute. The problem here is on cygwin’s side: It simply uses incompatible path names. In my opinion, Java + Cygwin should not be used together for several reasons: - Java is not a with Cygwin app, so it expects JAVA_HOME as full *windows* path – there is also no Cygwin-java available (thanks, J) - Your python version is expecting /cygdrive paths with forward slashes. ANT is not able to produce them unless you use a special Cygwin ANT version. The fix here is to make the python script check for Cygwin environment and fix those path using the dos2unix tool. It already does this, so maybe something is missing. The more correct variant is not to use Cygwin at all: The Jenkins server and I use a native win32 python. If you want a real POSIX/UNIX environment, I would use the official “Microsoft” one: Services for Unix, but also not in combination with Java. But I never tried the smoke tester with it (I don’t use the smoke tester, because I prefer to check the artifacts manually before giving my vote. The +1 is coming from me, not a script). Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen <http://www.thetaphi.de/> http://www.thetaphi.de eMail: [email protected] From: Shai Erera [mailto:[email protected]] Sent: Monday, April 22, 2013 7:50 AM To: [email protected] Subject: Re: nightly-smoke on Cygwin Changing <arg file= to <arg value= didn't help, I still got the same error. I should note that I am able to run smokeTestRelease fine, when validating RCs. Any ideas (besides moving to Linux :))? Shai On Mon, Apr 22, 2013 at 8:18 AM, Shai Erera <[email protected]> wrote: Hi I ran nightly-smoke on my Windows 7 via Cygwin, and ran into two errors as depicted below. JAVA7_HOME is set to /d/java/sun-java-70 and it seems to work with other ant targets, e.g. precommit, test etc. prepare-release-no-sign: [mkdir] Created dir: D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease [copy] Copying 401 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene [copy] Copying 194 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr [exec] JAVA7_HOME is D:\d\java\sun-java-70 This seems to be caused by this line: <env key="JAVA7_HOME" file="${JAVA7_HOME}"/> If I replace it by this, I don't get the error anymore (but there's a different one, see below): <env key="JAVA7_HOME" value="${JAVA7_HOME}"/> According to http://ant.apache.org/manual/Tasks/exec.html, 'value' denotes "The literal value for the environment variable." while 'file' is "The value for the environment variable. Will be replaced by the absolute filename of the file by Ant.". Anyone can confirm it is ok to make this change? After making this change, I hit another error: prepare-release-no-sign: [mkdir] Created dir: D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease [copy] Copying 401 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\lucene [copy] Copying 194 files to D:\dev\lucene\lucene-trunk\lucene\build\fakeRelease\solr [exec] JAVA7_HOME is /d/java/sun-java-70 [exec] NOTE: output encoding is UTF-8 [exec] Traceback (most recent call last): [exec] File "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line 1357, in <module> [exec] main() [exec] File "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line 1301, in main [exec] smokeTest(baseURL, version, tmpDir, isSigned) [exec] File "D:\dev\lucene\lucene-trunk\dev-tools\scripts\smokeTestRelease.py", line 1312, in smokeTest [exec] os.makedirs(tmpDir) [exec] File "/usr/lib/python3.2/os.py", line 152, in makedirs [exec] mkdir(name, mode) [exec] OSError: [Errno 2] No such file or directory: '/cygdrive/d/dev/lucene/lucene-trunk/D:\\dev\\lucene\\lucene-trunk\\lucene\\build\\fakeReleaseTmp' See the last line, how the fakeRelease location is corrupt. Could it be this line: <arg file="${fakeReleaseTmp}"/> The property fakeReleaseTmp is defined as relative path: <property name="fakeReleaseTmp" location="lucene/build/fakeReleaseTmp"/> In Eclipse, when I hover over the property, it displays the correct full path. But I wonder if Ant gets confused ... I'm running now with file= replaced by value=, but it will take some time for this to complete (the previous error I got was after 48 minutes). Again, can anyone confirm this change is safe? Shai
