Resending since original was sent to Reuti, but not the rest of the list.
On 11/29/2011 10:27 AM, Reuti wrote:
> Am 29.11.2011 um 15:57 schrieb Prentice Bisbal:
>
>> On 11/28/2011 07:01 PM, Reuti wrote:
>>> Am 28.11.2011 um 21:00 schrieb Prentice Bisbal:
>>>
>>>> On 11/23/2011 10:10 AM, Reuti wrote:
>>>>> Am 23.11.2011 um 16:01 schrieb Prentice Bisbal:
>>>>>
>>>>>> Schmidt,
>>>>>>
>>>>>> I've been having the same problem ever since I upgraded my cluster
>>>>>> nodes to RHEL 6. A few weeks ago. My SGE installation was not touched
>>>>>> during the upgrade, since it's on an NFS partition, and the head node
>>>>>> was not touched during the upgrade, either. The error seems harmless,
>>>>>> and I haven't tracked it down yet, but the existence of that error
>>>>>> message, harmless or not, is a problem, as users keep reporting it when
>>>>>> they see it.
>>>>> Interesting. Did the version of the bash change with the upgrade?
>>>>>
>>>>> -- Reuti
>>>>>
>>>> Sorry for the delayed response. Still behind on my mailing lists since
>>>> going to SC11.
>>>>
>>>> Yes, the version of bash changed with the upgrade.
>>>>
>>>> On a 5.7 system:
>>>>
>>>> $ bash --version
>>>> GNU bash, version 3.2.25(1)-release (x86_64-redhat-linux-gnu)
>>>> Copyright (C) 2005 Free Software Foundation, Inc.
>>>>
>>>> $ rpm -q bash
>>>> bash-3.2-32.el5
>>> Is this one and the same package. I mean the first looks like 64 bit the
>>> latter like 32bit. Or do they use a 32 bit shell on a 64 bit system?
>>>
>> I just double-checked. Yes, those are one and the same package. I have
>> no idea why the 64-bit package doesn't end in x86_64 in this case. To
>> verify:
>>
>> # which bash
>> /bin/bash
>>
>> # /bin/bash --version
>> GNU bash, version 3.2.25(1)-release (x86_64-redhat-linux-gnu)
>> Copyright (C) 2005 Free Software Foundation, Inc.
>>
>> # rpm -qf /bin/bash
>> bash-3.2-32.el5
>>
>> I'm using a local rebuild of RHEL, called PUIAS Linux (google it), so
>> maybe the maintainers (ie the guy down the hall) messed up the packaging
>> when rebuilding.
>>
>>
>>>> On a 6.1 system:
>>>>
>>>> $ bash --version
>>>>
>>>> GNU bash, version 4.1.2(1)-release (x86_64-unknown-linux-gnu)
>>>> Copyright (C) 2009 Free Software Foundation, Inc.
>>>> License GPLv3+: GNU GPL version 3 or later
>>>> <http://gnu.org/licenses/gpl.html>
>>>>
>>>> $ rpm -q bash
>>>> bash-4.1.2-8.el6.x86_64
>>> Do you use modules on the headnode of the cluster and forward all variables
>>> with -V like the OP? The environment is set by sourcing the created
>>> "environment" in the job's spool directory. You can peek into this file and
>>> check whether it's there.
>>>
>>> The question is for the OP too: do you need -V? Open MPI will use it
>>> automatically when going from the master node of the parallel job to the
>>> slaves.
>> I don't use modules on any of my systems (yet). After getting this
>> error, I did install the modules package, thinking it might fix the
>> problem, but it didn't. The environment-modules RPM isn't even installed
>> on the head node. I do instruct my users to use -V to forward
>> environment variables in their job scripts. Not all of the jobs
>> submitted on my cluster are Open MPI jobs.
>>
>> I did verify that the environment file is in the job's spool directory
>> for a test job, and I also saw this line was in there:
>>
>> module=() { eval `/usr/bin/modulecmd bash $*`
>>
>> I don't know where the missing closing brace disappeared to, or where
>> that module
> If you check with:
>
> $ env
>
> it might be there (on the headnode I assume - or any default variables
> defined for export in any of the sge_request files?). This is the bug in SGE
> 6.2u5, that it can't handle multi lines variables.
>
> But it's still curious: where it's coming from.
>
I finally had some time to dig around on my systems, and here's what I
found. Some of this will be obvious to experienced RHEL admins/users,
but I'm just trying to be complete, and provide a full explanation for
future googlers:
1 ./etc/bashrc or /etc/profile will source the files in /etc/profile.d,
which includes modules.sh
2. /etc/profile.d/modules.sh will call /usr/share/Modules/init/$shell or
/usr/share/Modules/init/sh:
$ more modules.sh
shell=`/bin/basename \`/bin/ps -p $$ -ocomm=\``
if [ -f /usr/share/Modules/init/$shell ]
then
. /usr/share/Modules/init/$shell
else
. /usr/share/Modules/init/sh
fi
Since my shell is bash, /usr/share/Modules/init/bash is called.
3. Amongst other things, /usr/share/Modules/init/bash defines the
function module() and then exports it in the first two lines of code:
module() { eval `/usr/bin/modulecmd bash $*`; }
export -f module
As you can see, there is a closing bracet, which is missing from the SGE
environment. Looking at the output of the 'env' command, I can confirm
that the full definition of module (from opening brace to closing brace)
makes it to my normal interactive, non-SGE environment. It looks like
this in the output of env:
module=() { eval `/usr/bin/modulecmd bash $*`
}
Notice, however, that the semicolon in the original definition is now a
new-line command. Which makes sense, but it must be tripping SGE due to
the known bug with multi-line environment variables in SGE 6.2u5, which
is what I'm using, as Reuti pointed out.
It looks like I could just remove the environment modules packages from
my systems, but I do plan on using them in the near future, so I need to
fix this. I didn't have modules installed when running RHEL 5, so that
explains why this error just popped up after the upgrade to RHEL 6. Will
upgrading to an SGE 6.2 update greater than 5 fix my problem? Is an
update after u5 even available now, post-Oracle?
--
Prentice
_______________________________________________
users mailing list
[email protected]
https://gridengine.org/mailman/listinfo/users
