[gt-user] Globus Toolkit website and GT6 repository temporary hosting this weekend

2016-06-16 Thread Stuart Martin
Hi All,

There is a scheduled power outage starting Friday June 17 at ~6am until 
sometime Sunday June 19 morning for the machines that host the Globus Toolkit 
website and package repos.  This includes Globus Connect Server installations.

Note: Globus Connect Personal software installations are unaffected by the 
outage and will remain available.

However, we have copied the GT6 website and repository to an alternate hosting 
location which should allow installations and updates for GT6 throughout this 
weekends outage.

We do not anticipate any website nor GT6 repository degradation, but in case 
there is some minor issues, we wanted to let you know.

Regards,
Stu

Re: [gt-user] Announcing GT 5.2.5 end-of-life

2016-05-12 Thread Stuart Martin
Hi All,

I’m reviving this thread.  It has been a while, but we had to get our
ducks in a row for how the Debian 7 and 8 upgrade path from GT 5.2.5 to
GT 6 will be handled.  In short, we will use Debian backports and have
added GT 6 to it.  New instructions for upgrading to Debian backports or to
Globus’ GT6 repo are here:
http://toolkit.globus.org/toolkit/docs/5.2-EOL.html 
<http://toolkit.globus.org/toolkit/docs/5.2-EOL.html>

I hope this thread (started in July 2015) has been enough of a notice
of our intentions, that at this time, we are officially setting GT 5.2.x
to EOL.

The docs and downloads web pages have been updated to show that all of the non 
gt6 versions are now unsupported.
http://toolkit.globus.org/toolkit/docs/ 
<http://toolkit.globus.org/toolkit/docs/>
http://toolkit.globus.org/toolkit/downloads/ 
<http://toolkit.globus.org/toolkit/downloads/>

Let us know if you have any questions.

The Globus team

> On Sep 1, 2015, at 2:49 PM, Stuart Martin  wrote:
> 
> Hi Tom,  Comments inline.
> 
>> On Aug 5, 2015, at 12:50 PM, Tom Downes > <mailto:dow...@uwm.edu>> wrote:
>> 
>> Hi Stuart:
>> 
>> Sorry for the delayed response as I was on vacation. A quick inspection of 
>> the repository structure for GT6 versus GT5 eliminates a number of my 
>> complaints.
>> 
>> http://toolkit.globus.org/ftppub/gt5/5.2/ 
>> <http://toolkit.globus.org/ftppub/gt5/5.2/> (e.g. I believe stable points to 
>> 5.2.2)
>> http://toolkit.globus.org/ftppub/gt6/stable/deb/ 
>> <http://toolkit.globus.org/ftppub/gt6/stable/deb/>
>> 
>> For example, GT5 had different directories for Debian 7 versus 8. You now do 
>> this correctly by combining all deb-based dists under a single directory 
>> structure. You haven't quite gotten all the way since the more Debian way 
>> for this:
>> 
>> deb http://toolkit.globus.org/ftppub/gt6/stable/deb 
>> <http://toolkit.globus.org/ftppub/gt6/stable/deb> wheezy contrib
>> deb http://toolkit.globus.org/ftppub/gt6/testing/deb 
>> <http://toolkit.globus.org/ftppub/gt6/testing/deb> wheezy contrib
>> 
>> would be 
>> 
>> deb http://toolkit.globus.org/ftppub/gt6/deb 
>> <http://toolkit.globus.org/ftppub/gt6/deb> wheezy contrib
>> deb http://toolkit.globus.org/ftppub/gt6/deb 
>> <http://toolkit.globus.org/ftppub/gt6/deb> wheezy testing
>> 
>> But this is a major improvement to the old directory structure. So kudos!
> 
> Thanks!
> 
>> Comment 1: when you're managing N machines, it's much easier to know the 
>> sources.list entries and the signing key. Automated provisioning software 
>> (Foreman + preseed, for example) tend to be unfriendly toward unsigned 
>> packages. Not impossible, but it's much easier when the documentation just 
>> tells you the right path and points you to the key and you don't have to 
>> export it manually from the GPG file. I can see why a .deb is easier for Joe 
>> Sixpack walking up with one computer, but it's not easier for me.
> 
> This is mostly a documentation issue. We can pubish a directory of the repo 
> configuration files and gpg key and include a link to that from the 
> documentation in addition to the repo packages.  I created an issue for this. 
>  We’ll add this to our todo’s.
> 
> https://globus.atlassian.net/browse/GT-626 
> <https://globus.atlassian.net/browse/GT-626>
>> Comment 2: Do you intend for the stable repositories to contain the past 
>> history of releases even when they have been superseded? I hope so as this 
>> facilitates version pinning without being accidentally left behind.
> 
> We keep older versions in the repository.  So, you can specify an older 
> version in case you need to back out from an update that goes bad, or stay 
> pinned to a specific version if need be.
> 
>> Comment 3: What is the closest repository to "proposed" in Debian-speak? 
>> i.e. a release that is going into the wild that, should no bug reports be 
>> filed, will be moved unchanged into stable.
> 
> We have “Stable”, “Unstable” and “Testing” repos for each GT release version. 
>  Described here:
>   
> http://toolkit.globus.org/toolkit/docs/latest-stable/admin/install/#idp36524752
>  
> <http://toolkit.globus.org/toolkit/docs/latest-stable/admin/install/#idp36524752>
> 
> The Testing repo would be the closest to Debian “proposed”.  Once we/Globus 
> developers determine packages are ready, packages move from Testing to 
> Stable.  “proposed” could be an email announcement and delay of those 
> packages in the Testing repo in order to allow the Globus communit

Re: [gt-user] Globus "strict mode" coming March 2016 - Action Required

2016-05-06 Thread Stuart Martin
Ok - this change has been released!  globus-gssapi-gsi 12.0 is now on globus’ 
stable repos.

-Stu

> On May 4, 2016, at 9:25 AM, Stuart Martin  wrote:
> 
> The change will be in package: globus-gssapi-gsi with version greater than or 
> equal to 12.0
> 
>> On May 4, 2016, at 9:07 AM, Steven C Timm > <mailto:t...@fnal.gov>> wrote:
>> 
>> What has not been clear to me is the following:
>> Stuart Martin's  E-mail says that new clients will be released but it does 
>> not say what Globus
>> version that those new clients correspond to.  How do we tell via the client 
>> versions which ones
>> are enforcing strict mode and which ones are not?  Furthermore those of us 
>> who get the globus clients through other distributions, how do we tell when 
>> our distribution is starting to use them?
>> 
>> Steve Timm
>> 
>> 
>> 
>> From: gt-user-boun...@lists.globus.org 
>> <mailto:gt-user-boun...@lists.globus.org> > <mailto:gt-user-boun...@lists.globus.org>> on behalf of Basney, Jim 
>> mailto:jbas...@illinois.edu>>
>> Sent: Wednesday, May 4, 2016 9:00:54 AM
>> To: GT User
>> Subject: Re: [gt-user] Globus ³strict mode² coming March 2016 - Action 
>> Required
>> 
>> Hi,
>> 
>> Did this change happen as planned? Is the transition going OK for everyone?
>> 
>> In my role as CA operator, I know we've been issuing a lot of host certs
>> with multiple subjectAltNames in preparation for this transition, so
>> hopefully everyone has the certs they need.
>> 
>> -Jim
>> 
>> On 2/4/16, 5:21 PM, Stuart Martin wrote:
>>> Hi All,
>>> 
>>> Here is a reminder about this deadline and upcoming change.
>>> 
>>> Admins should check their host certificates and update them if necessary.
>>> Replace any incompatible certificates by Mar 1, 2016.
>>> 
>>> To allow a bit of a buffer between the service-side certificate update
>>> deadline and clients beginning to use strict mode, updates will be made
>>> to the Globus Toolkit to default the client-side algorithm to strict mode
>>> on Tuesday, April 5, 2016.
>>> 
>>> - The Globus Team
>>> 
>>>> On Dec 1, 2015, at 2:50 PM, Stuart Martin >>> <mailto:smar...@mcs.anl.gov>> wrote:
>>>> 
>>>> Dear All,
>>>> 
>>>> Globus is planning to change the default client-side algorithm for
>>>> checking the server¹s identity used by GridFTP, MyProxy, GSI-OpenSSH,
>>>> and GRAM.  The new algorithm performs identity matching as described in
>>>> section 3.1 of RFC 2818
>>>> (https://tools.ietf.org/html/rfc2818#section-3.1 
>>>> <https://tools.ietf.org/html/rfc2818#section-3.1>), the standard
>>>> describing TLS use with HTTP.   This involves a change in the
>>>> globus-gssapi-gsi library, and will apply to any application that uses
>>>> the updated library.
>>>> 
>>>> The new ³strict mode² algorithm will be more strict in its enforcement,
>>>> checking that the server¹s certificate identity matches the hostname
>>>> that the client uses to contact the service.  Once clients are
>>>> configured for strict mode, client authentication (of any Globus
>>>> service) would fail if the service is using a certificate that does not
>>>> match the hostname that the client used to contact the service.
>>>> 
>>>> This change will bring our identity checking algorithm in line with RFC
>>>> 2818, and will also close the door to reverse DNS lookup related attack
>>>> vectors. As an example of why relying on reverse DNS for making security
>>>> related decisions is not recommended, see this link:
>>>> https://cwe.mitre.org/data/definitions/350.html 
>>>> <https://cwe.mitre.org/data/definitions/350.html>.
>>>> 
>>>> The Globus team has checked the host certificates used for a number of
>>>> GridFTP endpoints and found that many are _not_ RFC 2818 compatible.
>>>> These incompatible certificates will need to be replaced prior to
>>>> clients defaulting to the new strict mode algorithm.
>>>> 
>>>> We are reaching out to request that Globus service admins check their
>>>> host certificates and update them if necessary.  We are asking admins to
>>>> replace any incompatible certificates by Mar 1, 2016.  After March 1, we
>>>> will release

Re: [gt-user] Globus ģstrict modeē coming March 2016 - Action Required

2016-05-06 Thread Stuart Martin

> On May 5, 2016, at 10:59 AM, Cameron Harr  wrote:
> 
> Hi,
> The instructions here say if you're using Globus-provided certs, you aren't 
> affected. Does that include simple-CA certs?

No.  Globus-provided certs is referring to the certs included in the globus 
connect server package/setup.  Those certs don't have a hostnames in them so 
they don't matter for strict name checking.  Globus Transfer will always 
include the subject DN when connecting to a GCS endpoint configured with the 
included certificate.  GCS (or any GridFTP server) will not do a DNS lookup if 
the subject DN is included from the client.

> If so, are we still OK even if we see the "not ok" string at the end of 
> grid-cert-diagnostics? I'm seeing the following and want to know if I can 
> ignore this or not as it's Globus-provided:
> Checking subjectAltName extensions... no
> Verifying certificate chain... failed
> globus_credential: Error verifying credential: Failed to verify credential
> globus_gsi_callback_module: Could not verify credential
> globus_gsi_callback_module: Error with signing policy
> globus_gsi_callback_module: Error in OLD GAA code: The subject of the 
> certificate "/C=US/O=Globus Consortium/CN=Globus Connect CA" does not match 
> the signing policies defined in 
> /etc/grid-security/certificates/7a42187f.signing_policy
> 
> Performing name comparison... not ok
> 
> Thanks,
> Cameron
> 
> On 05/04/2016 07:25 AM, Stuart Martin wrote:
>> The change will be in package: globus-gssapi-gsi with version greater than 
>> or equal to 12.0
>> 
>>> On May 4, 2016, at 9:07 AM, Steven C Timm < 
>>> <mailto:t...@fnal.gov>t...@fnal.gov <mailto:t...@fnal.gov>> wrote:
>>> 
>>> What has not been clear to me is the following:
>>> Stuart Martin's  E-mail says that new clients will be released but it does 
>>> not say what Globus
>>> version that those new clients correspond to.  How do we tell via the 
>>> client versions which ones
>>> are enforcing strict mode and which ones are not?  Furthermore those of us 
>>> who get the globus clients through other distributions, how do we tell when 
>>> our distribution is starting to use them?
>>> 
>>> Steve Timm
>>> 
>>> 
>>> 
>>> From: gt-user-boun...@lists.globus.org 
>>> <mailto:gt-user-boun...@lists.globus.org> >> <mailto:gt-user-boun...@lists.globus.org>> on behalf of Basney, Jim 
>>> mailto:jbas...@illinois.edu>>
>>> Sent: Wednesday, May 4, 2016 9:00:54 AM
>>> To: GT User
>>> Subject: Re: [gt-user] Globus ³strict mode² coming March 2016 - Action 
>>> Required
>>> 
>>> Hi,
>>> 
>>> Did this change happen as planned? Is the transition going OK for everyone?
>>> 
>>> In my role as CA operator, I know we've been issuing a lot of host certs
>>> with multiple subjectAltNames in preparation for this transition, so
>>> hopefully everyone has the certs they need.
>>> 
>>> -Jim
>>> 
>>> On 2/4/16, 5:21 PM, Stuart Martin wrote:
>>>> Hi All,
>>>> 
>>>> Here is a reminder about this deadline and upcoming change.
>>>> 
>>>> Admins should check their host certificates and update them if necessary.
>>>> Replace any incompatible certificates by Mar 1, 2016.
>>>> 
>>>> To allow a bit of a buffer between the service-side certificate update
>>>> deadline and clients beginning to use strict mode, updates will be made
>>>> to the Globus Toolkit to default the client-side algorithm to strict mode
>>>> on Tuesday, April 5, 2016.
>>>> 
>>>> - The Globus Team
>>>> 
>>>>> On Dec 1, 2015, at 2:50 PM, Stuart Martin >>>> <mailto:smar...@mcs.anl.gov>> wrote:
>>>>> 
>>>>> Dear All,
>>>>> 
>>>>> Globus is planning to change the default client-side algorithm for
>>>>> checking the server¹s identity used by GridFTP, MyProxy, GSI-OpenSSH,
>>>>> and GRAM.  The new algorithm performs identity matching as described in
>>>>> section 3.1 of RFC 2818
>>>>> (https://tools.ietf.org/html/rfc2818#section-3.1 
>>>>> <https://tools.ietf.org/html/rfc2818#section-3.1>), the standard
>>>>> describing TLS use with HTTP.   This involves a change in the
>>>>> globus-gssapi-gsi library, and will apply to any a

Re: [gt-user] Globus ģstrict modeē coming March 2016 - Action Required

2016-05-04 Thread Stuart Martin
The change will be in package: globus-gssapi-gsi with version greater than or 
equal to 12.0

> On May 4, 2016, at 9:07 AM, Steven C Timm  wrote:
> 
> What has not been clear to me is the following:
> Stuart Martin's  E-mail says that new clients will be released but it does 
> not say what Globus
> version that those new clients correspond to.  How do we tell via the client 
> versions which ones
> are enforcing strict mode and which ones are not?  Furthermore those of us 
> who get the globus clients through other distributions, how do we tell when 
> our distribution is starting to use them?
> 
> Steve Timm
> 
> 
> 
> From: gt-user-boun...@lists.globus.org  on 
> behalf of Basney, Jim 
> Sent: Wednesday, May 4, 2016 9:00:54 AM
> To: GT User
> Subject: Re: [gt-user] Globus ³strict mode² coming March 2016 - Action 
> Required
> 
> Hi,
> 
> Did this change happen as planned? Is the transition going OK for everyone?
> 
> In my role as CA operator, I know we've been issuing a lot of host certs
> with multiple subjectAltNames in preparation for this transition, so
> hopefully everyone has the certs they need.
> 
> -Jim
> 
> On 2/4/16, 5:21 PM, Stuart Martin wrote:
>> Hi All,
>> 
>> Here is a reminder about this deadline and upcoming change.
>> 
>> Admins should check their host certificates and update them if necessary.
>> Replace any incompatible certificates by Mar 1, 2016.
>> 
>> To allow a bit of a buffer between the service-side certificate update
>> deadline and clients beginning to use strict mode, updates will be made
>> to the Globus Toolkit to default the client-side algorithm to strict mode
>> on Tuesday, April 5, 2016.
>> 
>> - The Globus Team
>> 
>>> On Dec 1, 2015, at 2:50 PM, Stuart Martin  wrote:
>>> 
>>> Dear All,
>>> 
>>> Globus is planning to change the default client-side algorithm for
>>> checking the server¹s identity used by GridFTP, MyProxy, GSI-OpenSSH,
>>> and GRAM.  The new algorithm performs identity matching as described in
>>> section 3.1 of RFC 2818
>>> (https://tools.ietf.org/html/rfc2818#section-3.1), the standard
>>> describing TLS use with HTTP.   This involves a change in the
>>> globus-gssapi-gsi library, and will apply to any application that uses
>>> the updated library.
>>> 
>>> The new ³strict mode² algorithm will be more strict in its enforcement,
>>> checking that the server¹s certificate identity matches the hostname
>>> that the client uses to contact the service.  Once clients are
>>> configured for strict mode, client authentication (of any Globus
>>> service) would fail if the service is using a certificate that does not
>>> match the hostname that the client used to contact the service.
>>> 
>>> This change will bring our identity checking algorithm in line with RFC
>>> 2818, and will also close the door to reverse DNS lookup related attack
>>> vectors. As an example of why relying on reverse DNS for making security
>>> related decisions is not recommended, see this link:
>>> https://cwe.mitre.org/data/definitions/350.html.
>>> 
>>> The Globus team has checked the host certificates used for a number of
>>> GridFTP endpoints and found that many are _not_ RFC 2818 compatible.
>>> These incompatible certificates will need to be replaced prior to
>>> clients defaulting to the new strict mode algorithm.
>>> 
>>> We are reaching out to request that Globus service admins check their
>>> host certificates and update them if necessary.  We are asking admins to
>>> replace any incompatible certificates by Mar 1, 2016.  After March 1, we
>>> will release updated Globus Toolkit components that will change the
>>> default client authorization algorithm to strict mode.  At that time,
>>> the Globus.org transfer service will also update its identity checking
>>> algorithm.  This should ensure no service disruptions for the Globus
>>> community.
>>> 
>>> Note: Globus Connect Server installations that use the Globus provided
>>> certificate are not affected and do not have to make any changes to
>>> their Globus Connect Server endpoint(s).
>>> 
>>> We have created a page where additional details about this change will
>>> be communicated:
>>> https://docs.globus.org/security-bulletins/2015-12-strict-mode/
>>> The above page includes common reasons for incompatibilities and how to
>>> check for compatibility.
>>> 
>>> If you have any questions or concerns regarding this planned change,
>>> please contact us at supp...@globus.org.
>>> 
>>> - The Globus team
> 



Re: [gt-user] Globus ģstrict modeē coming March 2016 - Action Required

2016-05-04 Thread Stuart Martin
Hi Jim,

Joe started the builds with this change yesterday, but hit an issue, so the 
stable repo has not yet been updated.  We anticipate the stable repo to be 
updated sometime today.  We’ll send email when the update is on stable.

As you know, only after users update their local installs would they have this 
change.  So, that might take some time for users to confirm all is good.

-Stu

> On May 4, 2016, at 9:00 AM, Basney, Jim  wrote:
> 
> Hi,
> 
> Did this change happen as planned? Is the transition going OK for everyone?
> 
> In my role as CA operator, I know we've been issuing a lot of host certs
> with multiple subjectAltNames in preparation for this transition, so
> hopefully everyone has the certs they need.
> 
> -Jim
> 
> On 2/4/16, 5:21 PM, Stuart Martin wrote:
>> Hi All,
>> 
>> Here is a reminder about this deadline and upcoming change.
>> 
>> Admins should check their host certificates and update them if necessary.
>> Replace any incompatible certificates by Mar 1, 2016.
>> 
>> To allow a bit of a buffer between the service-side certificate update
>> deadline and clients beginning to use strict mode, updates will be made
>> to the Globus Toolkit to default the client-side algorithm to strict mode
>> on Tuesday, April 5, 2016.
>> 
>> - The Globus Team
>> 
>>> On Dec 1, 2015, at 2:50 PM, Stuart Martin  wrote:
>>> 
>>> Dear All,
>>> 
>>> Globus is planning to change the default client-side algorithm for
>>> checking the server¹s identity used by GridFTP, MyProxy, GSI-OpenSSH,
>>> and GRAM.  The new algorithm performs identity matching as described in
>>> section 3.1 of RFC 2818
>>> (https://tools.ietf.org/html/rfc2818#section-3.1 
>>> <https://tools.ietf.org/html/rfc2818#section-3.1>), the standard
>>> describing TLS use with HTTP.   This involves a change in the
>>> globus-gssapi-gsi library, and will apply to any application that uses
>>> the updated library.
>>> 
>>> The new ³strict mode² algorithm will be more strict in its enforcement,
>>> checking that the server¹s certificate identity matches the hostname
>>> that the client uses to contact the service.  Once clients are
>>> configured for strict mode, client authentication (of any Globus
>>> service) would fail if the service is using a certificate that does not
>>> match the hostname that the client used to contact the service.
>>> 
>>> This change will bring our identity checking algorithm in line with RFC
>>> 2818, and will also close the door to reverse DNS lookup related attack
>>> vectors. As an example of why relying on reverse DNS for making security
>>> related decisions is not recommended, see this link:
>>> https://cwe.mitre.org/data/definitions/350.html.
>>> 
>>> The Globus team has checked the host certificates used for a number of
>>> GridFTP endpoints and found that many are _not_ RFC 2818 compatible.
>>> These incompatible certificates will need to be replaced prior to
>>> clients defaulting to the new strict mode algorithm.
>>> 
>>> We are reaching out to request that Globus service admins check their
>>> host certificates and update them if necessary.  We are asking admins to
>>> replace any incompatible certificates by Mar 1, 2016.  After March 1, we
>>> will release updated Globus Toolkit components that will change the
>>> default client authorization algorithm to strict mode.  At that time,
>>> the Globus.org transfer service will also update its identity checking
>>> algorithm.  This should ensure no service disruptions for the Globus
>>> community.
>>> 
>>> Note: Globus Connect Server installations that use the Globus provided
>>> certificate are not affected and do not have to make any changes to
>>> their Globus Connect Server endpoint(s).
>>> 
>>> We have created a page where additional details about this change will
>>> be communicated:
>>> https://docs.globus.org/security-bulletins/2015-12-strict-mode/
>>> The above page includes common reasons for incompatibilities and how to
>>> check for compatibility.
>>> 
>>> If you have any questions or concerns regarding this planned change,
>>> please contact us at supp...@globus.org.
>>> 
>>> - The Globus team



Re: [gt-user] Globus “strict mode” coming March 2016 - Action Required

2016-04-04 Thread Stuart Martin
Hi All,

I have been informed that some organizations need more time to update their 
host certificates to be compliant with the new strict mode algorithm.  To 
maintain compatibility throughout the Globus community, this update to the 
Globus Toolkit has been rescheduled for May 3.

I encourage everyone to test their host certificates now and update them if 
necessary.  See this page for details:
https://docs.globus.org/security-bulletins/2015-12-strict-mode/ 
<https://docs.globus.org/security-bulletins/2015-12-strict-mode/>

- The Globus Team

> On Feb 4, 2016, at 5:21 PM, Stuart Martin  wrote:
> 
> Hi All,
> 
> Here is a reminder about this deadline and upcoming change.
> 
> Admins should check their host certificates and update them if necessary.  
> Replace any incompatible certificates by Mar 1, 2016.
> 
> To allow a bit of a buffer between the service-side certificate update 
> deadline and clients beginning to use strict mode, updates will be made to 
> the Globus Toolkit to default the client-side algorithm to strict mode on 
> Tuesday, April 5, 2016.
> 
> - The Globus Team
> 
>> On Dec 1, 2015, at 2:50 PM, Stuart Martin  wrote:
>> 
>> Dear All,
>> 
>> Globus is planning to change the default client-side algorithm for checking 
>> the server’s identity used by GridFTP, MyProxy, GSI-OpenSSH, and GRAM.  The 
>> new algorithm performs identity matching as described in section 3.1 of RFC 
>> 2818 (https://tools.ietf.org/html/rfc2818#section-3.1), the standard 
>> describing TLS use with HTTP.   This involves a change in the 
>> globus-gssapi-gsi library, and will apply to any application that uses the 
>> updated library.
>> 
>> The new “strict mode” algorithm will be more strict in its enforcement, 
>> checking that the server’s certificate identity matches the hostname that 
>> the client uses to contact the service.  Once clients are configured for 
>> strict mode, client authentication (of any Globus service) would fail if the 
>> service is using a certificate that does not match the hostname that the 
>> client used to contact the service.
>> 
>> This change will bring our identity checking algorithm in line with RFC 
>> 2818, and will also close the door to reverse DNS lookup related attack 
>> vectors. As an example of why relying on reverse DNS for making security 
>> related decisions is not recommended, see this link: 
>> https://cwe.mitre.org/data/definitions/350.html.
>> 
>> The Globus team has checked the host certificates used for a number of 
>> GridFTP endpoints and found that many are _not_ RFC 2818 compatible.  These 
>> incompatible certificates will need to be replaced prior to clients 
>> defaulting to the new strict mode algorithm.
>> 
>> We are reaching out to request that Globus service admins check their host 
>> certificates and update them if necessary.  We are asking admins to replace 
>> any incompatible certificates by Mar 1, 2016.  After March 1, we will 
>> release updated Globus Toolkit components that will change the default 
>> client authorization algorithm to strict mode.  At that time, the Globus.org 
>> transfer service will also update its identity checking algorithm.  This 
>> should ensure no service disruptions for the Globus community.
>> 
>> Note: Globus Connect Server installations that use the Globus provided 
>> certificate are not affected and do not have to make any changes to their 
>> Globus Connect Server endpoint(s).
>> 
>> We have created a page where additional details about this change will be 
>> communicated:
>>  https://docs.globus.org/security-bulletins/2015-12-strict-mode/
>> The above page includes common reasons for incompatibilities and how to 
>> check for compatibility.
>> 
>> If you have any questions or concerns regarding this planned change, please 
>> contact us at supp...@globus.org.
>> 
>> - The Globus team
> 



Re: [gt-user] Globus “strict mode” coming March 2016 - Action Required

2016-02-04 Thread Stuart Martin
Hi All,

Here is a reminder about this deadline and upcoming change.

Admins should check their host certificates and update them if necessary.  
Replace any incompatible certificates by Mar 1, 2016.

To allow a bit of a buffer between the service-side certificate update deadline 
and clients beginning to use strict mode, updates will be made to the Globus 
Toolkit to default the client-side algorithm to strict mode on Tuesday, April 
5, 2016.

- The Globus Team

> On Dec 1, 2015, at 2:50 PM, Stuart Martin  wrote:
> 
> Dear All,
> 
> Globus is planning to change the default client-side algorithm for checking 
> the server’s identity used by GridFTP, MyProxy, GSI-OpenSSH, and GRAM.  The 
> new algorithm performs identity matching as described in section 3.1 of RFC 
> 2818 (https://tools.ietf.org/html/rfc2818#section-3.1), the standard 
> describing TLS use with HTTP.   This involves a change in the 
> globus-gssapi-gsi library, and will apply to any application that uses the 
> updated library.
> 
> The new “strict mode” algorithm will be more strict in its enforcement, 
> checking that the server’s certificate identity matches the hostname that the 
> client uses to contact the service.  Once clients are configured for strict 
> mode, client authentication (of any Globus service) would fail if the service 
> is using a certificate that does not match the hostname that the client used 
> to contact the service.
> 
> This change will bring our identity checking algorithm in line with RFC 2818, 
> and will also close the door to reverse DNS lookup related attack vectors. As 
> an example of why relying on reverse DNS for making security related 
> decisions is not recommended, see this link: 
> https://cwe.mitre.org/data/definitions/350.html.
> 
> The Globus team has checked the host certificates used for a number of 
> GridFTP endpoints and found that many are _not_ RFC 2818 compatible.  These 
> incompatible certificates will need to be replaced prior to clients 
> defaulting to the new strict mode algorithm.
> 
> We are reaching out to request that Globus service admins check their host 
> certificates and update them if necessary.  We are asking admins to replace 
> any incompatible certificates by Mar 1, 2016.  After March 1, we will release 
> updated Globus Toolkit components that will change the default client 
> authorization algorithm to strict mode.  At that time, the Globus.org 
> transfer service will also update its identity checking algorithm.  This 
> should ensure no service disruptions for the Globus community.
> 
> Note: Globus Connect Server installations that use the Globus provided 
> certificate are not affected and do not have to make any changes to their 
> Globus Connect Server endpoint(s).
> 
> We have created a page where additional details about this change will be 
> communicated:
>   https://docs.globus.org/security-bulletins/2015-12-strict-mode/
> The above page includes common reasons for incompatibilities and how to check 
> for compatibility.
> 
> If you have any questions or concerns regarding this planned change, please 
> contact us at supp...@globus.org.
> 
> - The Globus team



[gt-user] OpenSSL Security Advisory: CVE-2016-0701

2016-01-28 Thread Stuart Martin
Hi All,

On January 28th, a new vulnerability CVE-2016-0701 was announced.  The Globus 
team has reviewed the vulnerability and determined that the Globus services and 
client interactions to Globus services are not vulnerable.

As a precaution, it is recommended to upgrade any installations using the 
vulnerable openssl version.

References:
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-0701
https://www.openssl.org/news/secadv/20160128.txt

If you have any concerns about this issue, please contact us at 
supp...@globus.org.

- Globus Team

Re: [gt-user] [gt-dev] OpenSSH client vulnerability CVE-2016-0777

2016-01-20 Thread Stuart Martin
Hi All,

The GSI-OpenSSH 5.7 source update package is available here:
http://toolkit.globus.org/toolkit/advisories.html?version=6.0 
<http://toolkit.globus.org/toolkit/advisories.html?version=6.0>

It is available from the Globus repo for all RPM and Deb platforms.
http://toolkit.globus.org/toolkit/downloads/6.0/ 
<http://toolkit.globus.org/toolkit/downloads/6.0/>

It has been added to the Mac and Windows installers.

-Stu

> On Jan 14, 2016, at 3:03 PM, Stuart Martin  wrote:
> 
> Hi All,
> 
> On January 14th, a new vulnerability CVE-2016-0777 affecting OpenSSH clients 
> was announced.  Globus services and client interactions to Globus services 
> are not vulnerable.
> 
> This affects SSH and GSISSH clients when connecting to a malicious server.  
> Globus distributes GSI-OpenSSH, which is based on OpenSSH.  As such, we'll be 
> applying the security patch for this issue from the OpenSSH developers and 
> releasing updated gsi-openssh Globus Toolkit packages.
> 
> Note that the system installed ssh package is used by globus-ftp-client based 
> tools, such as globus-url-copy, when accessing sshftp:// URLs.  If you use 
> this feature, you should ensure your ssh package is up to date.
> 
> In the meantime, the problem can be avoided by adding the undocumented 
> "UseRoaming no" directive to the relevant config files.  The default 
> system-wide configuration file for ssh is  /etc/ssh/ssh_config, and for 
> gsissh is /etc/gsissh/ssh_config.
> 
> References:
>  
> https://www.qualys.com/2016/01/14/cve-2016-0777-cve-2016-0778/openssh-cve-2016-0777-cve-2016-0778.txt
>  
> <https://www.qualys.com/2016/01/14/cve-2016-0777-cve-2016-0778/openssh-cve-2016-0777-cve-2016-0778.txt>
>  http://www.openssh.com/txt/release-7.1p2 
> <http://www.openssh.com/txt/release-7.1p2>
> If you have any concerns about this issue, please contact us at 
> supp...@globus.org <mailto:supp...@globus.org>.
> 
> - Globus Team



[gt-user] OpenSSH client vulnerability CVE-2016-0777

2016-01-14 Thread Stuart Martin
Hi All,

On January 14th, a new vulnerability CVE-2016-0777 affecting OpenSSH clients 
was announced.  Globus services and client interactions to Globus services are 
not vulnerable.

This affects SSH and GSISSH clients when connecting to a malicious server.  
Globus distributes GSI-OpenSSH, which is based on OpenSSH.  As such, we'll be 
applying the security patch for this issue from the OpenSSH developers and 
releasing updated gsi-openssh Globus Toolkit packages.

Note that the system installed ssh package is used by globus-ftp-client based 
tools, such as globus-url-copy, when accessing sshftp:// URLs.  If you use this 
feature, you should ensure your ssh package is up to date.

In the meantime, the problem can be avoided by adding the undocumented 
"UseRoaming no" directive to the relevant config files.  The default 
system-wide configuration file for ssh is  /etc/ssh/ssh_config, and for gsissh 
is /etc/gsissh/ssh_config.

References:
 
https://www.qualys.com/2016/01/14/cve-2016-0777-cve-2016-0778/openssh-cve-2016-0777-cve-2016-0778.txt
 http://www.openssh.com/txt/release-7.1p2

If you have any concerns about this issue, please contact us at 
supp...@globus.org.

- Globus Team

[gt-user] Globus “strict mode” coming March 2016 - Action Required

2015-12-01 Thread Stuart Martin
Dear All,

Globus is planning to change the default client-side algorithm for checking the 
server’s identity used by GridFTP, MyProxy, GSI-OpenSSH, and GRAM.  The new 
algorithm performs identity matching as described in section 3.1 of RFC 2818 
(https://tools.ietf.org/html/rfc2818#section-3.1), the standard describing TLS 
use with HTTP.   This involves a change in the globus-gssapi-gsi library, and 
will apply to any application that uses the updated library.

The new “strict mode” algorithm will be more strict in its enforcement, 
checking that the server’s certificate identity matches the hostname that the 
client uses to contact the service.  Once clients are configured for strict 
mode, client authentication (of any Globus service) would fail if the service 
is using a certificate that does not match the hostname that the client used to 
contact the service.

This change will bring our identity checking algorithm in line with RFC 2818, 
and will also close the door to reverse DNS lookup related attack vectors. As 
an example of why relying on reverse DNS for making security related decisions 
is not recommended, see this link: 
https://cwe.mitre.org/data/definitions/350.html.

The Globus team has checked the host certificates used for a number of GridFTP 
endpoints and found that many are _not_ RFC 2818 compatible.  These 
incompatible certificates will need to be replaced prior to clients defaulting 
to the new strict mode algorithm.

We are reaching out to request that Globus service admins check their host 
certificates and update them if necessary.  We are asking admins to replace any 
incompatible certificates by Mar 1, 2016.  After March 1, we will release 
updated Globus Toolkit components that will change the default client 
authorization algorithm to strict mode.  At that time, the Globus.org transfer 
service will also update its identity checking algorithm.  This should ensure 
no service disruptions for the Globus community.

Note: Globus Connect Server installations that use the Globus provided 
certificate are not affected and do not have to make any changes to their 
Globus Connect Server endpoint(s).

We have created a page where additional details about this change will be 
communicated:
https://docs.globus.org/security-bulletins/2015-12-strict-mode/
The above page includes common reasons for incompatibilities and how to check 
for compatibility.

If you have any questions or concerns regarding this planned change, please 
contact us at supp...@globus.org.

- The Globus team

Re: [gt-user] GRAM Job submission failed because the job manager failed to create an internal script argument file

2015-11-04 Thread Stuart Martin
Did you have the "-save-logfile always" argument set in the config file?  If 
not, try setting that.

Here is an example:

https://twiki.opensciencegrid.org/bin/view/Documentation/Release3/TroubleshootingFaq#GRAM_How_to_save_the_jobmanager
 
<https://twiki.opensciencegrid.org/bin/view/Documentation/Release3/TroubleshootingFaq#GRAM_How_to_save_the_jobmanager>

-Stu

> On Nov 4, 2015, at 10:08 AM, Irena Johnson  wrote:
> 
> Stu,
> 
> Thank you for the link. I added 
> 
> more globus/globus-gram-job-manager.conf
> -globus-toolkit-version unknown
> -log-levels FATAL | ERROR | WARN | INFO | DEBUG
> -log-pattern /usr/pppl/globus/6.0_rh5/var/log/globus/gram_$(LOGNAME).log
> 
> 
> I am not sure why I am not getting any logs in 
> /usr/pppl/globus/6.0_rh5/var/log/globus/gram_tr_ijohnson.log
> 
> Could you please advise? Thank you
> 
> On Tue, Nov 3, 2015 at 5:22 PM, Stuart Martin  <mailto:smar...@mcs.anl.gov>> wrote:
> Ok.  Check the job manager's log file and see if there are any clues inside.
> http://toolkit.globus.org/toolkit/docs/latest-stable/gram5/admin/#idp34869296 
> <http://toolkit.globus.org/toolkit/docs/latest-stable/gram5/admin/#idp34869296>
> 
>> On Nov 3, 2015, at 3:34 PM, Irena Johnson > <mailto:ijohn...@pppl.gov>> wrote:
>> 
>> Hi Stu,
>> 
>> Thanks for your feedback. I confirm that the local user's home directory 
>> exists and is not full.
>> 
>> Regards,
>> Irena
>> 
>> 
>> 
>> On Tue, Nov 3, 2015 at 4:30 PM, Stuart Martin > <mailto:smar...@mcs.anl.gov>> wrote:
>> Hi Irena,
>> 
>> Here, it says to check that the local user’s home directory is writable and 
>> not full.  Either of those issues could prevent the job manager process from 
>> creating files in the mapped user’s home dir and cause this problem.
>> http://toolkit.globus.org/toolkit/docs/latest-stable/gram5/user/#gram5-error-codes
>>  
>> <http://toolkit.globus.org/toolkit/docs/latest-stable/gram5/user/#gram5-error-codes>
>> 
>> Cheers,
>> Stu
>> 
>>> On Nov 3, 2015, at 2:44 PM, Irena Johnson >> <mailto:ijohn...@pppl.gov>> wrote:
>>> 
>>> Dear Globus Support,
>>> 
>>> I have installed globus toolkit v. 6 (6.0.1443479657) from source on our 
>>> RHEL 5.10 system. 
>>> 
>>> I have configured different ports for gsiftp and gsigatekeeper (gsiftp6  
>>> 2194/tcp  and 
>>> gsigatekeeper6   2196/tcp).
>>> 
>>> I am able to run globus-url-copy successfully.
>>> 
>>> I can also run "globusrun -a -r globusserver.pppl.gov:2196 
>>> <http://globusserver.pppl.gov:2196/>"
>>> 
>>> However, I am getting an error when running:
>>> 
>>> globus-job-run  globusserver:2196/jobmanager-fork /bin/date
>>> GRAM Job submission failed because the job manager failed to create an 
>>> internal script argument file (error code 22)
>>> 
>>> I googled this error and found a claim that it occurs when the user does 
>>> not have a home directory on the gatekeeper. 
>>> 
>>> However, this is not the case. Please see below the log:
>>> 
>>> Nov  3 15:40:25 transpgrid1 xinetd[1157]: START: gsigatekeeper6 pid=32725 
>>> from=192.55.106.70
>>> Nov  3 15:40:25 transpgrid1 GRAM-gatekeeper[32725]: globus-gatekeeper 
>>> pid=32725 starting at Tue Nov  3 15:40:25 2015 
>>> Nov  3 15:40:25 transpgrid1 GRAM-gatekeeper[32725]: Got connection 
>>> 192.55.106.70 at Tue Nov  3 15:40:25 2015 
>>> Nov  3 15:40:25 transpgrid1 GRAM-gatekeeper[32725]: Authenticated globus 
>>> user: /DC=com/DC=DigiCert-Grid/O=Open Science Grid/OU=People/CN=Irena 
>>> Johnson 1116
>>> Nov  3 15:40:25 transpgrid1 GRAM-gatekeeper[32725]: Requested service: 
>>> jobmanager-fork 
>>> Nov  3 15:40:25 transpgrid1 GRAM-gatekeeper[32725]: Authorized as local 
>>> user: tr_ijohnson
>>> Nov  3 15:40:25 transpgrid1 GRAM-gatekeeper[32725]: Authorized as local 
>>> uid: 40491
>>> Nov  3 15:40:25 transpgrid1 GRAM-gatekeeper[32725]:   and local 
>>> gid: 28017
>>> Nov  3 15:40:25 transpgrid1 xinetd[1157]: EXIT: gsigatekeeper6 status=0 
>>> pid=32725 duration=0(sec)
>>> 
>>> 
>>> Could you please advise. Is this a bug? Is there a workaround? Thank you,
>>> 
>>> 
>>> Irena Johnson
>> 
>> 
>> 
>> 
>> 
> 
> 
> 
> 
> -- 
> Irena



Re: [gt-user] GRAM Job submission failed because the job manager failed to create an internal script argument file

2015-11-03 Thread Stuart Martin
Ok.  Check the job manager's log file and see if there are any clues inside.
http://toolkit.globus.org/toolkit/docs/latest-stable/gram5/admin/#idp34869296 
<http://toolkit.globus.org/toolkit/docs/latest-stable/gram5/admin/#idp34869296>
> On Nov 3, 2015, at 3:34 PM, Irena Johnson  wrote:
> 
> Hi Stu,
> 
> Thanks for your feedback. I confirm that the local user's home directory 
> exists and is not full.
> 
> Regards,
> Irena
> 
> 
> 
> On Tue, Nov 3, 2015 at 4:30 PM, Stuart Martin  <mailto:smar...@mcs.anl.gov>> wrote:
> Hi Irena,
> 
> Here, it says to check that the local user’s home directory is writable and 
> not full.  Either of those issues could prevent the job manager process from 
> creating files in the mapped user’s home dir and cause this problem.
> http://toolkit.globus.org/toolkit/docs/latest-stable/gram5/user/#gram5-error-codes
>  
> <http://toolkit.globus.org/toolkit/docs/latest-stable/gram5/user/#gram5-error-codes>
> 
> Cheers,
> Stu
> 
>> On Nov 3, 2015, at 2:44 PM, Irena Johnson > <mailto:ijohn...@pppl.gov>> wrote:
>> 
>> Dear Globus Support,
>> 
>> I have installed globus toolkit v. 6 (6.0.1443479657) from source on our 
>> RHEL 5.10 system. 
>> 
>> I have configured different ports for gsiftp and gsigatekeeper (gsiftp6  
>> 2194/tcp  and 
>> gsigatekeeper6   2196/tcp).
>> 
>> I am able to run globus-url-copy successfully.
>> 
>> I can also run "globusrun -a -r globusserver.pppl.gov:2196 
>> <http://globusserver.pppl.gov:2196/>"
>> 
>> However, I am getting an error when running:
>> 
>> globus-job-run  globusserver:2196/jobmanager-fork /bin/date
>> GRAM Job submission failed because the job manager failed to create an 
>> internal script argument file (error code 22)
>> 
>> I googled this error and found a claim that it occurs when the user does not 
>> have a home directory on the gatekeeper. 
>> 
>> However, this is not the case. Please see below the log:
>> 
>> Nov  3 15:40:25 transpgrid1 xinetd[1157]: START: gsigatekeeper6 pid=32725 
>> from=192.55.106.70
>> Nov  3 15:40:25 transpgrid1 GRAM-gatekeeper[32725]: globus-gatekeeper 
>> pid=32725 starting at Tue Nov  3 15:40:25 2015 
>> Nov  3 15:40:25 transpgrid1 GRAM-gatekeeper[32725]: Got connection 
>> 192.55.106.70 at Tue Nov  3 15:40:25 2015 
>> Nov  3 15:40:25 transpgrid1 GRAM-gatekeeper[32725]: Authenticated globus 
>> user: /DC=com/DC=DigiCert-Grid/O=Open Science Grid/OU=People/CN=Irena 
>> Johnson 1116
>> Nov  3 15:40:25 transpgrid1 GRAM-gatekeeper[32725]: Requested service: 
>> jobmanager-fork 
>> Nov  3 15:40:25 transpgrid1 GRAM-gatekeeper[32725]: Authorized as local 
>> user: tr_ijohnson
>> Nov  3 15:40:25 transpgrid1 GRAM-gatekeeper[32725]: Authorized as local uid: 
>> 40491
>> Nov  3 15:40:25 transpgrid1 GRAM-gatekeeper[32725]:   and local gid: 
>> 28017
>> Nov  3 15:40:25 transpgrid1 xinetd[1157]: EXIT: gsigatekeeper6 status=0 
>> pid=32725 duration=0(sec)
>> 
>> 
>> Could you please advise. Is this a bug? Is there a workaround? Thank you,
>> 
>> 
>> Irena Johnson
> 
> 
> 
> 
> 



Re: [gt-user] GRAM Job submission failed because the job manager failed to create an internal script argument file

2015-11-03 Thread Stuart Martin
Hi Irena,

Here, it says to check that the local user’s home directory is writable and not 
full.  Either of those issues could prevent the job manager process from 
creating files in the mapped user’s home dir and cause this problem.
http://toolkit.globus.org/toolkit/docs/latest-stable/gram5/user/#gram5-error-codes
 


Cheers,
Stu

> On Nov 3, 2015, at 2:44 PM, Irena Johnson  wrote:
> 
> Dear Globus Support,
> 
> I have installed globus toolkit v. 6 (6.0.1443479657) from source on our RHEL 
> 5.10 system. 
> 
> I have configured different ports for gsiftp and gsigatekeeper (gsiftp6  
> 2194/tcp  and 
> gsigatekeeper6   2196/tcp).
> 
> I am able to run globus-url-copy successfully.
> 
> I can also run "globusrun -a -r globusserver.pppl.gov:2196 
> "
> 
> However, I am getting an error when running:
> 
> globus-job-run  globusserver:2196/jobmanager-fork /bin/date
> GRAM Job submission failed because the job manager failed to create an 
> internal script argument file (error code 22)
> 
> I googled this error and found a claim that it occurs when the user does not 
> have a home directory on the gatekeeper. 
> 
> However, this is not the case. Please see below the log:
> 
> Nov  3 15:40:25 transpgrid1 xinetd[1157]: START: gsigatekeeper6 pid=32725 
> from=192.55.106.70
> Nov  3 15:40:25 transpgrid1 GRAM-gatekeeper[32725]: globus-gatekeeper 
> pid=32725 starting at Tue Nov  3 15:40:25 2015 
> Nov  3 15:40:25 transpgrid1 GRAM-gatekeeper[32725]: Got connection 
> 192.55.106.70 at Tue Nov  3 15:40:25 2015 
> Nov  3 15:40:25 transpgrid1 GRAM-gatekeeper[32725]: Authenticated globus 
> user: /DC=com/DC=DigiCert-Grid/O=Open Science Grid/OU=People/CN=Irena Johnson 
> 1116
> Nov  3 15:40:25 transpgrid1 GRAM-gatekeeper[32725]: Requested service: 
> jobmanager-fork 
> Nov  3 15:40:25 transpgrid1 GRAM-gatekeeper[32725]: Authorized as local user: 
> tr_ijohnson
> Nov  3 15:40:25 transpgrid1 GRAM-gatekeeper[32725]: Authorized as local uid: 
> 40491
> Nov  3 15:40:25 transpgrid1 GRAM-gatekeeper[32725]:   and local gid: 
> 28017
> Nov  3 15:40:25 transpgrid1 xinetd[1157]: EXIT: gsigatekeeper6 status=0 
> pid=32725 duration=0(sec)
> 
> 
> Could you please advise. Is this a bug? Is there a workaround? Thank you,
> 
> 
> Irena Johnson



[gt-user] test

2015-09-24 Thread Stuart Martin
Ignore


Re: [gt-user] Announcing GT 5.2.5 end-of-life

2015-09-24 Thread Stuart Martin
Hi Tom,  Comments inline.

> On Aug 5, 2015, at 12:50 PM, Tom Downes  wrote:
> 
> Hi Stuart:
> 
> Sorry for the delayed response as I was on vacation. A quick inspection of 
> the repository structure for GT6 versus GT5 eliminates a number of my 
> complaints.
> 
> http://toolkit.globus.org/ftppub/gt5/5.2/ 
> <http://toolkit.globus.org/ftppub/gt5/5.2/> (e.g. I believe stable points to 
> 5.2.2)
> http://toolkit.globus.org/ftppub/gt6/stable/deb/ 
> <http://toolkit.globus.org/ftppub/gt6/stable/deb/>
> 
> For example, GT5 had different directories for Debian 7 versus 8. You now do 
> this correctly by combining all deb-based dists under a single directory 
> structure. You haven't quite gotten all the way since the more Debian way for 
> this:
> 
> deb http://toolkit.globus.org/ftppub/gt6/stable/deb 
> <http://toolkit.globus.org/ftppub/gt6/stable/deb> wheezy contrib
> deb http://toolkit.globus.org/ftppub/gt6/testing/deb 
> <http://toolkit.globus.org/ftppub/gt6/testing/deb> wheezy contrib
> 
> would be 
> 
> deb http://toolkit.globus.org/ftppub/gt6/deb 
> <http://toolkit.globus.org/ftppub/gt6/deb> wheezy contrib
> deb http://toolkit.globus.org/ftppub/gt6/deb 
> <http://toolkit.globus.org/ftppub/gt6/deb> wheezy testing
> 
> But this is a major improvement to the old directory structure. So kudos!

Thanks!

> Comment 1: when you're managing N machines, it's much easier to know the 
> sources.list entries and the signing key. Automated provisioning software 
> (Foreman + preseed, for example) tend to be unfriendly toward unsigned 
> packages. Not impossible, but it's much easier when the documentation just 
> tells you the right path and points you to the key and you don't have to 
> export it manually from the GPG file. I can see why a .deb is easier for Joe 
> Sixpack walking up with one computer, but it's not easier for me.

This is mostly a documentation issue. We can pubish a directory of the repo 
configuration files and gpg key and include a link to that from the 
documentation in addition to the repo packages.  I created an issue for this.  
We’ll add this to our todo’s.

https://globus.atlassian.net/browse/GT-626 
<https://globus.atlassian.net/browse/GT-626>
> Comment 2: Do you intend for the stable repositories to contain the past 
> history of releases even when they have been superseded? I hope so as this 
> facilitates version pinning without being accidentally left behind.

We keep older versions in the repository.  So, you can specify an older version 
in case you need to back out from an update that goes bad, or stay pinned to a 
specific version if need be.

> Comment 3: What is the closest repository to "proposed" in Debian-speak? i.e. 
> a release that is going into the wild that, should no bug reports be filed, 
> will be moved unchanged into stable.

We have “Stable”, “Unstable” and “Testing” repos for each GT release version.  
Described here:

http://toolkit.globus.org/toolkit/docs/latest-stable/admin/install/#idp36524752

The Testing repo would be the closest to Debian “proposed”.  Once we/Globus 
developers determine packages are ready, packages move from Testing to Stable.  
“proposed” could be an email announcement and delay of those packages in the 
Testing repo in order to allow the Globus community to do additional testing.  
I don’t know if we actually want to do this delay, just thinking out loud.

> I am not sure how much consider gsisshd to be a part of the Toolkit, but it's 
> the most important part of the toolkit that I use, with gridftp being one 
> step down in importance. I believe that a lot of these problems still exist, 
> at least in the latest 5.2 series releases. We (LIGO, the scientific 
> collaboration for whom I work) repackage your packages to get around the bugs.
> 
> https://globus.atlassian.net/browse/GT-492 
> <https://globus.atlassian.net/browse/GT-492>

Yeah, we think we have these resolved in GT6.  Let us know if you find 
otherwise.

> I am dealing with a number of physical plant issues in the data center right 
> now, so I can't step through and re-create them on GT6. However, I was just 
> standing over my coworker's shoulder and was surprised to see that GT6's 
> gsi-openssh-server still installed it's startup/shutdown script into 
> /etc/init.d on Debian 8, otherwise now on systemd.

This has been in our backlog, but a low priority since the non-systemd methods 
still normally work.

> 
> --
> Tom Downes
> Senior Scientist and Data Center Manager
> Center for Gravitation, Cosmology and Astrophysics
> University of Wisconsin-Milwaukee
> 414.229.2678
> 
> On Tue, Jul 28, 2015 at 2:40 PM, Stuart Martin  <mailto:smar...@mcs.anl.gov&g

Re: [gt-user] directory transfers in RSL

2015-08-18 Thread Stuart Martin
Hi Adam,

Unfortunately, GRAM5 does not support creating a new directory as part of the 
file stage in or out, except via the scratch_dir RSL attribute. At present, 
stage in and out only support a list of file paths and not directories.

Do you have a sufficient workaround?  Is it working for you to make and remove 
the job directory outside of GRAM, e.g. before and after the GRAM job 
submission?

Regards,
Stu

> On Aug 13, 2015, at 11:32 AM, Adam Bazinet  wrote:
> 
> Dear Globus users/admins,
> 
> We've been trying to get directory transfers to work with RSL in GT6 (GRAM5), 
> but haven't had any luck so far. We haven't been able to find any examples 
> online of RSL that includes a directory transfer as part of file_stage_in or 
> file_stage_out, but we thought it might be possible because it does work in 
> our GT4-based system.
> 
> Here is an example attempted directory transfer:
> 
> (file_stage_in = 
> (gsiftp://arginine.umiacs.umd.edu/home/gt6admin/GT6Upgrade/GT4.2/garli/trunk/test/
>  
> emptyDir/
>  $(HOME)/2535472060.225141373834153/))
> 
> Note that $(HOME)/2535472060.225141373834153/ does not exist before this job 
> is executed --- transferring an empty directory and essentially renaming it 
> is the trick we use with GT4 to cause the working directory to be created 
> remotely before we stage additional files into it. However, we have also 
> tried non-empty directories, and those don't work either.
> 
> So anyway, the question's pretty simple: can you transfer entire directories 
> in GT6-compatible RSL? As an aside, we can get the behavior we want out of 
> globus-url-copy by adding the -cd and -r flags, but it's unclear if this 
> functionality is available through GRAM.
> 
> thanks,
> Adam
> 



Re: [gt-user] Announcing GT 5.2.5 end-of-life

2015-07-28 Thread Stuart Martin


> On Jul 18, 2015, at 6:49 PM, jcam...@upel.edu.ve wrote:
> 
> I agree with Tom.
> 
>  
>  
> El 2015-07-17 07:09, Tom Downes escribió:
> 
>> Stuart:
>>  
>> Maybe you could comment more specifically on what you intend for admins of 
>> OSes such as Debian 7 Wheezy which appear to include GT 5.2 based libraries 
>> but also promise security support through 201X, X>5?

Sorry for the delayed response.  I’ve been thinking about this some and plan to 
reply once the Globus team gets a chance to discuss this some more.

>>  
>> I have not always found the globus.org <http://globus.org/> repositories to 
>> be Debian-friendly so I'm not sure 5.2.X from debian.org 
>> <http://debian.org/> to 6.X from globus.org <http://globus.org/> is a clean 
>> upgrade path.

I’d encourage you to give the Globus GT6 repo a try.  We'd be very motivated to 
help work through any problems you run into.  We definitely want the Globus 
repos to be "Debian-friendly".

>> Tom
>> 
>> On Wed, Jul 15, 2015 at 2:33 PM, Stuart Martin > <mailto:smar...@mcs.anl.gov>> wrote:
>> Dear GT users,
>> 
>> We are announcing an end-of-life date for GT 5.2.5. Starting November 1, 
>> 2015, we will no longer support or maintain GT 5.2.5 or earlier versions of 
>> the Globus Toolkit. We believe this is sufficient notice for users of GT 
>> 5.2.5 to plan and complete their upgrade to GT6.
>> 
>> GT 6 was released in September 2014 and we are encouraging all users to move 
>> to this version as it provides multiple benefits over past releases. In 
>> particular, GT 6 has significantly simplified the building, testing, and 
>> distributing of the Toolkit, making it much easier for the developers to 
>> generate updates with bug fixes and new features, and for admins to install 
>> and maintain. We are currently working with package maintainers for EPEL, 
>> Fedora, Debian, and Ubuntu to ensure that the latest updates to GT6 are 
>> available quickly and included in these distributions. The latest GT release 
>> may be downloaded from: http://toolkit.globus.org/toolkit/downloads/6.0/ 
>> <http://toolkit.globus.org/toolkit/downloads/6.0/>.
>> 
>> Note that a GT release that is at end of life is: not supported, not 
>> receiving any updates (no minor, major, security fixes of any kind), and is 
>> not being built or tested on new OS versions. As with most open source 
>> projects, we have limited resources and many competing priorities so it is 
>> not possible for us to cost-effectively maintain multiple past releases if 
>> we are to continue delivering new features and improvements.
>> 
>> Thanks for your continued support of Globus.
>  
>  



[gt-user] Announcing GT 5.2.5 end-of-life

2015-07-15 Thread Stuart Martin
Dear GT users,

We are announcing an end-of-life date for GT 5.2.5. Starting November 1, 2015, 
we will no longer support or maintain GT 5.2.5 or earlier versions of the 
Globus Toolkit. We believe this is sufficient notice for users of GT 5.2.5 to 
plan and complete their upgrade to GT6.

GT 6 was released in September 2014 and we are encouraging all users to move to 
this version as it provides multiple benefits over past releases. In 
particular, GT 6 has significantly simplified the building, testing, and 
distributing of the Toolkit, making it much easier for the developers to 
generate updates with bug fixes and new features, and for admins to install and 
maintain. We are currently working with package maintainers for EPEL, Fedora, 
Debian, and Ubuntu to ensure that the latest updates to GT6 are available 
quickly and included in these distributions. The latest GT release may be 
downloaded from: http://toolkit.globus.org/toolkit/downloads/6.0/.

Note that a GT release that is at end of life is: not supported, not receiving 
any updates (no minor, major, security fixes of any kind), and is not being 
built or tested on new OS versions. As with most open source projects, we have 
limited resources and many competing priorities so it is not possible for us to 
cost-effectively maintain multiple past releases if we are to continue 
delivering new features and improvements.

Thanks for your continued support of Globus.

[gt-user] "Alternative chains certificate forgery" CVE-2015-1793 vulnerability

2015-07-09 Thread Stuart Martin
Hi All,

The Globus dev team has reviewed all Globus services and Globus Toolkit 
components to determine the impact of the vulnerability described in 
CVE-2015-1793  .  We have 
created a page where details about this issue will be communicated.

https://support.globus.org/entries/95308587 


Our assessment is that the severity of this vulnerability is extremely low.  
Only OpenSSL versions released since June 2015 (specifically, versions 1.0.2c, 
1.0.2b, 1.0.1n and 1.0.1o) are affected by this vulnerability.
Neither the Globus website, running services, nor software downloads use the 
vulnerable versions of OpenSSL.
Globus Toolkit clients and services may be vulnerable when used with an 
affected version of OpenSSL, though we are unaware of an attack vector.  
However, the currently supported platforms have not updated to the affected 
versions of OpenSSL.  Additionally, the versions of openssl distributed with 
Globus Connect Personal are not affected.
Actions We Have Taken to Close Attack Vector
None.  No action were required.
Recommended Actions for Globus Users and Administrators
We recommend any host with Globus services (e.g. Globus Connect Personal, 
Globus Connect Server, GridFTP, MyProxy, GSI-OpenSSH, GRAM) to review their 
host configuration and apply the advised OpenSSL updates if necessary.
Note: This is unlikely, as most major Linux distributions have not released an 
OpenSSL update since before June 2015.

Let us know if you have any questions.

- Globus Dev Team

Re: [gt-user] logjam vulnerability CVE-2015-4000

2015-06-09 Thread Stuart Martin
Correction.  Below is the correct forum link for this issue.

> On Jun 9, 2015, at 10:30 AM, Stuart Martin  wrote:
> 
> Hi All,
> 
> The Globus dev team has reviewed all Globus services and Globus Toolkit 
> components to determine the impact of the "logjam" vulnerability described in 
> CVE-2015-4000 
> <https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-4000>.  We have 
> created a page where details about this issue will be communicated.
> 

https://support.globus.org/entries/94083138

> 
> Our assessment is that there is a vulnerability for the Globus Toolkit 
> GridFTP and MyProxy components.  At present, these components do not prevent 
> the use of export ciphers for secure communication.  The exploit would 
> require a multi-step compromise on a network connection that would allow a 
> man-in-the-middle attack. This would be difficult to achieve but, since a 
> compromise is possible, we encourage all GridFTP and MyProxy services to be 
> updated as soon as possible.
> 
> For GSI-OpenSSH, we believe the impact is mitigated by the fact that the GSI 
> parts are protected inside the SSH protocol. Details from the OpenSSH 
> developers can be read here 
> <http://lists.mindrot.org/pipermail/openssh-unix-dev/2015-May/033896.html>.
> 
> GRAM is not impacted because it does not use ciphers for secure communication.
> Actions We Have Taken to Close Attack Vector
> An enhancement (GT-596 <https://globus.atlassian.net/browse/GT-596>) has been 
> implemented and made available for update for GT 6 and GT 5.2.5.
> The enhancement allows for an admin to set a specific cipher set to be used 
> for all Globus Toolkit components.
> The default ciphers configured for Globus Toolkit components will be the 
> OpenSSL defined “HIGH” ciphers.
> Documentation for the new configuration file is included in the GSIC admin 
> guide 
> <http://toolkit.globus.org/toolkit/docs/6.0/gsic/admin/#gsic-configuring-global-security-parameters>
> Recommended Actions for Globus Users and Administrators
> GridFTP Administrators
> Upgrading to the latest GT 6 
> <http://toolkit.globus.org/toolkit/advisories.html?version=6.0> or GT 5.2.5 
> <http://toolkit.globus.org/toolkit/advisories.html?version=5.2.5> packages 
> should be done ASAP.
> MyProxy Administrators
> Upgrading to the latest GT 6 
> <http://toolkit.globus.org/toolkit/advisories.html?version=6.0> or GT 5.2.5 
> <http://toolkit.globus.org/toolkit/advisories.html?version=5.2.5> packages 
> should be done ASAP.
> GSI-OpenSSH Administrators
> No action is needed at this time.
> However, we encourage upgrading to the latest GT 6 
> <http://toolkit.globus.org/toolkit/advisories.html?version=6.0> packages as a 
> precaution.
> GRAM Administrators
> No action is needed at this time.
> However, we encourage upgrading to the latest GT 6 
> <http://toolkit.globus.org/toolkit/advisories.html?version=6.0> packages as a 
> precaution.
> Globus Connect Server Administrators
> Upgrading to the latest version ASAP using your operating system’s package 
> manager, e.g. yum update, apt-get update/upgrade, etc.
> Globus Connect Personal users
> Upgrading to the latest version should be done ASAP.
> Update steps 
> <https://support.globus.org/entries/94287798-Updating-to-the-latest-version-of-Globus-Connect-Personal>
> 
> 
> Let us know if you have any questions.
> 
> - Globus Dev Team



[gt-user] logjam vulnerability CVE-2015-4000

2015-06-09 Thread Stuart Martin
Hi All,

The Globus dev team has reviewed all Globus services and Globus Toolkit 
components to determine the impact of the "logjam" vulnerability described in 
CVE-2015-4000 . 
 We have created a page where details about this issue will be communicated.

https://support.globus.org/entries/90923228 


Our assessment is that there is a vulnerability for the Globus Toolkit GridFTP 
and MyProxy components.  At present, these components do not prevent the use of 
export ciphers for secure communication.  The exploit would require a 
multi-step compromise on a network connection that would allow a 
man-in-the-middle attack. This would be difficult to achieve but, since a 
compromise is possible, we encourage all GridFTP and MyProxy services to be 
updated as soon as possible.

For GSI-OpenSSH, we believe the impact is mitigated by the fact that the GSI 
parts are protected inside the SSH protocol. Details from the OpenSSH 
developers can be read here 
.

GRAM is not impacted because it does not use ciphers for secure communication.
Actions We Have Taken to Close Attack Vector
An enhancement (GT-596 ) has been 
implemented and made available for update for GT 6 and GT 5.2.5.
The enhancement allows for an admin to set a specific cipher set to be used for 
all Globus Toolkit components.
The default ciphers configured for Globus Toolkit components will be the 
OpenSSL defined “HIGH” ciphers.
Documentation for the new configuration file is included in the GSIC admin 
guide 

Recommended Actions for Globus Users and Administrators
GridFTP Administrators
Upgrading to the latest GT 6 
 or GT 5.2.5 
 packages 
should be done ASAP.
MyProxy Administrators
Upgrading to the latest GT 6 
 or GT 5.2.5 
 packages 
should be done ASAP.
GSI-OpenSSH Administrators
No action is needed at this time.
However, we encourage upgrading to the latest GT 6 
 packages as a 
precaution.
GRAM Administrators
No action is needed at this time.
However, we encourage upgrading to the latest GT 6 
 packages as a 
precaution.
Globus Connect Server Administrators
Upgrading to the latest version ASAP using your operating system’s package 
manager, e.g. yum update, apt-get update/upgrade, etc.
Globus Connect Personal users
Upgrading to the latest version should be done ASAP.
Update steps 



Let us know if you have any questions.

- Globus Dev Team

Re: [gt-user] [gt-dev] Migrating GT services to TLS-only

2015-03-27 Thread Stuart Martin
Hi All,

April 2 is only 6 days away.  Everyone has had time to upgrade their GT 
installations in order to avoid any incompatibilities when services are 
configured to disallow SSLv3.  Starting Thursday, April 2, go ahead and make 
the change to prevent the use of SSLv3.  This can be done by setting the 
environment variable “GLOBUS_GSSAPI_FORCE_TLS” before starting any of the GT 
services: GridFTP, GRAM (gatekeeper), MyProxy, GSISSH.  Please see the service 
admin guides for details - http://toolkit.globus.org/toolkit/docs/latest-stable/

If after making the change you see errors coming from the services like this:

530-globus_xio: Authentication Error
530-globus_gsi_gssapi: Unable to verify remote side's credentials
530-globus_gsi_gssapi: SSLv3 handshake problems: Couldn't do ssl handshake
530-OpenSSL Error: s3_srvr.c:965: in library: SSL routines, function
SSL3_GET_CLIENT_HELLO: wrong version number
530 End.

That would indicate that some users are still using an old/incompatible version 
of the client.

Hopefully, there will be very few issues, since we have given everyone a good 
amount of time to prepare.

Cheers,
Stu

On Dec 8, 2014, at Dec 8, 11:30 AM, Stuart Martin  wrote:

> Hi All,
> 
> Here is an update on the first milestone for upgrading GRAM and MyProxy 
> client installations to be TLS-compatible prior to any GRAM and MyProxy 
> services being configured to be TLS-only.
> 
> Due to concerns shared from some organizations that they may not be able to 
> get their clients updated before Jan 1, 2015, we are now recommending all 
> users to delay configuring their Globus Toolkit services to be TLS-only until 
> after *April 1, 2015*. 
> 
> Prior to this April 1 deadline, we recommend all client installations upgrade 
> the GRAM and MyProxy clients to (at least) the following version numbers. 
> These add support for TLS to those components:
> 
> GT 6.0 GRAM TLS package: globus_gram_client-13.11
> GT 6.0 MyProxy TLS package: myproxy-6.1.8
> 
> GT 5.2 GRAM TLS package: globus_gram_client-12.5
> GT 5.2 MyProxy TLS package: None**
> 
> ** There are no plans to create a GT 5.2 MyProxy client update package, a 
> MyProxy client installation will have to be 6.0 to be fully compatible with a 
> TLS-only MyProxy service.
> 
> For Mac and Windows client installations, we will make available a new set of 
> GT 6.0 installers that contain the GRAM and MyProxy client updates. These 
> will be coming soon.
> 
> Let us know if you have any questions.
> 
> -Globus Dev Team
> 
> On Oct 21, 2014, at Oct 21, 1:54 PM, Stuart Martin  
> wrote:
> 
>> Hi All,
>> 
>> Due to the recently announced POODLE issue 
>> (https://support.globus.org/entries/101814643), we are planning to disable 
>> SSLv3 support in Globus Toolkit components.  All users maintaining GT 
>> installations older than 5.2 will need to upgrade to remain compatible with 
>> GT services that disable SSLv3 by July 1, 2015.
>> 
>> There is no immediate threat, so we can proceed with a priority on limiting 
>> the impact of incompatibility for end users.
>> 
>> (Now) The Globus team’s recommendation is for the entire ecosystem to 
>> upgrade to a supported release, either GT 6.0 or 5.2, both of which support 
>> TLS. This will allow a transition period where clients and services will be 
>> able to communicate with either TLS or SSLv3, with newer clients and 
>> services choosing TLS by default. We DO NOT recommend disabling SSLv3 for 
>> ANY installations during this transition time as it will cause 
>> incompatibility with older clients and services that haven’t completed the 
>> transition.
>> 
>> On January 1, 2015, we will begin the transition to configure Globus Toolkit 
>> clients and services as TLS-only by disabling SSLv3. We will provide 
>> documentation on how to update services to do so.
>> 
>> On July 1, 2015, we will update our security packages to disable SSLv3 and 
>> require TLS for all secure communication.
>> 
>> Note: Maintainers of non-GT clients and servers that are part of a 
>> community’s ecosystem should ensure their software can operate in the 
>> upcoming TLS-only environment.
>> 
>> Note: We will provide an update to the GRAM client remove use of SSLv3 prior 
>> to the transition period.
>> 
>> -Globus Dev Team
> 



[gt-user] OpenSSL vulnerability CVE-2015-0291

2015-03-19 Thread Stuart Martin
Hi All,

The Globus dev team has reviewed all Globus services and Globus Toolkit 
components to determine the impact of the OpenSSL vulnerability described in 
CVE-2015-0291.  We have created a page where details about this issue will be 
communicated.

https://support.globus.org/entries/90923228

Our assessment is that the Globus service and the Globus Toolkit clients and 
services are not vulnerable.

We recommend any host with Globus services (e.g. Globus Connect Personal, 
Globus Connect Server, GridFTP, MyProxy, GSI-OpenSSH, GRAM) to review their 
host configuration and apply the advised OpenSSL updates if necessary. 

Let us know if you have any questions.

- Globus Dev Team


[gt-user] test

2015-02-17 Thread Stuart Martin
ignore


[gt-user] GHOST vulnerability CVE-2015-0235

2015-01-28 Thread Stuart Martin
Hi All,

The Globus dev team has reviewed all Globus services and Globus Toolkit 
components to determine the impact of the “GHOST" vulnerability described in 
CVE-2015-0235.  We have created a page where details about this issue will be 
communicated.

https://support.globus.org/entries/87762727

Toolkit Summary: The Globus Toolkit contains some programs that call 
gethostbyname*() functions, but the conditions under which known exploits could 
result in any material damage are highly unlikely.

Please, see the above forum post for the recommended actions.

Let us know if you have any questions.

- Globus Dev Team

Re: [gt-user] [gt-dev] Migrating GT services to TLS-only

2014-12-08 Thread Stuart Martin
Hi All,

Here is an update on the first milestone for upgrading GRAM and MyProxy client 
installations to be TLS-compatible prior to any GRAM and MyProxy services being 
configured to be TLS-only.

Due to concerns shared from some organizations that they may not be able to get 
their clients updated before Jan 1, 2015, we are now recommending all users to 
delay configuring their Globus Toolkit services to be TLS-only until after 
*April 1, 2015*. 

Prior to this April 1 deadline, we recommend all client installations upgrade 
the GRAM and MyProxy clients to (at least) the following version numbers. These 
add support for TLS to those components:

GT 6.0 GRAM TLS package: globus_gram_client-13.11
GT 6.0 MyProxy TLS package: myproxy-6.1.8

GT 5.2 GRAM TLS package: globus_gram_client-12.5
GT 5.2 MyProxy TLS package: None**

** There are no plans to create a GT 5.2 MyProxy client update package, a 
MyProxy client installation will have to be 6.0 to be fully compatible with a 
TLS-only MyProxy service.

For Mac and Windows client installations, we will make available a new set of 
GT 6.0 installers that contain the GRAM and MyProxy client updates. These will 
be coming soon.

Let us know if you have any questions.

-Globus Dev Team

On Oct 21, 2014, at Oct 21, 1:54 PM, Stuart Martin  wrote:

> Hi All,
> 
> Due to the recently announced POODLE issue 
> (https://support.globus.org/entries/101814643), we are planning to disable 
> SSLv3 support in Globus Toolkit components.  All users maintaining GT 
> installations older than 5.2 will need to upgrade to remain compatible with 
> GT services that disable SSLv3 by July 1, 2015.
> 
> There is no immediate threat, so we can proceed with a priority on limiting 
> the impact of incompatibility for end users.
> 
> (Now) The Globus team’s recommendation is for the entire ecosystem to upgrade 
> to a supported release, either GT 6.0 or 5.2, both of which support TLS. This 
> will allow a transition period where clients and services will be able to 
> communicate with either TLS or SSLv3, with newer clients and services 
> choosing TLS by default. We DO NOT recommend disabling SSLv3 for ANY 
> installations during this transition time as it will cause incompatibility 
> with older clients and services that haven’t completed the transition.
> 
> On January 1, 2015, we will begin the transition to configure Globus Toolkit 
> clients and services as TLS-only by disabling SSLv3. We will provide 
> documentation on how to update services to do so.
> 
> On July 1, 2015, we will update our security packages to disable SSLv3 and 
> require TLS for all secure communication.
> 
> Note: Maintainers of non-GT clients and servers that are part of a 
> community’s ecosystem should ensure their software can operate in the 
> upcoming TLS-only environment.
> 
> Note: We will provide an update to the GRAM client remove use of SSLv3 prior 
> to the transition period.
> 
> -Globus Dev Team



[gt-user] Migrating GT services to TLS-only

2014-10-21 Thread Stuart Martin
Hi All,

Due to the recently announced POODLE issue 
(https://support.globus.org/entries/101814643), we are planning to disable 
SSLv3 support in Globus Toolkit components.  All users maintaining GT 
installations older than 5.2 will need to upgrade to remain compatible with GT 
services that disable SSLv3 by July 1, 2015.

There is no immediate threat, so we can proceed with a priority on limiting the 
impact of incompatibility for end users.

(Now) The Globus team’s recommendation is for the entire ecosystem to upgrade 
to a supported release, either GT 6.0 or 5.2, both of which support TLS. This 
will allow a transition period where clients and services will be able to 
communicate with either TLS or SSLv3, with newer clients and services choosing 
TLS by default. We DO NOT recommend disabling SSLv3 for ANY installations 
during this transition time as it will cause incompatibility with older clients 
and services that haven’t completed the transition.

On January 1, 2015, we will begin the transition to configure Globus Toolkit 
clients and services as TLS-only by disabling SSLv3. We will provide 
documentation on how to update services to do so.

On July 1, 2015, we will update our security packages to disable SSLv3 and 
require TLS for all secure communication.

Note: Maintainers of non-GT clients and servers that are part of a community’s 
ecosystem should ensure their software can operate in the upcoming TLS-only 
environment.

Note: We will provide an update to the GRAM client remove use of SSLv3 prior to 
the transition period.

-Globus Dev Team

[gt-user] SSLv3 POODLE vulnerability CVE-2014-3566

2014-10-16 Thread Stuart Martin
Hi All,

The Globus dev team has reviewed all Globus services and Globus Toolkit 
components to determine the impact of the SSLv3 “POODLE” vulnerability 
described in CVE-2014-3566.  We have created a page where details about this 
issue will be communicated.

 https://support.globus.org/entries/101814643

Toolkit Summary: After a thorough investigation, we cannot identify any attack 
vectors for the Globus Toolkit clients or services.  However, disabling SSLv3 
for GridFTP, MyProxy, GSI-OpenSSH or GRAM could cause incompatibility between 
clients and services.  Please, see the forum post for the recommended actions.

Let us know if you have any questions.

- Globus Dev Team

[gt-user] remote code execution through bash CVE-2014-6271

2014-09-25 Thread Stuart Martin
Hi All,

The Globus dev team has reviewed all Globus services and Globus Toolkit 
components to determine the impact of the remote code execution through bash 
described in CVE-2014-6271.  We have created a page where details about this 
issue will be communicated.

   https://support.globus.org/entries/99833293

Our initial assessment has found no possible exploits from this bash 
vulnerability.  However, as a precaution, we recommend that any host with 
Globus service (e.g. Globus Connect Personal, Globus Connect Server, GridFTP, 
MyProxy, GSI-OpenSSH, GRAM) to apply the advised patches ASAP.

GSI-OpenSSH users and administrators using OpenSSH's ForceCommand functionality 
to restrict the remote commands that a user can run should refer to the RedHat 
security blog 
(https://securityblog.redhat.com/2014/09/24/bash-specially-crafted-environment-variables-code-injection-attack/)
 and article (https://access.redhat.com/articles/1200223) which discusses the 
potential to bypass command restrictions using this vulnerability.

MyProxy server administrators using bash scripts with myproxy-server call-out 
functionality (passphrase_policy_program, proxy_extapp, 
certificate_issuer_program, certificate_extapp, certificate_mapapp, 
certificate_request_checker, certificate_issuer_checker, or 
accepted_credentials_mapapp) may be impacted and should promptly apply 
available patches.

Let us know if you have any questions.

- Globus Dev Team 

[gt-user] OpenSSL CCS Injection Vulnerability impact on Globus

2014-06-05 Thread Stuart Martin
We are aware and reviewing all Globus services and Globus Toolkit components to 
determine the impact of the OpenSSL vulnerability described in CVE-2014-0224 
(CCS Injection Vulnerability).  We have created a page where details about this 
issue will be communicated.

   https://support.globus.org/entries/71973746

Our initial assessment is that the nature of this CVE requires several unusual 
preconditions to be met and therefore the relative impact of this particular 
OpenSSL issue is low.  However, as a precaution, we recommend that any host 
with Globus services (e.g. Globus Connect Server, GridFTP, MyProxy, 
GSI-OpenSSH, GRAM) running OpenSSL version 1.0.1 or earlier to update ASAP.  
Additional details about possible impacts to specific Globus services will 
follow.

-Stu

Re: [gt-user] GridFTP DCSC transfer using different source and destination certs

2013-12-04 Thread Stuart Martin
Yeah.  Probably best in the gridftp user guide section, maybe here (even though 
this is not so "basic"):

http://toolkit.globus.org/toolkit/docs/latest-stable/gridftp/user/#gridftp-user-basic

I'll discuss with Mike to get it added.

On Dec 4, 2013, at Dec 4, 4:50 PM, Ian Foster  wrote:

> Should this be in our FAQ?
> 
> On Dec 4, 2013, at 3:53 PM, Stuart Martin  wrote:
> 
>> Hi All,
>> 
>> This was an off-list thread that may be helpful or informative to others, so 
>> I am posting it here.
>> 
>> Scott's use case from his original email is:
>> "Specifically, I am trying to transfer from Stampede to Blue Waters, using a 
>> community account certificate to authenticate to Stampede and my user cert 
>> to Blue Waters."
>> 
>> -Stu
>> 
>> Begin forwarded message:
>> 
>>> From: Scott Callaghan 
>>> Subject: RE: Using different source and destination certs
>>> Date: December 4, 2013 3:19:16 PM CST
>>> To: Michael Link , Stuart Martin 
>>> 
>>> Hi Mike,
>>> 
>>> That worked!  I used my user cert as -data-cred since both ends can handle 
>>> that one.  Thanks for your help!
>>> 
>>> -Scott
>>> 
>>> From: Michael Link 
>>> Sent: Wednesday, December 4, 2013 3:10 PM
>>> To: Scott Callaghan; Stuart Martin
>>> Subject: Re: Using different source and destination certs
>>> 
>>> Ah, sorry about that, auto may have been added in 5.0.5 or 5.2.x.  You
>>> can use either your src or dst cred for -data-cred as well -- it should
>>> be packaged in a way that both servers can accept it.
>>> 
>>> Mike
>>> 
>>> On 12/4/2013 2:39 PM, Scott Callaghan wrote:
>>>> Hi Mike,
>>>> 
>>>> I tried that, but it seems like -data-cred is expecting a file as an 
>>>> argument:
>>>> 
>>>> globus-url-copy -data-cred auto -dbg -vb -src-cred /tmp/x509up_u801878 
>>>> -dst-cred /tmp/x509up_u33527 
>>>> gsiftp://gridftp.stampede.tacc.utexas.edu/home1/00940/tera3d/test.txt 
>>>> gsiftp://bw-gridftp.ncsa.illinois.edu/u/sciteam/scottcal/test.txt
>>>> Error loading data channel credential: GSS Major Status: General failure
>>>> globus_gsi_gssapi: Unable to read credential for import: Couldn't open the 
>>>> file: auto
>>>> 
>>>> I'm running guc version 5.14, as part of GT 5.0.4, in case it's a version 
>>>> issue.  Thanks for your help with this!
>>>> 
>>>> -Scott
>>>> 
>>>> From: Michael Link 
>>>> Sent: Wednesday, December 4, 2013 2:21 PM
>>>> To: Scott Callaghan; Stuart Martin
>>>> Subject: Re: Using different source and destination certs
>>>> 
>>>> Hi Scott,
>>>> 
>>>> I thought using both -src-cred and -dst-cred would automatically use
>>>> DCSC, but you can force by adding '-data-cred auto'
>>>> 
>>>> Mike
>>>> 
>>>> On 12/4/2013 2:10 PM, Scott Callaghan wrote:
>>>>> Hi Stu,
>>>>> 
>>>>> I used the command:
>>>>> 
>>>>> globus-url-copy -dbg -vb -src-cred /tmp/x509up_u801878 -dst-cred 
>>>>> /tmp/x509up_u33527 
>>>>> gsiftp://gridftp.stampede.tacc.utexas.edu/home1/00940/tera3d/test.txt 
>>>>> gsiftp://bw-gridftp.ncsa.illinois.edu/u/sciteam/scottcal/test.txt
>>>>> 
>>>>> /tmp/x509up_u801878 is the community account proxy, /tmp/x509up_u33527 is 
>>>>> the scottcal proxy.
>>>>> 
>>>>> -Scott
>>>>> 
>>>>> From: Stuart Martin 
>>>>> Sent: Wednesday, December 4, 2013 2:05 PM
>>>>> To: Scott Callaghan; Mike Link
>>>>> Cc: Stuart Martin
>>>>> Subject: Re: Using different source and destination certs
>>>>> 
>>>>> Hey Scott,
>>>>> 
>>>>> You should be able to do this with guc.  Can you reply with the specific 
>>>>> options you are using on the guc command?  Here are the relevant options 
>>>>> to use.
>>>>> -cred 
>>>>> -src-cred | -sc 
>>>>> -dst-cred | -dc 
>>>>>Set the credentials to use for source, destination,
>>>>>or both ftp connections.

[gt-user] GridFTP DCSC transfer using different source and destination certs

2013-12-04 Thread Stuart Martin
Hi All,

This was an off-list thread that may be helpful or informative to others, so I 
am posting it here.

Scott's use case from his original email is:
"Specifically, I am trying to transfer from Stampede to Blue Waters, using a 
community account certificate to authenticate to Stampede and my user cert to 
Blue Waters."

-Stu

Begin forwarded message:

> From: Scott Callaghan 
> Subject: RE: Using different source and destination certs
> Date: December 4, 2013 3:19:16 PM CST
> To: Michael Link , Stuart Martin 
> 
> Hi Mike,
> 
> That worked!  I used my user cert as -data-cred since both ends can handle 
> that one.  Thanks for your help!
> 
> -Scott
> 
> From: Michael Link 
> Sent: Wednesday, December 4, 2013 3:10 PM
> To: Scott Callaghan; Stuart Martin
> Subject: Re: Using different source and destination certs
> 
> Ah, sorry about that, auto may have been added in 5.0.5 or 5.2.x.  You
> can use either your src or dst cred for -data-cred as well -- it should
> be packaged in a way that both servers can accept it.
> 
> Mike
> 
> On 12/4/2013 2:39 PM, Scott Callaghan wrote:
>> Hi Mike,
>> 
>> I tried that, but it seems like -data-cred is expecting a file as an 
>> argument:
>> 
>> globus-url-copy -data-cred auto -dbg -vb -src-cred /tmp/x509up_u801878 
>> -dst-cred /tmp/x509up_u33527 
>> gsiftp://gridftp.stampede.tacc.utexas.edu/home1/00940/tera3d/test.txt 
>> gsiftp://bw-gridftp.ncsa.illinois.edu/u/sciteam/scottcal/test.txt
>> Error loading data channel credential: GSS Major Status: General failure
>> globus_gsi_gssapi: Unable to read credential for import: Couldn't open the 
>> file: auto
>> 
>> I'm running guc version 5.14, as part of GT 5.0.4, in case it's a version 
>> issue.  Thanks for your help with this!
>> 
>> -Scott
>> 
>> From: Michael Link 
>> Sent: Wednesday, December 4, 2013 2:21 PM
>> To: Scott Callaghan; Stuart Martin
>> Subject: Re: Using different source and destination certs
>> 
>> Hi Scott,
>> 
>> I thought using both -src-cred and -dst-cred would automatically use
>> DCSC, but you can force by adding '-data-cred auto'
>> 
>> Mike
>> 
>> On 12/4/2013 2:10 PM, Scott Callaghan wrote:
>>> Hi Stu,
>>> 
>>> I used the command:
>>> 
>>> globus-url-copy -dbg -vb -src-cred /tmp/x509up_u801878 -dst-cred 
>>> /tmp/x509up_u33527 
>>> gsiftp://gridftp.stampede.tacc.utexas.edu/home1/00940/tera3d/test.txt 
>>> gsiftp://bw-gridftp.ncsa.illinois.edu/u/sciteam/scottcal/test.txt
>>> 
>>> /tmp/x509up_u801878 is the community account proxy, /tmp/x509up_u33527 is 
>>> the scottcal proxy.
>>> 
>>> -Scott
>>> 
>>> From: Stuart Martin 
>>> Sent: Wednesday, December 4, 2013 2:05 PM
>>> To: Scott Callaghan; Mike Link
>>> Cc: Stuart Martin
>>> Subject: Re: Using different source and destination certs
>>> 
>>> Hey Scott,
>>> 
>>> You should be able to do this with guc.  Can you reply with the specific 
>>> options you are using on the guc command?  Here are the relevant options to 
>>> use.
>>>   -cred 
>>>   -src-cred | -sc 
>>>   -dst-cred | -dc 
>>>  Set the credentials to use for source, destination,
>>>  or both ftp connections.
>>>   -data-cred 
>>>  Set the credential to use for data connection.  A value of 'auto' will
>>>  generate a temporary self-signed credential.  This may be used with
>>>  any authentication method, but the server must support the DCSC 
>>> command.
>>> 
>>> Also, Globus Transfer would do this for you after you activate each 
>>> endpoint with the credential.  So, you could let Globus do the work for you 
>>> :-)
>>> 
>>> including Mike for any additional followup.
>>> 
>>> Cheers,
>>> Stu
>>> 
>>> On Dec 4, 2013, at Dec 4, 1:18 PM, Scott Callaghan  wrote:
>>> 
>>>> Hi Stu,
>>>> 
>>>> Good to see you at SC.
>>>> 
>>>> I tried out using different certificates to authenticate to the source and 
>>>> destination, using a third-party transfer.  It looks like the 
>>>> authentication goes fine, but then, as I understand it, both hosts also 
>>>> have to be able to authenticate to the other certificate, and I think 
>>>>

Re: [gt-user] Enforce data Encryption on grid-ftp

2013-02-13 Thread Stuart Martin


What...?  Brock, you'll be giving a Globus World (http://www.globusworld.org/) 
talk which is this April 16 - 18?  That's great!  I'll be sure to register 
(http://www.globusworld.org/register.php) now and make my plans to attend!  
Thanks so much for letting me know!



On Feb 13, 2013, at Feb 13, 2:16 PM, Brock Palen wrote:

> Don't have them exactly yet, but wait for my Globus World talk ;-)
> 
> Brock Palen
> www.umich.edu/~brockp
> CAEN Advanced Computing
> bro...@umich.edu
> (734)936-1985
> 
> 
> 
> On Feb 13, 2013, at 2:59 PM, Raj Kettimuthu wrote:
> 
>> On Feb 13, 2013, at 1:33 PM, Brock Palen wrote:
>> 
>>> I think so.  GO would be our main client anyway. 
>>> 
>>> Is there a way for force it from GO?
>> 
>> Currently, there is no way for users (as endpoint owners) to set this. I 
>> have put in a Jira item for this. If you send me the endpoints that you want 
>> to force encryption, we can set this for you.
>> 
>> Raj
>> 
>>> 
>>> Brock Palen
>>> www.umich.edu/~brockp
>>> CAEN Advanced Computing
>>> bro...@umich.edu
>>> (734)936-1985
>>> 
>>> 
>>> 
>>> On Feb 13, 2013, at 2:28 PM, Raj Kettimuthu wrote:
>>> 
 Hi Brock,
 GridFTP server does not provide an option to enforce encryption for all 
 transfers. But here is what we can do:
 1. Use '-allow-from' option in the server to ensure that users can access 
 this server only from Globus Online
 2. Make sure that Globus Online always enables encryption for transfers to 
 and from this endpoint
 
 Will this work for your use case?
 
 Raj
 
 On Feb 13, 2013, at 12:18 PM, Brock Palen wrote:
 
> Is there a way to require encryption when talking to a grid-ftp server? 
> We have a case where data are sensitive and the requirement users are 
> that they never be transmitted in the clear. 
> 
> We would rather not leave setting encryption to every user request. 
> 
> Thanks!
> 
> Brock Palen
> www.umich.edu/~brockp
> CAEN Advanced Computing
> bro...@umich.edu
> (734)936-1985
> 
> 
> 
 
>>> 
>> 
> 



Re: [gt-user] *other* components that use the globus-ftp-client and globus-xio *libraries* (e.g. the FTS transfer agents), where the IPv6 option is *not* enabled by default.

2013-01-17 Thread Stuart Martin
Hi Adrian,

This was recently addressed and will be included in the next release, GT 5.2.4.

Here is the issue for details:
http://jira.globus.org/browse/GT-32

Please take a look and see if that will provide what you need.

Cheers,
Stu

On Jan 17, 2013, at Jan 17, 6:12 AM, Adrian Stelmaszyk wrote:

> There are *other* components that use the globus-ftp-client and globus-xio 
> *libraries* (e.g. the FTS transfer agents), where the IPv6 option is *not* 
> enabled by default, and can only be enabled via specific API calls 
> (globus_ftp_client_operationattr_set_allow_ipv6, 
> globus_io_attr_set_tcp_allow_ipv6). 
> It would be much preferable if IPv6 could be enabled (via an environment 
> variable, or the like) at the Globus library level, rather than having to 
> modify all existing users of the Globus libraries and add a configuration 
> option in each of them. The history of GGUS #80628 seems to imply that Globus 
> was willing to address this at the library level, but, as of GT5.2.3 (I just 
> checked it as well) nothing has occurred yet there. 
> 
> -- 
> Best Regards
> Adrian Stelmaszyk



Re: [gt-user] [gt-announce] GT 5.2.3 Release

2012-12-04 Thread Stuart Martin
We just realized the GridFTP highlights were incorrect.  Corrected below...

On Dec 3, 2012, at Dec 3, 8:43 AM, Joseph Bester wrote:

> The GT development team is pleased to make a new stable release of the Globus 
> Toolkit available for download.
> 
> Download links:
> http://www.globus.org/toolkit/downloads/5.2.3/
> 
> Quickstart and Detailed installation instructions:
> http://www.globus.org/toolkit/docs/5.2/5.2.3/admin/
> 
> If you've already installed the 5.2 stable repository package, you can get 
> the new packages via an apt or yum update without installing a new repo.
> 
> Highlights of this release include:

- GridFTP
- Fixed logging bugs found during XSEDE validation testing

> - GRAM5
> - Add support for LSF as a local resource manager.
> - Fix some stability bugs with SEG.
> 
> - MyProxy
> - Updated to MyProxy v5.9.
> 
> - GSI-Enabled OpenSSH
> - Updated to gsissh v5.5.
> 
> This release adds support for Ubuntu 12.04LTS and 12.10
> 
> Also supported with native RPM or Debian Packages: 
> - CentOS 4, 5, 6; 
> - Fedora 16, 17; 
> - RHEL 5, 6;
> - Scientific Linux 5, 6; 
> - Debian 6, 7 (testing); 
> - Ubuntu 10.04, 11.10, 12.04, 12.10; 
> 
> The toolkit is also tested on the following platforms: Solaris 11, MacOS 10.8 
> (Mountain Lion)



Re: [gt-user] who maintains the Globus Toolkit documentation?

2012-10-19 Thread Stuart Martin
Here is a jira for tracking http://jira.globus.org/browse/GT-308

On Oct 19, 2012, at Oct 19, 10:14 AM, Stuart Martin wrote:

> Sounds good.  As Vas says, we're looking to make some changes/improvements to 
> the Toolkit web pages.  That may be a longer term effort.  But, maybe there 
> are some simple, near term changes that can be made to get this going.  We'll 
> discuss internally here and let you know our plans.
> 
> Thanks for the input!
> 
> -Stu
> 
> On Oct 19, 2012, at Oct 19, 9:28 AM, Pete Eby wrote:
> 
>> I thinking having the install and quickstart guides in a wiki format
>> would be extremely helpful. The globus user community seems well
>> suited to help maintain this documentation. These guides are very
>> helpful, and allowing them to be updated might help to improve them
>> further.
>> 
>> Also, perhaps an additional community wiki page could be created for
>> trouble shooting common issues. There seems to be some "top 10" things
>> users encounter frequently, and this might help save them a
>> considerable amount of debug / research time.
>> 
>> Pete
>> 
>> On Fri, Oct 19, 2012 at 10:22 AM, Stuart Martin  wrote:
>>> I like that idea a lot.  We use docbook to generate most of the toolkit's 
>>> web content.  We'll have to think through if and where taking on a 
>>> transition like that would make sense.  Maybe there are some GT pages that 
>>> are best left to be generated via docbook and some that could be changed to 
>>> become wiki pages.
>>> 
>>> For example, I'm thinking (partially based on the input here) that the 
>>> install and quick start might be good candidates to be wiki pages.
>>>   
>>> http://www.globus.org/toolkit/docs/latest-stable/admin/install/#gtadmin
>>>   
>>> http://www.globus.org/toolkit/docs/latest-stable/admin/quickstart/#quickstart
>>> 
>>> Maybe this developer guide too?
>>>   http://www.globus.org/toolkit/docs/5.2/5.2.2/appendices/developer/
>>> 
>>> Thoughts?
>>> 
>>> -Stu
>>> 
>>> On Oct 19, 2012, at Oct 19, 6:09 AM, Pete Eby wrote:
>>> 
>>>> Has the idea of placing the documentation on a wiki where it could be
>>>> updated by community members been discussed? Perhaps by users with a
>>>> Globus Online login?
>>>> 
>>>> Pete
>>>> 
>>>> On Wed, Oct 17, 2012 at 10:58 AM, Stuart Martin  
>>>> wrote:
>>>>> Hi John,
>>>>> 
>>>>> Thanks for your thoughts.  We'll look to improve the GT doc based on your 
>>>>> feedback.
>>>>>  http://jira.globus.org/browse/GT-305
>>>>> 
>>>>> Cheers,
>>>>> Stu
>>>>> 
>>>>> On Oct 17, 2012, at Oct 17, 8:58 AM, john alexander sanabria ordonez 
>>>>> wrote:
>>>>> 
>>>>>> Hello,
>>>>>> 
>>>>>> I did try to follow the instructions given in the Globus Toolkit 
>>>>>> documentation web page but I found a couple of gaps that mislead the 
>>>>>> deployment of basic grid services (simpleCA + gram5 + gridftp), e.g.
>>>>>>• The documentation does not mention that the grid-ca-certs package 
>>>>>> should be installed
>>>>>>• The documentation can be more specific about the grid-ca-create 
>>>>>> command must be run by the globus user
>>>>>>• When this command is run by globus user some files (found in 
>>>>>> ~/.globus/simpleCA) must be copied to the /etc/grid-security directory
>>>>>>• The documentation should provide a hint which describes what [YUM 
>>>>>> and OSG specific] repositories should be installed for setting up a 
>>>>>> machine where Globus services  would be provisioned 
>>>>>> (https://twiki.grid.iu.edu/bin/view/Documentation/Release3/YumRepositories)
>>>>>> 
>>>>>> I have to thank to Marco Mambelli and Jose Caballero for their hints 
>>>>>> that allow me to define the correct set of steps for installing those 
>>>>>> services.
>>>>>> 
>>>>>> Regards,
>>>>>> 
>>>>>> PS:  I wrote some Chef recipes which allow to deploy the aforementioned 
>>>>>> services in a physical or virtual machine
>>>>> 
>>> 
> 



Re: [gt-user] who maintains the Globus Toolkit documentation?

2012-10-19 Thread Stuart Martin
Sounds good.  As Vas says, we're looking to make some changes/improvements to 
the Toolkit web pages.  That may be a longer term effort.  But, maybe there are 
some simple, near term changes that can be made to get this going.  We'll 
discuss internally here and let you know our plans.

Thanks for the input!

-Stu

On Oct 19, 2012, at Oct 19, 9:28 AM, Pete Eby wrote:

> I thinking having the install and quickstart guides in a wiki format
> would be extremely helpful. The globus user community seems well
> suited to help maintain this documentation. These guides are very
> helpful, and allowing them to be updated might help to improve them
> further.
> 
> Also, perhaps an additional community wiki page could be created for
> trouble shooting common issues. There seems to be some "top 10" things
> users encounter frequently, and this might help save them a
> considerable amount of debug / research time.
> 
> Pete
> 
> On Fri, Oct 19, 2012 at 10:22 AM, Stuart Martin  wrote:
>> I like that idea a lot.  We use docbook to generate most of the toolkit's 
>> web content.  We'll have to think through if and where taking on a 
>> transition like that would make sense.  Maybe there are some GT pages that 
>> are best left to be generated via docbook and some that could be changed to 
>> become wiki pages.
>> 
>> For example, I'm thinking (partially based on the input here) that the 
>> install and quick start might be good candidates to be wiki pages.
>>
>> http://www.globus.org/toolkit/docs/latest-stable/admin/install/#gtadmin
>>
>> http://www.globus.org/toolkit/docs/latest-stable/admin/quickstart/#quickstart
>> 
>> Maybe this developer guide too?
>>http://www.globus.org/toolkit/docs/5.2/5.2.2/appendices/developer/
>> 
>> Thoughts?
>> 
>> -Stu
>> 
>> On Oct 19, 2012, at Oct 19, 6:09 AM, Pete Eby wrote:
>> 
>>> Has the idea of placing the documentation on a wiki where it could be
>>> updated by community members been discussed? Perhaps by users with a
>>> Globus Online login?
>>> 
>>> Pete
>>> 
>>> On Wed, Oct 17, 2012 at 10:58 AM, Stuart Martin  wrote:
>>>> Hi John,
>>>> 
>>>> Thanks for your thoughts.  We'll look to improve the GT doc based on your 
>>>> feedback.
>>>>   http://jira.globus.org/browse/GT-305
>>>> 
>>>> Cheers,
>>>> Stu
>>>> 
>>>> On Oct 17, 2012, at Oct 17, 8:58 AM, john alexander sanabria ordonez wrote:
>>>> 
>>>>> Hello,
>>>>> 
>>>>> I did try to follow the instructions given in the Globus Toolkit 
>>>>> documentation web page but I found a couple of gaps that mislead the 
>>>>> deployment of basic grid services (simpleCA + gram5 + gridftp), e.g.
>>>>> • The documentation does not mention that the grid-ca-certs package 
>>>>> should be installed
>>>>> • The documentation can be more specific about the grid-ca-create 
>>>>> command must be run by the globus user
>>>>> • When this command is run by globus user some files (found in 
>>>>> ~/.globus/simpleCA) must be copied to the /etc/grid-security directory
>>>>> • The documentation should provide a hint which describes what [YUM 
>>>>> and OSG specific] repositories should be installed for setting up a 
>>>>> machine where Globus services  would be provisioned 
>>>>> (https://twiki.grid.iu.edu/bin/view/Documentation/Release3/YumRepositories)
>>>>> 
>>>>> I have to thank to Marco Mambelli and Jose Caballero for their hints that 
>>>>> allow me to define the correct set of steps for installing those services.
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> PS:  I wrote some Chef recipes which allow to deploy the aforementioned 
>>>>> services in a physical or virtual machine
>>>> 
>> 



Re: [gt-user] who maintains the Globus Toolkit documentation?

2012-10-19 Thread Stuart Martin
I like that idea a lot.  We use docbook to generate most of the toolkit's web 
content.  We'll have to think through if and where taking on a transition like 
that would make sense.  Maybe there are some GT pages that are best left to be 
generated via docbook and some that could be changed to become wiki pages.

For example, I'm thinking (partially based on the input here) that the install 
and quick start might be good candidates to be wiki pages.
http://www.globus.org/toolkit/docs/latest-stable/admin/install/#gtadmin

http://www.globus.org/toolkit/docs/latest-stable/admin/quickstart/#quickstart

Maybe this developer guide too?
http://www.globus.org/toolkit/docs/5.2/5.2.2/appendices/developer/

Thoughts?

-Stu

On Oct 19, 2012, at Oct 19, 6:09 AM, Pete Eby wrote:

> Has the idea of placing the documentation on a wiki where it could be
> updated by community members been discussed? Perhaps by users with a
> Globus Online login?
> 
> Pete
> 
> On Wed, Oct 17, 2012 at 10:58 AM, Stuart Martin  wrote:
>> Hi John,
>> 
>> Thanks for your thoughts.  We'll look to improve the GT doc based on your 
>> feedback.
>>http://jira.globus.org/browse/GT-305
>> 
>> Cheers,
>> Stu
>> 
>> On Oct 17, 2012, at Oct 17, 8:58 AM, john alexander sanabria ordonez wrote:
>> 
>>> Hello,
>>> 
>>> I did try to follow the instructions given in the Globus Toolkit 
>>> documentation web page but I found a couple of gaps that mislead the 
>>> deployment of basic grid services (simpleCA + gram5 + gridftp), e.g.
>>>  • The documentation does not mention that the grid-ca-certs package 
>>> should be installed
>>>  • The documentation can be more specific about the grid-ca-create 
>>> command must be run by the globus user
>>>  • When this command is run by globus user some files (found in 
>>> ~/.globus/simpleCA) must be copied to the /etc/grid-security directory
>>>  • The documentation should provide a hint which describes what [YUM 
>>> and OSG specific] repositories should be installed for setting up a machine 
>>> where Globus services  would be provisioned 
>>> (https://twiki.grid.iu.edu/bin/view/Documentation/Release3/YumRepositories)
>>> 
>>> I have to thank to Marco Mambelli and Jose Caballero for their hints that 
>>> allow me to define the correct set of steps for installing those services.
>>> 
>>> Regards,
>>> 
>>> PS:  I wrote some Chef recipes which allow to deploy the aforementioned 
>>> services in a physical or virtual machine
>> 



Re: [gt-user] who maintains the Globus Toolkit documentation?

2012-10-17 Thread Stuart Martin
Hi John,

Thanks for your thoughts.  We'll look to improve the GT doc based on your 
feedback.
http://jira.globus.org/browse/GT-305

Cheers,
Stu

On Oct 17, 2012, at Oct 17, 8:58 AM, john alexander sanabria ordonez wrote:

> Hello,
> 
> I did try to follow the instructions given in the Globus Toolkit 
> documentation web page but I found a couple of gaps that mislead the 
> deployment of basic grid services (simpleCA + gram5 + gridftp), e.g.
>   • The documentation does not mention that the grid-ca-certs package 
> should be installed
>   • The documentation can be more specific about the grid-ca-create 
> command must be run by the globus user
>   • When this command is run by globus user some files (found in 
> ~/.globus/simpleCA) must be copied to the /etc/grid-security directory
>   • The documentation should provide a hint which describes what [YUM and 
> OSG specific] repositories should be installed for setting up a machine where 
> Globus services  would be provisioned 
> (https://twiki.grid.iu.edu/bin/view/Documentation/Release3/YumRepositories)
> 
> I have to thank to Marco Mambelli and Jose Caballero for their hints that 
> allow me to define the correct set of steps for installing those services. 
> 
> Regards,
> 
> PS:  I wrote some Chef recipes which allow to deploy the aforementioned 
> services in a physical or virtual machine



[gt-user] Globus Toolkit 5.2.2 is now available

2012-07-24 Thread Stuart Martin
On behalf of the Globus Toolkit development team, I am pleased to announce that 
a new point release in the 5.2 series of the Globus Toolkit is now available - 
5.2.2.

Highlights of this release include:

GridFTP
* Added a new recommended "hybrid" server configuration option
- It's a split/single mode which only creates backend 
connections if client requests stripes
* Fixed a security bug where GridFTP acts as wrong user when user 
doesn't exist
* Improved memory management and process management
* Improved reliability

GRAM5
* Improved memory management and process management
* Improved scalability and reliability

MyProxy
* Updated to MyProxy version v5.8

GSI-Enabled OpenSSH
* Updated to gsissh version v5.5

Repositories Supported:
CentOS 4,5,6
Fedora 15,16,17
RHEL 5,6
Scientific Linux 5,6
SLES 11
Debian 6,7
Ubuntu 10.04, 10,10, 11.04, 11.10, 12.04

Note: For those that have 5.2.0 or 5.2.1 installed from the globus repo, doing 
an update will install the new 5.2.2 packages.

Relevant 5.2.2 links:

- Release notes: http://www.globus.org/toolkit/docs/5.2/5.2.2/rn/
- Software: http://www.globus.org/toolkit/downloads/5.2.2/
- Documentation: http://www.globus.org/toolkit/docs/5.2/5.2.2/
- Support: http://dev.globus.org/wiki/Globus_Toolkit#Mailing_Lists

We look forward to and encourage your feedback on GT 5.2.  Thanks for your 
support of Globus software.

Viva Globus!

-Stu Martin

[gt-user] Globus Toolkit 5.2.1 is now available

2012-04-26 Thread Stuart Martin
On behalf of the Globus Toolkit development team, I am pleased to announce that 
a new point release in the 5.2 series of the Globus Toolkit is now available - 
5.2.1.

Highlights of this release include:

GridFTP
* Added ability to restrict the directories/paths a server may access.
* Added server support for setting file modification time.
* Added ability (MLSC) to stream directory listings over the control 
channel.

GRAM5
* Fixed a number of bugs found by OSG during high scalability testing
* Added job expiration and automatic cleanup for jobs that finish, but 
client does not do a final commit for several hours. This is configurable via 
RSL.  Improves use of Condor-G for OSG.
* Added site .rvf file support.  This allows a site administrator to 
modify default RSL values and add or remove RSL attributes locally, without 
modifying packaged files. This allows configuration to remain when package 
updates are released.

MyProxy
* Updated to MyProxy version v5.6

GSI-Enabled OpenSSH
* Updated to gsissh version v5.4

Repo Support Update
* No longer supported: Debian 5; Fedora 13, 14
* Newly supported: Debian 7 (testing); Ubuntu 10.04LTS, 12.04 (testing)
* Supported: CentOS 5,6; Fedora 15, 16; RHEL 5,6; Scientific Linux 5,6; 
Debian 6,7; Ubuntu 10.04, 10,10, 11.04, 11.10, 12.04;

Note: For those that have 5.2.0 installed from the globus repo, doing an update 
will install the new 5.2.1 packages.

Relevant 5.2.1 links:

- Release notes: http://www.globus.org/toolkit/docs/5.2/5.2.1/rn/
- Software: http://www.globus.org/toolkit/downloads/5.2.1/
- Documentation: http://www.globus.org/toolkit/docs/5.2/5.2.1/
- Support: http://dev.globus.org/wiki/Globus_Toolkit#Mailing_Lists

We look forward to and encourage your feedback on GT 5.2.  Thanks for your 
support of Globus software.

Viva Globus!

-Stu Martin


Re: [gt-user] default values in GT5

2012-03-22 Thread Stuart Martin
On Mar 22, 2012, at Mar 22, 3:45 PM, Jose Caballero wrote:

> Hi,
> this is my first post in this list. I hope my first question is not an stupid 
> one :)
> 
> 
> 
> I have found the default values for all GT2 RSL values here:
> 
> http://www.globus.org/toolkit/docs/2.4/gram/gram_rsl_parameters.html
> 
> However, I am capable of finding the same info for GT5. 
> The closest one I found is this
> 
> http://globus.org/toolkit/docs/5.0/5.0.0/execution/gram5/pi/#gram5-rsl
> 
> But I don't see there the default values for every RSL variable. 
> Could some one send the link to the right URL if that is not the best one?

On that page is the list of RSL parameters, but as you point out it doesn't 
look like the defaults are indicated there.
http://globus.org/toolkit/docs/5.0/5.0.0/execution/gram5/pi/#id2597577

> Can I just assume the defaults are the same than for GT2 (e.g. default for 
> jobtype is multiple, and so on)?

Yes.  I'm pretty sure the defaults have not changed.

> 
> Thanks a lot in advance.
> Jose



[gt-user] Globus Toolkit 5.0.5 is now available

2012-03-08 Thread Stuart Martin
On behalf of the Globus Toolkit development team I am pleased to announce that 
a new point release in the 5.0 series of the Globus Toolkit is now available - 
5.0.5.

**Note** GT 5.0.5 will likely be the last release for the GT 5.0.x series.  
Soon we'll release GT 5.2.1, which will be a superset of the 5.0.5 source code. 
 5.2.x includes the convenience and simplicity of installing GT components 
using binary native packages.  We are encouraging users to move to the GT 5.2.x 
series at their convenience.  OSG has already moved to 5.2.  Soon 5.2 will be 
available directly in Debian, EPEL, RHEL, Ubuntu, ... repos.

Highlights of this release include:

GridFTP
* Added ability to restrict the directories/paths a server has access 
too
* Added server support for setting file modification time
* Added ability (MLSC) to stream directory listings over the control 
channel

GRAM5
* Fixed bugs affecting stability found by OSG during high scalability 
testing
* Added "log facility" option to support GRAM logging to syslog

MyProxy
* Updated to MyProxy version v5.6

GSI-Enabled OpenSSH
* Updated to gsissh version 5.4

Relevant 5.0.5 links:

- Release notes:http://www.globus.org/toolkit/docs/5.0/5.0.5/rn/
- Software: http://www.globus.org/toolkit/downloads/5.0.5/
- Documentation:http://www.globus.org/toolkit/docs/5.0/5.0.5/
- Support:  
http://dev.globus.org/wiki/Globus_Toolkit#Mailing_Lists

We look forward to and encourage your feedback on GT5.  Thanks for your support 
of Globus software.  

Viva Globus!

-Stu Martin

[gt-user] Globus Toolkit 5.2.0 is now available

2011-12-15 Thread Stuart Martin
On behalf of the Globus Toolkit development team, I am pleased to announce the 
production release of Globus Toolkit (GT) 5.2.
 
GT 5.2 is the culmination of more than a year of development effort to greatly 
enhance the ability to install, setup and update the GT client and service 
components.  This has been achieved by providing binary native packages for the 
following platforms: Red Hat, Fedora, Debian, Ubuntu, Scientific Linux, and 
CentOS.  We will maintain a Globus repository containing all packages.  We have 
started the process to become package maintainers for various RPM and Debian 
repos and will be working to include the 5.2 Globus packages into these repos 
soon.  
 
Many thanks to the IGE (Initiative for Globus in Europe) team for their 
collaboration and contributions on this native packaging effort.  Also, I'd 
like to thank Joe Bester for his significant development contributions on this 
release.
 
By following the Quickstart guide, you can now easily install and setup a full 
GT environment in minutes.
 
GT 5.2.0 includes the latest versions of all components: GRAM, GridFTP, MyProxy 
and GSISSH.  This latest version of GRAM includes a number of scalability and 
reliability improvements found from working with Open Science Grid (OSG) that 
are not in the 5.0 series.  GridFTP added support for the DCSC command, which 
allows the client to specify credentials used to secure the data channel 
connection.  Globus Online utilizes this command for seamless data movement 
across multiple security domains.
 
GT 5.2 is protocol and client API compatible with GT 5.0 (e.g. 5.2 clients will 
work with GT 5.0 services, and visa versa.)
 
Relevant 5.2 links:
 
- Release notes: http://www.globus.org/toolkit/docs/5.2/5.2.0/rn/
- Software: http://www.globus.org/toolkit/downloads/5.2.0/
- Documentation: http://www.globus.org/toolkit/docs/5.2/5.2.0/
- Support: http://dev.globus.org/wiki/Globus_Toolkit#Mailing_Lists
 
We look forward to and encourage your feedback on GT 5.2.  Thanks for your 
support of Globus software.  
 
Viva Globus!
 
-Stu Martin


Re: [gt-user] Globus Toolkit 5.1.3 (5.2 Beta) is now available

2011-11-17 Thread Stuart Martin
Hi Adam,

Reporting to the list here will work.  We're using jira for issue tracking.  
So, that would be another way to report issues.

I entered this issue here: http://jira.globus.org/browse/RIC-200

Thanks for the detailed report!

-Stu

On Nov 16, 2011, at Nov 16, 9:30 PM, Adam Mercer wrote:

> On Wed, Nov 16, 2011 at 12:00, Stuart Martin  wrote:
> 
> Hi
> 
>> On behalf of the Globus Toolkit 5.2 development team I am pleased to 
>> announce the 1st *Beta* version for GT 5.2. We are calling this release GT 
>> 5.1.3.  GT 5.1.3 is now feature complete.  The next release will be GT 
>> 5.2.0.  So, the time is now to jump in and test this 5.1.3 release and 
>> report any problems.  We're hoping to release GT 5.2.0 by the end of 
>> November.
> 
> Just tried building on Mac OS X 10.6.8 and ran into a few problems:
> 
> $ ./configure --prefix=$HOME/gt-5.1.3 --with-flavor=gcc64dbg
> checking build system type... i386-apple-darwin10.8.0
> configure: creating ./config.status
> config.status: creating Makefile
> config.status: creating config.site
> $
> 
> configure is recognising my system as 32bit. It is however, 64bit capable:
> 
> $ sysctl -n hw.cpu64bit_capable
> 1
> $
> 
> and as I am requesting a 64bit flavour this could cause problems with
> incorrect flags being passed to the compiler. This can be fixed with
> updating to a more recent config.guess and config.sub from upstream
> (the current version is about 5 years old),
> <http://git.savannah.gnu.org/cgit/config.git/tree/>, and updating it
> throughout the tree:
> 
> $ rm config.guess config.sub
> $ wget http://git.savannah.gnu.org/cgit/config.git/plain/config.guess
> $ wget http://git.savannah.gnu.org/cgit/config.git/plain/config.sub
> $ find . -name config.guess -exec cp config.guess {} \;
> $ find . -name config.sub -exec cp config.sub {} \;
> 
> With these in place the 64bit system is correctly identified:
> 
> $ ./configure --prefix=$HOME/gt-5.1.3 --with-flavor=gcc64dbg
> checking build system type... x86_64-apple-darwin10.8.0
> configure: creating ./config.status
> config.status: creating Makefile
> config.status: creating config.site
> $
> 
> The next problem comes when running 'make'
> 
> $ make
> 
> Making install in perl
> Making install in scripts
> test -z "/Users/ram/gt-5.1.3/sbin" || ../.././install-sh -c -d
> "/Users/ram/gt-5.1.3/sbin"
> /bin/sh: ../.././install-sh: Permission denied
> gnumake[4]: *** [install-sbinSCRIPTS] Error 126
> gnumake[3]: *** [install-am] Error 2
> gnumake[2]: *** [install-recursive] Error 1
> make[1]: *** [install-recursive] Error 1
> make[1]: Nothing to be done for `install-man'.
> DYLD_LIBRARY_PATH=/Users/ram/gt-5.1.3/lib
> /Users/ram/gt-5.1.3/sbin/gpt-build
> -srcdir=./source-trees/core/source gcc64dbg
> /bin/sh: /Users/ram/gt-5.1.3/sbin/gpt-build: No such file or directory
> make: *** [globus_core-only] Error 127
> $
> 
> the problem here is that the install-sh script is not executable.
> After ,arking all the install-sh scripts as executable, i.e. with
> something like:
> 
> $ find . -name install-sh -exec chmod +x {} \;
> 
> this gets further, but then fails, in gssapi-openssh, with:
> 
> /usr/bin/gcc -g   -m64 -fno-common  -Wall -g -m64 -fno-common -Wall
> -Wall -Wpointer-arith -Wsign-compare -Wformat-security
> -Wno-pointer-sign -fno-strict-aliasing -fno-builtin-memset
> -fstack-protector-all  -I. -I.. -I. -I./..  -I/include/globus -I/
> -no-cpp-precomp -I/Users/ram/gt-5.1.3/include/globus
> -I/Users/ram/gt-5.1.3/include/globus/gcc64dbg -no-cpp-precomp
> -I/Users/ram/gt-5.1.3/include/globus
> -I/Users/ram/gt-5.1.3/include/globus/
> -I/Users/ram/gt-5.1.3/include/globus  -DHAVE_CONFIG_H -c port-tun.c
> port-tun.c:111:20: error: /net/if.h: Input/output error
> port-tun.c: In function ‘sys_tun_open’:
> port-tun.c:120: error: storage size of ‘ifr’ isn’t known
> port-tun.c:174: error: invalid application of ‘sizeof’ to incomplete
> type ‘struct ifreq’
> port-tun.c:176: error: ‘IFF_UP’ undeclared (first use in this function)
> port-tun.c:176: error: (Each undeclared identifier is reported only once
> port-tun.c:176: error: for each function it appears in.)
> port-tun.c:178: error: invalid application of ‘sizeof’ to incomplete
> type ‘struct ifreq’
> port-tun.c:120: warning: unused variable ‘ifr’
> gnumake[2]: *** [port-tun.o] Error 1
> gnumake[1]: *** [openbsd-compat/libopenbsd-compat.a] Error 2
> 
> ERROR: Build has failed
> make: *** [gsi_openssh-only] Error 2
> $
> 
> This appears to be related to the -I/include/globus and -I/ being
> passed to the compiler. Looking at
> source-trees/gssapi-openssh/openssh/openbsd-comp

[gt-user] Globus Toolkit 5.1.3 (5.2 Beta) is now available

2011-11-16 Thread Stuart Martin
On behalf of the Globus Toolkit 5.2 development team I am pleased to announce 
the 1st *Beta* version for GT 5.2. We are calling this release GT 5.1.3.  GT 
5.1.3 is now feature complete.  The next release will be GT 5.2.0.  So, the 
time is now to jump in and test this 5.1.3 release and report any problems.  
We're hoping to release GT 5.2.0 by the end of November.

The 5.1.3 release comprises a source installer, binary RPM packages for all GT 
components, and binary Debian packages for non-GRAM components.  Soon, we'll be 
adding Debian packages for GRAM.

Also, I'm happy to let you know that GT 5.2 will be protocol and client API 
compatible with GT 5.0 (e.g. 5.2 clients will work with GT 5.0 services, and 
visa versa.)

Highlights of this GT 5.1.3 release as compared to 5.0.4 is largely the native 
packaging enhancements that simplify the installation, configuration and 
startup of the GT services: GRAM, GridFTP, and MyProxy.  For GRAM, 5.2 also 
includes a number of scalability and reliability improvements found from 
working with OSG.

Additional Notes:
1) *All* GT 5.1.3 package version numbers have been incremented in 
order to avoid any confusion with existing Globus Debs and RPMs that predate GT 
5.2.
2) We are providing a Globus repo for the various native packages.
3) We have started the process to become package managers / maintainers 
for various RPM and Debian repos and will be including the Globus packages into 
these repos as soon as we get that process worked out.

All information about GT 5.2 is here:  
http://dev.globus.org/wiki/Globus_Toolkit/5.2

We look forward to and encourage your feedback on GT5.2.  Thanks for your 
support of Globus software.  

Viva Globus!

-Stu Martin

Re: [gt-user] GRAM 5.0.4 job manager refuses to accept jobs

2011-09-16 Thread Stuart Martin
Ok - I've got it recorded here for Joe to look at.  He's off today, but will 
look into this early next week.
http://jira.globus.org/browse/GRAM-252

-Stu

On Sep 15, 2011, at Sep 15, 9:20 PM, Yuriy Halytskyy wrote:

> Hi Stu,
> 
> it is rather serious, normally there is a workaround for users (just
> request their old job statuses) but we cannot detect this issue
> automatically from client side as jglobus does not return errror on
> submission - no exception is thrown, and error code is set to zero.
> 
> Cheers,
> Yuriy
> 
> Excerpts from Stuart Martin's message of Fri Sep 16 03:57:20 +1200 2011:
>> Hi Yuriy,
>> 
>> We think a similar issue was hit and fixed in (soon to be released) GT 
>> 5.1.2.  It has not yet been back ported to 5.0.x
>> 
>> What is the priority on this?  How much is this affecting you / your users?
>> 
>> -Stu
>> 
>> On Sep 13, 2011, at Sep 13, 1:27 AM, Yuriy Halytskyy wrote:
>> 
>>> ok, now I can reproduce it. When proxy expires when job manager is
>>> waiting for COMMIT_END signal, it stops accepting new jobs. It seems I
>>> can restore it by sending commit_end, but this still looks like a bug
>>> to me as client may loose job id. 
>>> 
>>> 
>>> Cheers,
>>> Yuriy
>>> 
>>> Excerpts from Yuriy Halytskyy's message of Tue Sep 13 15:59:09 +1200 2011:
 Hi,
 
 user job manager gets into the state where the submission with globusrun 
 hangs and job is never submitted
 
 server logs say:
 Sep 13 15:35:30 gram5 gridinfo[30804]: ts=2011-09-13T03:35:30.104424Z 
 id=30804 event=gram.job.start level=INFO 
 gramid=/16145890501405029996/576663433152357309/
 peer=130.216.189.203:57672
 Sep 13 15:35:30 gram5 gridinfo[30804]: ts=2011-09-13T03:35:30.104582Z 
 id=30804 event=gram.add_request.end level=WARN 
 gramid=/16145890501405029996/576663433152357309/ status=-130
 reason="the job manager was sent a stop signal (job is still running)"
 Sep 13 15:35:30 gram5 gridinfo[30804]: ts=2011-09-13T03:35:30.104885Z 
 id=30804 event=gram.job.end level=INFO 
 gramid=/16145890501405029996/576663433152357309/ status=-130 msg="Request
 start failed" reason="the job manager was sent a stop signal (job is still 
 running)"
 
 submission with globusrun hangs:
 
 globusrun -batch -r gram5.ceres.auckland.ac.nz 
 '&(executable=echo)(arguments= 
 hello)(job_type=single)(count=1)(hostCount=1)(vo="/nz/nesi")(maxWalltime=10)(directory=/home/smas036)'
 globus_gram_client_callback_allow successful
 GRAM Job submission successful
 https://gram5.ceres.auckland.ac.nz:40398/16145891598704212781/576663433152357309/
 
 
 submission with two-phase does not hang and results in:
 globusrun  -batch -r gram5.ceres.auckland.ac.nz 
 '&(two_phase=5)(executable=echo)(arguments=
 hello)(job_type=single)(count=1)(hostCount=1)(vo="/nz/nesi")(maxWalltime=10)(directory=/home/smas036)'
 globus_gram_client_callback_allow successful
 GRAM Job submission failed because the job contact string does not match 
 any which the job manager is handling (error code 156)
 https://gram5.ceres.auckland.ac.nz:40398/16145891597960224316/576663433152357309/
 
 
 our users are getting into this problem all the time, but I cannot 
 reproduce putting job manager into that state. They can submit again when 
 I kill it.
 
 We haven't seen this, before our job submission software started 
 submitting jobs with two-phase.
 
 Cheers,
 Yuriy



Re: [gt-user] GRAM 5.0.4 job manager refuses to accept jobs

2011-09-15 Thread Stuart Martin
Hi Yuriy,

We think a similar issue was hit and fixed in (soon to be released) GT 5.1.2.  
It has not yet been back ported to 5.0.x

What is the priority on this?  How much is this affecting you / your users?

-Stu

On Sep 13, 2011, at Sep 13, 1:27 AM, Yuriy Halytskyy wrote:

> ok, now I can reproduce it. When proxy expires when job manager is
> waiting for COMMIT_END signal, it stops accepting new jobs. It seems I
> can restore it by sending commit_end, but this still looks like a bug
> to me as client may loose job id. 
> 
> 
> Cheers,
> Yuriy
> 
> Excerpts from Yuriy Halytskyy's message of Tue Sep 13 15:59:09 +1200 2011:
>> Hi,
>> 
>> user job manager gets into the state where the submission with globusrun 
>> hangs and job is never submitted
>> 
>> server logs say:
>> Sep 13 15:35:30 gram5 gridinfo[30804]: ts=2011-09-13T03:35:30.104424Z 
>> id=30804 event=gram.job.start level=INFO 
>> gramid=/16145890501405029996/576663433152357309/
>> peer=130.216.189.203:57672
>> Sep 13 15:35:30 gram5 gridinfo[30804]: ts=2011-09-13T03:35:30.104582Z 
>> id=30804 event=gram.add_request.end level=WARN 
>> gramid=/16145890501405029996/576663433152357309/ status=-130
>> reason="the job manager was sent a stop signal (job is still running)"
>> Sep 13 15:35:30 gram5 gridinfo[30804]: ts=2011-09-13T03:35:30.104885Z 
>> id=30804 event=gram.job.end level=INFO 
>> gramid=/16145890501405029996/576663433152357309/ status=-130 msg="Request
>> start failed" reason="the job manager was sent a stop signal (job is still 
>> running)"
>> 
>> submission with globusrun hangs:
>> 
>> globusrun -batch -r gram5.ceres.auckland.ac.nz 
>> '&(executable=echo)(arguments= 
>> hello)(job_type=single)(count=1)(hostCount=1)(vo="/nz/nesi")(maxWalltime=10)(directory=/home/smas036)'
>> globus_gram_client_callback_allow successful
>> GRAM Job submission successful
>> https://gram5.ceres.auckland.ac.nz:40398/16145891598704212781/576663433152357309/
>> 
>> 
>> submission with two-phase does not hang and results in:
>> globusrun  -batch -r gram5.ceres.auckland.ac.nz 
>> '&(two_phase=5)(executable=echo)(arguments=
>> hello)(job_type=single)(count=1)(hostCount=1)(vo="/nz/nesi")(maxWalltime=10)(directory=/home/smas036)'
>> globus_gram_client_callback_allow successful
>> GRAM Job submission failed because the job contact string does not match any 
>> which the job manager is handling (error code 156)
>> https://gram5.ceres.auckland.ac.nz:40398/16145891597960224316/576663433152357309/
>> 
>> 
>> our users are getting into this problem all the time, but I cannot reproduce 
>> putting job manager into that state. They can submit again when I kill it.
>> 
>> We haven't seen this, before our job submission software started submitting 
>> jobs with two-phase.
>> 
>> Cheers,
>> Yuriy



Re: [gt-user] Is there a way to limit number of GRAM jobs one userid can submit at a time in GT 5.0.4?

2011-09-06 Thread Stuart Martin
Hi Prakashan,

Is this for Fork only?  For the LRMs, you can limit the number of jobs in the 
queue (as well as other things).

If it for Fork, one thing VDT/OSG implemented for this was "managed fork".
https://twiki.grid.iu.edu/bin/view/ReleaseDocumentation/ManagedFork

http://vdt.cs.wisc.edu/releases/2.0.0/notes/Globus-ManagedFork-Setup.html

Basically, all "Fork" jobs go thru Condor.  And Condor is configured to submit 
jobs to the local/service host.  Using this you get Fork functionality, but the 
ability to control/limit the number of jobs per user.  You could probably 
implement this managed fork concept with any of the other LRMs too.  E.g. 
configure a PBS queue and all "Fork" jobs go to the PBS queue which run the 
jobs on the local/service host.

We have an issue for implementing this natively in GRAM, but have not gotten to 
it yet:
http://jira.globus.org/browse/GRAM-4

Cheers,
Stu

On Sep 5, 2011, at Sep 5, 11:43 AM, Prakashan Korambath wrote:

> Hi,
> 
> I was wondering whether there is a way to limit number of GRAM jobs 
> submission per userid in GT 5.04.  One way I can think of is limiting in 
> /etc/security/limits.conf using Linux controls to limit processes per userid. 
>  But if Globus has a way of controlling it that would be nice. Basically, we 
> are trying to limit number of runaway jobs.  Thanks,
> 
> Regards,
> 
> Prakashan



[gt-user] Fwd: [gtmanuals-user] restrict GRAM job manager's port to a certain range

2011-08-17 Thread Stuart Martin
I'm moving this to gt-user to get more help.  gtmanuals-user I think is more 
for basic web/doc questions.

-Stu

Begin forwarded message:

> From: Yu Huang 
> Date: August 16, 2011 2:38:34 PM CDT
> To: Stuart Martin 
> Cc: gtmanuals-u...@lists.globus.org
> Subject: Re: [gtmanuals-user] restrict GRAM job manager's port to a certain 
> range
> 
> Thanks, Stuart. The ucla grid here has something strange going on. The 
> gridftp file in xinetd.d directory has the GLOBUS_TCP_PORT_RANGE set to 
> 4,5. but not the gsigatekeeper.
> 
> The administrator modified it after i suspected there is a firewall around 
> it. My job will succeed if the job contact ID (controlled by gate keeper , i 
> think) from the server has its port between 4 and 5.
> 
> But after administrator modified it and restarted xinetd, port below 4 or 
> above 5 is still happening. Is restarting xinetd all it's needed? or 
> there is something else?
> 
> Also i noticed, after he added this GLOBUS_TCP_PORT_RANGE to gsigatekeeper, 
> globusrun or globus-job-run just hangs there forever after message "GRAM job 
> submission successful". Before there was no issue with globusrun or 
> globus-job-run, just condor-g. Any idea of what's going on there?
> 
> Thanks,
> yu
> 
> On Aug 15, 2011 7:32 AM, "Stuart Martin"  wrote:
> > Hi Yu,
> > 
> > Here is information on how to control the ports used by GRAM 
> > (gatekeeper/jobmanager)
> > 
> > http://dev.globus.org/wiki/FirewallHowTo#Controlling_the_Ephemeral_Port_Range
> > 
> > Let me know if this works for you or if you have further troubles.
> > 
> > Thanks,
> > -Stu
> > 
> > On Aug 12, 2011, at Aug 12, 6:03 PM, Yu Huang wrote:
> > 
> >> Hi,
> >> 
> >> How to restrict job manger on the GRAM server to have ports in the job 
> >> contact URLs within a certain range, say 4 to 5?
> >> 
> >> I'm using condor_g to talk to a server at ucla, 
> >> grid4.hoffman2.idre.ucla.edu, sometimes fail, sometimes succeed. The job 
> >> succeeds when the job contact URL has its port within 4 and 5 
> >> range, like job contact id is 
> >> https://grid4.hoffman2.idre.ucla.edu:41297/16145784839073078036/15589874525209144233/.
> >> 
> >> 08/11/11 13:33:44 [25832] Fetched 0 job ads from schedd 
> >> 08/11/11 13:33:44 [25832] Updating classad values for 94.0: 
> >> 08/11/11 13:33:44 [25832] DelegatedProxyExpiration = 1313134315 
> >> 08/11/11 13:33:44 [25832] GridJobId = "gt5 
> >> grid4.hoffman2.idre.ucla.edu/jobmanager-fork 
> >> https://grid4.hoffman2.idre.ucla.edu:412
> >> 97/16145784839073078036/15589874525209144233/" 
> >> 08/11/11 13:33:44 [25832] LastRemoteStatusUpdate = 1313094824 
> >> 08/11/11 13:33:44 [25832] leaving doContactSchedd() 
> >> 08/11/11 13:33:44 [25832] (94.0) doEvaluateState called: gmState 
> >> GM_SUBMIT_SAVE, globusState 32 
> >> 08/11/11 13:33:44 [25832] (94.0) gm state change: GM_SUBMIT_SAVE -> 
> >> GM_SUBMIT_COMMIT 
> >> 08/11/11 13:33:44 [25832] GAHP[25838] <- 'GRAM_JOB_SIGNAL 6 
> >> https://grid4.hoffman2.idre.ucla.edu:41297/16145784839073078036/1558987
> >> 4525209144233/ 5 NULL' 
> >> 08/11/11 13:33:44 [25832] GAHP[25838] -> 'S' 
> >> 08/11/11 13:33:44 [25832] GAHP[25838] <- 'RESULTS' 
> >> 08/11/11 13:33:44 [25832] GAHP[25838] -> 'R' 
> >> 08/11/11 13:33:44 [25832] GAHP[25838] -> 'S' '1' 
> >> 08/11/11 13:33:44 [25832] GAHP[25838] -> '6' '0' '0' '32' 
> >> 08/11/11 13:33:44 [25832] (94.0) doEvaluateState called: gmState 
> >> GM_SUBMIT_COMMIT, globusState 32 
> >> 08/11/11 13:33:44 [25832] (94.0) gm state change: GM_SUBMIT_COMMIT -> 
> >> GM_SUBMITTED 
> >> 08/11/11 13:33:44 [25832] GAHP[25838] <- 'RESULTS' 
> >> 08/11/11 13:33:44 [25832] GAHP[25838] -> 'R' 
> >> 08/11/11 13:33:44 [25832] GAHP[25838] -> 'S' '1' 
> >> 08/11/11 13:33:44 [25832] GAHP[25838] -> '2' 
> >> 'https://grid4.hoffman2.idre.ucla.edu:41297/16145784839073078036/15589874525209144233/
> >> ' '2' '0' 
> >> 08/11/11 13:33:44 [25832] (94.0) gram callback: state 2, errorcode 0 
> >> 08/11/11 13:33:44 [25832] (94.0) doEvaluateState called: gmState 
> >> GM_SUBMITTED, globusState 32 
> >> 08/11/11 13:33:44 [25832] (94.0) globus state change: UNSUBMITT

Re: [gt-user] Globus-Online makes non-RFC compliant assumptions about GridFTP

2011-08-16 Thread Stuart Martin
Actually, this was fixed in GO's command line interface (CLI)
https://www.globusonline.org/docs/clireference/

But, not yet fixed in the Transfer (REST) API which is used by GO's Web UI.  
This will be fixed and deployed in the next week or 2.  Here is the issue for 
it:
http://jira.globus.org/browse/KOA-1594

So, if you are using the transfer API or Web UI, you will still get this error.

-Stu

On Aug 16, 2011, at Aug 16, 10:07 AM, Stuart Martin wrote:

> Hi Sebastian,
> 
> This issue was reported to us by IGE.
>   http://jira.globus.org/browse/OPS-1
> 
> It was fixed ~3 month ago and released to production GO.
>   http://jira.globus.org/browse/KOA-1325
> 
> Are you still encountering this problem?  Or maybe there is a 
> separate/similar issue?
> 
> Cheers,
> Stu
> 
> On Aug 16, 2011, at Aug 16, 2:31 AM, Sebastian Czechowski wrote:
> 
>> Hello,
>> 
>> The problem is with the fact encoding for MLST and MLSD commands. The
>> spec says the fact names (the keys) are case insensitive; i.e., any
>> mixture of upper and lower case is allowed and represents the same fact
>> name.
>> 
>> Globus online on the other hand is picky about using an upper case first
>> character and lower case for the remaining characters. Eg. Type=dir and
>> type=dir is the same fact, but GO only recognizes Type=dir.
>> 
>> 
>> This request was issued to IGE by one of the users, can you please comment 
>> on that? Will GO support case insensitivity in commands?
>> 
>> Cheers,
>> Sebastian
>> 
>> -- 
>> Sebastian Czechowski   sebastian.czechow...@gridwisetech.com
>> IT Project Coordinator
>> GridwiseTechoffice/fax: +48 12 294 71 20
>> 
>> The Scalability Specialist  www.gridwisetech.com
>> 
>> 
> 



Re: [gt-user] Globus-Online makes non-RFC compliant assumptions about GridFTP

2011-08-16 Thread Stuart Martin
Hi Sebastian,

This issue was reported to us by IGE.
http://jira.globus.org/browse/OPS-1

It was fixed ~3 month ago and released to production GO.
http://jira.globus.org/browse/KOA-1325

Are you still encountering this problem?  Or maybe there is a separate/similar 
issue?

Cheers,
Stu

On Aug 16, 2011, at Aug 16, 2:31 AM, Sebastian Czechowski wrote:

> Hello,
> 
> The problem is with the fact encoding for MLST and MLSD commands. The
> spec says the fact names (the keys) are case insensitive; i.e., any
> mixture of upper and lower case is allowed and represents the same fact
> name.
> 
> Globus online on the other hand is picky about using an upper case first
> character and lower case for the remaining characters. Eg. Type=dir and
> type=dir is the same fact, but GO only recognizes Type=dir.
> 
> 
> This request was issued to IGE by one of the users, can you please comment on 
> that? Will GO support case insensitivity in commands?
> 
> Cheers,
> Sebastian
> 
> -- 
> Sebastian Czechowski   sebastian.czechow...@gridwisetech.com
> IT Project Coordinator
> GridwiseTechoffice/fax: +48 12 294 71 20
> 
> The Scalability Specialist  www.gridwisetech.com
> 
> 



[gt-user] Globus Toolkit 5.0.4 is now available

2011-05-18 Thread Stuart Martin
On behalf of the Globus Toolkit development team I am pleased to announce that 
a new point release in the 5.0 series of the Globus Toolkit is now available - 
5.0.4.

Highlights of this release include:

GridFTP
* Fixed GridFTP server bugs in striped or split configurations related 
to DCSC, setting driver stacks, and hanging backend processes.
* Fixed globus-url-copy bugs related to sync operations.
* Added globus-gridftp-server options -dc-default and -fs-default to 
make configurable the default xio driver stack for the network and filesystem. 
Previously, only the client was able to set this in a session. This makes it 
possible to always use drivers such as the rate limiting driver or the 
netlogger driver.

GRAM5
* Added RSL attributes to make it easier to debug an individual gram 
job.
* Added unique job manager log file names so that all log files for all 
users can be written to a central location instead of each user's home 
directory.

MyProxy
* Updated to MyProxy version v5.4

GSI-Enabled OpenSSH
* Updated to gsissh version 5.3

Relevant 5.0.4 links:

- Release notes:http://www.globus.org/toolkit/docs/5.0/5.0.4/rn/
- Software: http://www.globus.org/toolkit/downloads/5.0.4/
- Documentation:http://www.globus.org/toolkit/docs/5.0/5.0.4/
- Support:  
http://dev.globus.org/wiki/Globus_Toolkit#Mailing_Lists

We look forward to and encourage your feedback on GT5.  Thanks for your support 
of Globus software.  

Viva Globus!

-Stu Martin

[gt-user] Globus Toolkit 5.1.0 is now available

2011-05-04 Thread Stuart Martin
On behalf of the Globus Toolkit 5.2 development team I am pleased to announce 
the 2nd *Alpha* version for GT 5.2.  We are calling this release GT 5.1.0.  We 
have at least one more 5.1.x release before announcing a first stable 5.2.0 
release.

Highlights of this GT 5.1.0 (Alpha) release include:

* Added support for additional RPM-based linux platforms
- CentOS 5, Fedora 13 and 14, RedHat 5, and Scientific Linux 5.5
* Added support for MyProxy and GSI-Openssh native packages
* Added a new source installer for non RPM-based platforms

All information about GT 5.2 is here:  
http://dev.globus.org/wiki/Globus_Toolkit/5.2

We look forward to and encourage your feedback on GT5.2.  Thanks for your 
support of Globus software.  

Viva Globus!

-Stu Martin

Re: [gt-user] gt5.0.3 doubt

2011-04-03 Thread Stuart Martin
Hi Jorge,

Gridway has adapters for submitting to either gram2 or gram4.  gram5 is an 
improvement on the gram2 implementation.  gram5 is  backward compatible with 
gram2 with just a few exceptions 
(http://www.globus.org/toolkit/docs/5.0/5.0.3/execution/gram5/rn/#gram5-compatibility).

So yes, you should be able to use Gridway's gram2 adapter to submit jobs to a 
5.0.3 gram (gram5) service.

-Stu

On Mar 29, 2011, at Mar 29, 1:04 PM, Jorge Jaramillo wrote:

> Hello everybody,
> 
> I have a doubt, I read in a previous post that GT5.0.3 does not support ws. 
> The project we are developing is focused on the use of web apps, and java. 
> Can I use Gridway 5.6.1 to submit jobs to Globus gt5.0.3.



[gt-user] Globus Toolkit 5.0.3 is now available

2011-02-15 Thread Stuart Martin
On behalf of the Globus Toolkit development team I am pleased to announce that 
a new point release in the 5.0 series of the Globus Toolkit is now available - 
5.0.3.

Highlights of this release include:

GridFTP
* Added new command: Data Channel Security Context (DCSC)
- Useful for 3rd party transfers between GridFTP servers that 
use different CA certificates
* Added gridftp server chrooting
- Allows admin to limit the directories a gridftp server can 
access
* Added command strings for '-disable-command-list' option for gridftp 
server configuration
* Added Progress markers for stream mode
GRAM5
* Fixed a variety of bugs: PBS and Condor specific, improved 
reliability, improved usability
* Fixed bugs preventing build on Solaris
MyProxy
* Updated to MyProxy version v5.3
GSI-Enabled OpenSSH
* Updated to gsissh version 5.2

Relevant 5.0.3 links:

- Release notes:http://www.globus.org/toolkit/docs/5.0/5.0.3/rn/
- Software: http://www.globus.org/toolkit/downloads/5.0.3/
- Documentation:http://www.globus.org/toolkit/docs/5.0/5.0.3/
- Support:  
http://dev.globus.org/wiki/Globus_Toolkit#Mailing_Lists

We look forward to and encourage your feedback on GT5.  Thanks for your support 
of Globus software.  

Viva Globus!

-Stu Martin


Re: [gt-user] how about gt-sge adapter without mpich installation?

2011-02-07 Thread Stuart Martin
Hi,

GT5 come with support for the GRAM SGE adapter.  You would not have to install 
the AIST SGE adapter.

I'd recommend installing GT 5.0.2.  
http://www.globus.org/toolkit/downloads/5.0.2/

-Stu

On Feb 6, 2011, at Feb 6, 10:58 PM, 王海斌 wrote:

> Dear all, 
> 
> I build a cluster using SGE6.1 and prews-gram. Next step I want to install 
> the gt2-sge adapter software package. I have download it from 
> http://www.apgrid.org/globus-sge/gt2.html and read through the README file. 
> The file says MPICH 1.2.x is a prerequisite for the gt2-sge adapter 
> installation. But I do not intend to submit parallel jobs. How can i bypass 
> the MPICH installation and use the gt2-sge adapter normally? Please help me.
> 
> -- 
> 王海斌



[gt-user] planned downtime for globus.org website and ftp server this weekend

2010-12-02 Thread Stuart Martin
Globus Community,

There is a planned power outage at Argonne's TCS building from Friday 12/3 at 
~6pm thru Sunday ~8pm.  The globus.org website and ftp server is hosted in the 
TCS building.  As such, it will be down during this time.

During the power outage, access will be redirected to this page: 
outage.mcs.anl.gov.

The new Globus Online service will remain up and operational during this time. 
If you're new to Globus Online, please visit www.globusonline.org to learn how 
you can easily move data between GridFTP servers without setting up any custom 
IT infrastructure.

Thanks for your support of Globus software.

-Stu

Re: [gt-user] Globus GRAM reporting status for each task in a SGE job-job array submission

2010-10-01 Thread Stuart Martin
Adam,

Yes - please make these changes/patches available.  We'd like to take a look to 
see what changes might be necessary for GRAM in GT5.

Thanks,
Stu

On Oct 1, 2010, at Oct 1, 11:06 AM, Adam Bazinet wrote:

> Dear Prakashan,
> 
> Thanks for your reply.  Just confirming that we were running into the same 
> problem was very helpful.
> 
> Our solution involves modifying both sge.pm and sge's SEG to keep track, 
> using auxiliary files, of how many sub-jobs in a batch have completed, and to 
> only send the final "Done" notification when all sub-jobs have completed.
> 
> If anyone is interested in having this, we could send you our code, which is 
> basically working.  It consists of a modified sge.pm, sge SEG, and a separate 
> shell script.
> 
> thanks,
> Adam
> 
> 
> 
> On Tue, Sep 7, 2010 at 11:33 AM, Prakashan Korambath  
> wrote:
> Hi Adam,
> 
> It has been almost two years since we looked at this problem.  I have to 
> refresh my memories, but the problem is that it may be difficult for 
> reporting file to tell GRAM that last array job is done.  Say you submitted 
> 500 array jobs, and it is always possible that some of the array jobs like 
> 467, 468 (example) may be still running when job 500 is finished because some 
> nodes may be overloaded etc.  So we can't look at the last array job id even 
> if that is possible.
> 
> I think the best solution would be for GRAM to report back the JOBID to us 
> and we independently probe whether all jobs for that jobid is completed using 
> a Fork job submission periodically.  This will work for workflows.
> 
> Prakashan
> 
> 
> Adam Bazinet wrote:
> Just to follow up on this, here is some additional evidence:
> 
> Globus GRAM debugging:
> 
> 2010-09-06T19:50:00.980-04:00 DEBUG seg.SchedulerEventGenerator 
> [SEG-sge-Thread,run:171] seg input line: 001;1283816997;1708;8;0
> 2010-09-06T19:50:00.980-04:00 DEBUG seg.SchedulerEventGeneratorMonitor 
> [SEG-sge-Thread,addEvent:523]  JSM receiving scheduler event 1708 [Mon Sep 06 
> 19:49:57 EDT 2010] Done
> 2010-09-06T19:50:00.980-04:00 DEBUG seg.SchedulerEventGeneratorMonitor 
> [SEG-sge-Thread,addEvent:534] Dispatching event 1708 to job 
> 26b12c40-ba11-11df-a8c7-93aa2282a0f7
> 2010-09-06T19:50:00.980-04:00 DEBUG utils.GramExecutorService 
> [SEG-sge-Thread,execute:52] # tasks: 0
> 2010-09-06T19:50:00.980-04:00 DEBUG seg.SchedulerEventGenerator 
> [SEG-sge-Thread,run:171] seg input line: 001;1283816997;1708;8;0
> 2010-09-06T19:50:00.980-04:00 DEBUG exec.ManagedExecutableJobHome 
> [pool-2-thread-1,jobStateChanged:399] Receiving jobStateChange event for 
> resource key 
> {http://www.globus.org/namespaces/2008/03/gram/job}ResourceID=26b12c40-ba11-11df-a8c7-93aa2282a0f7
>  with:
> timestamp Mon Sep 06 19:49:57 EDT 2010
> (new) state Done
> exitCode 0
> 
> The "seg input line", above, occurs in the SGE reporting file here:
> 
> 1283816997:job_log:1283816997:deleted:1708:1:NONE:T:scheduler:topaz.si.edu:0:1024:1283816885:sge_job_script.38191:globus:staff::defaultdepartment:sge:job
>  deleted by schedd
> 1283816997:job_log:1283816997:deleted:1708:7:NONE:T:scheduler:topaz.si.edu:0:1024:1283816885:sge_job_script.38191:globus:staff::defaultdepartment:sge:job
>  deleted by schedd
> 
> (the only two lines in the file with this timestamp; these happen to be the 
> first 2/8 jobs in the batch to finish)
> 
> It's pretty clear that as soon as any sub-job finishes, Globus thinks the 
> whole batch is done and goes ahead with subsequent processing stages (e.g., 
> MergeStdout, StageOut).  I'm guessing the place to fix this is in the SEG 
> code; I'm willing to bet someone already has patched this.  If so, would you 
> be willing to share?
> 
> thanks,
> Adam
> 
> 
> 
> On Mon, Sep 6, 2010 at 5:17 PM, Adam Bazinet 
> mailto:pknut...@umiacs.umd.edu>> wrote:
> Hi everyone,
> 
> Was this issue ever resolved?  It is affecting our Globus installation 
> (4.2.1) and SGE cluster as well.  Specifically, the job seems to enter the 
> StageOut phase prematurely (say, when 6/8 jobs in a task array are 
> completed).  Any assistance is greatly appreciated.
> 
> thanks,
> Adam
> 
> 
> 
> On Tue, May 27, 2008 at 12:51 PM, Korambath, Prakashan 
> mailto:p...@ats.ucla.edu>> wrote:
> 
> Hi Martin,
> 
>  I am using gt4.0.6 on the client node.  I didn't try with Fork.  Let me see 
> how Fork behaves.  Thanks.
> 
> Prakashan
> 
> 
> 
> 
> -Original Message-
> From: Martin Feller [mailto:fel...@mcs.anl.gov]
> Sent: Tue 5/27/2008 9:48 AM
> To: Korambath, Prakashan
> Cc: gt-user; Jin, Kejian; Korambath, Prakashan
> Subject: Re: [gt-user] Globus GRAM reporting status for each task in a SGE 
> job-job array submission
> 
> Prakashan:
> 
> GRAM should send a Done notification if the last job is done, and not when
> the first job is done. I tried it here and it works as expected for me.
> What GT version are you using?
> This is probably not at all SGE related, but does it behave in the same way
> when you submit to, say, Fork instead of SGE?
>

Re: [gt-user] Recommondetion about Broker for Globus Toolkit

2010-08-19 Thread Stuart Martin
Here are some options for brokers.  They operate on top of GRAM and GridFTP:
Condor-G -> http://www.cs.wisc.edu/condor/condorg/
Swift -> http://www.ci.uchicago.edu/swift/index.php
GridWay -> www.gridway.org
Nimrod-G -> http://www.csse.monash.edu.au/~davida/nimrod/nimrodg.htm
CoG kit clients -> http://wiki.cogkit.org/wiki/Java_CoG_Kit, 
http://dev.globus.org/wiki/CoG_jglobus

GRMS does not appear to have been updated in a while.  Which version are you 
considering?

Any of these can be used as a "broker", can be extended to add new scheduling 
policies, and are actively used.  Depending on your goals/requirements, 
probably any of these can be used.

-Stu

On Aug 18, 2010, at Aug 18, 6:20 PM, Hasret Turkeri wrote:

> Hello!
> 
> I am Hasret Turkeri from Turkey.
> I am working in a school project in which we are working to construct a
> grid system which consist of three clusters.
> In order to achieve it, I have installed Globus toolkit on each cluster
> and they work properly.
> 
> At this point, I plan to setup a broker for the grid in order to make the
> user submit their jobs through the borker without considering the cluster
> which he/she should choice to submit their jobs.
> 
> I have made a quick search to obtain the convenient broker and I have
> found a number of brokers, GRMS, Gridway, etc.. But at this point I am not
> clear that which borker I should install.
> 
> At that point I need a recommendation about the broker system.
> Which broker system should I install?
> Which broker system is more convenient to the grid system which I have
> mentioned above?
> 
> Thanks for your helps.
> best regards.
> 
> Hasret Turkeri
> 
> 
> -- 
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
> 



[gt-user] Globus Toolkit 5.0.2 is now available

2010-07-16 Thread Stuart Martin
On behalf of the Globus Toolkit development team I am pleased to announce that 
a new point release in the 5.0 series of the Globus Toolkit is now available - 
5.0.2.

Highlights of this release include:

GridFTP
* Synchronization (globus-url-copy -sync) feature that transfers files 
only if they do not exist at the destination or differ from the source
* An offline mode for the server
GRAM5
* Improvements have been made to address all the known blocker issues 
for production deployment on TeraGrid and OSG
MyProxy
* Updated to MyProxy v5.2

Relevant 5.0.2 links:

- Release notes:http://www.globus.org/toolkit/docs/5.0/5.0.2/rn/
- Software: http://www.globus.org/toolkit/downloads/5.0.2/
- Documentation:http://www.globus.org/toolkit/docs/5.0/5.0.2/
- Support:  
http://dev.globus.org/wiki/Globus_Toolkit#Mailing_Lists

We look forward to and encourage your feedback on GT5.  Thanks for your support 
of Globus software.  

Viva Globus!

-Stu Martin

[gt-user] Globus Toolkit 5.0.1 is now available

2010-03-26 Thread Stuart Martin
On behalf of the Globus Toolkit development team I am pleased to announce that 
a new point release in the 5.0 series of the Globus Toolkit is now available - 
5.0.1.

Highlights of this release include:

GridFTP
* New globus-url-sync command for syncing individual files or 
directories
* New server option to control the default permissions of created files
* New server option to time out on slow or hanging filesystems
* New server logging level to include transfer statistics
GRAM5
* Improved reliability with Condor-G clients
* Fixed a number of bugs and memory leaks
MyProxy
* Updated to MyProxy v5.1
GSI-OpenSSH
* Updated to GSI-OpenSSH v5.2
GSI
* Added OpenSSL 1.0.0 Support

Relevant 5.0.1 links:

- Release notes:http://www.globus.org/toolkit/docs/5.0/5.0.1/rn/
- Software: http://www.globus.org/toolkit/downloads/5.0.1/
- Documentation:http://www.globus.org/toolkit/docs/5.0/5.0.1/
- Support:  
http://dev.globus.org/wiki/Globus_Toolkit#Mailing_Lists

We look forward to and encourage your feedback on GT5.  Thanks for your support 
of Globus software.  

Viva Globus!

-Stu Martin

Re: [gt-user] globus-mds-info service

2010-03-17 Thread Stuart Martin
This is still a problem on Big Red.  Any other ideas?

-Stu

On Jan 9, 2009, at Jan 9, 10:43 AM, Mike Lowe wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> Stuart Martin wrote:
>> Kate, Mike,
>> 
>> This is not a familiar error to the Globus folks I asked.
>> 
>> Eric Blau mentioned there was an update recently (December) to the list
>> of CAs that TG trusts, but I have no idea if its absence (or presence)
>> would cause something like that error.
>> So compare the ca dirs between the working host and the not working host
>> 
> 
> The CA dir on BigRed is up to date, the inca user on BigRed gets the
> error my user account on BigRed does not get the error.  Kate copied
> /etc/grid-security/certificates on a working host at SDSC to
> ~/.globus/certificates and still got the error.
> 
> I ran the following on BigRed as the inca user.
> 
> BigRed grid-security/certificates> openssl verify -CApath
> /etc/grid-security/certificates/  -verbose
> /N/soft/certificates/g1.hostcert.pem
> /N/soft/certificates/g1.hostcert.pem: OK
> 
> 
>> Charles suggested to check JAVA_HOME and check CLASSPATH
>> 
> JAVA_HOME and CLASSPATH are identical
> 
>> -Stu
>> 
>> On Jan 8, 2009, at Jan 8, 10:32 AM, Stuart Martin wrote:
>> 
>>> The situation is that a GT4 container is working on bigred except for
>>> the inca account from the bigred login host.  The inca account can
>>> successfully run commands from a different host.
>>> 
>>> Here is the error pulled from below:
>>>>>>> 
>>> BigRed inca/BigRed>  wsrf-query -a -s
>>> https://g1.bigred.teragrid.iu.edu:8446/wsrf/services/DefaultIndexService
>>> 
>>> Error: ; nested exception is:
>>>org.globus.common.ChainedIOException: Authentication failed
>>> [Caused by: org.ietf.jgss.GSSException, major code: 11, minor code: 0
>>>major string: General failure, unspecified at GSSAPI level
>>>minor string: None [Caused by: Bad certificate
>>> (java.security.SignatureException: SHA-1/RSA/PKCS#1: Not
>>> initialized)]]
>>> <<<<
>>> 
>>> 
>>> Any ideas?
>>> 
>>> Thanks,
>>> -Stu
>>> 
>>> Begin forwarded message:
>>> 
>>>> From: Mike Lowe 
>>>> Date: January 7, 2009 9:01:44 PM CST
>>>> To: Kate Ericson 
>>>> Cc: Stuart Martin , Inca Inca 
>>>> Subject: Re: globus-mds-info service
>>>> 
>> Good call on the hostname, I made the typo in the kit reg and repeatedly
>> missed it.  The iu.edu name is now in the reg and is the correct one.
>> 
>> g1:~ # openssl x509 -in /etc/grid-security/containercert.pem -noout
>> -subject
>> subject= /C=US/O=UTAustin/OU=TACC/CN=g1.bigred.teragrid.iu.edu
>> 
>> 
>> Kate Ericson wrote:
>>>>>> The mds-info service started failing on bigred on Wednesday, Novemeber
>>>>>> 5, 2008 at 4PM Pacific.  I can get it to pass on SDSC's cluster if I
>>>>>> use
>>>>>> "g1.bigred.teragrid.iu.edu" instead of "g1.bigred.iu.teragrid.org"
>>>>>> (which is what is registered in the core kit for the service).  It
>>>>>> still
>>>>>> fails on bigred though and there wasn't anything set for a trusted cert
>>>>>> location, I'm using the same cert as SDSC and the DN's in the
>>>>>> grid-mapfile.  I decided to copy SDSC's certs into bigred's
>>>>>> ~inca/.globus/certificates (see error below) so that I would have the
>>>>>> same cert directory that was succeeding at SDSC.  The only thing google
>>>>>> turned up for me with a similar error was
>>>>>> http://www.mail-archive.com/gt-user@lists.globus.org/msg00298.html. 
>>>>>> Any
>>>>>> ideas?  Here's the debug output:
>>>>>> 
>>>>>> 
>>>>>> BigRed BigRed/.globus> wsrf-query -d -a -s
>>>>>> "https://g1.bigred.teragrid.iu.edu:8446/wsrf/services/DefaultIndexService";
>>>>>> 
>>>>>> "//*[local-name()='V4KitsRP']/*//Kit[Name = 'core.teragrid.org' and
>>>>>> Version = '4.0.2']"
>>>>>> 
>>>>>> AxisFault
>>>>>> faultCode:
>>>>>> {http://schemas.xmlsoap.org/soap/envelope/}Server.userException
>>>>>> faultSubcode:
&g

Re: [gt-user] how to automatically divide a task?

2010-03-10 Thread Stuart Martin
Your assessment is correct.  GRAM provides more the plumbing.  There are some 
clients out there that provide this type of additional automated functionality 
on top of GRAM.  Or, you need to write a client to take advantage of it.

Here are a list of some GRAM clients for you to look at:
Condor-G -> http://www.cs.wisc.edu/condor/condorg/
Swift -> http://www.ci.uchicago.edu/swift/index.php
GridWay -> www.gridway.org
Nimrod-G -> http://www.csse.monash.edu.au/~davida/nimrod/nimrodg.htm
CoG kit clients -> http://wiki.cogkit.org/wiki/Java_CoG_Kit, 
http://dev.globus.org/wiki/CoG_jglobus

-Stu

On Mar 9, 2010, at Mar 9, 10:40 PM, Shuhan Xu wrote:

> Hi,
>> From concepts of grid computing, it is said that grid can find free
> resources and automatically allocate to tasks.
> But by using GRAM5, I found it is not that much "automatically". I
> have to design in detail. Like node1 process A1 and node2 process A2.
> 
> Is there any way that I can just throw the task and let Globus divide
> it and find free resources to run?
> 
> Thank you
> Shuhan Xu



Re: [gt-user] GRAM 5 and Java Jobs

2010-02-26 Thread Stuart Martin
You can submit a job that runs a java program.  Here is an example:

http://www.globus.org/toolkit/docs/5.0/5.0.0/execution/gram5/user/#id2522880

GT4 WS GRAM *service* had a build dependency on the Java Web Services Core 
code.  The GRAM5 service does not.  GRAM5 does not have any web service 
dependencies.

-Stu

On Feb 26, 2010, at Feb 26, 11:22 AM, Bruno Oliveira wrote:

> Hello,
> I have been reading the new globus 5 and i read that new globus doesn't have 
> java core, what is that means? With globus5 it's not possible submit java 
> jobs like the older GT4?
> 
> Best regards,
> Bruno Oliveira



Re: [gt-user] GRAM5

2010-02-17 Thread Stuart Martin
Hi Doru,

Good question.  I'm gonna include the gt-users here for others benefit too.

Yes.  You can only control it at the job manager configuration level that sets 
the directory location for all users.  See the JM config "-stdio-log 
LOG_DIRECTORY" parameter.  By using the $(HOME) variable, you get separate 
locations (directories) for each user.  Beware, the JM runs under the user's 
account, so all user's must be able to write to the location set by that 
parameter.  Also, a new JM log file is created each day, so the log files can 
be more easily removed/archived/...

I'd be curious to know what your specific plans are for this, so we could doc a 
concrete example for best practices.  The JM log location cannot be 
controlled/set for an individual job.

Details are here:  
http://www.globus.org/toolkit/docs/5.0/5.0.0/execution/gram5/admin/#gram5-cmd-globus-job-manager
>>>
-stdio-log LOG_DIRECTORY
Configure the job manager to write log messages to files in the LOG_DIRECTORY 
directory. Files will be named LOG_DIRECTORY/gram_MMDD.log. Logging is 
further controlled by the argument to the -log-levels parameter described 
below. The LOG_DIRECTORY value can include variables derived from the job 
manager environment using the same syntax as RSL substitutions. For example, 
-stdio-log $(HOME) would cause each user's logs to be stored in their 
individual home directories.
<<<

-Stu

On Feb 17, 2010, at Feb 17, 9:33 AM, Doru Marcusiu wrote:

> Hey Stu,
> 
> I was wondering if a particular feature made it in to GRAM5. Some time ago I 
> was asked if it is possible to define where the GRAM job manager log file 
> would be written. A reply from you and/or Joe Bester implied that GRAM5 may 
> provide a a way to do this either for all jobs or for individuals to define 
> for their jobs. Is that possible in GRAM5?
> 
> Doru
> 
> 
> 
> -- 
> =
> Doru Marcusiu
> Assistant Director, Production Cyber Environments
> National Center Supercomputing Applications
> 



Re: [gt-user] Breaking up the Globus modules

2010-02-09 Thread Stuart Martin
Hi Aaron,

Thanks for the feedback and sorry about the troubles.  Looks like we need to 
make some improvements in the GT doc.

I assume you looked at the quickstart guide for setting up the GT components.  
Maybe that is too basic and does not discuss well the options/recommendations 
for setting up a grid?  Seems like we need a new higher-level doc for that. 
   http://www.globus.org/toolkit/docs/5.0/5.0.0/admin/quickstart/#quickstart

For the GRAM LRMs supported by Globus, there is a list here in the admin guide. 
 SGE is listed there.  That seems ok to me.

http://www.globus.org/toolkit/docs/5.0/5.0.0/execution/gram5/admin/#gram5-admin-lrmAdapters

Cheers,
-Stu

On Feb 8, 2010, at Feb 8, 9:58 PM, Aaron Hicks wrote:

> Hi All,
> 
> Still working on this, having a very difficult time figuring out what 
> components of the Globus Toolkit I need to install, and where they ought to 
> be installed.
> 
> I have figured that I can just have GRAM5 on the Head Node (aka Service Node, 
> first node) of our cluster and use the LRM Adaptor and LRM Module to push 
> jobs into our existing Sun Grid Engine. A lot of reading needed to find that 
> GRAM5 supports SGE out of the box.
> 
> ...but do not want the GridFTP service managed or stored from the Head Node. 
> I've also found that you often have to drill very deeply into the 
> documentation before you discover that you do or don't need a component. Even 
> as far as reading the SRB site to figure out that I won't need that and the 
> SRB-DSI until much later.
> 
> Regards,
> 
> Aaron Hicks
> 
>> -Original Message-
>> From: gt-user-boun...@lists.globus.org [mailto:gt-user-
>> boun...@lists.globus.org] On Behalf Of Aaron Hicks
>> Sent: Tuesday, 2 February 2010 4:10 p.m.
>> To: gt-user@lists.globus.org
>> Subject: [gt-user] FW: Breaking up the Globus modules
>> 
>> Sorry everyone, posted this to the wrong list.
>> 
>> ...perhaps the announce list shouldn't be postable by anyone.
>> 
>> -Original Message-
>> From: announce-boun...@lists.globus.org [mailto:announce-
>> boun...@lists.globus.org] On Behalf Of Aaron Hicks
>> Sent: Tuesday, 2 February 2010 1:29 p.m.
>> To: annou...@lists.globus.org
>> Subject: [Globus] Breaking up the Globus modules
>> 
>> Hi all,
>> 
>> The Globus toolkit documentation states that Globus Toolkit 5.0.0 is
>> modular, and it gives some explanation about each component (MyProxy,
>> GridFTP, RLS, GRAM5, etc). What's not stated is which modules must be
>> installed on First Node ('elephant' in the Quickstart guide), and which
>> modules could be installed on other servers.
>> 
>> It looks like MyProxy and GridFTP could be set up on separate machines,
>> but I'm unsure about the other components.
>> 
>> We'd like to _not_ have a single monolithic installation on the head
>> node of our cluster, and instead distribute out various services (like
>> MyProxy) to virtual machines, or GridFTP/RLS onto a server/VM backed by
>> our SAN. This would allow us to tweak these components of Globus
>> without interfering with the others.
>> 
>> ...it's going to be complex enough setting up GRAM5 so that it pipes
>> jobs into SGE. It looks like it should just work.
>> 
>> Regards,
>> 
>> Aaron Hicks
>> 
>> 
>> Please consider the environment before printing this email
>> Warning:  This electronic message together with any attachments is
>> confidential. If you receive it in error: (i) you must not read, use,
>> disclose, copy or retain it; (ii) please contact the sender immediately
>> by reply email and then delete the emails.
>> The views expressed in this email may not be those of Landcare Research
>> New Zealand Limited. http://www.landcareresearch.co.nz
>> 
>> Please consider the environment before printing this email
>> Warning:  This electronic message together with any attachments is
>> confidential. If you receive it in error: (i) you must not read, use,
>> disclose, copy or retain it; (ii) please contact the sender immediately
>> by reply email and then delete the emails.
>> The views expressed in this email may not be those of Landcare Research
>> New Zealand Limited. http://www.landcareresearch.co.nz
> 
> Please consider the environment before printing this email
> Warning:  This electronic message together with any attachments is 
> confidential. If you receive it in error: (i) you must not read, use, 
> disclose, copy or retain it; (ii) please contact the sender immediately by 
> reply email and then delete the emails.
> The views expressed in this email may not be those of Landcare Research New 
> Zealand Limited. http://www.landcareresearch.co.nz



[gt-user] GlobusWORLD 2010

2010-02-08 Thread Stuart Martin
Hi All,

I wanted to remind and welcome everyone to come to GlobusWorld here at Argonne. 
 It is just 3 weeks away!

The program is full of many interesting talks and topics, including: Security, 
Data, BioMedical Grids, Workflow, Cloud computing, future plans (GRAM, 
Globus.Org, Crux) and a number of updates from the Globus community.

Then on Thursday, there are two 3 hours tutorials, both sessions will be 
interactive with instructors walking through how to use the Globus services.

Globus.Org Tutorial "An introduction to Globus.org, a new addition to the 
Globus software suite..."

GT5.0 Tutorial  "We will walk through the steps required for setting up 
a Grid infrastructure using the components of Globus Toolkit 5 (GT5) such as 
GRAM, GridFTP, GSI, GSI-OpenSSH and MyProxy..."

All the details are here: http://globusworld.org/program/

I hope to see you there!

-Stu

Begin forwarded message:

> From: Julie Wulf-Knoerzer 
> Date: February 1, 2010 3:36:14 PM CST
> To: Julie Wulf-Knoerzer 
> Subject: [gt-dev] GlobusWORLD 2010: Program Abstracts Now Available
> 
> A friendly reminder that GlobusWORLD 2010 will be held March 2-4 at Argonne 
> (near Chicago Illinois, USA). This year’s GlobusWORLD conference will focus 
> on Globus success stories, including discussions with Globus users about 
> their experiences, technical presentations, and tutorials on using new (and 
> old) Globus software.
> 
> GLOBUSWORLD 2010 PROGRAM
> The Preliminary Program with abstracts are online and can be accessed via 
> www.globusworld.org/program.
> 
> REGISTRATION
> Registration is available at www.globusworld.org.  The low registration fee 
> of $150 includes access to all program sessions, tutorials, meals, and social 
> functions.  Due to Argonne security restrictions, attendees are encouraged to 
> register as early as possible to ensure admittance to the site.
> 
> MORE INFORMATION
> The Globus community is comprised of individuals representing many 
> organizations and projects, and we are excited about engaging all 
> constituents in re-invigorating the Globus effort.  We are enthusiastic about 
> increasing the value that Globus can deliver to the many, multi-institutional 
> scientific and biomedical communities whose need for robust Grid computing 
> middleware continues to grow unabated.
> 
> See www.globusworld.org/about for details or to unsubscribe.



Re: [gt-user] prews-test

2010-01-22 Thread Stuart Martin
There is doc and examples here:

http://www.globus.org/toolkit/docs/5.0/5.0.0/execution/gram5/admin/#gram5-admin-testing

-Stu

On Jan 22, 2010, at Jan 22, 12:12 PM, Nikolay Kutovskiy wrote:

> It is written in documentation
> (http://www.globus.org/toolkit/docs/5.0/5.0.0/admin/install/#gtadmin-packaging)
> there is the target 'prews-test' which is "Tests for pre-webservices
> components".
> I wonder how those tests can be run?
> 
> Nikolay.



Re: [gt-user] globus-job-status: UNKNOWN JOB STATE 0

2010-01-22 Thread Stuart Martin
On Jan 22, 2010, at Jan 22, 12:00 PM, Nikolay Kutovskiy wrote:

> Hi Stuart
> 
> Stuart Martin wrote on 22/01/10 19:04:
>> Hi Nikolay,
>> 
>> Strange.  Yea - looks like a bug.  Is that repeatable / happens for every 
>> PBS job?
> I haven't tested hard yet but at least I saw such status twice (for
> different jobs).
> 
>> 
>> Are you using the SEG for PBS job monitoring?
> not yet. It wasn't clear for me how to configure SEG. It's written:
> It must be explicitly enabled by adding the -seg-module LRM option to
> the job manager configuration.
> But in what configuration file does it need to specify "-seg-module"
> option? in $GLOBUS_LOCATION/etc/globus-pbs.conf?

Doc is here: 
http://www.globus.org/toolkit/docs/5.0/5.0.0/execution/gram5/admin/#gram5-Interface_Config_Frag-seg_module

No - it goes in $GLOBUS_LOCATION/etc/grid-services/jobmanager-pbs

Mine looks like this:
% cat jobmanager-pbs
stderr_log,local_cred - 
/home/smartin/gt/5.0.0/INSTALL/libexec/globus-job-manager globus-job-manager 
-conf /home/smartin/gt/5.0.0/INSTALL/etc/globus-job-manager.conf -type pbs 
-seg-module pbs

> How? Just add
> '-seg-module' to new line? e.g.
> log_path=/var/spool/torque/server_logs
> -seg-module
> $ cat /usr/local/globus-5.0.0/etc/globus-pbs.conf
> log_path=/var/spool/torque/server_logs
> -seg-module
> 
> [root]$ globus-job-manager-event-generator -scheduler pbs -background
> -pidfile $GLOBUS_LOCATION/var/globus-job-manager-seg-pbs.pid
> Error: pbs not configured

Maybe that is a result of you adding the extra -seg-module line to the 
globus-pbs.conf file ??

> 
> So what exact steps needs to be performed to run SEG?
> 
> Nikolay.
> 
>> If not, try using the SEG and see what happens.
>>  
>> http://www.globus.org/toolkit/docs/5.0/5.0.0/execution/gram5/admin/#gram5-Interface_Config_Frag-seg_module
>>  
>> http://www.globus.org/toolkit/docs/5.0/5.0.0/execution/gram5/admin/#id2545819
>> 
>> -Stu
>> 
>> On Jan 22, 2010, at Jan 22, 7:43 AM, Nikolay Kutovskiy wrote:
>> 
>>> Hello list,
>>> 
>>> I have installed GRAM5 to use PBS and get the following output of
>>> globus-job-status command:
>>> $ globus-job-submit :2119/jobmanager-pbs /bin/hostname
>>> https://:51499/16073727533535086921/7782764993921513916/
>>> 
>>> [user]$ globus-job-status
>>> https://:51499/16073727533535086921/7782764993921513916/
>>> UNKNOWN JOB STATE 0
>>> 
>>> [user]$ globus-job-status
>>> https://:51499/16073727533535086921/7782764993921513916/
>>> UNKNOWN JOB STATE 0
>>> 
>>> [user]$ globus-job-status
>>> https://:51499/16073727533535086921/7782764993921513916/
>>> UNKNOWN JOB STATE 0
>>> [user]$ globus-job-status
>>> https://:51499/16073727533535086921/7782764993921513916/
>>> DONE
>>> 
>>> Is that a bug? some GRAM5|PBS misconfiguration?
>>> 
>>> Commands like globus-job-run work fine.
>>> Environment:
>>> gt5.0.0-all-source-installer.tar.bz2
>>> torque-2.3.7-1cri
>>> torque-docs-2.3.7-1cri
>>> torque-server-2.3.7-1cri
>>> torque-client-2.3.7-1cri
>>> torque-scheduler-2.3.7-1cri
>>> 
>>> Thanks in advance,
>>> Nikolay
>> 



Re: [gt-user] globus-job-status: UNKNOWN JOB STATE 0

2010-01-22 Thread Stuart Martin
Hi Nikolay,

Strange.  Yea - looks like a bug.  Is that repeatable / happens for every PBS 
job?

Are you using the SEG for PBS job monitoring?  If not, try using the SEG and 
see what happens.

http://www.globus.org/toolkit/docs/5.0/5.0.0/execution/gram5/admin/#gram5-Interface_Config_Frag-seg_module

http://www.globus.org/toolkit/docs/5.0/5.0.0/execution/gram5/admin/#id2545819

-Stu

On Jan 22, 2010, at Jan 22, 7:43 AM, Nikolay Kutovskiy wrote:

> Hello list,
> 
> I have installed GRAM5 to use PBS and get the following output of
> globus-job-status command:
> $ globus-job-submit :2119/jobmanager-pbs /bin/hostname
> https://:51499/16073727533535086921/7782764993921513916/
> 
> [user]$ globus-job-status
> https://:51499/16073727533535086921/7782764993921513916/
> UNKNOWN JOB STATE 0
> 
> [user]$ globus-job-status
> https://:51499/16073727533535086921/7782764993921513916/
> UNKNOWN JOB STATE 0
> 
> [user]$ globus-job-status
> https://:51499/16073727533535086921/7782764993921513916/
> UNKNOWN JOB STATE 0
> [user]$ globus-job-status
> https://:51499/16073727533535086921/7782764993921513916/
> DONE
> 
> Is that a bug? some GRAM5|PBS misconfiguration?
> 
> Commands like globus-job-run work fine.
> Environment:
> gt5.0.0-all-source-installer.tar.bz2
> torque-2.3.7-1cri
> torque-docs-2.3.7-1cri
> torque-server-2.3.7-1cri
> torque-client-2.3.7-1cri
> torque-scheduler-2.3.7-1cri
> 
> Thanks in advance,
> Nikolay



Re: [gt-user] GT 5.0.0 and RFT

2010-01-20 Thread Stuart Martin
Yes - that's right.

Cheers,
Stu

On Jan 20, 2010, at Jan 20, 2:38 PM, Arn wrote:

> ok, I assume this means that I can upgrade our current GridFTP
> machines to 5.0.0 and keep our (separate) RFT server at GT4.2.1.
> 
> Thanks for your prompt response.
> 
> Arn
> 
> On Wed, Jan 20, 2010 at 12:06 PM, Stuart Martin  wrote:
>> Hi Arn,
>> 
>> You can continue to use RFT from the 4.2 or 4.0 release for some time.  It 
>> will still work with gridftp from gt5.
>> 
>> We are working on a new globus.org (online) file transfer service.  It is 
>> still in development.  Some high-level details about it are here: 
>> http://globus.org/service/  We are hoping to be adding a few "beta" testers 
>> in the March - April timeframe.
>> 
>> -Stu
>> 
>> On Jan 20, 2010, at Jan 20, 1:53 PM, Arn wrote:
>> 
>>> Hi,
>>> 
>>> Why is RFT not included in GT 5.0.0 ? What if I need a reliable file
>>> transfer system for GridFTP in 5.0 , what can I do ?
>>> 
>>> According to the release notes some replacement is in development,
>>> specifically which component will replace RFT.
>>> However, the release notes for GridFTP in 5.0.0 indicate that some
>>> reliability is built into GridFTP. So can someone please clarify about
>>> what to do if a reliable file transfer is needed with GridFTP in GT
>>> 5.0.0 ?
>>> 
>>> Thanks in advance
>>> Arn
>> 
>> 



Re: [gt-user] GT 5.0.0 and RFT

2010-01-20 Thread Stuart Martin
Hi Arn,

You can continue to use RFT from the 4.2 or 4.0 release for some time.  It will 
still work with gridftp from gt5.

We are working on a new globus.org (online) file transfer service.  It is still 
in development.  Some high-level details about it are here: 
http://globus.org/service/  We are hoping to be adding a few "beta" testers in 
the March - April timeframe.

-Stu

On Jan 20, 2010, at Jan 20, 1:53 PM, Arn wrote:

> Hi,
> 
> Why is RFT not included in GT 5.0.0 ? What if I need a reliable file
> transfer system for GridFTP in 5.0 , what can I do ?
> 
> According to the release notes some replacement is in development,
> specifically which component will replace RFT.
> However, the release notes for GridFTP in 5.0.0 indicate that some
> reliability is built into GridFTP. So can someone please clarify about
> what to do if a reliable file transfer is needed with GridFTP in GT
> 5.0.0 ?
> 
> Thanks in advance
> Arn



[gt-user] GT 5.0.0 is now available

2010-01-20 Thread Stuart Martin
On behalf of the Globus Toolkit development team I am pleased to announce that 
a new stable release of the Globus Toolkit is now available.

Most components of GT5 are incremental updates (numerous bug fixes and new 
features) over their GT4 counter-parts (e.g. GridFTP, RLS, MyProxy, 
GSI-OpenSSH). Some components of GT4 are not included in GT5 (e.g. GT4 Java 
Core, WS-GRAM4, RFT), to be replaced by new software under development (e.g. 
Crux, Globus.org Service).  And GRAM has been vastly improved by returning to 
the pre-ws GRAM2 code base and making substantial backward-compatible 
improvements.  Details about each component are available in the release notes. 

GT4.x releases will continue to be maintained and supported at least through 
the end of 2010.  New users should immediately adopt 5.0.0.  Existing users are 
encouraged to begin evaluating and migrating  to GT5.  Please contact us if you 
have questions or problems along the way.

We look forward to and encourage your feedback on GT5.

Relevant 5.0.0 links:

- Release notes:http://www.globus.org/toolkit/docs/5.0/5.0.0/rn/
- Software: http://www.globus.org/toolkit/downloads/5.0.0/
- Documentation:http://www.globus.org/toolkit/docs/5.0/5.0.0/
- Support:  
http://dev.globus.org/wiki/Globus_Toolkit#Mailing_Lists
- Background:   
http://lists.globus.org/pipermail/announce/2009-March/54.html

http://lists.globus.org/pipermail/gt-user/2009-October/008578.html.

Thanks for your support of Globus software.

Viva Globus!

-Stu Martin

Re: [gt-user] Issues with Fedora Core 12 and Globus Toolkit 4.2.1?

2010-01-07 Thread Stuart Martin
On Jan 7, 2010, at Jan 7, 11:33 AM, Craig E. Ward wrote:

> Thanks for the reply.
> 
> I don't know that I need the latest and greatest Globus Toolkit for what I'm 
> doing. The Fedora Core 12 repositories have Globus Toolkit packages. Do you 
> have any sense that these are "good enough" for most uses?

Yup - I think so.

> 
> Thanks,
> 
> Craig
> 
> Stuart Martin wrote:
>> Looks like it's an issue with the changes in openssl 1.0, which has already 
>> been reported here:
>>  https://bugzilla.mcs.anl.gov/globus/show_bug.cgi?id=6841
>> No GT version supports openssl 1.0 - even the upcoming GT 5.0.0.  But, we 
>> plan to add support for it in the first quarter of 2010.  So, probably GT 
>> 5.0.1.
>> As for a workaround, can you install on older version of openssl and use 
>> that for the globus build?
>> -Stu
>> On Jan 6, 2010, at Jan 6, 5:05 PM, Craig E. Ward wrote:
>>> I'm attempting to build from source version 4.2.1 of the toolkit on a 
>>> Fedora Core 12 Linux system. It's failing during compilation:
>>> 
>>> /usr/lib64/ccache/gcc -DPACKAGE_NAME=\"\" -DPACKAGE_TARNAME=\"\" 
>>> -DPACKAGE_VERSION=\"\" -DPACKAGE_STRING=\"\" -DPACKAGE_BUGREPORT=\"\" 
>>> -DPACKAGE=\"globus_gsi_proxy_ssl\" -DVERSION=\"1.5\" -I. 
>>> -I/users/wardc/Build/gt4.2.1-all-source-installer/source-trees-thr/gsi/proxy/proxy_ssl/source/library
>>>  -I/usr/local/globus-4.2.1/include/gcc64dbgpthr -I. 
>>> -I/usr/local/globus-4.2.1/include 
>>> -I/usr/local/globus-4.2.1/include/gcc64dbgpthr -g -m64 -Wall -c 
>>> proxycertinfo.c  -fPIC -DPIC -o .libs/proxycertinfo.o
>>> In file included from proxycertinfo.h:36,
>>>from proxycertinfo.c:34:
>>> proxypolicy.h:105: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ 
>>> before ‘*’ token
>>> In file included from proxycertinfo.c:34:
>>> proxycertinfo.h:124: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or 
>>> ‘__attribute__’ before ‘*’ token
>>> proxycertinfo.c:52: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ 
>>> before ‘*’ token
>>> proxycertinfo.c: In function ‘PROXYCERTINFO_new’:
>>> proxycertinfo.c:87: warning: cast from pointer to integer of different size
>>> proxycertinfo.c: In function ‘PROXYCERTINFO_dup’:
>>> proxycertinfo.c:139: warning: passing argument 2 of ‘ASN1_dup’ from 
>>> incompatible pointer type
>>> /usr/include/openssl/asn1.h:955: note: expected ‘void * (*)(void **, const 
>>> unsigned char **, long int)’ but argument is of type ‘char * (*)()’
>>> proxycertinfo.c: In function ‘d2i_PROXYCERTINFO’:
>>> proxycertinfo.c:465: warning: passing argument 2 of ‘d2i_PROXYPOLICY’ from 
>>> incompatible pointer type
>>> proxypolicy.h:146: note: expected ‘unsigned char **’ but argument is of 
>>> type ‘const unsigned char **’
>>> proxycertinfo.c: In function ‘d2i_PROXYCERTINFO_OLD’:
>>> proxycertinfo.c:560: warning: passing argument 2 of ‘d2i_PROXYPOLICY’ from 
>>> incompatible pointer type
>>> proxypolicy.h:146: note: expected ‘unsigned char **’ but argument is of 
>>> type ‘const unsigned char **’
>>> make[2]: *** [proxycertinfo.lo] Error 1
>>> make[2]: Leaving directory 
>>> `/users/wardc/Build/gt4.2.1-all-source-installer/source-trees-thr/gsi/proxy/proxy_ssl/source/library'
>>> make[1]: *** [all-recursive] Error 1
>>> make[1]: Leaving directory 
>>> `/users/wardc/Build/gt4.2.1-all-source-installer/source-trees-thr/gsi/proxy/proxy_ssl/source'
>>> 
>>> ERROR: Build has failed
>>> make: *** [globus_gsi_proxy_ssl-thr] Error 2
>>> 
>>> Anyone see this error before? Any suggestions for a workaround?
>>> 
>>> Thanks,
>>> 
>>> Craig
>>> 
>>> -- 
>>> Craig E. Ward
>>> USC Information Sciences Institute
>>> 310-448-8271
>>> cw...@isi.edu
> 
> -- 
> Craig E. Ward
> USC Information Sciences Institute
> 310-448-8271
> cw...@isi.edu



Re: [gt-user] Issues with Fedora Core 12 and Globus Toolkit 4.2.1?

2010-01-06 Thread Stuart Martin
Looks like it's an issue with the changes in openssl 1.0, which has already 
been reported here:
https://bugzilla.mcs.anl.gov/globus/show_bug.cgi?id=6841

No GT version supports openssl 1.0 - even the upcoming GT 5.0.0.  But, we plan 
to add support for it in the first quarter of 2010.  So, probably GT 5.0.1.

As for a workaround, can you install on older version of openssl and use that 
for the globus build?

-Stu

On Jan 6, 2010, at Jan 6, 5:05 PM, Craig E. Ward wrote:

> I'm attempting to build from source version 4.2.1 of the toolkit on a Fedora 
> Core 12 Linux system. It's failing during compilation:
> 
> /usr/lib64/ccache/gcc -DPACKAGE_NAME=\"\" -DPACKAGE_TARNAME=\"\" 
> -DPACKAGE_VERSION=\"\" -DPACKAGE_STRING=\"\" -DPACKAGE_BUGREPORT=\"\" 
> -DPACKAGE=\"globus_gsi_proxy_ssl\" -DVERSION=\"1.5\" -I. 
> -I/users/wardc/Build/gt4.2.1-all-source-installer/source-trees-thr/gsi/proxy/proxy_ssl/source/library
>  -I/usr/local/globus-4.2.1/include/gcc64dbgpthr -I. 
> -I/usr/local/globus-4.2.1/include 
> -I/usr/local/globus-4.2.1/include/gcc64dbgpthr -g -m64 -Wall -c 
> proxycertinfo.c  -fPIC -DPIC -o .libs/proxycertinfo.o
> In file included from proxycertinfo.h:36,
> from proxycertinfo.c:34:
> proxypolicy.h:105: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ 
> before ‘*’ token
> In file included from proxycertinfo.c:34:
> proxycertinfo.h:124: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ 
> before ‘*’ token
> proxycertinfo.c:52: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ 
> before ‘*’ token
> proxycertinfo.c: In function ‘PROXYCERTINFO_new’:
> proxycertinfo.c:87: warning: cast from pointer to integer of different size
> proxycertinfo.c: In function ‘PROXYCERTINFO_dup’:
> proxycertinfo.c:139: warning: passing argument 2 of ‘ASN1_dup’ from 
> incompatible pointer type
> /usr/include/openssl/asn1.h:955: note: expected ‘void * (*)(void **, const 
> unsigned char **, long int)’ but argument is of type ‘char * (*)()’
> proxycertinfo.c: In function ‘d2i_PROXYCERTINFO’:
> proxycertinfo.c:465: warning: passing argument 2 of ‘d2i_PROXYPOLICY’ from 
> incompatible pointer type
> proxypolicy.h:146: note: expected ‘unsigned char **’ but argument is of type 
> ‘const unsigned char **’
> proxycertinfo.c: In function ‘d2i_PROXYCERTINFO_OLD’:
> proxycertinfo.c:560: warning: passing argument 2 of ‘d2i_PROXYPOLICY’ from 
> incompatible pointer type
> proxypolicy.h:146: note: expected ‘unsigned char **’ but argument is of type 
> ‘const unsigned char **’
> make[2]: *** [proxycertinfo.lo] Error 1
> make[2]: Leaving directory 
> `/users/wardc/Build/gt4.2.1-all-source-installer/source-trees-thr/gsi/proxy/proxy_ssl/source/library'
> make[1]: *** [all-recursive] Error 1
> make[1]: Leaving directory 
> `/users/wardc/Build/gt4.2.1-all-source-installer/source-trees-thr/gsi/proxy/proxy_ssl/source'
> 
> ERROR: Build has failed
> make: *** [globus_gsi_proxy_ssl-thr] Error 2
> 
> Anyone see this error before? Any suggestions for a workaround?
> 
> Thanks,
> 
> Craig
> 
> -- 
> Craig E. Ward
> USC Information Sciences Institute
> 310-448-8271
> cw...@isi.edu



Re: [gt-user] where can i find the API and source code

2009-11-24 Thread Stuart Martin
Sorry about that.  We've been making some improvements to the  
globus.org website.  This now works for me.


-Stu

On Nov 24, 2009, at Nov 24, 8:32 AM, Peter Pan wrote:


Hello all,

where can i find the API for the current stable version 4.2.1? The  
link like: http://www.globus.org/api/javadoc-4.2.1/globus_wsrf_rft_service_java/ 
 is invalid. It seems like that all links for javadoc 4.2.1 are  
invalid. And the links for downloading the source code are also  
unavailable. I'll be very appreciate if somebody can send me the  
correct links or the downloaded source code etc...I need it very  
urgently.:(


Thanks in advance!

Peter

--
Dipl.-Inf. Xueming(Peter) Pan http://www.cgv.tugraz.at
Email: x@cgv.tugraz.at Tel: +43 (316) 873-5416
Institute of Computer Graphics and Knowledge Visualization
TU-Graz   Inffeldgasse 16c, 8010 Graz, Austria





Re: [gt-user] Globus Toolkit Series 5 (GT5) Plans

2009-11-11 Thread Stuart Martin

Hi Benjamin,

Ian explained this in his Oct 21 email: 
http://lists.globus.org/pipermail/gt-user/2009-October/008578.html

"... Note that GT5 will not include Java Core. Instead, we will  
continue to support GT4 Java Core, and will work with our users to  
migrate GT4 Java Core services to Crux when it becomes available."

As far as GRAM goes...
"... We will continue to support GRAM4 at least through December 2010  
(perhaps longer, depending upon demand and funding), but have begun to  
assist GRAM4 users in migrating to GRAM5. If you are an existing GRAM4  
user and would like to discuss migration issues to GRAM5, please  
contact us."
Hope this helps makes things clear.  Ask away if it doesn't, or if you  
have other specific questions/concerns.


-Stu

On Nov 11, 2009, at Nov 11, 9:50 AM, Benjamin Henne wrote:


Hi Stuart,

At 08.10.09 21:37, Stuart Martin wrote:
The components in GT5 will be GridFTP, GRAM5 (greatly enhanced  
version

of GRAM2), GSI-OpenSSH, MyProxy and RLS, along with the various
supporting libraries that these components use. The GT4 Web Services
components are not included in GT5, including WS GRAM (aka GRAM4),  
RFT,

Delegation, Java WS Core, C WS core components. (Of course, we will
continue to support and maintain these GT4 Web Services components.)


will you continue to support and maintain GT 4.0.x and or or GT  
4.2.x Web Services components?


Regards,
Benjamin





[gt-user] GRAM V5 *Beta* 1 now available

2009-11-03 Thread Stuart Martin

Hi All,

We are happy to make available a GRAM V5 *Beta* 1 version for testing.
http://dev.globus.org/wiki/GRAM/GRAM5#Beta_1

GRAM5 is now feature complete and ready for the upcoming GT 5.0.0  
release which will be coming out shortly.


New features in this Beta1 release:

* New Job Manager logging implementation
		Using the CEPDS logging format (http://www.cedps.net/index.php/LoggingBestPractices 
)

Can now distinguish log message for each jobs
Make it possible to rotate the log files safely
		Reduce the amount of information written, while improve ability to  
debug and identify problems


* Usage statistics implementation
Each job will send a usage packet to the Globus collector

* Improved compatibility with Condor-G when job managers are restarted

* Improved documentation for the GRAM5 command-line tools

* Improved installation for LRMs
		Added Make targets for LRM adapters. e.g. make gram5-pbs; or make  
gram5-condor;


We are in the process of rerunning the scalability tests that were run  
using Alpha1 (http://dev.globus.org/wiki/GRAM5_Scalability_Results).   
We'll produce a new report for GRAM5 Beta1.


We would encourage GRAM users to install and test this new Beta 1  
version.  Please send your feedback to gram-...@globus.org or to me  
directly.


Thanks for your support!

- GRAM development team


Re: [gt-user] Globus Toolkit Series 5 (GT5) Plans

2009-10-15 Thread Stuart Martin

Hi Kilian,

Thanks for the question.  The short answer is no.  The long answer is  
that the Globus effort for providing this functionality is evolving.   
The new effort is called "CRUX" (http://confluence.globus.org/display/whi/Crux+for+GT+Developers 
) that is focused on providing the same type of functionality as java  
WS core.  e.g. ability to easily create java web services.


This in the early stages of development.  Please send any use cases  
that you may have.  For further questions and details on CRUX, contact  
Ravi (cc'ed).


-Stu

On Oct 9, 2009, at Oct 9, 2:59 AM, Kilian CAVALOTTI wrote:


Hi Stu,

On Thu, Oct 8, 2009 at 9:37 PM, Stuart Martin   
wrote:

The GT4 Web Services components are
not included in GT5, including WS GRAM (aka GRAM4), RFT,  
Delegation, Java WS
Core, C WS core components.  (Of course, we will continue to  
support and

maintain these GT4 Web Services components.)


Sorry for the naive question, but does that mean that Web Services
will be discontinued at some point?

Thanks,
--
Kilian




Re: [gt-user] Globus Toolkit Series 5 (GT5) Plans

2009-10-15 Thread Stuart Martin
Perhaps in a future release.  It does not look like that will be ready  
in time for GT 5.0.0.  It could be added in the GT 5.0.x series though  
when it becomes available.


-Stu

On Oct 9, 2009, at Oct 9, 7:37 AM, Silvio Costa wrote:


Hi Stuart,

Recently I read about a new command "globus-url-sync" that will be  
used to synchronze files using GridFTP in a similar way to "rsync".

Do you know if this command is included in GridFTP component of GT5?

Thanks in advance,

Silvio Costa





Re: [gt-user] [gt-dev] Globus Toolkit Series 5 (GT5) Plans

2009-10-15 Thread Stuart Martin

Do you mean this?  http://sourceforge.net/projects/myca/
I don't think this will be added for GT5.  Perhaps it would be simple  
enough to add, but I think it makes more sense to keep gt5 to be just  
a small set of components for now.


We do want to add support for SGE in GRAM5.  But, I don't think we'll  
be able to add it in gt 5.0.0.  We should be able to be provide an  
update package for it in the near future and then also include it in a  
5.0.x point release.


-Stu

On Oct 8, 2009, at Oct 8, 7:35 PM, Gaurang Mehta wrote:


Please include myCA, and the jobmanager-adapters and seg for SGE also.

-Gaurang


On Oct 8, 2009, at 12:37 PM, Stuart Martin wrote:


Hi All,

We have been planning and preparing a Globus Toolkit version 5.0  
release.  More detailed information will be coming soon regarding  
Globus plans resulting from the feedback to Ian's previous email  
asking for help in shaping Globus' future (http://www.mail-archive.com/gt-user@lists.globus.org/msg00927.html 
).  But for now, I want to let you know about our plans and  
schedule for the GT5 release.


The components in GT5 will be GridFTP, GRAM5 (greatly enhanced  
version of GRAM2), GSI-OpenSSH, MyProxy and RLS, along with the  
various supporting libraries that these components use.  The GT4  
Web Services components are not included in GT5, including WS GRAM  
(aka GRAM4), RFT, Delegation, Java WS Core, C WS core components.   
(Of course, we will continue to support and maintain these GT4 Web  
Services components.)  This reduction in number of components and  
overall build dependencies should allow us to more easily  
coordinate and create this release than past GT releases.


We are hoping to release GT 5.0.0 in early November.  The current  
schedule is:

Oct 16  Code Freeze
Oct 19 - 30 Release Candidate creation, testing, doc
Nov 2   Announce GT 5.0.0

Questions and feedback welcome.

Regards,
Stu Martin


Gaurang Mehta
Research Programmer
Collaborative Computing Group
USC Information Sciences Institute
4676 Admiralty Way, Ste 1001,
Marina Del Rey, CA - 90292
Ph: 310-448-9101





[gt-user] Globus Toolkit Series 5 (GT5) Plans

2009-10-08 Thread Stuart Martin

Hi All,

We have been planning and preparing a Globus Toolkit version 5.0  
release.  More detailed information will be coming soon regarding  
Globus plans resulting from the feedback to Ian's previous email  
asking for help in shaping Globus' future (http://www.mail-archive.com/gt-user@lists.globus.org/msg00927.html 
).  But for now, I want to let you know about our plans and schedule  
for the GT5 release.


The components in GT5 will be GridFTP, GRAM5 (greatly enhanced version  
of GRAM2), GSI-OpenSSH, MyProxy and RLS, along with the various  
supporting libraries that these components use.  The GT4 Web Services  
components are not included in GT5, including WS GRAM (aka GRAM4),  
RFT, Delegation, Java WS Core, C WS core components.  (Of course, we  
will continue to support and maintain these GT4 Web Services  
components.)  This reduction in number of components and overall build  
dependencies should allow us to more easily coordinate and create this  
release than past GT releases.


We are hoping to release GT 5.0.0 in early November.  The current  
schedule is:

Oct 16  Code Freeze
Oct 19 - 30 Release Candidate creation, testing, doc
Nov 2   Announce GT 5.0.0

Questions and feedback welcome.

Regards,
Stu Martin


Re: [gt-user] Submiting java job

2009-10-08 Thread Stuart Martin
Try adding the java environment variable(s) in the job description.  A  
basic example is here:

http://www.globus.org/toolkit/docs/4.2/4.2.1/execution/gram4/user/#gram4-user-specifyingsimplejob-jdd

Something like:


JAVA_HOME
PUT PATH HERE


-Stu

On Oct 8, 2009, at Oct 8, 9:06 AM, Bruno Oliveira wrote:


Thanks for the answers.
I executed the which command and set the correct path of java.  Now  
i have my rsl file like this:



/usr/lib/jdk1.5.0_17/bin/java
-jar 
Projeccao.jar
${GLOBUS_USER_HOME}/stdout
${GLOBUS_USER_HOME}/stderr


gsiftp://debian9.estgf.ipp.pt:2811/home/vasco/Desktop/HelloWorld.jar 

file:///${GLOBUS_USER_HOME}/HelloWorld.jardestinationUrl>






file:///${GLOBUS_USER_HOME}/HelloWorld.jar





The job ran without problems, but  i don't have any stdout. In  
stderr file I have the error:


Unrecognized option: -jar
Could not create the Java virtual machine.

The machine have the environment variable JAVA_HOME correctly  
defined and i have this variable in the path.


thanks,
Best regards,
Bruno Oliveira



2009/10/1 Bruno Oliveira 
Thanks for your answer.
Now i got the message:
globusrun-ws: Job failed: Invalid executable path "java".

I have the JAVA_HOME environment variable correctly set, i can also  
run the jar file by typing this: java -jar HelloWorld.jar, but when  
i submit to globus i got the message:


globusrun-ws: Job failed: Invalid executable path "java".

My RSL now is:


java
-jar
/home/vasco/Desktop/HelloWorld.jar
${GLOBUS_USER_HOME}/stdout
 ${GLOBUS_USER_HOME}/stderr
  
 
 gsiftp://debian9.estgf.ipp.pt:2811/home/vasco/Desktop/HelloWorld.jar 

file:///${GLOBUS_USER_HOME}/output_javadestinationUrl>





file:///${GLOBUS_USER_HOME}/output_java




Thanks for the help,
Best regards,
Bruno Oliveira


2009/9/30 Bruno Oliveira 

Hi,
I want to submit a java job to Globus. I have a simple Hello World  
program, and i build the jar file.


I have the follow rsl:

/home/vasco/Desktop/HelloWorld.jar
   ${GLOBUS_USER_HOME}
${GLOBUS_USER_HOME}/stdout
 ${GLOBUS_USER_HOME}/stderr
  
 
 gsiftp://debian9.estgf.ipp.pt:2811/home/vasco/Desktop/HelloWorld.jar 

file:///${GLOBUS_USER_HOME}/output_javadestinationUrl>





file:///${GLOBUS_USER_HOME}/output_java




It is a simple test, but i got problems when i submit:

globusrun-ws -submit -S -f helloworld.rsl
Delegating user credentials...Done.
Submitting job...Done.
Job ID: uuid:ddbfef30-addc-11de-9c77-000874ac6b63
Termination time: 09/30/3009 16:18 GMT
Current job state: StageIn
Current job state: Failed
Destroying job...Done.
Cleaning up any delegated credentials...Done.
globusrun-ws: Job failed: The executable could not be started.  
starter: the job failed when the job manager attempted to run it



Apparently globus found the executable, but i dont know why this  
error happens.


Thanks for your help,
Best Regards,
Bruno Oliveira






[gt-user] GRAM V5 Alpha 3 now available

2009-09-17 Thread Stuart Martin

Hi All,

We are happy to make available a new GRAM V5 alpha 3 version for  
testing.

http://dev.globus.org/wiki/GRAM/GRAM5#Alpha_3

New features in this release:
- Support for clients to get the remote application's exit code
- Support for clients to get the version of the job manager

Alpha 2 was deployed at 2 TeraGrid sites: LSU's Queen Bee and NCSA's  
Abe.  Thanks to Lukasz Lacinski and Doru Marcusiu.  Also, Alpha 2 got  
some excellent testing and feedback from Jaime Frey and Igor Sfiligoi  
using condor-g.  Thanks Jaime and Igor.  Igor ran a number of tests  
submitting 5000 jobs at a time.  Here is a comment from Igor on the  
performance comparison to GRAM2:


"GRAM5 can easily keep a job turnaround of around 1Hz (50 jobs/min).  
Compare this to the 7.5 jobs/min of GT2.
Plus, it can start jobs at a 7Hz rate (450 jobs/min); GT2 best case  
scenario was 0.5Hz (33 jobs/min).
(note: the comparison is not really apple to apple due to missing file  
transfer)"


We would encourage GRAM users to install and test this new Alpha 3  
version.  Please send your feedback to gram-...@globus.org or to me  
directly.


Thanks for your support!

- GRAM development team


Re: [gt-user] catching the output of job

2009-07-22 Thread Stuart Martin

Hi JuanPablo,

With GRAM5 there are parameters to:
- set the working directory
	- stage files in, transfer files from a gridftp/GASS server to a  
local file path
	- stage files out, transfer files from a local file path to a gridftp/ 
GASS server


All parameters are here:

http://www.globus.org/api/c-globus-4.2.1/globus_gram_job_manager/html/globus_job_manager_rsl.html

The "sandboxes" that are described here:
https://grid.ct.infn.it/twiki/bin/view/GILDA/WmProxyUse
 seem to have a similar file staging functionality, but not the same  
as in GRAM.


Why do you ask?

Cheers,
-Stu

On Jul 22, 2009, at Jul 22, 2:01 PM, JuanPablo wrote:


hi,
I have a  question with the file catalog, is posible launch a globus
jobs with files that are located on a gridftp server?
or, send the output of globus to gridftp server and register in rls ?

in glite is posible with jdl arguments
InputSandbox = {
"gsiftp://aliserv6.ct.infn.it/dpm/ct.infn.it/home/gilda/generated/2006-05-26/filea5f5a851-8cec-4494-9afb-0c59e0625380 
",

"script_gridftp_input_data.sh" };

https://grid.ct.infn.it/twiki/bin/view/GILDA/WmProxyUse


exists a similar option for globus ?

Many thanks.
--
JuanPablo




[gt-user] GRAM5 Alpha 2 is now available

2009-07-13 Thread Stuart Martin

GRAM users:

We are happy to make available a second GRAM V5 alpha quality version  
for testing.


There are 2 main improvements to highlight in this release.
	1) Fixes to GRAM5 that would occasionally cause condor-g to get stuck  
when using the grid-monitor

- Jira Issues: http://jira.globus.org/browse/GRAM-20 and 
http://jira.globus.org/browse/GRAM-21
2) Auditing support for TeraGrid Science Gateway user attribute.
- More can be read about this here - 
http://dev.globus.org/wiki/GRAM2_Support_for_TeraGrid_Science_Gateways

The main GRAM5 web page is here:
http://dev.globus.org/wiki/GRAM/GRAM5

The Alpha 2 download and details are here:
http://dev.globus.org/wiki/GRAM/GRAM5#Alpha_2

Thanks to those that have tested the first Alpha.  We look forward to  
more GRAM5 testing and feedback.


- GRAM development team


Re: [gt-user] Running a job without any certificates

2009-07-09 Thread Stuart Martin
One complication is that GRAM relies on a delegated proxy to perform  
service-side file stage in and out for the user's job and also make it  
available for the user's application in order to interact with other  
grid services.  This delegation is required in the protocol between  
the gram client and gatekeeper.  As a first step, you could simplify  
and focus on non-staging jobs, then you can avoid needing a delegated  
proxy on the service-side.  You could modify the protocol between  
client and gatekeeper as needed.  Then as a second step, you could  
reintroduce delegation using a separate delegation service.  The gram  
client would be modified to create a delegated proxy and then the gram  
service would be modified to fetch the delegated proxy from this new  
delegation service.


-Stu

On Jul 9, 2009, at Jul 9, 9:13 AM, Nakkapt Boonsri wrote:


Regarding to running a job without proxy certificate,
I'm currently writing my bachelor thesis on authentication in Globus
Toolkit without proxy certicate, but with SAML assertion.
At this moment, an idea is to deactivate the proof of proxy  
certificate,

which depends on many components to the last step at GateKeeper.
Still many questions about possibility and its consequent.

-Nakkapat

Stuart Martin schrieb:

Right.  Submitting a job with either GRAM4 (WS GRAM) or GRAM5 (the
latest pre-ws based gram) requires security to be setup.

-Stu

On Jul 9, 2009, at Jul 9, 8:46 AM, Vanja Milosevski wrote:

Well the whole point of my question is because I don't want to get  
any

security overhead.

I did try the -nosec switch already but it still requires me to
initiate a proxy certificate before proceeding even though the
transmition goes through http and not through https. I assume some
sort of encryption is still taking place (maybe message-level?).

So ultimately, as I understand from the discussions above, it is not
possible (at least not straight forward) to submit a job without a
proxy certificate initialised?


-Vanja.






On Thu, Jul 9, 2009 at 7:53 AM, Luis wrote:

Hello!

You only need to start the globus container with the option -nosec

For example:

$GLOBUS_LOCATION/sbin/globus-start-container-detached -p 8443 - 
nosec


Regards

El mié, 08-07-2009 a las 19:47 +0100, Vanja Milosevski escribió:

Hello,

Is it possible to submit a job to globus without setting up any
security (i.e. without proxy certificates)?


Thanks,
-Vanja








--
Nakkapat Boonsri
-
Südstr. 152
74072 Heilbronn
Germany
-
email : nhtg...@gmail.com
tel.: +49(0)176/6110-4890
-






Re: [gt-user] Running a job without any certificates

2009-07-09 Thread Stuart Martin
Right.  Submitting a job with either GRAM4 (WS GRAM) or GRAM5 (the  
latest pre-ws based gram) requires security to be setup.


-Stu

On Jul 9, 2009, at Jul 9, 8:46 AM, Vanja Milosevski wrote:


Well the whole point of my question is because I don't want to get any
security overhead.

I did try the -nosec switch already but it still requires me to
initiate a proxy certificate before proceeding even though the
transmition goes through http and not through https. I assume some
sort of encryption is still taking place (maybe message-level?).

So ultimately, as I understand from the discussions above, it is not
possible (at least not straight forward) to submit a job without a
proxy certificate initialised?


-Vanja.






On Thu, Jul 9, 2009 at 7:53 AM, Luis wrote:

Hello!

You only need to start the globus container with the option -nosec

For example:

$GLOBUS_LOCATION/sbin/globus-start-container-detached -p 8443 -nosec

Regards

El mié, 08-07-2009 a las 19:47 +0100, Vanja Milosevski escribió:

Hello,

Is it possible to submit a job to globus without setting up any
security (i.e. without proxy certificates)?


Thanks,
-Vanja







[gt-user] Fwd: GRAM V5 Alpha (scalability) Release

2009-06-04 Thread Stuart Martin
Forwarding here in case there are some GRAM users have not seen this  
yet.


Cheers,
Stu

Begin forwarded message:


From: Stuart Martin 
Date: May 12, 2009 9:08:07 AM CDT
To: GRAM developer 
Cc: Stuart Martin 
Subject: GRAM V5 Alpha (scalability) Release

GRAM2 (aka Pre-WS GRAM) users:

We are happy to make available a new GRAM V5 alpha quality version  
for testing. http://dev.globus.org/wiki/GRAM/Scalability_Alpha_20090504


GRAM5 is built from the GT4 GRAM2 code base.  GRAM5 removes some  
features and alters some behaviors, though it remains protocol- 
compatible with existing GRAM2 deployments. Specifically, file  
streaming has been replaced by end-of-job file staging  
(transparently to the user), and MPICH-G2 multijob coordination is  
removed from the service.  Preliminary compatibility testing has  
been successful with existing GRAM2 clients to the new GRAM5  
service: globusrun, COG-jglobus, and Condor-G clients submitting and  
monitoring jobs. For Condor-G and GRAM5, we recommend not using the  
grid monitor.


This GRAM5 alpha release improves the scalability of the GRAM system  
by reducing the CPU and memory use.  In addition to scalability- 
related changes, this alpha includes fixes for bugs in GSSAPI, GASS  
Cache, and GRAM that impact the performance and reliability of GRAM.  
This is the first public alpha release of GRAM5. It is not  
recommended for production environments, though we do request  
feedback and bug reports.


Significant Design Changes
-
There are 2 significant modifications that account for the  
scalability improvements in GRAM5:
	1) All job management and processing is done with a single Job  
Manager process per user (instead of one per job).  This coupled  
with throttling the amount of work each user's job manager will do,  
makes it so the system load average is independent from the number  
of jobs in the system.  This is done without sacrificing performance.
	2) Monitoring of the LRM jobs is done using the scheduler event  
generator (SEG).  The SEG has been used for a number of years in WS  
GRAM for scalability reasons.  The SEG is more efficient than  
individual job querying using the LRM's CLI.  And it is also more  
efficient than using condor-g's grid-monitor approach.


A complete list of all changes is here: http://tinyurl.com/pyhfy5

Performance and Scalability testing
-
In addition to functional and compatibility testing, we ran series  
of performance and scalability tests with GRAM5 with our own java  
throughput testing client.  And last, we ran a test using condor-g  
using the same job load/scenario to both GRAM2 and GRAM5.  All test  
results are here: http://dev.globus.org/wiki/GRAM5_Scalability_Results


Observations from the 5-client test, http://tinyurl.com/qklpco:
	In the 1 hour test period, 4944 job were submitted, with more than  
4500 pending in the PBS queue at once. This demonstrates that the  
number of jobs being monitored by GRAM5 does not adversely affect  
its performance. This also shows that a naive client implementation  
with 5 x 50 concurrent threads does not cause the high load average  
on the cluster head node often seen with GRAM2.  With GRAM2, these  
levels of scalability were only achievable when using Condor-G grid- 
monitor, but GRAM5 is able to do so by itself and with less resource  
use.


Observations from the 2000 job condor-g tests:
	Results for GRAM2: http://dev.globus.org/wiki/GRAM5_Scalability_Results#Test_6 
:_gram2-condor-g
	Results for GRAM5: http://dev.globus.org/wiki/GRAM5_Scalability_Results#Test_7 
:_gram5-condor-g


	The service host's cpu load average was significantly less for  
GRAM5 2.3 (peak 3.8) than GRAM2 29.2 (peak 35.8).  The service host  
memory profile was similar and reasonable for both services.  Both  
services processed the 2000 jobs successfully and in roughly the  
same duration (GRAM5's was less).


We are encouraged by these results and we hope you are too.  We  
encourage community testing and feedback (positive or negative) on  
this GRAM5 Alpha.  For example, we'd like to know how GRAM5 behaves  
for your use cases/scenarios.  E.g. where there any errors?  How did  
it perform?  What client did you use?  Describe your scenarios like  
we did on our scalability results page. The more detail the better.   
If you are a GRAM4 user, we encourage you to try GRAM5.  If there is  
a particular feature you depend upon in GRAM4 that is not available  
in GRAM5, please let us know.


Please send your feedback to gram-...@globus.org.

Download and install instructions for the GRAM5 Alpha is here: 
http://dev.globus.org/wiki/GRAM/Scalability_Alpha_20090504

- GRAM development team




Re: [gt-user] [gram-user] Getting PBS details with GRAM

2009-05-29 Thread Stuart Martin


On May 28, 2009, at May 28, 6:02 PM, Cihan Altinay wrote:


Hi Stuart,

Stuart Martin wrote:
TeraGrid is using OGSA-DAI to create a virtual DB on top of gram  
auditing DB and TeraGrid's central DB (TGCDB).

Here is a paper on that:
   
http://www.globus.org/alliance/publications/papers/TG_GRAM_Auditing_and_LEAD_Gateway.pdf


Cool, thanks.

Alternatively, is it possible to get the PBS job ID to use PBS  
tools for

this task?

The "localJobId" was added as a resource property in GT 4.2.
   http://www.globus.org/toolkit/docs/4.2/4.2.0/execution/gram4/developer/gram4-wsdl.html#gram4-rps-mejpt 
 But this was not available in ws gram in GT 4.0.


I see. Unfortunately we are still using 4.0. A related question  
which might solve my problem is: How to use the email on job  
execution/completion feature of PBS with Gram? I noticed that the  
JobDescription perl file contains the relevant checks to add -m and - 
M options to the PBS script but how can I actually set the address  
and options from my RSL or Java API?


That would be an extension to the job description, here is the doc on  
that:


http://www-unix.globus.org/toolkit/docs/4.0/execution/wsgram/user-index.html#s-wsgram-user-specifyingextensions

I think something like this would work:


bob...@altinay.de
yes




Cheers,
Cihan




Re: [gt-user] [gram-user] Getting PBS details with GRAM

2009-05-26 Thread Stuart Martin

On May 25, 2009, at May 25, 10:06 AM, Tom Scavo wrote:

On Sun, May 24, 2009 at 11:34 PM, Cihan Altinay   
wrote:


we are setting up a web portal which allows users to submit jobs and
retrieve results using WS-GRAM. I was wondering if there is a way to
obtain PBS statistics such as CPU time and memory usage of a  
submitted

job through GRAM?


I don't know, so I've cc'd the gram-user mailing list.


No.  GRAM does not gather this information from the local resource  
managers (e.g. pbs) and make it available directly via GRAM.  But,  
TeraGrid is using OGSA-DAI to create a virtual DB on top of gram  
auditing DB and TeraGrid's central DB (TGCDB).


Here is a paper on that:

http://www.globus.org/alliance/publications/papers/TG_GRAM_Auditing_and_LEAD_Gateway.pdf



Alternatively, is it possible to get the PBS job ID to use PBS  
tools for

this task?


The "localJobId" was added as a resource property in GT 4.2.

http://www.globus.org/toolkit/docs/4.2/4.2.0/execution/gram4/developer/gram4-wsdl.html#gram4-rps-mejpt

But this was not available in ws gram in GT 4.0.





Yes.  Assuming GRAM audit is enabled at the resource, you can query
the GRAM audit database to obtain the local_job_id (which is what the
PBS job ID is called in the gram_audit_table):  For example, using the
GRAM Audit Tools, here's how you obtain the local_job_id given a
particular job_grid_id (GJID):

$ $GRIDSHIB_HOME/bin/get-audit-record --debug \
   --GJID $GJID | grep local_job_id
http://globus.org/names/attribute/gram/audit_v1/local_job_id 12345

The output is a name-value pair for the "attribute" local_job_id.

The GRAM Audit Tools include a Java API to build your own if the
provided tools (such get-audit-record) don't do what you want.

Hope this helps,

Tom




[gt-user] Fwd: Help shape Globus' future: A call for use cases

2009-04-06 Thread Stuart Martin



Begin forwarded message:


From: Ian Foster 
Date: March 31, 2009 1:05:35 AM CDT
To: annou...@globus.org
Cc: Ann Chervenak , Lisa Childers  
, Ravi Madduri , Stuart  
Martin , Raj Kettimuthu ,  
Lee Liming , cdigs-t...@mcs.anl.gov

Subject: Help shape Globus' future: A call for use cases

Summary

Over the past few months, within the context of the NSF Community  
Driven Improvement of Globus Software (CDIGS: www.cdigs.org)  
project, we have begun re-evaluating how we should invest toward  
Globus' future.  We believe we are starting to understand  
alternatives and trade-offs.  Of course, our understanding of the  
needs of the Globus community is imperfect, and we do not want to  
make decisions without broader consultation. Thus, we would now like  
to engage with the Globus community to help us work through this  
evaluation.


A key concern at this point in the process is that we may not have a  
complete enough set of uses cases that represent what people are  
doing with Globus, or hope to do with Globus.  Thus, we ask that you  
help us by providing a description of how you use (or hope to use)  
Globus, along with contact information so that we can follow up with  
you to clarify details and/or participate in subsequent stages of  
this effort. We provide below more details on what we are looking for.


Now the longer version...


Where are we?

We have had tremendous successes with Globus over the past 13 years,  
along with some trying times.  A challenging issue over the years  
has been adapting to the changing technology landscape.  Globus  
Toolkit version 4 (GT4) was introduced approximately 5 years ago,  
and much has changed since then.  So what are the circumstances that  
factor into our decisions on how to evolve?


Users: Many healthy, active, growing communities are relying on  
Globus for their day-to-day work, including TeraGrid, OSG, ESG,  
LIGO, caBIG, and many others.  (Based on our anonymized usage  
reporting, it is clear there are also many more Globus users than  
those we know about.)  These communities challenge us with the  
constant, healthy tension between demands for stability and new  
functionality.  There are also new, large communities that are  
adopting Globus, such as BIRN and other biomedical communities,  
which will continue to press these sometimes conflicting demands.


Funding: We are fortunate to have significant NSF CDIGS funding for  
a couple more years, which allows us to focus on support and  
development of Globus to meet user needs.  We also have a rich mix  
of grants for a wide range of activities, ranging from computer  
science research, to infrastructure deployment and support, to  
application integration, to complete application development.   
However, these efforts are of course finite in duration, and we must  
be constantly looking to develop new programs to carry Globus forward.


Technology trends: The technology landscape in which we operate  
continues to evolve at a dizzying pace, with technologies and  
approaches such as Cloud Computing, Software-as-a-Service (SaaS),  
OpenID, OpenCL, multi-core, etc.  In the meantime, technology trends  
that we have embraced in Globus, such as SOA and Web services,  
continue to mature, for example with the widespread adoption of  
lighter-weight REST approaches.


Aging foundations: Globus relies heavily on open source components  
from other communities.  Some component choices, such as OpenSSL,  
have stood the test of time.  Others have not: in particular, on the  
Java web services front (e.g. GT4 Java Core), we find ourselves  
highly dependent on libraries such as Apache Axis v1.x and PureTLS,  
which most of the rest of the world has moved away from.  Thus, not  
only do we have to support these technologies entirely on our own,  
but it becomes increasingly difficult to support our users and our  
own newer activities that (rightly) want to exploit new open source  
software libraries and approaches.  These aging GT4 Java Core  
foundations are also inhibiting our ability to effectively address a  
variety of issues (e.g., scalability, REST support) in other Globus  
components, such as GRAM4 (aka WS-GRAM), RFT, and MDS4.


Community: The Globus Alliance (dev.globus.org) community of  
developers has made many valuable contributions that have seen wide  
adoption, such as MyProxy, GridShib, Gaards, and GridWay.  However,  
the dev.globus.org community has not helped significantly with the  
sustainability of existing components by bringing new developers  
into to the community who add features or maintain those components.



Where do we go from here?

Thanks in particular to our NSF CDIGS funding, we have some time and  
resources to address these issues, and we will not abruptly leave  
our users out in the cold now or in coming years.  One CDIGS mandate  
is to continue supporting existing users, which we will clearly do.   
So regardless of what decisions ar

Re: [gt-user] A Design Question about Portal Development

2009-03-23 Thread Stuart Martin

On Mar 23, 2009, at Mar 23, 9:24 AM, tracy_luofengji wrote:


Dear All,
Hello. Here I want to ask a question which is nothing to do with the  
detail of grid technology, but a design pattern related to grid  
portal development.


I have established a computational grid infrastracture in my lab  
using GT and relavant technology, and now I want to develop a portal  
prototype. I plan to use a machine to serve as the portal server,  
and end users access the grid resources through the client tool,  
which use the services provided by the portal server. Currently I do  
not plan to develop the Web-based Portal due to the time  
limitation(I am more familiar with desktop GUI developing).


Obviously there are some issues must be addressed in the portal  
server, such as certificates, job submission, access the web mds...  
Well, here is my question:


Should I encapsulate those functions as several web servers which  
can be deployed in the GT container (such as  
SingleJobSubmissionService, ArrayJobSubmissionService,  
ComputeResourcesListService...) and let the client tool access those  
web services directly, or should I write a daemon on the portal  
server, which listens the requests of the clients and then access  
the grid infrastructure, and finally send the response to the  
client, just as the classical C-S model.


Typically, people will create an application specific web service  
instead of another higher-level "generic" web service.  For example,  
creating a web service that can remotely execute blast, see the open  
life science gateway - http://lsgw.uc.teragrid.org:8080/gridsphere/gridsphere


There is some relevant documentation here on TeraGrid Gateways that I  
think will be helpful to you.

http://www.teragrid.org/gateways/

I'll point out a couple links from that page, one about building a  
gateway in a day using simpleGrid:

http://www.teragrid.org/gateways/developers/
http://www.teragrid.org/gateways/developers/inaday.php

There are some TG specific things in there, but in the end many use  
GRAM and GridFTP underneath, so this info/code  should be useful to  
you too.




So, I want to disscuss it with you. Any suggestion will be much  
appraciated.


Thanks!
Tracy



网易邮箱,中国第一大电子邮件服务商




Re: [gt-user] problem in transferring file

2009-02-27 Thread Stuart Martin
It looks to me like you may have the source and dest mixed up.  For  
stage out, the source would typically be the file:/// url which would  
get replaced by ws gram with the service-side gridftp server host and  
port.  Then the gsiftp url would be to the gridftp server running on  
the client-side.



   
 gsiftp://gridftp.frost.ncar.teragrid.org/ptmp/xyz/dummy01.dat 

 file:///Users/xyz/Desktop/dummy01.datdestinationUrl>

   


-Stu

On Feb 27, 2009, at Feb 27, 10:58 AM, Ufuk Utku Turuncoglu wrote:


Hi,

I try to submit a globus job with file transfer but i got following  
error. The local file appears as null in the log.


Submission ID: uuid:af0a91f0-0494-11de-bdbe-fda4d1871b6e
delegation level: gsilimited
delegation level: gsifull
WAITING FOR JOB TO FINISH:
== State Notification ==
State : Failed
Holding: false

Exit Code: 0
Failed >Failed<
Fault:
fault type: org.globus.exec.generated.StagingFaultType:
attribute: fileStageOut
description:
Staging error for RSL element fileStageOut, from gsiftp://gridftp.frost.ncar.teragrid.org:2811/ptmp/xyz/dummy01.dat 
 to null.

destination: null
faultReason:
faultString:
gt2ErrorCode: 0
originator: Address: 
https://fr0103ge.ncar.teragrid.org:8443/wsrf/services/ManagedJobFactoryService
Reference property[0]:
http://www.globus.org/namespaces/2004/10/gram/job 
">cd2d9100-0494-11de-97de-d94967efe41a


source: gsiftp://gridftp.frost.ncar.teragrid.org:2811/ptmp/xyz/dummy01.dat
stackTrace:
org.globus.exec.generated.StagingFaultType: Staging error for RSL  
element fileStageOut, from gsiftp://gridftp.frost.ncar.teragrid.org:2811/ptmp/xyz/dummy01.dat 
 to null.

Timestamp: Thu Feb 26 23:06:41 MST 2009
Originator: Address: 
https://fr0103ge.ncar.teragrid.org:8443/wsrf/services/ManagedJobFactoryService
Reference property[0]:
http://www.globus.org/namespaces/2004/10/gram/job 
">cd2d9100-0494-11de-97de-d94967efe41a


I also check delegation level and it seems as full. I am using  
gt4.0.8 java libraries. The jobs work correctly without data  
transfer part.


Any suggestion will be helpful,

--ufuk

RSL Script ---



/bin/sh
/ptmp/xyz
-c
/bin/uname -a
/ptmp/xyz/1235714747266.stdout
/ptmp/xyz/1235714747266.stderr
1
single

   
 gsiftp://gridftp.frost.ncar.teragrid.org/ptmp/xyz/dummy01.dat 

 file:///Users/xyz/Desktop/dummy01.datdestinationUrl>

   







Re: [gt-user] building grid with globus

2009-02-26 Thread Stuart Martin


On Feb 25, 2009, at Feb 25, 11:50 AM, Sherif Nagy wrote:


Hello,

I am a little bit new into grid computing and globus, I have been
reading @ http://globus.org/toolkit/docs/4.2/4.2.1/admin/install/ and
@ http://www.globusconsortium.org/tutorial/ for few days now. but i am
a little bit confused , as my setup now having a job batch and
scheduling system that is already working " Torque/PBS with Maui " one
machine running pbs_server " which will be running globus as far as I
understand " and 4 machines running the pbs mini server pbs_mom which
connects to the PBS and everything works perfect, so if i understand
correctly I'll install globus toolkit into the PBS server with
--enable-wmfs-pbs flag


Yes.  But it is "--enable-wsgram-pbs"


and no need to install anything on the compute
nodes "the other 4 machines" ?


Right.


not even globus-ftp gridftp services
for file management ?


Well, GRAM assumes there is a shared file system for the compute  
cluster (head node and worker nodes).  Typically, there is a gridftp  
server running on the same host as the gram service and used for file  
transfers from/to remote clients to/from the head node.  But you can  
configure the gridftp server to run on a separate host and configure  
ws gram to use it.  If there are any file system path differences  
between these 2 hosts, then add those filesystem mapping file.


http://www.globus.org/toolkit/docs/4.0/admin/docbook/ch11.html#s-wsgram-Interface_Config_Frag-filesysmap


the globus will just submit the job to the pbs
and the pbs will handle everything else ?


With a shared file system, then PBS will schedule and execute the  
jobs.  Then gram will perform and remote file transfers before and  
after the job, e.g. file stage in and out (and cleanup).



also if i have another
cluster i'll have to install globus on the master node and link both
globus setup together right ?


You would do the same installation on the 2nd cluster.  And now you  
can submit gram jobs to each.


Then there are client workload tools that will facilitate the  
execution to multiple gram installations.


Some populate tools for this are: Condor-G, GridWay, Swift, GEMLCA, AHE

http://dev.globus.org/wiki/Incubator/Swift
http://www.gridway.org
http://www.cs.wisc.edu/condor/condorg/
http://wiki.realitygrid.org/wiki/AHE_GENIUS
http://dev.globus.org/wiki/Incubator/GEMLCA


 and what about the clients do they need
to deploy globus also ?  please advice and if there is any extra links
will help me understand the big picture will be appreciated.


The GRAM documentation is here:
http://www-unix.globus.org/toolkit/docs/4.0/execution/wsgram/



Thank you
regards,
Sherif




Re: [gt-user] A question about applications on grid

2009-02-26 Thread Stuart Martin
multi-cluster MPI support exists for the grid.  mpich-g2 provides this  
for pre-ws gram (gram2) installations.  MPIg is the more recent  
version and provides this functionality for both pre-ws gram and ws  
gram (gram4).


Information on MPICH-G2 is here:
http://www3.niu.edu/mpi/

Information on MPIg is here:
ftp://ftp.cs.niu.edu/pub/karonis/MPIg/index.html

Work has been done in these implementations (but maybe only in MPIg,  
not sure) to optimize the inter process message communication given  
the network topology.  e.g. use GEthernet when available between 2  
processes.  Process broadcast messages in an efficient manner.


Nick Karonis (cc'ed) could help you with any additional questions.

-Stu

On Feb 25, 2009, at Feb 25, 11:35 PM, Le Trung Kien wrote:


Hi,
I have a question about the ability of deploying parallel  
application on a large grid system and

optimizing the use of available resources for sure.
We saw that a "classic" parallel application which worked well on  
clustering systems, which have
their best communicating condition between cluster nodes (with speed  
of Fabric or GEthernet).
And now, when the grid computing is widely developed and deployed,  
the old clusters have been
joined in grids. For simple, we consider a cluster of computer as a  
grid node; so, in general, I wonder if
one such classic parallel application could be executed on two or  
more grid nodes concurrently
and give us the best performance in time consuming, assuming grid  
nodes communicate with

others through DSL line (about 1 or 2 Mb/s of speed).

Thank you, any ideas please.
Best Regards.


--
Le Trung Kien.




Re: [gt-user] GT4 dependencies between subjobs of a multijob

2009-02-18 Thread Stuart Martin

Hi Sebastian,

Multi jobs in both GRAM versions, GRAM2 and GRAM4, were created to  
support multi-site MPI jobs (mpich-g2 and MPIg).  There is no support  
for dependencies between jobs.  Other higher-level services have  
support for this, like condor-g DAGs.  I think Swift and GridWay  
support job dependencies too.  These tools can then use GRAM  
underneath for remote job submission to the LRM.


With the above tools, the 2nd job is not submitted until the first is  
complete.  But, some LRMs support dependencies, like SGE.  If both  
jobs will be run on the same compute resource, then giving them to the  
LRM at the beginning may be more efficient.  GRAM does not expose this  
functionality at present.  Extending GRAM to support job dependencies  
is another possibility here.  I don't see us being able to implement  
this ourselves right now, but we could work with you in case you want  
to contribute that functionality.


-Stu

On Feb 18, 2009, at Feb 18, 7:05 AM, Sebastian Breuers wrote:


Dear list members,

I wondered, if it is possible to write a job description that fits  
the following case:


I have two jobs 'Job1' and 'Job2'. These jobs depend sequentially on  
each other, i.e. 'Job2' should only be executed when 'Job1' has  
(successfully) finished.


I've found the multijob tag for the xml description but I could not  
figure out, if there is a possibility to specify that one of the  
subjobs has to wait for the other job to finish.


I've searched the mailing list archive but could not get the clue or  
a hint.
So the question is, if it is at least possible. And if yes, how can  
I accomplish this behavior.


Thanks in advance.

Greetings

Sebastian Breuers

Dipl.-Biol. Sebastian Breuers   Tel: +49-221-478-7021
eMail: s.breu...@uni-koeln.de   Fax: +49-221-478-5568
web:   http://www.uni-koeln.de/rrzk/autoren/breuers

Universität zu Köln  University of Cologne
Zentrum für Angewandte InformatikCenter for Applied Computer  
Science

Robert-Koch-Str. 10  Robert-Koch-Str. 10
D-50931 Köln D-50931 Cologne
 Federal Rep. of Germany





  1   2   >