From: Joseph Bester <[EMAIL PROTECTED]>
Subject: Re: [gt-user] GRAM2 problem(globusrun fails)
Date: Thu, 24 Jul 2008 08:14:58 -0400
> On Jul 23, 2008, at 11:48 PM, Tatsuhiko Inoue wrote:
> > Hello
> >
> > A problem occurred in GRAM2 of GT4.2.0.
> > globusrun command fails with the following message on Ubuntu 8.04.
>
> Did these problems occur with GT4.2.0 from binary installer (which did
> you use) or from a source build? x86 or x86_64 or ia64?
>
Them occur with GT4.2.0 from source build and x86.
I attach gatekeeper log and jobmanager log.
(ubuntu-gatekeeper.log and gram_job_mgr_24836.log)
I modified to source code for seeing SSL error, and I see the following error.
0:error:1408F06B:SSL routines:SSL3_GET_RECORD:bad decompression:s3_pkt.c:438:
> > $ globusrun -r example.org "&(executable=/bin/hostname)"
> > globus_gram_client_callback_allow successful
> > GRAM Job submission failed because data transfer to the server
> > failed (error code 10)
>
> This is sometimes the symptom of the gatekeeper crash problem fixed by
> the globus_gatekeeper-4.0 advisory on
> http://www.globus.org/toolkit/advisories.html
>
I use that advisory, but problems are not solved.
> Otherwise, see if there's something in the gatekeeper log.
>
> > Also on MacOS X 10.4, globusrun command fails.
> > Then globusrun print the following message.
> >
> > $ globusrun -r example.org "&(executable=/bin/hostname)"
> > globus_gram_client_callback_allow successful
> > GRAM Job submission failed because the connection to the server
> > failed (check host and port) (error code 12)
>
> Could be a ssl issue or a tcp/ip issue. Again check the gatekeeper
> log. Is this a PPC or Intel Mac?
> Again, from a binary or source installer?
>
I think ssl issue. I attach gatekeeper log.(mac-gatekeeper.log)
This is a PCC Mac and I use source installer.
TIME: Fri Jul 25 11:22:14 2008
PID: 24681 -- Notice: 6: /usr/local/gt4/sbin/globus-gatekeeper pid=24681
starting at Fri Jul 25 11:22:14 2008
TIME: Fri Jul 25 11:22:14 2008
PID: 24682 -- Notice: 6: /usr/local/gt4/sbin/globus-gatekeeper pid=24682
starting at Fri Jul 25 11:22:14 2008
TIME: Fri Jul 25 11:22:14 2008
PID: 24682 -- Notice: 6: GRAM contact:
akiba119.apgrid.org:51989:/C=JP/O=AIST/OU=GRID/CN=Tatsuhiko Inoue
TIME: Fri Jul 25 11:22:45 2008
PID: 24835 -- Notice: 6: Got connection 192.50.74.119 at Fri Jul 25 11:22:45
2008
TIME: Fri Jul 25 11:22:45 2008
PID: 24835 -- Notice: 5: Authenticated globus user:
/C=JP/O=AIST/OU=GRID/CN=Tatsuhiko Inoue
TIME: Fri Jul 25 11:22:45 2008
PID: 24835 -- Notice: 0: GRID_SECURITY_HTTP_BODY_FD=7
TIME: Fri Jul 25 11:22:45 2008
PID: 24835 -- Notice: 5: Requested service: jobmanager
TIME: Fri Jul 25 11:22:45 2008
PID: 24835 -- Notice: 5: Authorized as local user: tatuhiko
TIME: Fri Jul 25 11:22:45 2008
PID: 24835 -- Notice: 5: Authorized as local uid: 500
TIME: Fri Jul 25 11:22:45 2008
PID: 24835 -- Notice: 5: and local gid: 500
TIME: Fri Jul 25 11:22:45 2008
PID: 24835 -- Notice: 0: executing /usr/local/gt4/libexec/globus-job-manager
TIME: Fri Jul 25 11:22:45 2008
PID: 24835 -- Notice: 0: GRID_SECURITY_CONTEXT_FD=10
TIME: Fri Jul 25 11:22:45 2008
PID: 24835 -- Notice: 0: Child 24836 started
7/25 11:22:45 JM: TARGET_GLOBUS_LOCATION = /usr/local/gt4
7/25 11:22:45 JM: Security context imported
7/25 11:22:45 JM: Adding new callback contact
(url=https://akiba119.apgrid.org:59394/, mask=1048575)
7/25 11:22:45 JM: Added successfully
7/25 11:22:45 Pre-parsed RSL string: &("executable" = "/bin/hostname" )
7/25 11:22:45
<<<<<Job Request RSL
&("executable" = "/bin/hostname" )
>>>>>Job Request RSL
7/25 11:22:45
<<<<<Job Request RSL (canonical)
&("executable" = "/bin/hostname" )
>>>>>Job Request RSL (canonical)
7/25 11:22:45 Job Manager State Machine (entering):
GLOBUS_GRAM_JOB_MANAGER_STATE_MAKE_SCRATCHDIR
7/25 11:22:45
<<<<<Job RSL
&("environment" = ("HOME" "/home/tatuhiko" ) ("LOGNAME" "tatuhiko" )
)("executable" = "/bin/hostname" )
>>>>>Job RSL
7/25 11:22:45
<<<<<Job RSL (post-eval)
&("environment" = ("HOME" "/home/tatuhiko" ) ("LOGNAME" "tatuhiko" )
)("executable" = "/bin/hostname" )
>>>>>Job RSL (post-eval)
Adding default RSL of proxy_timeout = 60
Adding default RSL of dry_run = no
Adding default RSL of gram_my_job = collective
Adding default RSL of job_type = multiple
Adding default RSL of count = 1
Adding default RSL of stderr = /dev/null
Adding default RSL of stdout = /dev/null
Adding default RSL of stdin = /dev/null
Adding default RSL of directory = $(HOME)
7/25 11:22:45
<<<<<Job RSL (post-validation)
&("directory" = $("HOME") )("stdin" = "/dev/null" )("stdout" = "/dev/null"
)("stderr" = "/dev/null" )("count" = "1" )("job_type" = "multiple"
)("gram_my_job" = "collective" )("dry_run" = "no" )("proxy_timeout" = "60"
)("environment" = ("HOME" "/home/tatuhiko" ) ("LOGNAME" "tatuhiko" )
)("executable" = "/bin/hostname" )
>>>>>Job RSL (post-validation)
7/25 11:22:45
<<<<<Job RSL (post-validation-eval)
&("directory" = "/home/tatuhiko" )("stdin" = "/dev/null" )("stdout" =
"/dev/null" )("stderr" = "/dev/null" )("count" = "1" )("job_type" = "multiple"
)("gram_my_job" = "collective" )("dry_run" = "no" )("proxy_timeout" = "60"
)("environment" = ("HOME" "/home/tatuhiko" ) ("LOGNAME" "tatuhiko" )
)("executable" = "/bin/hostname" )
>>>>>Job RSL (post-validation-eval)
7/25 11:22:45 JMI: Getting RSL output value
7/25 11:22:45 JMI: Processing output positions
7/25 11:22:45 JMI: Getting RSL output value
7/25 11:22:45 JMI: Processing output positions
7/25 11:22:45 Job Manager State Machine (entering):
GLOBUS_GRAM_JOB_MANAGER_STATE_REMOTE_IO_FILE_CREATE
7/25 11:22:45 JM: Opening output destinations
7/25 11:22:45 JM: stdout goes to
/home/tatuhiko/.globus/job/akiba119.apgrid.org/24836.1216952565/stdout
7/25 11:22:45 JM: stderr goes to
/home/tatuhiko/.globus/job/akiba119.apgrid.org/24836.1216952565/stderr
7/25 11:22:45 ignoring stdout and stderr
7/25 11:22:45 no opens in progress, registering state machine callback
7/25 11:22:45 JM: Finished opening output destinations
7/25 11:22:45 Job Manager State Machine (entering):
GLOBUS_GRAM_JOB_MANAGER_STATE_OPEN_OUTPUT
7/25 11:22:45 JM: GSSAPI type is GSI.. relocating proxy
7/25 11:22:45 JMI: testing job manager scripts for type fork exist and
permissions are ok.
7/25 11:22:45 JMI: completed script validation: job manager type is fork.
7/25 11:22:45 JMI: in globus_gram_job_manager_script_proxy_relocate()
7/25 11:22:45 JMI: cmd = proxy_relocate
Fri Jul 25 11:22:45 2008 JM_SCRIPT: New Perl JobManager created.
Fri Jul 25 11:22:45 2008 JM_SCRIPT: Using jm supplied job dir:
/home/tatuhiko/.globus/job/akiba119.apgrid.org/24836.1216952565
Fri Jul 25 11:22:45 2008 JM_SCRIPT: proxy_relocate(enter)
7/25 11:22:45 JMI: while return_buf = GRAM_SCRIPT_X509_USER_PROXY =
/home/tatuhiko/.globus/job/akiba119.apgrid.org/24836.1216952565/x509_up
7/25 11:22:45 Job Manager State Machine (entering):
GLOBUS_GRAM_JOB_MANAGER_STATE_PROXY_RELOCATE
7/25 11:22:45 JM: Relocated Proxy to
/home/tatuhiko/.globus/job/akiba119.apgrid.org/24836.1216952565/x509_up
7/25 11:22:45 JM: before sending to client: rc=0 (Success)
7/25 11:22:45 Job Manager State Machine (exiting):
GLOBUS_GRAM_JOB_MANAGER_STATE_TWO_PHASE
7/25 11:22:45 Job Manager State Machine (entering):
GLOBUS_GRAM_JOB_MANAGER_STATE_TWO_PHASE
7/25 11:22:45 Job Manager State Machine (entering):
GLOBUS_GRAM_JOB_MANAGER_STATE_TWO_PHASE_COMMITTED
7/25 11:22:45 Job Manager State Machine (entering):
GLOBUS_GRAM_JOB_MANAGER_STATE_STAGE_IN
7/25 11:22:45 JMI: testing job manager scripts for type fork exist and
permissions are ok.
7/25 11:22:45 JMI: completed script validation: job manager type is fork.
7/25 11:22:45 JMI: in globus_gram_job_manager_submit()
7/25 11:22:45 JMI: local stdout filename = /dev/null.
7/25 11:22:45 JMI: local stderr filename = /dev/null.
7/25 11:22:45 JMI: cmd = submit
7/25 11:22:45 JMI: returning with success
Fri Jul 25 11:22:46 2008 JM_SCRIPT: New Perl JobManager created.
Fri Jul 25 11:22:46 2008 JM_SCRIPT: Using jm supplied job dir:
/home/tatuhiko/.globus/job/akiba119.apgrid.org/24836.1216952565
7/25 11:22:46 JMI: while return_buf = GRAM_SCRIPT_JOB_ID = 24846
7/25 11:22:46 JMI: while return_buf = GRAM_SCRIPT_JOB_STATE = 2
7/25 11:22:46 Job Manager State Machine (entering):
GLOBUS_GRAM_JOB_MANAGER_STATE_SUBMIT
7/25 11:22:46 JM: in globus_gram_job_manager_reporting_file_create()
7/25 11:22:46 JM: not reporting job information
7/25 11:22:46 JM: in globus_gram_job_manager_history_file_create()
7/25 11:22:46 JM: NOT empty client callback list.
7/25 11:22:46 JM: sending callback of status 2 (failure code 0) to
https://akiba119.apgrid.org:59394/.
7/25 11:22:46 Job Manager State Machine (entering):
GLOBUS_GRAM_JOB_MANAGER_STATE_POLL2
7/25 11:22:46 JMI: testing job manager scripts for type fork exist and
permissions are ok.
7/25 11:22:46 JMI: completed script validation: job manager type is fork.
7/25 11:22:46 JMI: in globus_gram_job_manager_poll()
7/25 11:22:46 JMI: local stdout filename = /dev/null.
7/25 11:22:46 JMI: local stderr filename = /dev/null.
7/25 11:22:46 JMI: poll: seeking:
https://akiba119.apgrid.org:60781/24836/1216952565/
7/25 11:22:46 JMI: poll_fast: returning -1 = GLOBUS_FAILURE (try Perl scripts)
7/25 11:22:46 JMI: cmd = poll
7/25 11:22:46 JMI: returning with success
Fri Jul 25 11:22:46 2008 JM_SCRIPT: New Perl JobManager created.
Fri Jul 25 11:22:46 2008 JM_SCRIPT: Using jm supplied job dir:
/home/tatuhiko/.globus/job/akiba119.apgrid.org/24836.1216952565
Fri Jul 25 11:22:46 2008 JM_SCRIPT: polling job 24846
7/25 11:22:46 JMI: while return_buf = GRAM_SCRIPT_JOB_STATE = 8
7/25 11:22:46 Job Manager State Machine (entering):
GLOBUS_GRAM_JOB_MANAGER_STATE_POLL1
7/25 11:22:46 JM: in globus_gram_job_manager_history_file_create()
7/25 11:22:46 Job Manager State Machine (entering):
GLOBUS_GRAM_JOB_MANAGER_STATE_PRE_CLOSE_OUTPUT
7/25 11:22:46 Job Manager State Machine (entering):
GLOBUS_GRAM_JOB_MANAGER_STATE_CLOSE_OUTPUT
7/25 11:22:46 Job Manager State Machine (entering):
GLOBUS_GRAM_JOB_MANAGER_STATE_STAGE_OUT
7/25 11:22:46 JM: NOT empty client callback list.
7/25 11:22:46 JM: sending callback of status 8 (failure code 0) to
https://akiba119.apgrid.org:59394/.
7/25 11:22:46 Job Manager State Machine (entering):
GLOBUS_GRAM_JOB_MANAGER_STATE_TWO_PHASE_END
7/25 11:22:46 Job Manager State Machine (entering):
GLOBUS_GRAM_JOB_MANAGER_STATE_TWO_PHASE_END_COMMITTED
7/25 11:22:46 Job Manager State Machine (entering):
GLOBUS_GRAM_JOB_MANAGER_STATE_FILE_CLEAN_UP
7/25 11:22:46 Job Manager State Machine (entering):
GLOBUS_GRAM_JOB_MANAGER_STATE_SCRATCH_CLEAN_UP
7/25 11:22:46 JMI: testing job manager scripts for type fork exist and
permissions are ok.
7/25 11:22:46 JMI: completed script validation: job manager type is fork.
7/25 11:22:46 JMI: cmd = cache_cleanup
Fri Jul 25 11:22:46 2008 JM_SCRIPT: New Perl JobManager created.
Fri Jul 25 11:22:46 2008 JM_SCRIPT: Using jm supplied job dir:
/home/tatuhiko/.globus/job/akiba119.apgrid.org/24836.1216952565
Fri Jul 25 11:22:46 2008 JM_SCRIPT: Using jm supplied job dir:
/home/tatuhiko/.globus/job/akiba119.apgrid.org/24836.1216952565
Fri Jul 25 11:22:46 2008 JM_SCRIPT: cache_cleanup(enter)
Fri Jul 25 11:22:46 2008 JM_SCRIPT: Cleaning files in job dir
/home/tatuhiko/.globus/job/akiba119.apgrid.org/24836.1216952565
Fri Jul 25 11:22:46 2008 JM_SCRIPT: Removed 2 files from
/home/tatuhiko/.globus/job/akiba119.apgrid.org/24836.1216952565
Fri Jul 25 11:22:46 2008 JM_SCRIPT: cache_cleanup(exit)
7/25 11:22:46 Job Manager State Machine (entering):
GLOBUS_GRAM_JOB_MANAGER_STATE_CACHE_CLEAN_UP
7/25 11:22:46 JM: in globus_gram_job_manager_reporting_file_remove()
7/25 11:22:46 JM: exiting globus_gram_job_manager.
TIME: Fri Jul 25 11:31:03 2008
PID: 8657 -- Notice: 6: /usr/local/GT/gt4.2.0/sbin/globus-gatekeeper pid=8657
starting at Fri Jul 25 11:31:03 2008
TIME: Fri Jul 25 11:31:03 2008
PID: 8658 -- Notice: 6: /usr/local/GT/gt4.2.0/sbin/globus-gatekeeper pid=8658
starting at Fri Jul 25 11:31:03 2008
TIME: Fri Jul 25 11:31:03 2008
PID: 8658 -- Notice: 6: GRAM contact:
jonagold.hpcc.jp:55932:/C=JP/O=AIST/OU=GRID/CN=Tatsuhiko Inoue
TIME: Fri Jul 25 11:31:13 2008
PID: 8671 -- Notice: 6: Got connection 192.50.74.149 at Fri Jul 25 11:31:13
2008
Failed reading length 0
GSS authentication failure
globus_gss_assist token :3: read failure: Connection closed
Failure: GSS failed Major:01090000 Minor:00000000 Token:00000003
TIME: Fri Jul 25 11:31:13 2008
PID: 8671 -- Failure: GSS failed Major:01090000 Minor:00000000 Token:00000003
