Hi,
I've been experimenting with hpx for a hobby project, with a small
virtualized cluster of 3 debian 8.2 machines running on esxi server. When I
run any hpx code on a single locality, it appears to be working; however,
whenever I try to use more than one locality, I invariably get a
segmentation fault, regardless of which code I am using. I first encountered
the trouble with my own code, but it also happens when running any of the
example apps as well. I am somewhat new to all of this and I cannot figure
out how to attach a debugger to try and identify the cause of these errors.
I'm using hpx 0.9.11 on my small cluster using the latest version of slurm .
(I chose slurm as it appears to provide support for Intel Phi nodes running
applications in native mode). To the best of my knowledge, slurm is
configured correctly. However, it is certainly possible that I have done
something wrong configuring slurm.
I have tried using boost 1.58, 1.59, and 1.60. I have tried with clang 3.7
and with Intel C++ 16 and 16 update 1. In all cases, I get the same
segmentation fault whenever I try and run on more than a single locality. I
have played around with single vs. multiple network interfaces, single vs.
multiple networks, etc.
Lately, I have re-built boost using the Intel compiler to ensure that there
was no issue caused by hpx and boost having been compiled with different
compilers. I have been trying to troubleshoot this only based on the
example code, rather than my own code, so that I can be confident that the
problems are not caused by my own code errors/bugs.
I know there is a command-line option to attach a debugger but I cannot
figure out how to use this.
I've attached a copy of my slurm.conf for reference, and the output of
--hpx:dump-config and --hpx:debug-clp
HPX stack trace / complete error message is copied below.
I'm really stuck here and honestly have no idea how to resolve this issue.
I greatly appreciate any help that you can offer. Furthermore, I'd really
appreciate some guidance as to how to use a debugger to debug my own hpx
code to identify and resolve issues with that code. Please let me know if
there's any additional information that I should provide.
Thank you very much in advance,
Shmuel
shmuel@ssh01:/usr/local/lib
> srun -n1 -N1 1d_stencil_7
Localities,OS_Threads,Execution_Time_sec,Points_per_Partition,Partitions,Tim
e_Steps
1, 1, 0.093138849, 10, 10, 45
shmuel@ssh01:~
> srun -n2 -N1 1d_stencil_7
{stack-trace}: 4 frames:
0x7f09a45e9840 : hpx::detail::backtrace(unsigned long) + 0x80 in
/usr/local/lib/libhpx.so.0
0x7f09a45eeced : boost::exception_ptr
hpx::detail::get_exception<hpx::exception>(hpx::exception const&,
std::string const&, std::string const&, long, std::string const&) + 0x23d in
/usr/local/lib/libhpx.so.0
0x7f09a45ee8bc : void
hpx::detail::throw_exception<hpx::exception>(hpx::exception const&,
std::string const&, std::string const&, long) + 0x10c in
/usr/local/lib/libhpx.so.0
0x7f09a4a0de3d :
hpx::agas::server::primary_namespace::resolve_free_list(boost::unique_lock<h
px::lcos::local::spinlock>&,
std::list<std::_Rb_tree_iterator<std::pair<hpx::naming::gid_type const,
long> >,
std::allocator<std::_Rb_tree_iterator<std::pair<hpx::naming::gid_type const,
long> > > > const&,
std::list<hpx::agas::server::primary_namespace::free_entry,
std::allocator<hpx::agas::server::primary_namespace::free_entry> >&,
hpx::naming::gid_type const&, hpx::naming::gid_type const&,
hpx::error_code&) + 0x137d in /usr/local/lib/libhpx.so.0
{env}: 85 entries:
ALTERNATE_EDITOR=
CPATH=/opt/intel/compilers_and_libraries_2016.1.150/linux/mkl/include
CXX=clang++
DIRHISTORY_SIZE=30
DISPLAY=localhost:10.0
EDITOR=/usr/bin/vim
FPATH=/home/shmuel/.oh-my-zsh/plugins/wd:/home/shmuel/.oh-my-zsh/plugins/tmu
x:/home/shmuel/.oh-my-zsh/plugins/dirhistory:/home/shmuel/.oh-my-zsh/plugins
/colorize:/home/shmuel/.oh-my-zsh/plugins/history:/home/shmuel/.oh-my-zsh/pl
ugins/sudo:/home/shmuel/.oh-my-zsh/plugins/command-not-found:/home/shmuel/.o
h-my-zsh/plugins/tmux:/home/shmuel/.oh-my-zsh/plugins/mosh:/home/shmuel/.oh-
my-zsh/plugins/git-extras:/home/shmuel/.oh-my-zsh/plugins/battery:/home/shmu
el/.oh-my-zsh/plugins/git-flow-avh:/home/shmuel/.oh-my-zsh/plugins/git:/home
/shmuel/.oh-my-zsh/functions:/home/shmuel/.oh-my-zsh/completions:/usr/local/
share/zsh/site-functions:/usr/share/zsh/vendor-functions:/usr/share/zsh/vend
or-completions:/usr/share/zsh/functions/Calendar:/usr/share/zsh/functions/Ch
pwd:/usr/share/zsh/functions/Completion:/usr/share/zsh/functions/Completion/
AIX:/usr/share/zsh/functions/Completion/BSD:/usr/share/zsh/functions/Complet
ion/Base:/usr/share/zsh/functions/Completion/Cygwin:/usr/share/zsh/functions
/Completion/Darwin:/usr/share/zsh/functions/Completion/Debian:/usr/share/zsh
/functions/Completion/Linux:/usr/share/zsh/functions/Completion/Mandriva:/us
r/share/zsh/functions/Completion/Redhat:/usr/share/zsh/functions/Completion/
Solaris:/usr/share/zsh/functions/Completion/Unix:/usr/share/zsh/functions/Co
mpletion/X:/usr/share/zsh/functions/Completion/Zsh:/usr/share/zsh/functions/
Completion/openSUSE:/usr/share/zsh/functions/Exceptions:/usr/share/zsh/funct
ions/MIME:/usr/share/zsh/functions/Misc:/usr/share/zsh/functions/Newuser:/us
r/share/zsh/functions/Prompts:/usr/share/zsh/functions/TCP:/usr/share/zsh/fu
nctions/VCS_Info:/usr/share/zsh/functions/VCS_Info/Backends:/usr/share/zsh/f
unctions/Zftp:/usr/share/zsh/functions/Zle:/home/shmuel/bin/funcs
GDBSERVER_MIC=/opt/intel/debugger_2016/gdb/targets/mic/bin/gdbserver
GDB_CROSS=/opt/intel/debugger_2016/gdb/intel64_mic/bin/gdb-mic
HOME=/home/shmuel
INFOPATH=/opt/intel/documentation_2016/en/debugger//gdb-ia/info/:/opt/intel/
documentation_2016/en/debugger//gdb-mic/info/:/opt/intel/documentation_2016/
en/debugger//gdb-igfx/info/
INTEL_LICENSE_FILE=/opt/intel/compilers_and_libraries_2016.1.150/linux/licen
ses:/opt/intel/licenses:/home/shmuel/intel/licenses
INTEL_PYTHONHOME=/opt/intel/debugger_2016/python/intel64/
I_MPI_ROOT=/opt/intel/compilers_and_libraries_2016.1.150/linux/mpi
LANG=en_US.utf8
LANGUAGE=en_CA:en
LC_ALL=en_CA.UTF-8
LC_CTYPE=en_CA.UTF-8
LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2016.1.150/linux/compiler
/lib/intel64:/opt/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64
/lib:/opt/intel/compilers_and_libraries_2016.1.150/linux/mpi/mic/lib:/opt/in
tel/compilers_and_libraries_2016.1.150/linux/tbb/lib/intel64/gcc4.4:/opt/int
el/compilers_and_libraries_2016.1.150/linux/compiler/lib/intel64:/opt/intel/
compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64:/opt/intel/debugger
_2016/libipt/intel64/lib:/home/shmuel/src/fx/lib/:
LESS=-R
LIBRARY_PATH=/opt/intel/compilers_and_libraries_2016.1.150/linux/tbb/lib/int
el64/gcc4.4:/opt/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib
/intel64:/opt/intel/compilers_and_libraries_2016.1.150/linux/mkl/lib/intel64
LOGNAME=shmuel
LSCOLORS=Gxfxcxdxbxegedabagacad
MAIL=/var/mail/shmuel
MANPATH=/opt/intel/man/common:/opt/intel/compilers_and_libraries_2016.1.150/
linux/mpi/man:/opt/intel/compilers_and_libraries_2016.1.150/linux/man/en_US:
/opt/intel/documentation_2016/en/debugger//gdb-ia/man/:/opt/intel/documentat
ion_2016/en/debugger//gdb-mic/man/:/opt/intel/documentation_2016/en/debugger
//gdb-igfx/man/::/home/shmuel/src/tup/
MIC_LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2016.1.150/linux/comp
iler/lib/mic:/opt/intel/compilers_and_libraries_2016.1.150/linux/mpi/mic/lib
:/opt/intel/compilers_and_libraries_2016.1.150/linux/tbb/lib/mic:/opt/intel/
compilers_and_libraries_2016.1.150/linux/compiler/lib/mic:/opt/intel/compile
rs_and_libraries_2016.1.150/linux/mkl/lib/mic
MIC_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2016.1.150/linux/compile
r/lib/mic:/opt/intel/compilers_and_libraries_2016.1.150/linux/mpi/mic/lib
MKLROOT=/opt/intel/compilers_and_libraries_2016.1.150/linux/mkl
MPM_LAUNCHER=/opt/intel/debugger_2016/mpm/mic/bin/start_mpm.sh
NLSPATH=/opt/intel/compilers_and_libraries_2016.1.150/linux/compiler/lib/int
el64/locale/%l_%t/%N:/opt/intel/compilers_and_libraries_2016.1.150/linux/mkl
/lib/intel64/locale/%l_%t/%N:/opt/intel/debugger_2016/gdb/intel64_mic/share/
locale/%l_%t/%N:/opt/intel/debugger_2016/gdb/intel64/share/locale/%l_%t/%N
OLDPWD=/home/shmuel
PAGER=less
PATH=/opt/intel/compilers_and_libraries_2016.1.150/linux/bin/intel64:/opt/in
tel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/bin:/opt/intel/debu
gger_2016/gdb/intel64_mic/bin:/usr/local/texlive/2014/bin/x86_64-linux:/usr/
local/bin:/usr/bin:/bin:/usr/games:/usr/local/sbin:/sbin:/usr/local/games:/h
ome/shmuel/src/tup/
PWD=/home/shmuel
REPORTTIME=2
SHELL=/bin/zsh
SHLVL=1
SLURMD_NODENAME=hpc02
SLURM_CHECKPOINT_IMAGE_DIR=/var/lib/slurm-llnl/checkpoint
SLURM_CLUSTER_NAME=cluster
SLURM_CPUS_ON_NODE=2
SLURM_DISTRIBUTION=block
SLURM_GTIDS=0,1
SLURM_JOBID=2165
SLURM_JOB_CPUS_PER_NODE=2
SLURM_JOB_ID=2165
SLURM_JOB_NAME=1d_stencil_7
SLURM_JOB_NODELIST=hpc02
SLURM_JOB_NUM_NODES=1
SLURM_JOB_PARTITION=debug
SLURM_JOB_UID=1000
SLURM_JOB_USER=shmuel
SLURM_LAUNCH_NODE_IPADDR=192.168.1.125
SLURM_LOCALID=1
SLURM_NNODES=1
SLURM_NODEID=0
SLURM_NODELIST=hpc02
SLURM_NPROCS=2
SLURM_NTASKS=2
SLURM_PRIO_PROCESS=0
SLURM_PROCID=1
SLURM_SRUN_COMM_HOST=192.168.1.125
SLURM_SRUN_COMM_PORT=45712
SLURM_STEPID=0
SLURM_STEP_ID=0
SLURM_STEP_LAUNCHER_PORT=45712
SLURM_STEP_NODELIST=hpc02
SLURM_STEP_NUM_NODES=1
SLURM_STEP_NUM_TASKS=2
SLURM_STEP_TASKS_PER_NODE=2
SLURM_SUBMIT_DIR=/home/shmuel
SLURM_SUBMIT_HOST=ssh01.thelevines.ca
SLURM_TASKS_PER_NODE=2
SLURM_TASK_PID=18855
SLURM_TOPOLOGY_ADDR=hpc02
SLURM_TOPOLOGY_ADDR_PATTERN=node
SRUN_DEBUG=3
SSH_CLIENT=193.90.12.86 38280 22
SSH_CONNECTION=193.90.12.86 38280 192.168.1.125 22
SSH_TTY=/dev/pts/0
TERM=xterm
USER=shmuel
ZSH_TMUX_TERM=screen
_=/usr/local/bin/srun
_ZSH_TMUX_FIXED_CONFIG=/home/shmuel/.oh-my-zsh/plugins/tmux/tmux.only.conf
{locality-id}: 1
{hostname}: [ (tcp:192.168.1.72:7911) ]
{process-id}: 18855
{function}: primary_namespace::resolve_free_list
{file}: /usr/src/hpx/src/runtime/agas/server/primary_namespace_server.cpp
{line}: 1021
{os-thread}: 0, worker-thread#0
{thread-id}: 00000000020813c0
{thread-description}: <unknown>
{state}: state_running
{auxinfo}:
{config}:
HPX_HAVE_NATIVE_TLS=ON
HPX_HAVE_STACKTRACES=ON
HPX_HAVE_COMPRESSION_BZIP2=OFF
HPX_HAVE_COMPRESSION_SNAPPY=OFF
HPX_HAVE_COMPRESSION_ZLIB=OFF
HPX_HAVE_PARCEL_COALESCING=ON
HPX_HAVE_PARCELPORT_TCP=ON
HPX_HAVE_PARCELPORT_MPI=OFF
HPX_HAVE_PARCELPORT_IPC=OFF
HPX_HAVE_PARCELPORT_IBVERBS=OFF
HPX_HAVE_VERIFY_LOCKS=OFF
HPX_HAVE_HWLOC=ON
HPX_HAVE_ITTNOTIFY=OFF
HPX_HAVE_RUN_MAIN_EVERYWHERE=OFF
HPX_LIMIT=5
HPX_PARCEL_MAX_CONNECTIONS=512
HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
HPX_INITIAL_AGAS_LOCAL_CACHE_SIZE=256
HPX_AGAS_LOCAL_CACHE_SIZE_PER_THREAD=32
HPX_HAVE_MALLOC=tcmalloc
HPX_PREFIX (configured)=/usr/local
HPX_PREFIX=/usr/local
{version}: V0.9.11 (AGAS: V3.0), Git: 4c96a9b3b3
{boost}: V1.60.0
{build-type}: release
{date}: Jan 3 2016 23:53:54
{platform}: linux
{compiler}: Intel C++ C++0x mode version 1600
{stdlib}: GNU libstdc++ version 20141220
{what}: primary_namespace::resolve_free_list, failed to resolve gid,
gid({0000000200000001, 0000000000001002}): HPX(internal_server_error)
{stack-trace}: 2 frames:
0x7f09a4670d79 : hpx::termination_handler(int) + 0x159 in
/usr/local/lib/libhpx.so.0
0x7f09a11e78d0 : ??? + 0x7f09a11e78d0 in
/lib/x86_64-linux-gnu/libpthread.so.0
{what}: Segmentation fault
{config}:
HPX_HAVE_NATIVE_TLS=ON
HPX_HAVE_STACKTRACES=ON
HPX_HAVE_COMPRESSION_BZIP2=OFF
HPX_HAVE_COMPRESSION_SNAPPY=OFF
HPX_HAVE_COMPRESSION_ZLIB=OFF
HPX_HAVE_PARCEL_COALESCING=ON
HPX_HAVE_PARCELPORT_TCP=ON
HPX_HAVE_PARCELPORT_MPI=OFF
HPX_HAVE_PARCELPORT_IPC=OFF
HPX_HAVE_PARCELPORT_IBVERBS=OFF
HPX_HAVE_VERIFY_LOCKS=OFF
HPX_HAVE_HWLOC=ON
HPX_HAVE_ITTNOTIFY=OFF
HPX_HAVE_RUN_MAIN_EVERYWHERE=OFF
HPX_LIMIT=5
HPX_PARCEL_MAX_CONNECTIONS=512
HPX_PARCEL_MAX_CONNECTIONS_PER_LOCALITY=4
HPX_INITIAL_AGAS_LOCAL_CACHE_SIZE=256
HPX_AGAS_LOCAL_CACHE_SIZE_PER_THREAD=32
HPX_HAVE_MALLOC=tcmalloc
HPX_PREFIX (configured)=/usr/local
HPX_PREFIX=/usr/local
{version}: V0.9.11 (AGAS: V3.0), Git: 4c96a9b3b3
{boost}: V1.60.0
{build-type}: release
{date}: Jan 3 2016 23:53:54
{platform}: linux
{compiler}: Intel C++ C++0x mode version 1600
{stdlib}: GNU libstdc++ version 20141220
srun: error: hpc02: task 1: Aborted
slurm.conf
Description: Binary data
shmuel@ssh01:~
> srun -n2 -N1 1d_stencil_7 --hpx:dump-config
Configuration after runtime start:
----------------------------------
============================
[application]
Configuration after runtime start:
----------------------------------
============================
[application]
[hpx]
'affinity' : 'pu'
'cmd_line' : '/usr/local/bin/1d_stencil_7 --hpx:dump-config'
'component_path' : '$[hpx.location]:$[system.executable_prefix]' ->
'/usr/local:/usr/local'
'component_path_suffixes' : '/lib/hpx:/bin/hpx'
'cores' : '1'
'expect_connecting_localities' : '0'
'finalize_wait_time' : '-1.0'
'first_pu' : '0'
'first_used_core' : '1'
'ini_path' :
':/local/src/fx_hpx/lib:/local/src/fx_hpx/lib:/local/src/fx_hpx/lib:/local/src/fx_hpx/lib:/local/src/fx_hpx/lib:/local/src/fx_hpx/lib'
'localities' : '2'
'locality' : '1'
'location' : '$[system.prefix]' -> '/usr/local'
'master_ini_path' : '$[hpx.location]:$[system.executable_prefix]/' ->
'/usr/local:/usr/local/'
'master_ini_path_suffixes' : '/share/hpx-0.9.11:/../share/hpx-0.9.11'
'max_background_threads' : '$[hpx.os_threads]' -> '1'
'nodes' : 'hpc02'
'numa_sensitive' : '0'
'os_threads' : '1'
'program_name' : '/usr/local/bin/1d_stencil_7'
'pu_offset' : '0'
'pu_step' : '1'
'reconstructed_cmd_line' : '/usr/local/bin/1d_stencil_7 --dt=1 --dx=1
--hpx:config --hpx:dump-config --k=0.5'
'runtime_mode' : 'worker'
'scheduler' : 'local-priority'
'shutdown_timeout' : '-1.0'
[agas]
'address' : '192.168.1.72'
'dedicated_server' : '0'
'local_cache_size' : '256'
'local_cache_size_per_thread' : '32'
'max_pending_refcnt_requests' : '4096'
'port' : '7910'
'service_mode' : 'hosted'
'use_caching' : '1'
'use_range_caching' : '1'
[commandline]
'aliasing' : '1'
'allow_unknown' : '0'
[aliases]
'-0' : '--hpx:node=0'
'-1' : '--hpx:node=1'
'-2' : '--hpx:node=2'
'-3' : '--hpx:node=3'
'-4' : '--hpx:node=4'
'-5' : '--hpx:node=5'
'-6' : '--hpx:node=6'
'-7' : '--hpx:node=7'
'-8' : '--hpx:node=8'
'-9' : '--hpx:node=9'
'-I' : '--hpx:ini'
'-a' : '--hpx:agas'
'-c' : '--hpx:console'
'-h' : '--hpx:help'
'-l' : '--hpx:localities'
'-p' : '--hpx:app-config'
'-q' : '--hpx:queuing'
'-r' : '--hpx:run-agas-server'
'-t' : '--hpx:threads'
'-v' : '--hpx:version'
'-w' : '--hpx:worker'
'-x' : '--hpx:hpx'
[components]
'load_external' : '1'
[adding_counter]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[average_count_counter]
'enabled' : '1'
'name' : 'hpx'
'path' : '$[hpx.location]/bin/libhpx.so' -> '/usr/local/bin/libhpx.so'
'static' : '1'
[barrier]
'enabled' : '1'
'name' : 'hpx'
'path' : '$[hpx.location]/bin/libhpx.so' -> '/usr/local/bin/libhpx.so'
'static' : '1'
[component_namespace]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[dividing_counter]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[elapsed_time_counter]
'enabled' : '1'
'name' : 'hpx'
'path' : '$[hpx.location]/bin/libhpx.so' -> '/usr/local/bin/libhpx.so'
'static' : '1'
[hpx_io_counters]
'enabled' : '1'
'name' : 'hpx_io_counters'
'no_factory' : '1'
'path' : '/usr/local/lib/hpx'
[hpx_lcos_server_latch]
'enabled' : '1'
'name' : 'hpx'
'path' : '$[hpx.location]/bin/libhpx.so' -> '/usr/local/bin/libhpx.so'
'static' : '1'
[hpx_memory]
'enabled' : '1'
'name' : 'hpx_memory'
'no_factory' : '1'
'path' : '/usr/local/lib/hpx'
[locality_namespace]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[max_count_counter]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[median_count_counter]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[min_count_counter]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[multiplying_counter]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[output_stream_factory]
'enabled' : '1'
'name' : 'hpx_iostreams'
'static' : '1'
[partition_server]
'enabled' : '$[hpx.components.load_external]' -> '1'
'name' : 'hpx'
'static' : '1'
[primary_namespace]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[raw_counter]
'enabled' : '1'
'name' : 'hpx'
'path' : '$[hpx.location]/bin/libhpx.so' -> '/usr/local/bin/libhpx.so'
'static' : '1'
[rolling_mean_count_counter]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[sine_counter]
'enabled' : '$[hpx.components.load_external]' -> '1'
'name' : 'hpx_sine'
'path' : '/usr/local/lib/hpx'
[subtracting_counter]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[symbol_namespace]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[variance_count_counter]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[logging]
'destination' : 'console'
'format' : '(T%locality%/%hpxthread%.%hpxphase%/%hpxcomponent%)
P%parentloc%/%hpxparent%.%hpxparentphase% %time%($hh:$mm.$ss.$mili) [%idx%]|\n'
'level' : '0'
[agas]
'destination' : 'file(hpx.agas.$[system.pid].log)' ->
'file(hpx.agas.18925.log)'
'format' : '(T%locality%/%hpxthread%.%hpxphase%/%hpxcomponent%)
P%parentloc%/%hpxparent%.%hpxparentphase% %time%($hh:$mm.$ss.$mili)
[%idx%][AGAS] |\n'
'level' : '-1'
[application]
'destination' : 'console'
'format' : '(T%locality%/%hpxthread%.%hpxphase%/%hpxcomponent%)
P%parentloc%/%hpxparent%.%hpxparentphase% %time%($hh:$mm.$ss.$mili) [%idx%]
[APP] |\n'
'level' : '-1'
[console]
'destination' : ' file(hpx.$[system.pid].log)' -> '
file(hpx.18925.log)'
'format' : '|'
'level' : '$[hpx.logging.level]' -> '0'
[agas]
'destination' : 'file
(hpx.agas.$[system.pid].log)' -> 'file (hpx.agas.18925.log)'
'format' : '|'
'level' : '$[hpx.logging.agas.level]' -> '-1'
[application]
'destination' : 'file
(hpx.application.$[system.pid].log)' -> 'file
(hpx.application.18925.log)'
'format' : '|'
'level' : '$[hpx.logging.application.level]' -> '-1'
[hpx]
'affinity' : 'pu'
'cmd_line' : '/usr/local/bin/1d_stencil_7 --hpx:dump-config'
'component_path' : '$[hpx.location]:$[system.executable_prefix]' ->
'/usr/local:/usr/local'
'component_path_suffixes' : '/lib/hpx:/bin/hpx'
'cores' : '1'
'expect_connecting_localities' : '0'
'finalize_wait_time' : '-1.0'
'first_pu' : '0'
'first_used_core' : '0'
'ini_path' :
':/local/src/fx_hpx/lib:/local/src/fx_hpx/lib:/local/src/fx_hpx/lib:/local/src/fx_hpx/lib:/local/src/fx_hpx/lib:/local/src/fx_hpx/lib'
'localities' : '2'
'locality' : '0'
'location' : '$[system.prefix]' -> '/usr/local'
'master_ini_path' : '$[hpx.location]:$[system.executable_prefix]/' ->
'/usr/local:/usr/local/'
'master_ini_path_suffixes' : '/share/hpx-0.9.11:/../share/hpx-0.9.11'
'max_background_threads' : '$[hpx.os_threads]' -> '1'
'nodes' : 'hpc02'
'numa_sensitive' : '0'
'os_threads' : '1'
'program_name' : '/usr/local/bin/1d_stencil_7'
'pu_offset' : '0'
'pu_step' : '1'
'reconstructed_cmd_line' : '/usr/local/bin/1d_stencil_7 --dt=1 --dx=1
--hpx:config --hpx:dump-config --k=0.5'
'runtime_mode' : 'console'
'scheduler' : 'local-priority'
'shutdown_timeout' : '-1.0'
[agas]
'address' : '192.168.1.72'
'dedicated_server' : '0'
'local_cache_size' : '256'
'local_cache_size_per_thread' : '32'
'max_pending_refcnt_requests' : '4096'
'port' : '7910'
'service_mode' : 'bootstrap'
'use_caching' : '1'
'use_range_caching' : '1'
[debuglog]
'destination' : 'file
(hpx.debuglog.$[system.pid].log)' -> 'file
(hpx.debuglog.18925.log)'
'format' : '|'
'level' : '$[hpx.logging.debuglog.level]' -> '-1'
[parcel]
'destination' : 'file
(hpx.parcel.$[system.pid].log)' -> 'file
(hpx.parcel.18925.log)'
'format' : '|'
'level' : '$[hpx.logging.parcel.level]' -> '-1'
[timing]
'destination' : 'file(hpx.timing.$[system.pid].log)' ->
'file(hpx.timing.18925.log)'
'format' : '|'
'level' : '$[hpx.logging.timing.level]' -> '-1'
[debuglog]
'destination' : 'console'
'format' : '(T%locality%/%hpxthread%.%hpxphase%/%hpxcomponent%)
P%parentloc%/%hpxparent%.%hpxparentphase% %time%($hh:$mm.$ss.$mili) [%idx%]
[DEB] |\n'
'level' : '-1'
[commandline]
'aliasing' : '1'
'allow_unknown' : '0'
[aliases]
'-0' : '--hpx:node=0'
'-1' : '--hpx:node=1'
'-2' : '--hpx:node=2'
'-3' : '--hpx:node=3'
'-4' : '--hpx:node=4'
'-5' : '--hpx:node=5'
'-6' : '--hpx:node=6'
'-7' : '--hpx:node=7'
'-8' : '--hpx:node=8'
'-9' : '--hpx:node=9'
'-I' : '--hpx:ini'
'-a' : '--hpx:agas'
'-c' : '--hpx:console'
'-h' : '--hpx:help'
'-l' : '--hpx:localities'
'-p' : '--hpx:app-config'
'-q' : '--hpx:queuing'
'-r' : '--hpx:run-agas-server'
'-t' : '--hpx:threads'
'-v' : '--hpx:version'
'-w' : '--hpx:worker'
'-x' : '--hpx:hpx'
[components]
'load_external' : '1'
[adding_counter]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[average_count_counter]
'enabled' : '1'
'name' : 'hpx'
'path' : '$[hpx.location]/bin/libhpx.so' -> '/usr/local/bin/libhpx.so'
'static' : '1'
[barrier]
'enabled' : '1'
'name' : 'hpx'
'path' : '$[hpx.location]/bin/libhpx.so' -> '/usr/local/bin/libhpx.so'
'static' : '1'
[parcel]
'destination' : 'file (hpx.parcel.$[system.pid].log)' ->
'file (hpx.parcel.18925.log)'
'format' : '(T%locality%/%hpxthread%.%hpxphase%/%hpxcomponent%)
P%parentloc%/%hpxparent%.%hpxparentphase% %time%($hh:$mm.$ss.$mili) [%idx%][
PT] |\n'
'level' : '-1'
[timing]
'destination' : 'console'
'format' : '(T%locality%/%hpxthread%.%hpxphase%/%hpxcomponent%)
P%parentloc%/%hpxparent%.%hpxparentphase% %time%($hh:$mm.$ss.$mili) [%idx%]
[TIM] |\n'
'level' : '-1'
[component_namespace]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[dividing_counter]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[elapsed_time_counter]
'enabled' : '1'
'name' : 'hpx'
'path' : '$[hpx.location]/bin/libhpx.so' -> '/usr/local/bin/libhpx.so'
'static' : '1'
[hpx_io_counters]
'enabled' : '1'
'name' : 'hpx_io_counters'
'no_factory' : '1'
'path' : '/usr/local/lib/hpx'
[hpx_lcos_server_latch]
'enabled' : '1'
'name' : 'hpx'
'path' : '$[hpx.location]/bin/libhpx.so' -> '/usr/local/bin/libhpx.so'
'static' : '1'
[hpx_memory]
'enabled' : '1'
'name' : 'hpx_memory'
'no_factory' : '1'
'path' : '/usr/local/lib/hpx'
[locality_namespace]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[max_count_counter]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[median_count_counter]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[parcel]
'address' : '192.168.1.72'
'array_optimization' : '1'
'async_serialization' : '1'
'bootstrap' : 'tcp'
'enable_security' : '0'
'endian_out' : 'little'
'max_connections' : '512'
'max_connections_per_locality' : '4'
'max_message_size' : '1000000000'
'max_outbound_message_size' : '1000000'
'message_handlers' : '0'
'port' : '7911'
'zero_copy_optimization' : '$[hpx.parcel.array_optimization]' -> '1'
[tcp]
'array_optimization' : '$[hpx.parcel.array_optimization]' -> '1'
'async_serialization' : '$[hpx.parcel.async_serialization]' -> '1'
'enable' : '1'
'enable_security' : '$[hpx.parcel.enable_security]' -> '0'
'max_connections' : '$[hpx.parcel.max_connections]' -> '512'
'max_connections_per_locality' :
'$[hpx.parcel.max_connections_per_locality]' -> '4'
'max_message_size' : '$[hpx.parcel.max_message_size]' -> '1000000000'
'max_outbound_message_size' : '$[hpx.parcel.max_outbound_message_size]'
-> '1000000'
'name' : 'hpx'
'parcel_pool_size' : '$[hpx.threadpools.parcel_pool_size]' -> '2'
'path' : '/usr/local/lib/hpx:'
'priority' : '1'
'zero_copy_optimization' : '$[hpx.parcel.zero_copy_optimization]' -> '1'
[stacks]
'huge_size' : '0x2000000'
'large_size' : '0x0200000'
'medium_size' : '0x0020000'
'small_size' : '0x8000'
'use_guard_pages' : '1'
[threadpools]
'io_pool_size' : '2'
'parcel_pool_size' : '2'
'timer_pool_size' : '2'
[system]
'executable_prefix' : '/usr/local'
'pid' : '18925'
'prefix' : '/usr/local'
============================
[min_count_counter]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[multiplying_counter]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[output_stream_factory]
'enabled' : '1'
'name' : 'hpx_iostreams'
'static' : '1'
[partition_server]
'enabled' : '$[hpx.components.load_external]' -> '1'
'name' : 'hpx'
'static' : '1'
[primary_namespace]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[raw_counter]
'enabled' : '1'
'name' : 'hpx'
'path' : '$[hpx.location]/bin/libhpx.so' -> '/usr/local/bin/libhpx.so'
'static' : '1'
[rolling_mean_count_counter]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[sine_counter]
'enabled' : '$[hpx.components.load_external]' -> '1'
'name' : 'hpx_sine'
'path' : '/usr/local/lib/hpx'
[subtracting_counter]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[symbol_namespace]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[variance_count_counter]
'enabled' : '1'
'name' : 'hpx'
'static' : '1'
[logging]
'destination' : 'console'
'format' : '(T%locality%/%hpxthread%.%hpxphase%/%hpxcomponent%)
P%parentloc%/%hpxparent%.%hpxparentphase% %time%($hh:$mm.$ss.$mili) [%idx%]|\n'
'level' : '0'
[agas]
'destination' : 'file(hpx.agas.$[system.pid].log)' ->
'file(hpx.agas.18924.log)'
'format' : '(T%locality%/%hpxthread%.%hpxphase%/%hpxcomponent%)
P%parentloc%/%hpxparent%.%hpxparentphase% %time%($hh:$mm.$ss.$mili)
[%idx%][AGAS] |\n'
'level' : '-1'
[application]
'destination' : 'console'
'format' : '(T%locality%/%hpxthread%.%hpxphase%/%hpxcomponent%)
P%parentloc%/%hpxparent%.%hpxparentphase% %time%($hh:$mm.$ss.$mili) [%idx%]
[APP] |\n'
'level' : '-1'
[console]
'destination' : ' file(hpx.$[system.pid].log)' -> '
file(hpx.18924.log)'
'format' : '|'
'level' : '$[hpx.logging.level]' -> '0'
[agas]
'destination' : 'file
(hpx.agas.$[system.pid].log)' -> 'file (hpx.agas.18924.log)'
'format' : '|'
'level' : '$[hpx.logging.agas.level]' -> '-1'
[application]
'destination' : 'file
(hpx.application.$[system.pid].log)' -> 'file
(hpx.application.18924.log)'
'format' : '|'
'level' : '$[hpx.logging.application.level]' -> '-1'
[debuglog]
'destination' : 'file
(hpx.debuglog.$[system.pid].log)' -> 'file
(hpx.debuglog.18924.log)'
'format' : '|'
'level' : '$[hpx.logging.debuglog.level]' -> '-1'
[parcel]
'destination' : 'file
(hpx.parcel.$[system.pid].log)' -> 'file
(hpx.parcel.18924.log)'
'format' : '|'
'level' : '$[hpx.logging.parcel.level]' -> '-1'
[timing]
'destination' : 'file(hpx.timing.$[system.pid].log)' ->
'file(hpx.timing.18924.log)'
'format' : '|'
'level' : '$[hpx.logging.timing.level]' -> '-1'
[debuglog]
'destination' : 'console'
'format' : '(T%locality%/%hpxthread%.%hpxphase%/%hpxcomponent%)
P%parentloc%/%hpxparent%.%hpxparentphase% %time%($hh:$mm.$ss.$mili) [%idx%]
[DEB] |\n'
'level' : '-1'
[parcel]
'destination' : 'file (hpx.parcel.$[system.pid].log)' ->
'file (hpx.parcel.18924.log)'
'format' : '(T%locality%/%hpxthread%.%hpxphase%/%hpxcomponent%)
P%parentloc%/%hpxparent%.%hpxparentphase% %time%($hh:$mm.$ss.$mili) [%idx%][
PT] |\n'
'level' : '-1'
[timing]
'destination' : 'console'
'format' : '(T%locality%/%hpxthread%.%hpxphase%/%hpxcomponent%)
P%parentloc%/%hpxparent%.%hpxparentphase% %time%($hh:$mm.$ss.$mili) [%idx%]
[TIM] |\n'
'level' : '-1'
[parcel]
'address' : '192.168.1.72'
'array_optimization' : '1'
'async_serialization' : '1'
'bootstrap' : 'tcp'
'enable_security' : '0'
'endian_out' : 'little'
'max_connections' : '512'
'max_connections_per_locality' : '4'
'max_message_size' : '1000000000'
'max_outbound_message_size' : '1000000'
'message_handlers' : '0'
'port' : '7910'
'zero_copy_optimization' : '$[hpx.parcel.array_optimization]' -> '1'
[tcp]
'array_optimization' : '$[hpx.parcel.array_optimization]' -> '1'
'async_serialization' : '$[hpx.parcel.async_serialization]' -> '1'
'enable' : '1'
'enable_security' : '$[hpx.parcel.enable_security]' -> '0'
'max_connections' : '$[hpx.parcel.max_connections]' -> '512'
'max_connections_per_locality' :
'$[hpx.parcel.max_connections_per_locality]' -> '4'
'max_message_size' : '$[hpx.parcel.max_message_size]' -> '1000000000'
'max_outbound_message_size' : '$[hpx.parcel.max_outbound_message_size]'
-> '1000000'
'name' : 'hpx'
'parcel_pool_size' : '$[hpx.threadpools.parcel_pool_size]' -> '2'
'path' : '/usr/local/lib/hpx:'
'priority' : '1'
'zero_copy_optimization' : '$[hpx.parcel.zero_copy_optimization]' -> '1'
[stacks]
'huge_size' : '0x2000000'
'large_size' : '0x0200000'
'medium_size' : '0x0020000'
'small_size' : '0x8000'
'use_guard_pages' : '1'
[threadpools]
'io_pool_size' : '2'
'parcel_pool_size' : '2'
'timer_pool_size' : '2'
[system]
'executable_prefix' : '/usr/local'
'pid' : '18924'
'prefix' : '/usr/local'
============================
----------------------------------
shmuel@ssh01:~ > srun -n2 -N1 1d_stencil_7 --hpx:debug-clp SLURM tasks per node found (SLURM_TASKS_PER_NODE): 2 SLURM nodelist found (SLURM_NODELIST): hpc02 got node list extracted: 'hpc02' incrementing agas_node_num using AGAS host: 'hpc02' (node number 0) Nodes from nodelist: hpc02: 1 (192.168.1.72:0) agas host_name: hpc02 asio host_name: hpc02.thelevines.ca host_name: hpc02.thelevines.ca resolved: 'hpc02.thelevines.ca' to: 192.168.1.72 resolved: 'hpc02' to: 192.168.1.72 Configuration before runtime start: ----------------------------------- hpx.nodes!=hpc02 hpx.expect_connecting_localities=0 hpx.locality!=0 hpx.scheduler=local-priority hpx.affinity=pu hpx.pu_step=1 hpx.pu_offset=0 hpx.numa_sensitive=0 hpx.parcel.address=192.168.1.72 hpx.parcel.port=7910 hpx.agas.address=192.168.1.72 hpx.agas.port=7910 hpx.agas.service_mode=bootstrap hpx.os_threads=1 hpx.cores=1 hpx.localities=2 hpx.runtime_mode=console ----------------------------------- SLURM tasks per node found (SLURM_TASKS_PER_NODE): 2 SLURM nodelist found (SLURM_NODELIST): hpc02 got node list extracted: 'hpc02' incrementing agas_node_num using AGAS host: 'hpc02' (node number 0) Nodes from nodelist: hpc02: 1 (192.168.1.72:0) agas host_name: hpc02 asio host_name: hpc02.thelevines.ca host_name: hpc02.thelevines.ca resolved: 'hpc02.thelevines.ca' to: 192.168.1.72 resolved: 'hpc02' to: 192.168.1.72 Configuration before runtime start: ----------------------------------- hpx.nodes!=hpc02 hpx.expect_connecting_localities=0 hpx.locality!=1 hpx.scheduler=local-priority hpx.affinity=pu hpx.pu_step=1 hpx.pu_offset=0 hpx.numa_sensitive=0 hpx.parcel.address=192.168.1.72 hpx.parcel.port=7911 hpx.agas.address=192.168.1.72 hpx.agas.port=7910 hpx.os_threads=1 hpx.cores=1 hpx.localities=2 hpx.runtime_mode=worker -----------------------------------
_______________________________________________ hpx-users mailing list [email protected] https://mail.cct.lsu.edu/mailman/listinfo/hpx-users
