[lttng-dev] show the ratio of reads from page cache vs reads from disk, for a user space file read of a C program

[MMM via lttng-dev]

Dear Users,
I am beginner in LTTng. I am going to trace a C program and show the ratio of 
reads from page cache vs reads from disk, for a user space file read using 
Trace Compass. While I can not write appropriate commands in 
trace_point_provider.h and in the C program. Could you please guide me or 
introduce a sufficient and simple reference to help me do it.
Best regards,Mehdi.
___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

[lttng-dev] [PATCH lttng-tools] Fix: wrong parameter to fcntl in pipe_set_flag

[Julien Desfossez]

Depending on the flags passed, fcntl must be called with F_SETFD or
F_SETFL. This fix checks the flag passed and ensure it is valid and
calls fcntl with the right parameter.

Also, for CLOEXEC, we need to pass FD_CLOEXEC, not O_CLOEXEC.

Signed-off-by: Julien Desfossez 
---
 src/bin/lttng-sessiond/notification-thread.c |  2 +-
 src/common/pipe.c| 25 ++---
 2 files changed, 23 insertions(+), 4 deletions(-)

diff --git a/src/bin/lttng-sessiond/notification-thread.c 
b/src/bin/lttng-sessiond/notification-thread.c
index c47b365..3ef7df1 100644
--- a/src/bin/lttng-sessiond/notification-thread.c
+++ b/src/bin/lttng-sessiond/notification-thread.c
@@ -96,7 +96,7 @@ struct notification_thread_handle 
*notification_thread_handle_create(
goto end;
}
 
-   event_pipe = lttng_pipe_open(O_CLOEXEC);
+   event_pipe = lttng_pipe_open(FD_CLOEXEC);
if (!event_pipe) {
ERR("event_pipe creation");
goto error;
diff --git a/src/common/pipe.c b/src/common/pipe.c
index 4220a40..4fe45ef 100644
--- a/src/common/pipe.c
+++ b/src/common/pipe.c
@@ -154,9 +154,28 @@ static int _pipe_set_flags(struct lttng_pipe *pipe, int 
flags)
}
 
for (i = 0; i < 2; i++) {
-   ret = fcntl(pipe->fd[i], F_SETFD, flags);
-   if (ret < 0) {
-   PERROR("fcntl lttng pipe %d", flags);
+   if (flags & O_NONBLOCK) {
+   ret = fcntl(pipe->fd[i], F_SETFL, O_NONBLOCK);
+   if (ret < 0) {
+   PERROR("fcntl lttng pipe %d", flags);
+   goto end;
+   }
+   }
+   if (flags & FD_CLOEXEC) {
+   ret = fcntl(pipe->fd[i], F_SETFD, FD_CLOEXEC);
+   if (ret < 0) {
+   PERROR("fcntl lttng pipe %d", flags);
+   goto end;
+   }
+   }
+   /*
+* We only check for O_NONBLOCK or FD_CLOEXEC, if another flag 
is
+* needed, we can add it, but for now just make sure we don't 
make
+* mistakes with the parameters we pass.
+*/
+   if (!(flags & O_NONBLOCK) && !(flags & FD_CLOEXEC)) {
+   fprintf(stderr, "Unsupported flag\n");
+   ret = -1;
goto end;
}
}
-- 
2.7.4

___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] [PATCH urcu] Tests fix: errors in shell scripts

[Mathieu Desnoyers]

Merged into master, thanks!

Mathieu

- On Jul 26, 2017, at 1:31 PM, Michael Jeanson mjean...@efficios.com wrote:

> Fix all shellcheck errors in the test scripts, switch to posix
> compatible syntax. Remove duplicated code already included in common.sh.
> Call the tap.sh cleanup code from our exit trap instead of overriding
> it.
> 
> Signed-off-by: Michael Jeanson 
> ---
> tests/benchmark/common.sh | 38 +--
> tests/benchmark/run-urcu-tests.sh | 55 ++-
> tests/benchmark/runhash.sh| 47 ++---
> tests/benchmark/runtests-batch.sh | 29 +++--
> tests/benchmark/runtests.sh   | 27 +++
> 5 files changed, 84 insertions(+), 112 deletions(-)
> 
> diff --git a/tests/benchmark/common.sh b/tests/benchmark/common.sh
> index 1f5f07f..4dbc567 100755
> --- a/tests/benchmark/common.sh
> +++ b/tests/benchmark/common.sh
> @@ -2,25 +2,35 @@
> # This file is meant to be sourced from other tests scripts.
> #
> 
> -if [ -x "$URCU_TEST_TIME_BIN" ]; then
> - test_time_bin="$URCU_TEST_TIME_BIN"
> -elif [ -x "/usr/bin/time" ]; then
> - test_time_bin="/usr/bin/time"
> -else
> - test_time_bin=""
> -fi
> +cleanup() {
> + if [ x"$TMPFILE" != "x" ]; then
> + rm -f "$TMPFILE"
> + fi
> 
> -function cleanup()
> -{
> -if [ x"$tmpfile" != x"" ]; then
> -rm -f $tmpfile
> -fi
> + # Call the tap.sh exit cleanup code
> + _exit
> }
> 
> -function xseq () {
> +xseq() {
>   i=$1
>   while [[ "$i" -le "$2" ]]; do
>   echo "$i"
> - i=$(expr $i + 1)
> + i=$(( i + 1 ))
>   done
> }
> +
> +# Set TEST_TIME_BIN
> +if [ -x "$URCU_TEST_TIME_BIN" ]; then
> + TEST_TIME_BIN="$URCU_TEST_TIME_BIN"
> +elif [ -x "/usr/bin/time" ]; then
> + TEST_TIME_BIN="/usr/bin/time"
> +else
> + TEST_TIME_BIN=""
> +fi
> +export TEST_TIME_BIN
> +
> +# Create a temporary file for tests output
> +TMPFILE=$(mktemp)
> +
> +# Set traps to delete the temporary file on exit
> +trap cleanup EXIT
> diff --git a/tests/benchmark/run-urcu-tests.sh
> b/tests/benchmark/run-urcu-tests.sh
> index 1df988a..751514a 100755
> --- a/tests/benchmark/run-urcu-tests.sh
> +++ b/tests/benchmark/run-urcu-tests.sh
> @@ -3,7 +3,7 @@
> #first parameter: seconds per test
> DURATION=$1
> 
> -if [ "x$DURATION" = "x" ]; then
> +if [ "x${DURATION}" = "x" ]; then
>   echo "usage: $0 [DURATION]"
>   exit 1
> fi
> @@ -30,16 +30,13 @@ fi
> # fraction: 15 * 29 =
> # scalabilit NUM_CPUS * 15
> # reader 15 * 23 =
> -NUM_TESTS=$(( 19 + 435 + ( ${NUM_CPUS} * 15 ) + 345 ))
> +NUM_TESTS=$(( 19 + 435 + ( NUM_CPUS * 15 ) + 345 ))
> 
> plan_tests${NUM_TESTS}
> 
> #run all tests
> diag "Executing URCU tests"
> 
> -tmpfile=
> -trap cleanup SIGINT SIGTERM EXIT
> -tmpfile=$(mktemp)
> 
> #extra options, e.g. for setting affinity on even CPUs :
> #EXTRA_OPTS=$(for a in $(seq 0 2 127); do echo -n "-a ${a} "; done)
> @@ -67,16 +64,16 @@ BATCH_ARRAY="1 2 4 8 16 32 64 128 256 512 1024 2048 4096
> 8192 16384 32768 65536
>131072 262144"
> BATCH_TEST_ARRAY="test_urcu_gc"
> 
> -NR_WRITERS=$((${NUM_CPUS} / 2))
> +NR_WRITERS=$((NUM_CPUS / 2))
> +NR_READERS=$((NUM_CPUS - NR_WRITERS))
> 
> -NR_READERS=$((${NUM_CPUS} - ${NR_WRITERS}))
> for BATCH_SIZE in ${BATCH_ARRAY}; do
>   for TEST in ${BATCH_TEST_ARRAY}; do
> - okx $test_time_bin ./${TEST} ${NR_READERS} ${NR_WRITERS} 
> ${DURATION} \
> - -d 0 -b ${BATCH_SIZE} ${EXTRA_OPTS} 2>${tmpfile}
> - cat $tmpfile | while read line; do
> - echo "# $line"
> - done
> + okx ${TEST_TIME_BIN} ./"${TEST}" "${NR_READERS}" "${NR_WRITERS}"
> "${DURATION}" \
> + -d 0 -b "${BATCH_SIZE}" ${EXTRA_OPTS} 2>"${TMPFILE}"
> + while read line; do
> + echo "## $line"
> + done <"${TMPFILE}"
>   done
> done
> 
> @@ -93,16 +90,16 @@ diag "Executing update fraction test"
> WDELAY_ARRAY="0 1 2 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 32768
>   65536 131072 262144 524288 1048576 2097152 4194304 8388608
>   16777216 33554432 67108864 134217728"
> -NR_WRITERS=$((${NUM_CPUS} / 2))
> +NR_WRITERS=$((NUM_CPUS / 2))
> +NR_READERS=$((NUM_CPUS - NR_WRITERS))
> 
> -NR_READERS=$((${NUM_CPUS} - ${NR_WRITERS}))
> for WDELAY in ${WDELAY_ARRAY}; do
>   for TEST in ${TEST_ARRAY}; do
> - okx $test_time_bin ./${TEST} ${NR_READERS} ${NR_WRITERS} 
> ${DURATION} \
> - -d ${WDELAY} ${EXTRA_OPTS} 2>$tmpfile
> - cat $tmpfile | while read line; do
> - echo "# $line"
> - done
> + okx ${TEST_TIME_BIN} ./"${TEST}" "${NR_READERS}" "${NR_WRITERS}"
> "${DURATION}" \
> + -d "${WDELAY}" ${EXTRA_OPTS} 2>"${TMPFILE}"
> + while read 

Re: [lttng-dev] [PATCH urcu v2] Fix: don't use overlapping mmap mappings on Cygwin

[Mathieu Desnoyers]

Merged into master, 0.10, 0.9, 0.8, thanks!

Mathieu

- On Jul 28, 2017, at 11:51 AM, Michael Jeanson mjean...@efficios.com wrote:

> The allocation scheme used by the mmap based RCU hash table is to make a
> large unaccessible mapping to reserve memory without allocating it.
> Then smaller chunks are allocated by overlapping read/write mappings which
> do allocate memory. Deallocation is done by an overlapping unaccessible
> mapping.
> 
> This scheme was tested on Linux, macOS and Solaris. However, on Cygwin the
> mmap wrapper is based on the Windows NtMapViewOfSection API which doesn't
> support overlapping mappings.
> 
> An alternative to the overlapping mappings is to use mprotect to change the
> protection on chunks of the large mapping, read/write to allocate and none
> to deallocate. This works perfecty on Cygwin and Solaris but on Linux a
> call to madvise is also required to deallocate and it just doesn't work on
> macOS.
> 
> For this reason, we keep to original scheme on all platforms except Cygwin.
> 
> Signed-off-by: Michael Jeanson 
> ---
> src/rculfhash-mm-mmap.c | 61 +
> 1 file changed, 56 insertions(+), 5 deletions(-)
> 
> diff --git a/src/rculfhash-mm-mmap.c b/src/rculfhash-mm-mmap.c
> index 3cc3fa0..a8fadf0 100644
> --- a/src/rculfhash-mm-mmap.c
> +++ b/src/rculfhash-mm-mmap.c
> @@ -28,8 +28,30 @@
> #define MAP_ANONYMOUS MAP_ANON
> #endif
> 
> -/* reserve inaccessible memory space without allocation any memory */
> -static void *memory_map(size_t length)
> +/*
> + * The allocation scheme used by the mmap based RCU hash table is to make a
> + * large unaccessible mapping to reserve memory without allocating it.
> + * Then smaller chunks are allocated by overlapping read/write mappings which
> + * do allocate memory. Deallocation is done by an overlapping unaccessible
> + * mapping.
> + *
> + * This scheme was tested on Linux, macOS and Solaris. However, on Cygwin the
> + * mmap wrapper is based on the Windows NtMapViewOfSection API which doesn't
> + * support overlapping mappings.
> + *
> + * An alternative to the overlapping mappings is to use mprotect to change 
> the
> + * protection on chunks of the large mapping, read/write to allocate and none
> + * to deallocate. This works perfecty on Cygwin and Solaris but on Linux a
> + * call to madvise is also required to deallocate and it just doesn't work on
> + * macOS.
> + *
> + * For this reason, we keep to original scheme on all platforms except 
> Cygwin.
> + */
> +
> +
> +/* Reserve inaccessible memory space without allocating it */
> +static
> +void *memory_map(size_t length)
> {
>   void *ret = mmap(NULL, length, PROT_NONE,
>   MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
> @@ -38,7 +60,8 @@ static void *memory_map(size_t length)
>   return ret;
> }
> 
> -static void memory_unmap(void *ptr, size_t length)
> +static
> +void memory_unmap(void *ptr, size_t length)
> {
>   int ret __attribute__((unused));
> 
> @@ -47,7 +70,33 @@ static void memory_unmap(void *ptr, size_t length)
>   assert(ret == 0);
> }
> 
> -static void memory_populate(void *ptr, size_t length)
> +#ifdef __CYGWIN__
> +/* Set protection to read/write to allocate a memory chunk */
> +static
> +void memory_populate(void *ptr, size_t length)
> +{
> + int ret __attribute__((unused));
> +
> + ret = mprotect(ptr, length, PROT_READ | PROT_WRITE);
> +
> + assert(!ret);
> +}
> +
> +/* Set protection to none to deallocate a memory chunk */
> +static
> +void memory_discard(void *ptr, size_t length)
> +{
> + int ret __attribute__((unused));
> +
> + ret = mprotect(ptr, length, PROT_NONE);
> +
> + assert(!ret);
> +}
> +
> +#else /* __CYGWIN__ */
> +
> +static
> +void memory_populate(void *ptr, size_t length)
> {
>   void *ret __attribute__((unused));
> 
> @@ -61,7 +110,8 @@ static void memory_populate(void *ptr, size_t length)
>  * Discard garbage memory and avoid system save it when try to swap it out.
>  * Make it still reserved, inaccessible.
>  */
> -static void memory_discard(void *ptr, size_t length)
> +static
> +void memory_discard(void *ptr, size_t length)
> {
>   void *ret __attribute__((unused));
> 
> @@ -70,6 +120,7 @@ static void memory_discard(void *ptr, size_t length)
> 
>   assert(ret == ptr);
> }
> +#endif /* __CYGWIN__ */
> 
> static
> void cds_lfht_alloc_bucket_table(struct cds_lfht *ht, unsigned long order)
> --
> 2.7.4

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] [PATCH lttng-tools] Use initial-exec TLS model

[Mathieu Desnoyers]

Please disregard this patch. We need to investigate glibc's behavior with
initial-exec's TLS initialization further before doing those changes.

Thanks,

Mathieu

- On Oct 19, 2017, at 10:51 AM, Mathieu Desnoyers 
mathieu.desnoy...@efficios.com wrote:

> The initial-exec TLS model is async-signal-safe, whereas the
> global-dynamic is not. This is especially important for the logging
> facility, because the first time a thread touches the TLS could be
> from a signal handler (correctness).
> 
> Moreover, IE is faster than GD model.
> 
> Also change the health state to the IE model, just in case we end up
> putting health progress reporting statements in signal handlers in the
> future. Given that we link against, but don't dlopen, that library, it
> is not using any of the IE backup pool, so there is no good reason for
> using the GD model.
> 
> Link: https://docs.oracle.com/cd/E23824_01/html/819-0690/chapter8-20.html
> Link:
> https://gcc.gnu.org/onlinedocs/gcc/Common-Variable-Attributes.html#Common-Variable-Attributes
> Signed-off-by: Mathieu Desnoyers 
> ---
> configure.ac   | 7 ++-
> src/common/error.c | 2 +-
> src/common/health/health.c | 2 +-
> 3 files changed, 8 insertions(+), 3 deletions(-)
> 
> diff --git a/configure.ac b/configure.ac
> index 016c56ec..77316a28 100644
> --- a/configure.ac
> +++ b/configure.ac
> @@ -440,7 +440,7 @@ AC_SUBST(UUID_LIBS)
> AC_CHECK_FUNC([clock_gettime], [AC_DEFINE_UNQUOTED([LTTNG_HAVE_CLOCK_GETTIME],
> 1, [Has clock_gettime() support.])])
> 
> # URCU library version needed or newer
> -m4_define([WRONG_LIBURCU_MSG], [Userspace RCU (liburcu) >= 0.9.0 is needed])
> +m4_define([WRONG_LIBURCU_MSG], [Userspace RCU (liburcu) >= 0.11.0 is needed])
> 
> # Check liburcu needed function calls
> AC_CHECK_DECL([cds_list_add], [],
> @@ -470,6 +470,11 @@ AC_CHECK_DECL([urcu_ref_get_unless_zero], [],
> [AC_MSG_ERROR([WRONG_LIBURCU_MSG])], [[#include ]]
> )
> 
> +#Macro added in urcu 0.11.0
> +AC_CHECK_DECL([DEFINE_URCU_TLS_IE], [],
> + [AC_MSG_ERROR([WRONG_LIBURCU_MSG])], [[#include ]]
> +)
> +
> # Check for libkmod, it will be auto-neabled if found but won't fail if it's
> not,
> # it can be explicitly disabled with --without-kmod
> AH_TEMPLATE([HAVE_KMOD], [Define if you have kmod support])
> diff --git a/src/common/error.c b/src/common/error.c
> index 2215886d..5c45fc70 100644
> --- a/src/common/error.c
> +++ b/src/common/error.c
> @@ -36,7 +36,7 @@
> static int lttng_opt_abort_on_error = -1;
> 
> /* TLS variable that contains the time of one single log entry. */
> -DEFINE_URCU_TLS(struct log_time, error_log_time);
> +DEFINE_URCU_TLS_IE(struct log_time, error_log_time);
> 
> LTTNG_HIDDEN
> const char *log_add_time(void)
> diff --git a/src/common/health/health.c b/src/common/health/health.c
> index 830b6f0e..b87c70b4 100644
> --- a/src/common/health/health.c
> +++ b/src/common/health/health.c
> @@ -54,7 +54,7 @@ struct health_app {
> };
> 
> /* Define TLS health state. */
> -DEFINE_URCU_TLS(struct health_state, health_state);
> +DEFINE_URCU_TLS_IE(struct health_state, health_state);
> 
> /*
>  * Initialize health check subsytem.
> --
> 2.11.0

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
___
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev