[jira] [Commented] (MESOS-920) Set GLOG_drop_log_memory=false in environment prior to logging initialization.
[
https://issues.apache.org/jira/browse/MESOS-920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117765#comment-16117765
]
Benjamin Mahler commented on MESOS-920:
---
Looks like a fix was upstreamed, possibly making it safe to use use the
drop_log_memory flag:
https://github.com/google/glog/pull/145
> Set GLOG_drop_log_memory=false in environment prior to logging initialization.
> --
>
> Key: MESOS-920
> URL: https://issues.apache.org/jira/browse/MESOS-920
> Project: Mesos
> Issue Type: Improvement
>Affects Versions: 0.15.0, 0.16.0
>Reporter: Benjamin Mahler
>Assignee: Kapil Arya
>Priority: Blocker
> Labels: tech-debt
> Fix For: 0.24.2, 0.25.1, 0.26.1, 0.27.0
>
>
> We've observed issues where the masters are slow to respond. Two perf traces
> collected while the masters were slow to respond:
> {noformat}
> 25.84% [kernel][k] default_send_IPI_mask_sequence_phys
> 20.44% [kernel][k] native_write_msr_safe
> 4.54% [kernel][k] _raw_spin_lock
> 2.95% libc-2.5.so [.] _int_malloc
> 1.82% libc-2.5.so [.] malloc
> 1.55% [kernel][k] apic_timer_interrupt
> 1.36% libc-2.5.so [.] _int_free
> {noformat}
> {noformat}
> 29.03% [kernel][k] default_send_IPI_mask_sequence_phys
> 9.64% [kernel][k] _raw_spin_lock
> 7.38% [kernel][k] native_write_msr_safe
> 2.43% libc-2.5.so [.] _int_malloc
> 2.05% libc-2.5.so [.] _int_free
> 1.67% [kernel][k] apic_timer_interrupt
> 1.58% libc-2.5.so [.] malloc
> {noformat}
> These have been found to be attributed to the posix_fadvise calls made by
> glog. We can disable these via the environment:
> {noformat}
> GLOG_DEFINE_bool(drop_log_memory, true, "Drop in-memory buffers of log
> contents. "
> "Logs can grow very quickly and they are rarely read before
> they "
> "need to be evicted from memory. Instead, drop them from
> memory "
> "as soon as they are flushed to disk.");
> {noformat}
> {code}
> if (FLAGS_drop_log_memory) {
> if (file_length_ >= logging::kPageSize) {
> // don't evict the most recent page
> uint32 len = file_length_ & ~(logging::kPageSize - 1);
> posix_fadvise(fileno(file_), 0, len, POSIX_FADV_DONTNEED);
> }
> }
> {code}
> We should set GLOG_drop_log_memory=false prior to making our call to
> google::InitGoogleLogging, to avoid others running into this issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MESOS-920) Set GLOG_drop_log_memory=false in environment prior to logging initialization.
[
https://issues.apache.org/jira/browse/MESOS-920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15116430#comment-15116430
]
Neil Conway commented on MESOS-920:
---
FYI, I opened https://github.com/google/glog/issues/84 for this issue with glog
upstream.
> Set GLOG_drop_log_memory=false in environment prior to logging initialization.
> --
>
> Key: MESOS-920
> URL: https://issues.apache.org/jira/browse/MESOS-920
> Project: Mesos
> Issue Type: Improvement
> Components: technical debt
>Affects Versions: 0.15.0, 0.16.0
>Reporter: Benjamin Mahler
>Assignee: Kapil Arya
>Priority: Blocker
> Labels: mesosphere
>
> We've observed issues where the masters are slow to respond. Two perf traces
> collected while the masters were slow to respond:
> {noformat}
> 25.84% [kernel][k] default_send_IPI_mask_sequence_phys
> 20.44% [kernel][k] native_write_msr_safe
> 4.54% [kernel][k] _raw_spin_lock
> 2.95% libc-2.5.so [.] _int_malloc
> 1.82% libc-2.5.so [.] malloc
> 1.55% [kernel][k] apic_timer_interrupt
> 1.36% libc-2.5.so [.] _int_free
> {noformat}
> {noformat}
> 29.03% [kernel][k] default_send_IPI_mask_sequence_phys
> 9.64% [kernel][k] _raw_spin_lock
> 7.38% [kernel][k] native_write_msr_safe
> 2.43% libc-2.5.so [.] _int_malloc
> 2.05% libc-2.5.so [.] _int_free
> 1.67% [kernel][k] apic_timer_interrupt
> 1.58% libc-2.5.so [.] malloc
> {noformat}
> These have been found to be attributed to the posix_fadvise calls made by
> glog. We can disable these via the environment:
> {noformat}
> GLOG_DEFINE_bool(drop_log_memory, true, "Drop in-memory buffers of log
> contents. "
> "Logs can grow very quickly and they are rarely read before
> they "
> "need to be evicted from memory. Instead, drop them from
> memory "
> "as soon as they are flushed to disk.");
> {noformat}
> {code}
> if (FLAGS_drop_log_memory) {
> if (file_length_ >= logging::kPageSize) {
> // don't evict the most recent page
> uint32 len = file_length_ & ~(logging::kPageSize - 1);
> posix_fadvise(fileno(file_), 0, len, POSIX_FADV_DONTNEED);
> }
> }
> {code}
> We should set GLOG_drop_log_memory=false prior to making our call to
> google::InitGoogleLogging, to avoid others running into this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-920) Set GLOG_drop_log_memory=false in environment prior to logging initialization.
[
https://issues.apache.org/jira/browse/MESOS-920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15114785#comment-15114785
]
Jian Qiu commented on MESOS-920:
When I run make check on OSX, it appears a error message.
{code}
./mesos-tests
dyld: Symbol not found: __ZN3fLB21FLAGS_drop_log_memoryE
Referenced from:
/Users/qiujian/Documents/mesos/build/src/.libs/libmesos-0.27.0.dylib
Expected in: flat namespace
in /Users/qiujian/Documents/mesos/build/src/.libs/libmesos-0.27.0.dylib
make[3]: *** [check-local] Trace/BPT trap: 5
make[2]: *** [check-am] Error 2
make[1]: *** [check] Error 2
make: *** [check-recursive] Error 1
{code}
Seems to be related to this ticket?
> Set GLOG_drop_log_memory=false in environment prior to logging initialization.
> --
>
> Key: MESOS-920
> URL: https://issues.apache.org/jira/browse/MESOS-920
> Project: Mesos
> Issue Type: Improvement
> Components: technical debt
>Affects Versions: 0.15.0, 0.16.0
>Reporter: Benjamin Mahler
>Assignee: Kapil Arya
>Priority: Blocker
> Labels: mesosphere
>
> We've observed issues where the masters are slow to respond. Two perf traces
> collected while the masters were slow to respond:
> {noformat}
> 25.84% [kernel][k] default_send_IPI_mask_sequence_phys
> 20.44% [kernel][k] native_write_msr_safe
> 4.54% [kernel][k] _raw_spin_lock
> 2.95% libc-2.5.so [.] _int_malloc
> 1.82% libc-2.5.so [.] malloc
> 1.55% [kernel][k] apic_timer_interrupt
> 1.36% libc-2.5.so [.] _int_free
> {noformat}
> {noformat}
> 29.03% [kernel][k] default_send_IPI_mask_sequence_phys
> 9.64% [kernel][k] _raw_spin_lock
> 7.38% [kernel][k] native_write_msr_safe
> 2.43% libc-2.5.so [.] _int_malloc
> 2.05% libc-2.5.so [.] _int_free
> 1.67% [kernel][k] apic_timer_interrupt
> 1.58% libc-2.5.so [.] malloc
> {noformat}
> These have been found to be attributed to the posix_fadvise calls made by
> glog. We can disable these via the environment:
> {noformat}
> GLOG_DEFINE_bool(drop_log_memory, true, "Drop in-memory buffers of log
> contents. "
> "Logs can grow very quickly and they are rarely read before
> they "
> "need to be evicted from memory. Instead, drop them from
> memory "
> "as soon as they are flushed to disk.");
> {noformat}
> {code}
> if (FLAGS_drop_log_memory) {
> if (file_length_ >= logging::kPageSize) {
> // don't evict the most recent page
> uint32 len = file_length_ & ~(logging::kPageSize - 1);
> posix_fadvise(fileno(file_), 0, len, POSIX_FADV_DONTNEED);
> }
> }
> {code}
> We should set GLOG_drop_log_memory=false prior to making our call to
> google::InitGoogleLogging, to avoid others running into this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-920) Set GLOG_drop_log_memory=false in environment prior to logging initialization.
[
https://issues.apache.org/jira/browse/MESOS-920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15114786#comment-15114786
]
Kapil Arya commented on MESOS-920:
--
Looks like a platform issue. I'll create a RR right away.
> Set GLOG_drop_log_memory=false in environment prior to logging initialization.
> --
>
> Key: MESOS-920
> URL: https://issues.apache.org/jira/browse/MESOS-920
> Project: Mesos
> Issue Type: Improvement
> Components: technical debt
>Affects Versions: 0.15.0, 0.16.0
>Reporter: Benjamin Mahler
>Assignee: Kapil Arya
>Priority: Blocker
> Labels: mesosphere
>
> We've observed issues where the masters are slow to respond. Two perf traces
> collected while the masters were slow to respond:
> {noformat}
> 25.84% [kernel][k] default_send_IPI_mask_sequence_phys
> 20.44% [kernel][k] native_write_msr_safe
> 4.54% [kernel][k] _raw_spin_lock
> 2.95% libc-2.5.so [.] _int_malloc
> 1.82% libc-2.5.so [.] malloc
> 1.55% [kernel][k] apic_timer_interrupt
> 1.36% libc-2.5.so [.] _int_free
> {noformat}
> {noformat}
> 29.03% [kernel][k] default_send_IPI_mask_sequence_phys
> 9.64% [kernel][k] _raw_spin_lock
> 7.38% [kernel][k] native_write_msr_safe
> 2.43% libc-2.5.so [.] _int_malloc
> 2.05% libc-2.5.so [.] _int_free
> 1.67% [kernel][k] apic_timer_interrupt
> 1.58% libc-2.5.so [.] malloc
> {noformat}
> These have been found to be attributed to the posix_fadvise calls made by
> glog. We can disable these via the environment:
> {noformat}
> GLOG_DEFINE_bool(drop_log_memory, true, "Drop in-memory buffers of log
> contents. "
> "Logs can grow very quickly and they are rarely read before
> they "
> "need to be evicted from memory. Instead, drop them from
> memory "
> "as soon as they are flushed to disk.");
> {noformat}
> {code}
> if (FLAGS_drop_log_memory) {
> if (file_length_ >= logging::kPageSize) {
> // don't evict the most recent page
> uint32 len = file_length_ & ~(logging::kPageSize - 1);
> posix_fadvise(fileno(file_), 0, len, POSIX_FADV_DONTNEED);
> }
> }
> {code}
> We should set GLOG_drop_log_memory=false prior to making our call to
> google::InitGoogleLogging, to avoid others running into this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-920) Set GLOG_drop_log_memory=false in environment prior to logging initialization.
[
https://issues.apache.org/jira/browse/MESOS-920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15114790#comment-15114790
]
Kapil Arya commented on MESOS-920:
--
Here is the RR: https://reviews.apache.org/r/42704/
[~jieyu]: Can you take a look and see if this fixes the build issue? I don't
have a OS X machine handy.
Thanks!
> Set GLOG_drop_log_memory=false in environment prior to logging initialization.
> --
>
> Key: MESOS-920
> URL: https://issues.apache.org/jira/browse/MESOS-920
> Project: Mesos
> Issue Type: Improvement
> Components: technical debt
>Affects Versions: 0.15.0, 0.16.0
>Reporter: Benjamin Mahler
>Assignee: Kapil Arya
>Priority: Blocker
> Labels: mesosphere
>
> We've observed issues where the masters are slow to respond. Two perf traces
> collected while the masters were slow to respond:
> {noformat}
> 25.84% [kernel][k] default_send_IPI_mask_sequence_phys
> 20.44% [kernel][k] native_write_msr_safe
> 4.54% [kernel][k] _raw_spin_lock
> 2.95% libc-2.5.so [.] _int_malloc
> 1.82% libc-2.5.so [.] malloc
> 1.55% [kernel][k] apic_timer_interrupt
> 1.36% libc-2.5.so [.] _int_free
> {noformat}
> {noformat}
> 29.03% [kernel][k] default_send_IPI_mask_sequence_phys
> 9.64% [kernel][k] _raw_spin_lock
> 7.38% [kernel][k] native_write_msr_safe
> 2.43% libc-2.5.so [.] _int_malloc
> 2.05% libc-2.5.so [.] _int_free
> 1.67% [kernel][k] apic_timer_interrupt
> 1.58% libc-2.5.so [.] malloc
> {noformat}
> These have been found to be attributed to the posix_fadvise calls made by
> glog. We can disable these via the environment:
> {noformat}
> GLOG_DEFINE_bool(drop_log_memory, true, "Drop in-memory buffers of log
> contents. "
> "Logs can grow very quickly and they are rarely read before
> they "
> "need to be evicted from memory. Instead, drop them from
> memory "
> "as soon as they are flushed to disk.");
> {noformat}
> {code}
> if (FLAGS_drop_log_memory) {
> if (file_length_ >= logging::kPageSize) {
> // don't evict the most recent page
> uint32 len = file_length_ & ~(logging::kPageSize - 1);
> posix_fadvise(fileno(file_), 0, len, POSIX_FADV_DONTNEED);
> }
> }
> {code}
> We should set GLOG_drop_log_memory=false prior to making our call to
> google::InitGoogleLogging, to avoid others running into this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-920) Set GLOG_drop_log_memory=false in environment prior to logging initialization.
[
https://issues.apache.org/jira/browse/MESOS-920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15114803#comment-15114803
]
Jian Qiu commented on MESOS-920:
It works for me, thanks!
> Set GLOG_drop_log_memory=false in environment prior to logging initialization.
> --
>
> Key: MESOS-920
> URL: https://issues.apache.org/jira/browse/MESOS-920
> Project: Mesos
> Issue Type: Improvement
> Components: technical debt
>Affects Versions: 0.15.0, 0.16.0
>Reporter: Benjamin Mahler
>Assignee: Kapil Arya
>Priority: Blocker
> Labels: mesosphere
>
> We've observed issues where the masters are slow to respond. Two perf traces
> collected while the masters were slow to respond:
> {noformat}
> 25.84% [kernel][k] default_send_IPI_mask_sequence_phys
> 20.44% [kernel][k] native_write_msr_safe
> 4.54% [kernel][k] _raw_spin_lock
> 2.95% libc-2.5.so [.] _int_malloc
> 1.82% libc-2.5.so [.] malloc
> 1.55% [kernel][k] apic_timer_interrupt
> 1.36% libc-2.5.so [.] _int_free
> {noformat}
> {noformat}
> 29.03% [kernel][k] default_send_IPI_mask_sequence_phys
> 9.64% [kernel][k] _raw_spin_lock
> 7.38% [kernel][k] native_write_msr_safe
> 2.43% libc-2.5.so [.] _int_malloc
> 2.05% libc-2.5.so [.] _int_free
> 1.67% [kernel][k] apic_timer_interrupt
> 1.58% libc-2.5.so [.] malloc
> {noformat}
> These have been found to be attributed to the posix_fadvise calls made by
> glog. We can disable these via the environment:
> {noformat}
> GLOG_DEFINE_bool(drop_log_memory, true, "Drop in-memory buffers of log
> contents. "
> "Logs can grow very quickly and they are rarely read before
> they "
> "need to be evicted from memory. Instead, drop them from
> memory "
> "as soon as they are flushed to disk.");
> {noformat}
> {code}
> if (FLAGS_drop_log_memory) {
> if (file_length_ >= logging::kPageSize) {
> // don't evict the most recent page
> uint32 len = file_length_ & ~(logging::kPageSize - 1);
> posix_fadvise(fileno(file_), 0, len, POSIX_FADV_DONTNEED);
> }
> }
> {code}
> We should set GLOG_drop_log_memory=false prior to making our call to
> google::InitGoogleLogging, to avoid others running into this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-920) Set GLOG_drop_log_memory=false in environment prior to logging initialization.
[
https://issues.apache.org/jira/browse/MESOS-920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15114799#comment-15114799
]
Neil Conway commented on MESOS-920:
---
That patch fixes the build issue for me.
> Set GLOG_drop_log_memory=false in environment prior to logging initialization.
> --
>
> Key: MESOS-920
> URL: https://issues.apache.org/jira/browse/MESOS-920
> Project: Mesos
> Issue Type: Improvement
> Components: technical debt
>Affects Versions: 0.15.0, 0.16.0
>Reporter: Benjamin Mahler
>Assignee: Kapil Arya
>Priority: Blocker
> Labels: mesosphere
>
> We've observed issues where the masters are slow to respond. Two perf traces
> collected while the masters were slow to respond:
> {noformat}
> 25.84% [kernel][k] default_send_IPI_mask_sequence_phys
> 20.44% [kernel][k] native_write_msr_safe
> 4.54% [kernel][k] _raw_spin_lock
> 2.95% libc-2.5.so [.] _int_malloc
> 1.82% libc-2.5.so [.] malloc
> 1.55% [kernel][k] apic_timer_interrupt
> 1.36% libc-2.5.so [.] _int_free
> {noformat}
> {noformat}
> 29.03% [kernel][k] default_send_IPI_mask_sequence_phys
> 9.64% [kernel][k] _raw_spin_lock
> 7.38% [kernel][k] native_write_msr_safe
> 2.43% libc-2.5.so [.] _int_malloc
> 2.05% libc-2.5.so [.] _int_free
> 1.67% [kernel][k] apic_timer_interrupt
> 1.58% libc-2.5.so [.] malloc
> {noformat}
> These have been found to be attributed to the posix_fadvise calls made by
> glog. We can disable these via the environment:
> {noformat}
> GLOG_DEFINE_bool(drop_log_memory, true, "Drop in-memory buffers of log
> contents. "
> "Logs can grow very quickly and they are rarely read before
> they "
> "need to be evicted from memory. Instead, drop them from
> memory "
> "as soon as they are flushed to disk.");
> {noformat}
> {code}
> if (FLAGS_drop_log_memory) {
> if (file_length_ >= logging::kPageSize) {
> // don't evict the most recent page
> uint32 len = file_length_ & ~(logging::kPageSize - 1);
> posix_fadvise(fileno(file_), 0, len, POSIX_FADV_DONTNEED);
> }
> }
> {code}
> We should set GLOG_drop_log_memory=false prior to making our call to
> google::InitGoogleLogging, to avoid others running into this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-920) Set GLOG_drop_log_memory=false in environment prior to logging initialization.
[
https://issues.apache.org/jira/browse/MESOS-920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15114041#comment-15114041
]
Neil Conway commented on MESOS-920:
---
Can we report this issue to glog upstream?
> Set GLOG_drop_log_memory=false in environment prior to logging initialization.
> --
>
> Key: MESOS-920
> URL: https://issues.apache.org/jira/browse/MESOS-920
> Project: Mesos
> Issue Type: Improvement
> Components: technical debt
>Affects Versions: 0.15.0, 0.16.0
>Reporter: Benjamin Mahler
>Assignee: Kapil Arya
>Priority: Blocker
> Labels: mesosphere
>
> We've observed issues where the masters are slow to respond. Two perf traces
> collected while the masters were slow to respond:
> {noformat}
> 25.84% [kernel][k] default_send_IPI_mask_sequence_phys
> 20.44% [kernel][k] native_write_msr_safe
> 4.54% [kernel][k] _raw_spin_lock
> 2.95% libc-2.5.so [.] _int_malloc
> 1.82% libc-2.5.so [.] malloc
> 1.55% [kernel][k] apic_timer_interrupt
> 1.36% libc-2.5.so [.] _int_free
> {noformat}
> {noformat}
> 29.03% [kernel][k] default_send_IPI_mask_sequence_phys
> 9.64% [kernel][k] _raw_spin_lock
> 7.38% [kernel][k] native_write_msr_safe
> 2.43% libc-2.5.so [.] _int_malloc
> 2.05% libc-2.5.so [.] _int_free
> 1.67% [kernel][k] apic_timer_interrupt
> 1.58% libc-2.5.so [.] malloc
> {noformat}
> These have been found to be attributed to the posix_fadvise calls made by
> glog. We can disable these via the environment:
> {noformat}
> GLOG_DEFINE_bool(drop_log_memory, true, "Drop in-memory buffers of log
> contents. "
> "Logs can grow very quickly and they are rarely read before
> they "
> "need to be evicted from memory. Instead, drop them from
> memory "
> "as soon as they are flushed to disk.");
> {noformat}
> {code}
> if (FLAGS_drop_log_memory) {
> if (file_length_ >= logging::kPageSize) {
> // don't evict the most recent page
> uint32 len = file_length_ & ~(logging::kPageSize - 1);
> posix_fadvise(fileno(file_), 0, len, POSIX_FADV_DONTNEED);
> }
> }
> {code}
> We should set GLOG_drop_log_memory=false prior to making our call to
> google::InitGoogleLogging, to avoid others running into this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
