Re: New committers?
On 09/02/2012 08:42 AM, Jim Jagielski wrote: On Sep 2, 2012, at 12:02 AM, Martin Sebormse...@gmail.com wrote: On 08/31/2012 02:38 PM, Liviu Nicoara wrote: My input below. On 08/31/12 09:42, Wojciech Meyer wrote: The two significant ones (as far as I can understand): - as I heard from Christopher Bergström that it's hard to push the stdcxx to FreeBSD ports repository (I can understand it and that sounds pretty bad, if that's the case then the board should consider re-licensing as advised; I agree in general it's a hard decision for the board, but imagine the project would benefit, IANAL tho) Christopher's wishes and goals may be different from others'. I do not believe he has ulterior motives that would be detrimental to the rest of us but AFAICT he has not made a compelling argument. Even with one, it stretches the imagination what could possibly convince Apache to give up on STDCXX ownership. Just a point of clarity: the ASF doesn't own stdcxx. They license it from Rogue Wave which still has the copyright. (Not that anyone there realizes it or would know what to do with it if they did.) IIUC, that's also why they can't relicense it under different terms. FWIW, the ASF never requires copyright assignment... Just a copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works. Also, there is nothing in our bylaws or in the various license agreements that *exclude* the ASF ever releasing code not under the ALv2 (how could it? After all, that would prevent us from ever being able to move to ALv3). Again, we could, if we wanted to (which we never will, btw) actually make our code under the GPLv2... So what would it take to change the license to BSD as Christopher asks (IIUC)? Martin
Re: stdcxx Wikipedia update
On 09/02/2012 10:16 AM, Wojciech Meyer wrote: According to the wikipedia we are dead? The status was added by Wikipedia user 188.50.90.23. Based on past contributions, it looks like someone or something with a brief two-day interest in the status of Apache projects: http://en.wikipedia.org/wiki/Special:Contributions/188.50.90.23 http://en.wikipedia.org/wiki/Apache_C%2B%2B_Standard_Library are we? :-) If somebody support this, I will update the relevant field. I'm not sure what the status is meant to reflect. Clearly, even though the project has been mostly inactive for years, it's still an ASF project with a community of developers and users around it. Changing it to Active or whatever makes the most sense seems more appropriate at this stage than what's there now. Martin -- Wojciech Meyer http://danmey.org
Re: STDCXX-1056 [was: Re: STDCXX forks]
On Sep 4, 2012, at 10:34 AM, Stefan Teleman wrote:
On Tue, Sep 4, 2012 at 7:35 AM, Liviu Nicoara nikko...@hates.ms wrote:
By no means I am dismissing it. It is at the very least an issue of
efficiency. I will try an Intel C++ build on x86_64 at some point today.
What build type was that? I also notice that you have 4.2.1 in your path?
Are you building out of 4.2.1. tag? I built off 4.2.x branch which also has
support for custom timeouts (--soft-timeout) in rw_test.
Yes, this is 4.2.1 with all the Solaris patches which can be applied
on Linux. The environment is:
# INFO (S1) (10 lines):
# TEXT:
# COMPILER: Intel C++, __INTEL_COMPILER = 1210,
__INTEL_COMPILER_BUILD_DATE = 20111011, __EDG_VERSION__ = 403
# ENVIRONMENT: pentiumpro running linux-elf (Fedora release 17 (Beefy
Miracle) (3.5.0-2.fc17.x86_64)) with glibc 2.15
# FILE: 22.locale.numpunct.mt.cpp
# COMPILED: Sep 4 2012, 09:11:36
# COMMENT: thread safety
# CLAUSE: lib.locale.numpunct
[ ... snip ... ]
With all the thread-safety Solaris patches for 1056 applied. The test
runs to completion:
# NOTE (S2) (5 lines):
# TEXT: executing locale -a /tmp/tmpfile-Cej8JU
# CLAUSE: lib.locale.numpunct
# FILE: process.cpp
# LINE: 276
# INFO (S1) (3 lines):
# TEXT: testing std::numpunctcharT with 8 threads, 20 iterations
each, in 32 locales { C aa_DJ aa_DJ.iso88591 aa_DJ.utf8
aa_ER aa_ER@saaho aa_ER.utf8 aa_ER.utf8@saaho aa_ET
aa_ET.utf8 af_ZA af_ZA.iso88591 af_ZA.utf8 am_ET
am_ET.utf8 an_ES an_ES.iso885915 an_ES.utf8 ar_AE
ar_AE.iso88596 ar_AE.utf8 ar_BH ar_BH.iso88596 ar_BH.utf8
ar_DZ ar_DZ.iso88596 ar_DZ.utf8 ar_EG ar_EG.iso88596
ar_EG.utf8 ar_IN ar_IN.utf8 }
# CLAUSE: lib.locale.numpunct
[ ... snip ... ]
# +---+--+--+--+
# | DIAGNOSTIC| ACTIVE | TOTAL | INACTIVE |
# +---+--+--+--+
# | (S1) INFO | 11 | 11 | 0% |
# | (S2) NOTE |1 |1 | 0% |
# | (S8) ERROR|0 |3 | 100% |
# | (S9) FATAL|0 |1 | 100% |
# +---+--+--+--+
real 2139.31
user 2406.09
sys 155.61
[steleman@darthvader][/src/steleman/programming/stdcxx-intel/stdcxx-4.2.1-thread-safe/build/tests][09/04/2012
9:50:14][1300]
These tests running to completion (with the patches applied) are 100%
reproducible on every single run, on Linux and Solaris (Intel and
SPARC).
That is good, right?
The timings are the output of /usr/bin/time -p.
Yes, ~40 minutes wall clock run time for one test is quite a lot. But
it runs to completion, with zero errors. And when taking into
consideration that there are 8 threads with 20 iterations, maybe
it's not that much. FWIW, SELinux is also fully enabled in my
environment.
Ok, so that clears (this build at least), maybe inefficient but runs to
completion with no crashes.
The timeouts can also be a symptom of race conditions/deadlocks or
attempting to access invalid thread stacks, or partially written data.
I would look at timeouts from the standpoint of the progression of the run: if
the run does not progress at all, as in threads are deadlocked, then I would
consider that a defect. If the run progresses, it is a mere inefficiency,
however bad.
It's very platform specific. On Solaris SPARC (either 32-bit or
64-bit) I never get timeouts, I always get either SEGV or SIGBUS. On
Intel it appears that timeouts are the prevailing symptom.
Could not get an Intel build off the ground on the account of the LIMITS.cpp
test not running to completion because of a possible compiler bug. I posted
earlier an account of it. Do you have a support account that allows posting bug
reports for Intel's C++ compiler?
I am not sure what an explicit run timeout would add, except hiding
the hang/deadlock/memory corruption problem behind a timeout.
The timeout option allowed me to run the program with a larger value, such that
the alarm would not trigger and allow the program to run to completion. Before
enlarging the timeout value, the individual tests would time out.
I did not have access to a Solaris box since I left Rogue Wave. Does Sun/Oracle
have any program that would allow one to test drive their compiler on a shared
box somewhere? I vaguely remember that HP had something like that a while ago.
Thanks!
Liviu
Re: STDCXX-1056 [was: Re: STDCXX forks]
I recall discussing this when you opened the defect but I'm not
sure what the outcome of the discussion was.
I looked at the code some more just now and I agree with you that
(at least the numpunct) facet isn't thread safe. The premise was
that we didn't need any locking because the facet never changes
after it's initialized, and the same data member can safely be
assigned multiple times.
The problem is that the string data members must be initialized
before they can be assigned, and assignment to the same member
must be guarded against concurrent accesses, there seems to be
no mechanism that guarantees that this will be true when the
same facet is used from within multiple threads.
For example, the function below tries to avoid re-initialization
of _C_grouping but the |= expression can be reordered either by
the compiler or by the hardware to complete before the assignment
to the member. The function also fails to protect the assignment
to the member from taking place simultaneously in multiple threads.
template class _CharT
inline string numpunct_CharT::grouping () const
{
if (!(_C_flags _RW::__rw_gr)) {
numpunct* const __self = _RWSTD_CONST_CAST (numpunct*, this);
// [try to] get the grouping first (may throw)
// then set a flag to avoid future initializations
__self-_C_grouping = do_grouping ();
__self-_C_flags|= _RW::__rw_gr;
}
return _C_grouping;
}
We need a fix but I don't think the one in the patch attached
to the issue is a good solution. Locking all objects of the
template specialization is far too coarse. Even having a lock
per object would kill iostream performance. Creating a deep
copy on return from each of the functions would also slow
things down noticeably.
The efficient solution must avoid locking in the common case
(i.e., after the facet has been initialized). It can avoid
locking around the POD data members altogether since those can
safely be assigned multiple times (on common hardware), but it
needs to ensure that the string data members are initialized
before they are assigned to and are assigned at most once
simultaneously. If we introduce a lock, it must be per object
and not per type.
With that, I would expect the following function to fix the
problem in the one above. Let me know what you think.
template class _CharT
inline string numpunct_CharT::grouping () const
{
if (!(_C_flags _RW::__rw_gr)) {
numpunct* const __self = _RWSTD_CONST_CAST (numpunct*, this);
_RWSTD_MT_GUARD (_C_mutex);
// [try to] get the grouping first (may throw)
// then set a flag to avoid future initializations
__self-_C_grouping = do_grouping ();
__self-_C_flags|= _RW::__rw_gr;
}
return _C_grouping;
}
Martin
On 09/04/2012 07:40 PM, Stefan Teleman wrote:
On Tue, Sep 4, 2012 at 9:15 PM, Liviu Nicoaranikko...@hates.ms wrote:
That is good, right?
Yes, it is good. :-)
Could not get an Intel build off the ground on the account of the LIMITS.cpp
test not running to completion because of a possible compiler bug. I posted
earlier an account of it. Do you have a support account that allows posting bug
reports for Intel's C++ compiler?
Oh, yes I remember that now, it happened to me too. I worked around it
by copying the LIMITS executable from a gcc build.
I don't have a support contract with Intel - I'm using the free
compilers at home. I'm guessing Intel must have an open forum for
Intel developers using their compilers, so that might be a way to
report the bug. But I haven't looked for it.
I did not have access to a Solaris box since I left Rogue Wave. Does Sun/Oracle
have any program that would allow one to test drive their compiler on a shared
box somewhere? I vaguely remember that HP had something like that a while ago.
I know Sun had such a program. Oracle I don't really know but my
educated guess is not a chance. ;-)
One way of testing with SunPro is to download the free version of
SunPro from Oracle, and use it on Linux (yes, SunPro exists for Linux
as well). It works very well on OpenSUSE and Ubuntu (tested and used
on both myself, debugged stdcxx on OpenSUSE with SunPro 12.2. Doesn't
work at all on Fedora 17 because of the TLS glibc bug (however it used
to on earlier Fedora releases).
And now for the latest about 1056:
I managed (after many, MANY tries) to get a run of
22.locale.numpunct.mt on Solaris SPARC 32-bit build (at work), and
without the thread-safety patches applied, and I hit the jackpot:
# INFO (S1) (10 lines):
# TEXT:
# COMPILER: SunPro, __SUNPRO_CC = 0x5100
# ENVIRONMENT: sparc-v8 running sunos-5.11
# FILE: 22.locale.numpunct.mt.cpp
# COMPILED: Sep 4 2012, 16:10:03
# COMMENT: thread safety
# CLAUSE: lib.locale.numpunct
# NOTE (S2) (5 lines):
# TEXT: executing /usr/bin/locale -a /var/tmp/tmpfile-MWa4bQ
# CLAUSE:
Re: STDCXX-1056 [was: Re: STDCXX forks]
On Tue, Sep 4, 2012 at 10:49 PM, Martin Sebor mse...@gmail.com wrote:
template class _CharT
inline string numpunct_CharT::grouping () const
{
if (!(_C_flags _RW::__rw_gr)) {
numpunct* const __self = _RWSTD_CONST_CAST (numpunct*, this);
_RWSTD_MT_GUARD (_C_mutex);
// [try to] get the grouping first (may throw)
// then set a flag to avoid future initializations
__self-_C_grouping = do_grouping ();
__self-_C_flags|= _RW::__rw_gr;
}
return _C_grouping;
}
That's what I wanted to do originally - use a per-object mutext.
Unfortunately the _C_mutex member in rw::__rw_synchronized is static:
struct __rw_synchronized
{
// static so that it takes up no space
static _RWSTD_EXPORT __rw_mutex _C_mutex;
void _C_lock () { }
void _C_unlock () { }
__rw_guard _C_guard () {
return __rw_guard (_C_mutex);
}
and __rw::rw_guard doesn't have an appropriate constructor.
Intel C++ complains about it too:
/src/steleman/programming/stdcxx-intel/stdcxx-4.2.1-thread-safe/include/loc/_numpunct.h(181):
error: no instance of constructor __rw::__rw_guard::__rw_guard
matches the argument list
argument types are: (const __rw::__rw_mutex)
_RWSTD_MT_GUARD (_C_mutex);
^
This works:
template class _CharT
inline string numpunct_CharT::grouping () const
{
if (!(_C_flags _RW::__rw_gr)) {
numpunct* const __self = _RWSTD_CONST_CAST (numpunct*, this);
_RWSTD_MT_STATIC_GUARD (_Type);
// [try to] get the grouping first (may throw)
// then set a flag to avoid future initializations
__self-_C_grouping = do_grouping ();
__self-_C_flags|= _RW::__rw_gr;
}
return _C_grouping;
}
Although I'm not sure of the performance implications difference
between _RWSTD_MT_STATIC_GUARD and _RWSTD_MT_CLASS_GUARD for this
particular problem. I'm going with nothing in real life. :-)
And even so, this is still not thread-safe:
Two different threads [ T1 and T2 ], seeking two different locales
[en_US.UTF-8 and ja_JP.UTF-8], enter std::numpunct_CharT::grouping()
at the same time - because they are running on two different cores.
They both test for
if (!(_C_flags _RW::__rw_gr))
and then -- assuming the expression above evaluates to true -- one of
them wins the mutex [T1], and the other one [T2] blocks on the mutex.
When T1 is done and exits the function, the grouping is set to
en_US.UTF-8 and the mutex is released. Now T2 acquires the mutex, and
proceeds to setting grouping to ja-JP.UTF-8. Woe on the caller running
from T1 who now thinks he got en_US.UTF-8, but instead he gets
ja_JP.UTF-8, which was duly set so by T2, but T1 had no clue about it
(remember, the std::string grouping _charT buffer is shared by the
caller from T1 and the caller from T2).
So at a minimum, the locking must happen before evaluating the
if (!(_C_flags _RW::__rw_gr))
expression.
This still doesn't solve what ends up being returned in grouping. If
we lock at the top of the function, then, when T2 acquires the mutex,
the test expression will evaluate to false. Therefore T2 will return
whatever is in grouping right now, which happens to be en_US.UTF-8 as
set by T1, when T2 really wanted ja_JP.UTF-8.
I really think the appropriate fix here -- which would address the
performance implications -- is more complex than this. I am thinking
about creating and using a (non-publicly accessible) internal locale
cache:
typedef std::mapstd::string, std::locale locale_cache;
where all the locales are stored fully initialized, on demand. There
is only one locale instantiation and initialization overhead cost per
locale. After a locale has been instantiated and placed into the
cache, the caller of any specfic locale gets a copy from the cache,
fully instantiated and initialized. But this breaks ABI, so I'm
thinking it's for stdcxx 5.
Thoughts?
--Stefan
--
Stefan Teleman
KDE e.V.
stefan.tele...@gmail.com
