Re: Regression problem, call for advice Re: Call for advice and testing of nss (and nspr) and intention to upload correction

2016-11-06 Thread Ola Lundqvist
Hi

The old nss version will be available in wheezy. The new version is
available in wheezy+security only. So it should be available already now
without me doing any changes.

This actually look like a problem for jessie too. See this post here:
http://forums.debian.net/viewtopic.php?f=6=129904=625402

But I have not checked that myself.

Anyway I'm going to put aside this issue for now. I have to save some hours
for front-desk work later.

// Ola

On 4 November 2016 at 17:07, Markus Koschany  wrote:

> On 04.11.2016 15:38, J. R. Okajima wrote:
> [...]
> > I'd suggest a variation of 4, which is
> > - keep the latest NSS pkgs as is, which is equivalent to your option 1.
> > - for the oldstable users who suffer from this problem (like me),
> >   provide the previous NSS pkgs so that they can downgrade as their own
> >   choice.
> > - "provide" here means just to put the previous versions somewhere on
> >   Debian site as "hold-pkgs-for-oldstable" or something.
>
> [...]
>
> You can always find a previous version of any Debian package at
> snapshot.debian.org, e.g
>
> http://snapshot.debian.org/package/nss/
>
>
>
>
>


-- 
 --- Inguza Technology AB --- MSc in Information Technology 
/  o...@inguza.comFolkebogatan 26\
|  o...@debian.org   654 68 KARLSTAD|
|  http://inguza.com/Mobile: +46 (0)70-332 1551 |
\  gpg/f.p.: 7090 A92B 18FE 7994 0C36 4FE4 18A1 B1CF 0FE5 3DD9  /
 ---


Re: Regression problem, call for advice Re: Call for advice and testing of nss (and nspr) and intention to upload correction

2016-11-04 Thread J. R. Okajima

Hello all,

Ola Lundqvist:
> As I can see it there are the following options:
> 1) Do nothing. Let it be like this. We have a regression problem but only
> for software that fork and use nss in several threads.
> 2) Try to reverse the library split. This is a non-trivial task.
> 3) Try to fix the dlopen problem. I have tried in many ways but always
> fail. If anyone have a really good idea about this, please let me know.
> 4) Reverse the whole nss update. I'm not 100% sure how to do that as we did
> a version update and it is hard to "downgrade". We can certainly fix the
> CVE that this update solved. It should not be too hard.
>
> What do you all think is the best option?

I'd suggest a variation of 4, which is
- keep the latest NSS pkgs as is, which is equivalent to your option 1.
- for the oldstable users who suffer from this problem (like me),
  provide the previous NSS pkgs so that they can downgrade as their own
  choice.
- "provide" here means just to put the previous versions somewhere on
  Debian site as "hold-pkgs-for-oldstable" or something.


> The investigation have taken a considerable amount of time so I do not want
> to continue with this unless you really think it is important.

I appricate your effort, Ola.


J. R. Okajima



Re: Regression problem, call for advice Re: Call for advice and testing of nss (and nspr) and intention to upload correction

2016-11-04 Thread Mike Hommey
On Fri, Nov 04, 2016 at 01:17:36PM +0100, Ola Lundqvist wrote:
> Hi all
> 
> I have now analyzed the problem and the problem is that libfreebl3.so have
> been split into a libfreebl3.so that is pre-loaded and a libfreeblpriv3.so
> that is dynamically loaded by libfreebl3.so. This works well in many
> situations but apparently not in google chrome. I guess this is because of
> some kind of forking mechanism.

Are you sure it's not caused by chromium's sandboxing?

Surely more recent versions of chromium can handle the split, and maybe
they had to do something for that to work. It would be worth looking at
the chromium source history to find out what they did.

Mike



Re: Regression problem, call for advice Re: Call for advice and testing of nss (and nspr) and intention to upload correction

2016-11-04 Thread Guido Günther
Hi Ola,

On Fri, Nov 04, 2016 at 01:17:36PM +0100, Ola Lundqvist wrote:
[..snip analysis..]
> As I can see it there are the following options:
> 1) Do nothing. Let it be like this. We have a regression problem but only
> for software that fork and use nss in several threads.
> 2) Try to reverse the library split. This is a non-trivial task.
> 3) Try to fix the dlopen problem. I have tried in many ways but always
> fail. If anyone have a really good idea about this, please let me know.
> 4) Reverse the whole nss update. I'm not 100% sure how to do that as we did
> a version update and it is hard to "downgrade". We can certainly fix the
> CVE that this update solved. It should not be too hard.
> 
> What do you all think is the best option?

I would neither do 4 (it's good to have newer nspr/nss, see #824872) or
2 (would deviate us from stretch, jessie and upstream). Given we don't
find other regressions it'd go for 3 or 1.

Although chromium is unsupported there might be people using it to
access "trusted" hosts for things that dont work with Firefix (at least
VCenter comes to mind).

Did you check what upstream chromium did when they updated nss? Maybe we
can cherry-pick a simple fix from there? If not this just leaves us with 1.

Cheers,
 -- Guido



Re: Regression problem, call for advice Re: Call for advice and testing of nss (and nspr) and intention to upload correction

2016-11-04 Thread Ola Lundqvist
Hi all

I have now analyzed the problem and the problem is that libfreebl3.so have
been split into a libfreebl3.so that is pre-loaded and a libfreeblpriv3.so
that is dynamically loaded by libfreebl3.so. This works well in many
situations but apparently not in google chrome. I guess this is because of
some kind of forking mechanism.

The library split was mentioned here:
https://bugzilla.redhat.com/show_bug.cgi?id=1165603

I have tried to patch so that the dlopen call is first tested
with RTLD_NOLOAD option, but that just give the result that it is not
loaded and then when loaded it returns null because it is already loaded (I
guess that is the reason).

I have also tried without RTLD_NOW option but then it does not work at all.

I have also tried to make static variables to check for the handle but
apparently they are not in the same memory space because it is not set the
second time.

I can see that the library is first loaded here:
[nssinit.c:556(nss_Init)] ENTER nss_Init(sql:/home/ola/.pki/nssdb, ...)

And then later (by another thready I guess) it is loaded by
[nssinit.c:556(nss_Init)] ENTER nss_Init(, ...)

>From the NSS code there is a function that should only be called once and
it is apparently called twice.

As I can see it there are the following options:
1) Do nothing. Let it be like this. We have a regression problem but only
for software that fork and use nss in several threads.
2) Try to reverse the library split. This is a non-trivial task.
3) Try to fix the dlopen problem. I have tried in many ways but always
fail. If anyone have a really good idea about this, please let me know.
4) Reverse the whole nss update. I'm not 100% sure how to do that as we did
a version update and it is hard to "downgrade". We can certainly fix the
CVE that this update solved. It should not be too hard.

What do you all think is the best option?

The investigation have taken a considerable amount of time so I do not want
to continue with this unless you really think it is important.
Also I think I need some help as I'm running load on time.

I have made a heavy patch for this that you can have if you are interested.
It just adds debug code.
The result of that is seen below.

(gdb) run
Starting program: /usr/lib/chromium/chromium --password-store=detect
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffeb8b1700 (LWP 19564)]
[1:1:1104/115422:ERROR:image_metadata_extractor.cc(106)] Couldn't load
libexif.
[New Thread 0x7fffeb0b0700 (LWP 19571)]
[New Thread 0x7fffe9e75700 (LWP 19572)]
[New Thread 0x7fffe9674700 (LWP 19573)]
[New Thread 0x77ea9700 (LWP 19574)]
[New Thread 0x77e88700 (LWP 19575)]
[New Thread 0x7fffe8e73700 (LWP 19576)]
[Thread 0x7fffe8e73700 (LWP 19576) exited]
[New Thread 0x7fffe8e73700 (LWP 19577)]
[New Thread 0x77e67700 (LWP 19578)]
[nssinit.c:556(nss_Init)] ENTER nss_Init(sql:/home/ola/.pki/nssdb, ...)
[genload.c:155(loader_LoadLibrary)] Enter libfreeblpriv3.so =>
/usr/lib/x86_64-linux-gnu/nss/libsoftokn3.so
[genload.c:108(loader_LoadLibInReferenceDir)] 'dlopen'
/usr/lib/x86_64-linux-gnu/nss/libfreeblpriv3.so
[genload.c:113(loader_LoadLibInReferenceDir)] It was not loaded before!
Loading now.
[pk11load.c:455(secmod_LoadPKCS11Module)] library == NULL
[nssinit.c:798(nss_Init)] RET Success
[New Thread 0x7fffe2ed4700 (LWP 19579)]
[New Thread 0x7fffe26d3700 (LWP 19580)]
[New Thread 0x7fffe1ed2700 (LWP 19581)]
[New Thread 0x7fffe16d1700 (LWP 19582)]
[New Thread 0x7fffe0ed0700 (LWP 19583)]
[New Thread 0x7fffe06cf700 (LWP 19584)]
[New Thread 0x7fffdfece700 (LWP 19585)]
[New Thread 0x7fffdf6cd700 (LWP 19586)]
[New Thread 0x7fffde6cb700 (LWP 19588)]
[New Thread 0x7fffddeca700 (LWP 19589)]
[New Thread 0x7fffdeecc700 (LWP 19587)]
[New Thread 0x7fffdd6c9700 (LWP 19591)]
[New Thread 0x7fffdcec8700 (LWP 19593)]
[New Thread 0x7fffd7fff700 (LWP 19595)]
[19590:19590:1104/115423:ERROR:sandbox_linux.cc(308)] InitializeSandbox()
called with multiple threads in process gpu-process
[New Thread 0x7fffd77fe700 (LWP 19603)]
[New Thread 0x7fffd6ffd700 (LWP 19604)]
[New Thread 0x7fffd67fc700 (LWP 19607)]
[nssinit.c:556(nss_Init)] ENTER nss_Init(, ...)
[genload.c:155(loader_LoadLibrary)] Enter libfreeblpriv3.so =>
/usr/lib/x86_64-linux-gnu/nss/libsoftokn3.so
[genload.c:108(loader_LoadLibInReferenceDir)] 'dlopen'
/usr/lib/x86_64-linux-gnu/nss/libfreeblpriv3.so
[genload.c:113(loader_LoadLibInReferenceDir)] It was not loaded before!
Loading now.
[genload.c:186(loader_LoadLibrary)] Loading failed: libfreeblpriv3.so
[loader.c:89(freebl_LoadDSO)] Other failure for name libfreeblpriv3.so
[loader.c:746(RNG_RNGInit)] !vector && PR_SUCCESS != freebl_RunLoaderOnce()
[pkcs11.c:2921(nsc_CommonInitialize)] RNG_RNGInit() -1
[loader.c:746(RNG_RNGInit)] !vector && PR_SUCCESS != freebl_RunLoaderOnce()
[pkcs11.c:2921(nsc_CommonInitialize)] RNG_RNGInit() -1
[pk11load.c:301(secmod_ModuleInit)] PK11_MapError 48

Re: Regression problem, call for advice Re: Call for advice and testing of nss (and nspr) and intention to upload correction

2016-11-02 Thread Ben Hutchings
On Wed, 2016-11-02 at 20:41 +0100, Jiří Jánský wrote:
> Hello all,
> there is still one thing, that is unclear for me. Chromium is security
> unsupported package. But does it also mean, that it is unsupported at all
> (can be non-function after install by apt-get install chromium)?
[...]

I don't think we've specifically discussed this yet.  That's what is
happening now.

Ben.

-- 
Ben Hutchings
The world is coming to an end.  Please log off.



signature.asc
Description: This is a digitally signed message part


Re: Regression problem, call for advice Re: Call for advice and testing of nss (and nspr) and intention to upload correction

2016-11-02 Thread Ola Lundqvist
Hi Jiri

As I understand it is not supported at all. It does not mean that it will
break automatically (as in the ruby-rails-2.3 example you gave) but we can
not maintain it in a good way. One of the main reason for it not being
supported is that the later versions do not even build on wheezy and hence
it is hard to maintain. There could be many reasons for a software to no
longer work. Environment changes is one example (as this nss update), but
there are other cases too, like that the service the software uses has
upgraded API or similar.

I'll make a new try to find out what the reason for this problem really is,
mainly in order to understand whether this can impact some other
application or not. But I will not spend too much time on it unless more
people from the LTS team think this (chrome not usable) is a serious
problem that must be fixed.

As I understand it, this is the situation:
- After the NSS upgrade chromium no longer work.
- Before the NSS upgrade chromium do work fine.
- Chromium has no Debian LTS support.
- Latest version of chrome can not install on wheezy (requires libstdc++6
(>= 4.8.0))

My suggestion is to upgrade to current Debian stable release, but there may
of course be things that prevent you from doing that.
An alternative is that you downgrade your libnss3 package and "hold" it so
it is not upgraded next time.
http://askubuntu.com/questions/18654/how-to-prevent-updating-of-a-specific-package

Best regards

// Ola

On 2 November 2016 at 20:41, Jiří Jánský  wrote:

> Hello all,
> there is still one thing, that is unclear for me. Chromium is security
> unsupported package. But does it also mean, that it is unsupported at all
> (can be non-function after install by apt-get install chromium)?
>
> I am aware, that wheezy chromium package is quite outdated for today web.
> But isn't this decision precedence for another similar packages? Security
> support for ruby-rails-2.3 was also ceased. Let's say, that I have
> application that is tested against that. I can still easily install it and
> run it under that, even if there are some described and non-fixed
> security vulnerabilities. And that wouldn't be true if some library, that
> it depand on would be upgraded to non-compatible version. I see it as a
> problem, that wasn't here in pre-lts verions. If I'll install woody, all
> packaged would be outdated, but fully function. That will not be true with
>  wheezy anymore (at least until libnss3 downgrade).
>
> Is there some debian policy regarding it?
>
> Regards
> Jiri Jansky
>
> P.S.: Current chrome-stable has unment dependencies in wheezy
>
>
> 2016-11-02 0:18 GMT+01:00 Ben Hutchings :
>
>> On Tue, 2016-11-01 at 23:37 +0100, Ola Lundqvist wrote:
>> > Hi Ben, Balint and others
>> >
>> > I'd like to have some advice on this regression.
>> >
>> > 1) Is this worth investigating?
>> >  - Chrome is not supported, however we have now made it to crash. Ben
>> > obviously like that but maybe others do not have the same opinion.
>>
>> I like that the outdated Debian package of chromium for wheezy is now
>> unusable.  If the current upstream version of Chromium (or Chrome) is
>> also broken then that *is* a problem - but I didn't think it could
>> still run on wheezy, or that it would use nss as a system library.
>>
>> > 2) Is this severe enough for me to revert the nss 3.26 upload?
>> [...]
>>
>> If *only* the outdated Debian package of chromium is affected, then
>> obviously I don't think it is.
>>
>> Ben.
>>
>> --
>> Ben Hutchings
>> Horngren's Observation:
>>Among economists, the real world is often a special
>> case.
>>
>>
>


-- 
 --- Inguza Technology AB --- MSc in Information Technology 
/  o...@inguza.comFolkebogatan 26\
|  o...@debian.org   654 68 KARLSTAD|
|  http://inguza.com/Mobile: +46 (0)70-332 1551 |
\  gpg/f.p.: 7090 A92B 18FE 7994 0C36 4FE4 18A1 B1CF 0FE5 3DD9  /
 ---


Re: Regression problem, call for advice Re: Call for advice and testing of nss (and nspr) and intention to upload correction

2016-11-02 Thread Jiří Jánský
Hello all,
there is still one thing, that is unclear for me. Chromium is security
unsupported package. But does it also mean, that it is unsupported at all
(can be non-function after install by apt-get install chromium)?

I am aware, that wheezy chromium package is quite outdated for today web.
But isn't this decision precedence for another similar packages? Security
support for ruby-rails-2.3 was also ceased. Let's say, that I have
application that is tested against that. I can still easily install it and
run it under that, even if there are some described and non-fixed
security vulnerabilities. And that wouldn't be true if some library, that
it depand on would be upgraded to non-compatible version. I see it as a
problem, that wasn't here in pre-lts verions. If I'll install woody, all
packaged would be outdated, but fully function. That will not be true with
 wheezy anymore (at least until libnss3 downgrade).

Is there some debian policy regarding it?

Regards
Jiri Jansky

P.S.: Current chrome-stable has unment dependencies in wheezy

2016-11-02 0:18 GMT+01:00 Ben Hutchings :

> On Tue, 2016-11-01 at 23:37 +0100, Ola Lundqvist wrote:
> > Hi Ben, Balint and others
> >
> > I'd like to have some advice on this regression.
> >
> > 1) Is this worth investigating?
> >  - Chrome is not supported, however we have now made it to crash. Ben
> > obviously like that but maybe others do not have the same opinion.
>
> I like that the outdated Debian package of chromium for wheezy is now
> unusable.  If the current upstream version of Chromium (or Chrome) is
> also broken then that *is* a problem - but I didn't think it could
> still run on wheezy, or that it would use nss as a system library.
>
> > 2) Is this severe enough for me to revert the nss 3.26 upload?
> [...]
>
> If *only* the outdated Debian package of chromium is affected, then
> obviously I don't think it is.
>
> Ben.
>
> --
> Ben Hutchings
> Horngren's Observation:
>Among economists, the real world is often a special
> case.
>
>


Re: Regression problem, call for advice Re: Call for advice and testing of nss (and nspr) and intention to upload correction

2016-11-02 Thread Ola Lundqvist
Hi Holger and Ben

Thanks for the feedback. I'll check whether upstream chrome or chromium can
be installed and is usable on wheezy.

If chrome or chromium is uninstallable on wheezy, then is this an issue? I
would see it as that, but maybe outside LTS scope.

// Ola

On 2 November 2016 at 00:50, Holger Levsen  wrote:

> On Tue, Nov 01, 2016 at 05:18:45PM -0600, Ben Hutchings wrote:
> > I like that the outdated Debian package of chromium for wheezy is now
> > unusable.  If the current upstream version of Chromium (or Chrome) is
> > also broken then that *is* a problem
>
> seconded. (and thanks for wording this so well, Ben.)
>
>
> --
> cheers,
> Holger
>



-- 
 --- Inguza Technology AB --- MSc in Information Technology 
/  o...@inguza.comFolkebogatan 26\
|  o...@debian.org   654 68 KARLSTAD|
|  http://inguza.com/Mobile: +46 (0)70-332 1551 |
\  gpg/f.p.: 7090 A92B 18FE 7994 0C36 4FE4 18A1 B1CF 0FE5 3DD9  /
 ---


Re: Regression problem, call for advice Re: Call for advice and testing of nss (and nspr) and intention to upload correction

2016-11-01 Thread Holger Levsen
On Tue, Nov 01, 2016 at 05:18:45PM -0600, Ben Hutchings wrote:
> I like that the outdated Debian package of chromium for wheezy is now
> unusable.  If the current upstream version of Chromium (or Chrome) is
> also broken then that *is* a problem

seconded. (and thanks for wording this so well, Ben.)


-- 
cheers,
Holger


signature.asc
Description: Digital signature


Re: Regression problem, call for advice Re: Call for advice and testing of nss (and nspr) and intention to upload correction

2016-11-01 Thread Ben Hutchings
On Tue, 2016-11-01 at 23:37 +0100, Ola Lundqvist wrote:
> Hi Ben, Balint and others
> 
> I'd like to have some advice on this regression.
> 
> 1) Is this worth investigating?
>  - Chrome is not supported, however we have now made it to crash. Ben
> obviously like that but maybe others do not have the same opinion.

I like that the outdated Debian package of chromium for wheezy is now
unusable.  If the current upstream version of Chromium (or Chrome) is
also broken then that *is* a problem - but I didn't think it could
still run on wheezy, or that it would use nss as a system library.

> 2) Is this severe enough for me to revert the nss 3.26 upload?
[...]

If *only* the outdated Debian package of chromium is affected, then
obviously I don't think it is.

Ben.

-- 
Ben Hutchings
Horngren's Observation:
   Among economists, the real world is often a special
case.



signature.asc
Description: This is a digitally signed message part


Re: Regression problem, call for advice Re: Call for advice and testing of nss (and nspr) and intention to upload correction

2016-11-01 Thread Mike Hommey
On Tue, Nov 01, 2016 at 11:37:29PM +0100, Ola Lundqvist wrote:
> Hi Ben, Balint and others
> 
> I'd like to have some advice on this regression.
> 
> 1) Is this worth investigating?
>  - Chrome is not supported, however we have now made it to crash. Ben
> obviously like that but maybe others do not have the same opinion.
> 2) Is this severe enough for me to revert the nss 3.26 upload?
>  If yes, how do I do that in the best way? I can not easily replace the
> .orig file, but I can make a version without an orig file maybe. And
> replace the source with the old one. Not sure how I should name the
> versions in that case.
> 3) If we think it is worth investigating do anyone have a good idea on how
> to do that. This is a multi-threaded program and gdb did not really help.
> When I run strace chrome fails as I'm running it with strace... So I end up
> in another problem. The only remaining thing seem to be to add quite a few
> printf statements. I'll do that tomorrow unless someone have told me to do
> otherwise.

http://www.rr-project.org

Mike



Regression problem, call for advice Re: Call for advice and testing of nss (and nspr) and intention to upload correction

2016-11-01 Thread Ola Lundqvist
Hi Ben, Balint and others

I'd like to have some advice on this regression.

1) Is this worth investigating?
 - Chrome is not supported, however we have now made it to crash. Ben
obviously like that but maybe others do not have the same opinion.
2) Is this severe enough for me to revert the nss 3.26 upload?
 If yes, how do I do that in the best way? I can not easily replace the
.orig file, but I can make a version without an orig file maybe. And
replace the source with the old one. Not sure how I should name the
versions in that case.
3) If we think it is worth investigating do anyone have a good idea on how
to do that. This is a multi-threaded program and gdb did not really help.
When I run strace chrome fails as I'm running it with strace... So I end up
in another problem. The only remaining thing seem to be to add quite a few
printf statements. I'll do that tomorrow unless someone have told me to do
otherwise.
4) Have anyone seen a problem on anything else than chrome?
 I'm a little worried that this is a bigger problem than chrome.

I have now got two other reports about this problem. One from Okajima and
one from Jiri (cc to this mail). Both had the same problem.
As I understand it is not possible to upgrade chrome browser only. You have
to upgrade more than that. I may be wrong though.

I have put in some effort to understand why it fails, but I do not yet do
that. I'm a little worried that there may be other applications having the
same problem.

The error number is: SEC_ERROR_PKCS11_DEVICE_ERROR =   (SEC_ERROR_BASE +
169) = -0x2000 + 169 = -8192 + 169 = -8023

Yesterday I found this article:
http://stackoverflow.com/questions/22303198/why-would-pk11-generaterandom-return-an-error-8023

So I tried to reverse that fix but it did not help.

The chromium nss_util.cc code looks like this:

// Force a crash with error info on NSS_NoDB_Init failure.
void CrashOnNSSInitFailure() {
  int nss_error = PR_GetError();
  int os_error = PR_GetOSError();
  base::debug::Alias(_error);
  base::debug::Alias(_error);
  LOG(ERROR) << "Error initializing NSS without a persistent database: "
 << GetNSSErrorMessage();
  LOG(FATAL) << "nss_error=" << nss_error << ", os_error=" << os_error;
}

And then this function is called by
if (nodb_init) {
  status = NSS_NoDB_Init(NULL);
  if (status != SECSuccess) {
CrashOnNSSInitFailure();
return;
  }

and
status = NSS_InitReadWrite(nss_config_dir.c_str());
if (status != SECSuccess) {
  LOG(ERROR) << "Error initializing NSS with a persistent "
"database (" << nss_config_dir
 << "): " << GetNSSErrorMessage();
}
  }
  if (status != SECSuccess) {
VLOG(1) << "Initializing NSS without a persistent database.";
status = NSS_NoDB_Init(NULL);
if (status != SECSuccess) {
  CrashOnNSSInitFailure();
  return;
}
  }

I tried to compare the NSS_NoDB_Init functions and the nss_init function
but they are very similar. So it have to be some other deeper function.

I'll investigate more tomorrow to see what I can find out. It is a little
too late now.

// Ola

On 1 November 2016 at 22:41, Ben Hutchings  wrote:

> On Tue, 2016-11-01 at 17:53 +0100, Bálint Réczey wrote:
> > Hi,
> >
> > It seems the nss update broke chromium:
> > https://lists.debian.org/debian-user/2016/10/msg00981.html
>
> Good.
>
> > Maybe when we update gcc for firefox we can also continue supporting
> > chromium:
> > https://lists.debian.org/debian-security-announce/2015/msg00031.html
> [...]
>
> What would be the point?  Anyone using chromium on wheezy should have
> upgraded already.
>
> Ben.
>
> --
> Ben Hutchings
> Horngren's Observation:
>Among economists, the real world is often a special
> case.
>
>


-- 
 --- Inguza Technology AB --- MSc in Information Technology 
/  o...@inguza.comFolkebogatan 26\
|  o...@debian.org   654 68 KARLSTAD|
|  http://inguza.com/Mobile: +46 (0)70-332 1551 |
\  gpg/f.p.: 7090 A92B 18FE 7994 0C36 4FE4 18A1 B1CF 0FE5 3DD9  /
 ---