Re: [regression] Newer gits cannot clone any remote repos

2012-11-21 Thread Douglas Mencken
> The threaded index-pack code did not fail for
> me on cygwin at all during development, including tests, but failed
> immediately I installed v1.7.11. On real repositories, it failed
> intermittently. On some repos it always failed, on some it never
> failed and on some others it would sometimes fail, sometimes not.

Then why did you commit it? If it has so high random failure rate.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [regression] Newer gits cannot clone any remote repos

2012-11-15 Thread Ramsay Jones
Torsten Bögershausen wrote:
> On 13.11.12 19:55, Ramsay Jones wrote:
>> Douglas Mencken wrote:
>>> *Any* git clone fails with:
>>>
>>> fatal: premature end of pack file, 106 bytes missing
>>> fatal: index-pack failed
>>>
>>> At first, I tried 1.8.0, and it failed. Then I tried to build 1.7.10.5
>>> then, and it worked. Then I tried 1.7.12.2, but it fails the same way
>>> as 1.8.0.
>>> So I decided to git bisect.
>>>
>>> b8a2486f1524947f232f657e9f2ebf44e3e7a243 is the first bad commit
>>> ``index-pack: support multithreaded delta resolving''
>>
>> This looks like the same problem I had on cygwin, which lead to
>> commit c0f86547c ("index-pack: Disable threading on cygwin", 26-06-2012).
>>
>> I didn't notice which platform you are on, but maybe you also have a
>> thread-unsafe pread()? Could you try re-building git with the
>> NO_THREAD_SAFE_PREAD build variable set?
>>
>> HTH.
>>
>> ATB,
>> Ramsay Jones
> 
> This is interesting.
> I had the same problem on a PowerPC 
> (Old PowerBook G4 running Linux).
> 
> Using NO_THREAD_SAFE_PREAD helped, thanks for the hint.
> (After recompiling without NO_THREAD_SAFE_PREAD I could clone
> from this machine again, so the problem is not really reproducable)

Yes, the failures would be intermittent (and often not easily
reproducible). The threaded index-pack code did not fail for
me on cygwin at all during development, including tests, but failed
immediately I installed v1.7.11. On real repositories, it failed
intermittently. On some repos it always failed, on some it never
failed and on some others it would sometimes fail, sometimes not.

ATB,
Ramsay Jones

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [regression] Newer gits cannot clone any remote repos

2012-11-15 Thread Ramsay Jones
Douglas Mencken wrote:
>>  Could you try re-building git with the
>> NO_THREAD_SAFE_PREAD build variable set?
> 
> Yeah! It works!!!
> 
> --- evil/Makefile
> +++ good/Makefile
> @@ -957,6 +957,7 @@
>   HAVE_PATHS_H = YesPlease
>   LIBC_CONTAINS_LIBINTL = YesPlease
>   HAVE_DEV_TTY = YesPlease
> + NO_THREAD_SAFE_PREAD = YesPlease
>  endif
>  ifeq ($(uname_S),GNU/kFreeBSD)
>   NO_STRLCPY = YesPlease
> 
> With this, I do have correctly working git clone.

OK, good.

You didn't mention which platform you are on; from the above Makefile
hunk, however, I can deduce that you are on *some* version of Linux.

Hmm, it doesn't seem too likely that your pread() is thread-unsafe
(possible, just unlikely), so it could be a more fundamental problem
with the threaded index-pack code (ie commit b8a2486f1 et. seq.).

However, as a first step could you try running the test program
(given below) on your system to determine if your pread() is thread-safe
or not. (gcc -I. -o test-pread test-pread.c; ./test-pread)

Also, what is the output of "uname -a".

ATB,
Ramsay Jones

-- >8 --
#include "git-compat-util.h"
#include "thread-utils.h"

#define DATA_FILE "junk.data"
#define MAX_DATA 256 * 1024
#define NUM_THREADS 3
#define TRIALS 50

struct thread_data {
pthread_t t;
int fd;
int cnt;
int fails;
unsigned long n;
};

static struct thread_data t[NUM_THREADS+1];

int create_data_file(void)
{
int i, fd = open(DATA_FILE, O_CREAT | O_TRUNC | O_WRONLY, 0600);
if (fd < 0)
return -1;
for (i = 0; i < MAX_DATA; i++)
if (write(fd, &i, sizeof(int)) < 0) {
close(fd);
unlink(DATA_FILE);
return -1;
}
close(fd);
return 0;
}

void *read_thread(void *data)
{
struct thread_data *d = (struct thread_data *)data;
int i, j, rd;
for (i = 0; i < TRIALS; i += MAX_DATA) {
for (j = 0; j < MAX_DATA; j++) {
ssize_t sz = read(d->fd, &rd, sizeof(int));
if (sz < 0 || rd != j)
d->fails++;
d->cnt++;
}
lseek(d->fd, 0, SEEK_SET);
}
return NULL;
}

void *pread_thread(void *data)
{
struct thread_data *d = (struct thread_data *)data;
int i, j, rd;
for (i = 0; i < TRIALS; i++) {
ssize_t sz;
d->n = d->n * 1103515245 + 12345;
j = d->n % MAX_DATA;
sz = pread(d->fd, &rd, sizeof(int), j * sizeof(int));
if (sz < 0 || rd != j)
d->fails++;
d->cnt++;
}
return NULL;
}

int main(int argc, char *argv[])
{
int fd, i;

if (create_data_file() < 0) {
printf("can't create data file\n");
return 1;
}

if ((fd = open(DATA_FILE, O_RDONLY)) < 0) {
printf("can't open data file\n");
unlink(DATA_FILE);
return 1;
}

for (i = 0; i < NUM_THREADS+1; i++) {
int ret;

t[i].fd = fd;
t[i].cnt = 0;
t[i].fails = 0;
t[i].n = i * 16381;
ret = pthread_create(&t[i].t, NULL,
(i == 0) ? read_thread : pread_thread,
&t[i]);
if (ret) {
printf("can't create thread %d (%s)\n", i, 
strerror(ret));
unlink(DATA_FILE);
return 1;
}
}

for (i = 0; i < NUM_THREADS+1; i++)
pthread_join(t[i].t, NULL);
close(fd);

for (i = 0; i < NUM_THREADS+1; i++)
printf("%2d: trials %d, failed %d\n", i, t[i].cnt, t[i].fails);

unlink(DATA_FILE);
return 0;
}





--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [regression] Newer gits cannot clone any remote repos

2012-11-15 Thread Nguyen Thai Ngoc Duy
On Wed, Nov 14, 2012 at 2:55 AM, Douglas Mencken  wrote:
>>  Could you try re-building git with the
>> NO_THREAD_SAFE_PREAD build variable set?
>
> Yeah! It works!!!
>
> --- evil/Makefile
> +++ good/Makefile
> @@ -957,6 +957,7 @@
> HAVE_PATHS_H = YesPlease
> LIBC_CONTAINS_LIBINTL = YesPlease
> HAVE_DEV_TTY = YesPlease
> +   NO_THREAD_SAFE_PREAD = YesPlease
>  endif
>  ifeq ($(uname_S),GNU/kFreeBSD)
> NO_STRLCPY = YesPlease
>
> With this, I do have correctly working git clone.

Sorry you had to figure that out the hard way. Could you make it a
proper patch? I'm surprised that Linux pread does not behave the same
way across platforms though. Or maybe it only happens with certain
Linux versions. What version are you using?
-- 
Duy
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [regression] Newer gits cannot clone any remote repos

2012-11-14 Thread Andreas Schwab
Douglas Mencken  writes:

 I cannot reproduce the problem (on openSUSE 12.2).
>>>
>>> You do need multiple CPU/multi-core machine, as I got it.
>>
>> Which is what I have.
>
> Then try to build *vanilla* git 1.8.0,

Which is what I did.

> not OpenSuSE's one (with a lot of patches inside srcrpm).

Which lot of patches?

Andreas.

-- 
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [regression] Newer gits cannot clone any remote repos

2012-11-14 Thread Douglas Mencken
>>> I cannot reproduce the problem (on openSUSE 12.2).
>>
>> You do need multiple CPU/multi-core machine, as I got it.
>
> Which is what I have.

Then try to build *vanilla* git 1.8.0, not OpenSuSE's one (with a lot
of patches inside srcrpm).
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [regression] Newer gits cannot clone any remote repos

2012-11-14 Thread Andreas Schwab
Douglas Mencken  writes:

>> I cannot reproduce the problem (on openSUSE 12.2).
>
> You do need multiple CPU/multi-core machine, as I got it.

Which is what I have.

Andreas.

-- 
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [regression] Newer gits cannot clone any remote repos

2012-11-14 Thread Douglas Mencken
> I cannot reproduce the problem (on openSUSE 12.2).

You do need multiple CPU/multi-core machine, as I got it.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [regression] Newer gits cannot clone any remote repos

2012-11-13 Thread Andreas Schwab
Torsten Bögershausen  writes:

> Are there more people running PowerPC (on the server side) ?

I cannot reproduce the problem (on openSUSE 12.2).

Andreas.

-- 
Andreas Schwab, sch...@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [regression] Newer gits cannot clone any remote repos

2012-11-13 Thread Torsten Bögershausen
On 13.11.12 19:55, Ramsay Jones wrote:
> Douglas Mencken wrote:
>> *Any* git clone fails with:
>>
>> fatal: premature end of pack file, 106 bytes missing
>> fatal: index-pack failed
>>
>> At first, I tried 1.8.0, and it failed. Then I tried to build 1.7.10.5
>> then, and it worked. Then I tried 1.7.12.2, but it fails the same way
>> as 1.8.0.
>> So I decided to git bisect.
>>
>> b8a2486f1524947f232f657e9f2ebf44e3e7a243 is the first bad commit
>> ``index-pack: support multithreaded delta resolving''
> 
> This looks like the same problem I had on cygwin, which lead to
> commit c0f86547c ("index-pack: Disable threading on cygwin", 26-06-2012).
> 
> I didn't notice which platform you are on, but maybe you also have a
> thread-unsafe pread()? Could you try re-building git with the
> NO_THREAD_SAFE_PREAD build variable set?
> 
> HTH.
> 
> ATB,
> Ramsay Jones

This is interesting.
I had the same problem on a PowerPC 
(Old PowerBook G4 running Linux).

Using NO_THREAD_SAFE_PREAD helped, thanks for the hint.
(After recompiling without NO_THREAD_SAFE_PREAD I could clone
from this machine again, so the problem is not really reproducable)

Are there more people running PowerPC (on the server side) ?
/Torsten

 


--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [regression] Newer gits cannot clone any remote repos

2012-11-13 Thread Ramsay Jones
Douglas Mencken wrote:
> *Any* git clone fails with:
> 
> fatal: premature end of pack file, 106 bytes missing
> fatal: index-pack failed
> 
> At first, I tried 1.8.0, and it failed. Then I tried to build 1.7.10.5
> then, and it worked. Then I tried 1.7.12.2, but it fails the same way
> as 1.8.0.
> So I decided to git bisect.
> 
> b8a2486f1524947f232f657e9f2ebf44e3e7a243 is the first bad commit
> ``index-pack: support multithreaded delta resolving''

This looks like the same problem I had on cygwin, which lead to
commit c0f86547c ("index-pack: Disable threading on cygwin", 26-06-2012).

I didn't notice which platform you are on, but maybe you also have a
thread-unsafe pread()? Could you try re-building git with the
NO_THREAD_SAFE_PREAD build variable set?

HTH.

ATB,
Ramsay Jones

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [regression] Newer gits cannot clone any remote repos

2012-11-12 Thread Douglas Mencken
> On Mon, Nov 12, 2012 at 12:12 PM, Kevin  wrote:
> > Maybe handy to say that you're on a Powerpc platform.

Oh, and yes, I'm on 2 x 2-core ("4-core") machine.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [regression] Newer gits cannot clone any remote repos

2012-11-12 Thread Kevin
Maybe handy to say that you're on a Powerpc platform.

On Mon, Nov 12, 2012 at 5:32 PM, Douglas Mencken  wrote:
> *Any* git clone fails with:
>
> fatal: premature end of pack file, 106 bytes missing
> fatal: index-pack failed
>
> At first, I tried 1.8.0, and it failed. Then I tried to build 1.7.10.5
> then, and it worked. Then I tried 1.7.12.2, but it fails the same way
> as 1.8.0.
> So I decided to git bisect.
>
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html