Re: [RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

Andreas Dilger wrote:
> On Jan 26, 2008  08:27 +0300, Al Boldi wrote:
> > Jan Kara wrote:
> > > > data=ordered mode has proven reliable over the years, and it does
> > > > this by ordering filedata flushes before metadata flushes.  But this
> > > > sometimes causes contention in the order of a 10x slowdown for
> > > > certain apps, either due to the misuse of fsync or due to inherent
> > > > behaviour like db's, as well as inherent starvation issues exposed
> > > > by the data=ordered mode.
> > > >
> > > > data=writeback mode alleviates data=order mode slowdowns, but only
> > > > works per-mount and is too dangerous to run as a default mode.
> > > >
> > > > This RFC proposes to introduce a tunable which allows to disable
> > > > fsync and changes ordered into writeback writeout on a per-process
> > > > basis like this:
> > > >
> > > >   echo 1 > /proc/`pidof process`/softsync
> > >
> > >   I guess disabling fsync() was already commented on enough. Regarding
> > > switching to writeback mode on per-process basis - not easily possible
> > > because sometimes data is not written out by the process which stored
> > > them (think of mmaped file).
> >
> > Do you mean there is a locking problem?
> >
> > > And in case of DB, they use direct-io
> > > anyway most of the time so they don't care about journaling mode
> > > anyway.
> >
> > Testing with sqlite3 and mysql4 shows that performance drastically
> > improves with writeback writeout.
> >
> > >  But as Diego wrote, there is definitely some room for improvement in
> > > current data=ordered mode so the difference shouldn't be as big in the
> > > end.
> >
> > Yes, it would be nice to get to the bottom of this starvation problem,
> > but even then, the proposed tunable remains useful for misbehaving apps.
>
> Al, can you try a patch posted to linux-fsdevel and linux-ext4 from
> Hisashi Hifumi <[EMAIL PROTECTED]> to see if this improves
> your situation?  Dated Mon, 04 Feb 2008 19:15:25 +0900.
>
> [PATCH] ext3,4:fdatasync should skip metadata writeout when
> overwriting
>
> It may be that we already have a solution in that patch for database
> workloads where the pages are already allocated by avoiding the need
> for ordered mode journal flushing in that case.

Well, it seems that it does have a positive effect for the 'konqueror hangs' 
case, but doesn't improve the db case.

This shouldn't be surprising, as the db redundant writeout problem is 
localized not in fsync but rather in ext3_ordered_write_end.

Maybe some form of a staged merged commit could help.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

Andreas Dilger wrote:
 On Jan 26, 2008  08:27 +0300, Al Boldi wrote:
  Jan Kara wrote:
data=ordered mode has proven reliable over the years, and it does
this by ordering filedata flushes before metadata flushes.  But this
sometimes causes contention in the order of a 10x slowdown for
certain apps, either due to the misuse of fsync or due to inherent
behaviour like db's, as well as inherent starvation issues exposed
by the data=ordered mode.
   
data=writeback mode alleviates data=order mode slowdowns, but only
works per-mount and is too dangerous to run as a default mode.
   
This RFC proposes to introduce a tunable which allows to disable
fsync and changes ordered into writeback writeout on a per-process
basis like this:
   
  echo 1  /proc/`pidof process`/softsync
  
 I guess disabling fsync() was already commented on enough. Regarding
   switching to writeback mode on per-process basis - not easily possible
   because sometimes data is not written out by the process which stored
   them (think of mmaped file).
 
  Do you mean there is a locking problem?
 
   And in case of DB, they use direct-io
   anyway most of the time so they don't care about journaling mode
   anyway.
 
  Testing with sqlite3 and mysql4 shows that performance drastically
  improves with writeback writeout.
 
But as Diego wrote, there is definitely some room for improvement in
   current data=ordered mode so the difference shouldn't be as big in the
   end.
 
  Yes, it would be nice to get to the bottom of this starvation problem,
  but even then, the proposed tunable remains useful for misbehaving apps.

 Al, can you try a patch posted to linux-fsdevel and linux-ext4 from
 Hisashi Hifumi [EMAIL PROTECTED] to see if this improves
 your situation?  Dated Mon, 04 Feb 2008 19:15:25 +0900.

 [PATCH] ext3,4:fdatasync should skip metadata writeout when
 overwriting

 It may be that we already have a solution in that patch for database
 workloads where the pages are already allocated by avoiding the need
 for ordered mode journal flushing in that case.

Well, it seems that it does have a positive effect for the 'konqueror hangs' 
case, but doesn't improve the db case.

This shouldn't be surprising, as the db redundant writeout problem is 
localized not in fsync but rather in ext3_ordered_write_end.

Maybe some form of a staged merged commit could help.


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

Jan Kara wrote:
> On Tue 05-02-08 10:07:44, Al Boldi wrote:
> > Jan Kara wrote:
> > > On Sat 02-02-08 00:26:00, Al Boldi wrote:
> > > > Chris Mason wrote:
> > > > > Al, could you please compare the write throughput from vmstat for
> > > > > the data=ordered vs data=writeback runs?  I would guess the
> > > > > data=ordered one has a lower overall write throughput.
> > > >
> > > > That's what I would have guessed, but it's actually going up 4x fold
> > > > for mysql from 559mb to 2135mb, while the db-size ends up at 549mb.
> > >
> > >   So you say we write 4-times as much data in ordered mode as in
> > > writeback mode. Hmm, probably possible because we force all the dirty
> > > data to disk when committing a transation in ordered mode (and don't
> > > do this in writeback mode). So if the workload repeatedly dirties the
> > > whole DB, we are going to write the whole DB several times in ordered
> > > mode but in writeback mode we just keep the data in memory all the
> > > time. But this is what you ask for if you mount in ordered mode so I
> > > wouldn't consider it a bug.
> >
> > Ok, maybe not a bug, but a bit inefficient.  Check out this workload:
> >
> > sync;
> >
> > while :; do
> >   dd < /dev/full > /mnt/sda2/x.dmp bs=1M count=20
> >   rm -f /mnt/sda2/x.dmp
> >   usleep 1
> > done
:
:
> > Do you think these 12mb redundant writeouts could be buffered?
>
>   No, I don't think so. At least when I run it, number of blocks written
> out varies which confirms that these 12mb are just data blocks which
> happen to be in the file when transaction commits (which is every 5
> seconds).

Just a thought, but maybe double-buffering can help?

> And to satisfy journaling gurantees in ordered mode you must
> write them so you really have no choice...

Making this RFC rather useful.

What we need now is an implementation, which should be easy.

Maybe something on these lines:

<< in ext3_ordered_write_end >>
  if (current->soft_sync & 1)
return ext3_writeback_write_end;

<< in ext3_ordered_writepage >>
  if (current->soft_sync & 2)
return ext3_writeback_writepage;

<< in ext3_sync_file >>
  if (current->soft_sync & 4)
return ret;

<< in ext3_file_write >>
  if (current->soft_sync & 8)
return ret;

As you can see soft_sync is masked and bits are ordered by importance.

It would be neat if somebody interested could cook-up a patch.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

Jan Kara wrote:
 On Tue 05-02-08 10:07:44, Al Boldi wrote:
  Jan Kara wrote:
   On Sat 02-02-08 00:26:00, Al Boldi wrote:
Chris Mason wrote:
 Al, could you please compare the write throughput from vmstat for
 the data=ordered vs data=writeback runs?  I would guess the
 data=ordered one has a lower overall write throughput.
   
That's what I would have guessed, but it's actually going up 4x fold
for mysql from 559mb to 2135mb, while the db-size ends up at 549mb.
  
 So you say we write 4-times as much data in ordered mode as in
   writeback mode. Hmm, probably possible because we force all the dirty
   data to disk when committing a transation in ordered mode (and don't
   do this in writeback mode). So if the workload repeatedly dirties the
   whole DB, we are going to write the whole DB several times in ordered
   mode but in writeback mode we just keep the data in memory all the
   time. But this is what you ask for if you mount in ordered mode so I
   wouldn't consider it a bug.
 
  Ok, maybe not a bug, but a bit inefficient.  Check out this workload:
 
  sync;
 
  while :; do
dd  /dev/full  /mnt/sda2/x.dmp bs=1M count=20
rm -f /mnt/sda2/x.dmp
usleep 1
  done
:
:
  Do you think these 12mb redundant writeouts could be buffered?

   No, I don't think so. At least when I run it, number of blocks written
 out varies which confirms that these 12mb are just data blocks which
 happen to be in the file when transaction commits (which is every 5
 seconds).

Just a thought, but maybe double-buffering can help?

 And to satisfy journaling gurantees in ordered mode you must
 write them so you really have no choice...

Making this RFC rather useful.

What we need now is an implementation, which should be easy.

Maybe something on these lines:

 in ext3_ordered_write_end 
  if (current-soft_sync  1)
return ext3_writeback_write_end;

 in ext3_ordered_writepage 
  if (current-soft_sync  2)
return ext3_writeback_writepage;

 in ext3_sync_file 
  if (current-soft_sync  4)
return ret;

 in ext3_file_write 
  if (current-soft_sync  8)
return ret;

As you can see soft_sync is masked and bits are ordered by importance.

It would be neat if somebody interested could cook-up a patch.


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

Jan Kara wrote:
> On Sat 02-02-08 00:26:00, Al Boldi wrote:
> > Chris Mason wrote:
> > > Al, could you please compare the write throughput from vmstat for the
> > > data=ordered vs data=writeback runs?  I would guess the data=ordered
> > > one has a lower overall write throughput.
> >
> > That's what I would have guessed, but it's actually going up 4x fold for
> > mysql from 559mb to 2135mb, while the db-size ends up at 549mb.
>
>   So you say we write 4-times as much data in ordered mode as in writeback
> mode. Hmm, probably possible because we force all the dirty data to disk
> when committing a transation in ordered mode (and don't do this in
> writeback mode). So if the workload repeatedly dirties the whole DB, we
> are going to write the whole DB several times in ordered mode but in
> writeback mode we just keep the data in memory all the time. But this is
> what you ask for if you mount in ordered mode so I wouldn't consider it a
> bug.

Ok, maybe not a bug, but a bit inefficient.  Check out this workload:

sync;

while :; do
  dd < /dev/full > /mnt/sda2/x.dmp bs=1M count=20
  rm -f /mnt/sda2/x.dmp
  usleep 1
done

vmstat 1 ( with mount /dev/sda2 /mnt/sda2 -o data=writeback) << note io-bo >>

procs ---memory-- ---swap-- -io --system-- cpu
 r  b   swpd   free   buff  cache   si   sobibo   incs us sy id wa
 2  0  0 293008   5232  5743600 0 0   18   206  4 80 16  0
 1  0  0 282840   5232  6762000 0 0   18   238  3 81 16  0
 1  0  0 297032   5244  5336400 0   152   21   211  4 79 17  0
 1  0  0 285236   5244  6522400 0 0   18   232  4 80 16  0
 1  0  0 299464   5244  5088000 0 0   18   222  4 80 16  0
 1  0  0 290156   5244  6017600 0 0   18   236  3 80 17  0
 0  0  0 302124   5256  4778800 0   152   21   213  4 80 16  0
 1  0  0 292180   5256  5824800 0 0   18   239  3 81 16  0
 1  0  0 287452   5256  6244400 0 0   18   202  3 80 17  0
 1  0  0 293016   5256  5739200 0 0   18   250  4 80 16  0
 0  0  0 302052   5256  4778800 0 0   19   194  3 81 16  0
 1  0  0 297536   5268  5292800 0   152   20   233  4 79 17  0
 1  0  0 286468   5268  6387200 0 0   18   212  3 81 16  0
 1  0  0 301572   5268  4881200 0 0   18   267  4 79 17  0
 1  0  0 292636   5268  5777600 0 0   18   208  4 80 16  0
 1  0  0 302124   5280  4778800 0   152   21   237  4 80 16  0
 1  0  0 291436   5280  5897600 0 0   18   205  3 81 16  0
 1  0  0 302068   5280  4778800 0 0   18   234  3 81 16  0
 1  0  0 293008   5280  5738800 0 0   18   221  4 79 17  0
 1  0  0 297288   5292  5253200 0   156   22   233  2 81 16  1
 1  0  0 294676   5292  5572400 0 0   19   199  3 81 16  0


vmstat 1 (with mount /dev/sda2 /mnt/sda2 -o data=ordered)

procs ---memory-- ---swap-- -io --system-- cpu
 r  b   swpd   free   buff  cache   si   sobibo   incs us sy id wa
 2  0  0 291052   5156  5901600 0 0   19   223  3 82 15  0
 1  0  0 291408   5156  5870400 0 0   18   218  3 81 16  0
 1  0  0 291888   5156  5827600 020   23   229  3 80 17  0
 1  0  0 300764   5168  4947200 0 12864   91   235  3 69 13 15
 1  0  0 300740   5168  4945600 0 0   19   215  3 80 17  0
 1  0  0 301088   5168  4904400 0 0   18   241  4 80 16  0
 1  0  0 298220   5168  5187200 0 0   18   225  3 81 16  0
 0  1  0 289168   5168  6075200 0 12712   45   237  3 77 15  5
 1  0  0 300260   5180  4985200 0   152   68   211  4 72 15  9
 1  0  0 298616   5180  5146000 0 0   18   237  3 81 16  0
 1  0  0 296988   5180  5309200 0 0   18   223  3 81 16  0
 1  0  0 296608   5180  5348000 0 0   18   223  3 81 16  0
 0  0  0 301640   5192  4803600 0 12868   93   206  4 67 13 16
 0  0  0 301624   5192  4803600 0 0   21   218  3 81 16  0
 0  0  0 301600   5192  4803600 0 0   18   212  3 81 16  0
 0  0  0 301584   5192  4803600 0 0   18   209  4 80 16  0
 0  0  0 301568   5192  4803600 0 0   18   208  3 81 16  0
 1  0  0 285520   5204  6454800 0 12864   95   216  3 69 13 15
 2  0  0 285124   5204  6492400 0 0   18   222  4 80 16  0
 1  0  0 283612   5204  6639200 0 0   18   231  3 81 16  0
 1  0  0 284216   5204  6573600 0 0   18 

Re: [RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

Jan Kara wrote:
 On Sat 02-02-08 00:26:00, Al Boldi wrote:
  Chris Mason wrote:
   Al, could you please compare the write throughput from vmstat for the
   data=ordered vs data=writeback runs?  I would guess the data=ordered
   one has a lower overall write throughput.
 
  That's what I would have guessed, but it's actually going up 4x fold for
  mysql from 559mb to 2135mb, while the db-size ends up at 549mb.

   So you say we write 4-times as much data in ordered mode as in writeback
 mode. Hmm, probably possible because we force all the dirty data to disk
 when committing a transation in ordered mode (and don't do this in
 writeback mode). So if the workload repeatedly dirties the whole DB, we
 are going to write the whole DB several times in ordered mode but in
 writeback mode we just keep the data in memory all the time. But this is
 what you ask for if you mount in ordered mode so I wouldn't consider it a
 bug.

Ok, maybe not a bug, but a bit inefficient.  Check out this workload:

sync;

while :; do
  dd  /dev/full  /mnt/sda2/x.dmp bs=1M count=20
  rm -f /mnt/sda2/x.dmp
  usleep 1
done

vmstat 1 ( with mount /dev/sda2 /mnt/sda2 -o data=writeback)  note io-bo 

procs ---memory-- ---swap-- -io --system-- cpu
 r  b   swpd   free   buff  cache   si   sobibo   incs us sy id wa
 2  0  0 293008   5232  5743600 0 0   18   206  4 80 16  0
 1  0  0 282840   5232  6762000 0 0   18   238  3 81 16  0
 1  0  0 297032   5244  5336400 0   152   21   211  4 79 17  0
 1  0  0 285236   5244  6522400 0 0   18   232  4 80 16  0
 1  0  0 299464   5244  5088000 0 0   18   222  4 80 16  0
 1  0  0 290156   5244  6017600 0 0   18   236  3 80 17  0
 0  0  0 302124   5256  4778800 0   152   21   213  4 80 16  0
 1  0  0 292180   5256  5824800 0 0   18   239  3 81 16  0
 1  0  0 287452   5256  6244400 0 0   18   202  3 80 17  0
 1  0  0 293016   5256  5739200 0 0   18   250  4 80 16  0
 0  0  0 302052   5256  4778800 0 0   19   194  3 81 16  0
 1  0  0 297536   5268  5292800 0   152   20   233  4 79 17  0
 1  0  0 286468   5268  6387200 0 0   18   212  3 81 16  0
 1  0  0 301572   5268  4881200 0 0   18   267  4 79 17  0
 1  0  0 292636   5268  5777600 0 0   18   208  4 80 16  0
 1  0  0 302124   5280  4778800 0   152   21   237  4 80 16  0
 1  0  0 291436   5280  5897600 0 0   18   205  3 81 16  0
 1  0  0 302068   5280  4778800 0 0   18   234  3 81 16  0
 1  0  0 293008   5280  5738800 0 0   18   221  4 79 17  0
 1  0  0 297288   5292  5253200 0   156   22   233  2 81 16  1
 1  0  0 294676   5292  5572400 0 0   19   199  3 81 16  0


vmstat 1 (with mount /dev/sda2 /mnt/sda2 -o data=ordered)

procs ---memory-- ---swap-- -io --system-- cpu
 r  b   swpd   free   buff  cache   si   sobibo   incs us sy id wa
 2  0  0 291052   5156  5901600 0 0   19   223  3 82 15  0
 1  0  0 291408   5156  5870400 0 0   18   218  3 81 16  0
 1  0  0 291888   5156  5827600 020   23   229  3 80 17  0
 1  0  0 300764   5168  4947200 0 12864   91   235  3 69 13 15
 1  0  0 300740   5168  4945600 0 0   19   215  3 80 17  0
 1  0  0 301088   5168  4904400 0 0   18   241  4 80 16  0
 1  0  0 298220   5168  5187200 0 0   18   225  3 81 16  0
 0  1  0 289168   5168  6075200 0 12712   45   237  3 77 15  5
 1  0  0 300260   5180  4985200 0   152   68   211  4 72 15  9
 1  0  0 298616   5180  5146000 0 0   18   237  3 81 16  0
 1  0  0 296988   5180  5309200 0 0   18   223  3 81 16  0
 1  0  0 296608   5180  5348000 0 0   18   223  3 81 16  0
 0  0  0 301640   5192  4803600 0 12868   93   206  4 67 13 16
 0  0  0 301624   5192  4803600 0 0   21   218  3 81 16  0
 0  0  0 301600   5192  4803600 0 0   18   212  3 81 16  0
 0  0  0 301584   5192  4803600 0 0   18   209  4 80 16  0
 0  0  0 301568   5192  4803600 0 0   18   208  3 81 16  0
 1  0  0 285520   5204  6454800 0 12864   95   216  3 69 13 15
 2  0  0 285124   5204  6492400 0 0   18   222  4 80 16  0
 1  0  0 283612   5204  6639200 0 0   18   231  3 81 16  0
 1  0  0 284216   5204  6573600 0 0   18   218  4 80 16  0
 0  1  0 289160   5204  6075200 0 12712   56   213  3 74 15  8
 1  0  0 285884   5216  64128

Re: [RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

Chris Mason wrote:
> On Thursday 31 January 2008, Jan Kara wrote:
> > On Thu 31-01-08 11:56:01, Chris Mason wrote:
> > > On Thursday 31 January 2008, Al Boldi wrote:
> > > > The big difference between ordered and writeback is that once the
> > > > slowdown starts, ordered goes into ~100% iowait, whereas writeback
> > > > continues 100% user.
> > >
> > > Does data=ordered write buffers in the order they were dirtied?  This
> > > might explain the extreme problems in transactional workloads.
> >
> >   Well, it does but we submit them to block layer all at once so
> > elevator should sort the requests for us...
>
> nr_requests is fairly small, so a long stream of random requests should
> still end up being random IO.
>
> Al, could you please compare the write throughput from vmstat for the
> data=ordered vs data=writeback runs?  I would guess the data=ordered one
> has a lower overall write throughput.

That's what I would have guessed, but it's actually going up 4x fold for 
mysql from 559mb to 2135mb, while the db-size ends up at 549mb.

This may mean that data=ordered isn't buffering redundant writes; or worse.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

Andreas Dilger wrote:
> On Wednesday 30 January 2008, Al Boldi wrote:
> > And, a quick test of successive 1sec delayed syncs shows no hangs until
> > about 1 minute (~180mb) of db-writeout activity, when the sync abruptly
> > hangs for minutes on end, and io-wait shows almost 100%.
>
> How large is the journal in this filesystem?  You can check via
> "debugfs -R 'stat <8>' /dev/XXX".

32mb.

> Is this affected by increasing
> the journal size?  You can set the journal size via "mke2fs -J size=400"
> at format time, or on an unmounted filesystem by running
> "tune2fs -O ^has_journal /dev/XXX" then "tune2fs -J size=400 /dev/XXX".

Setting size=400 doesn't help, nor does size=4.

> I suspect that the stall is caused by the journal filling up, and then
> waiting while the entire journal is checkpointed back to the filesystem
> before the next transaction can start.
>
> It is possible to improve this behaviour in JBD by reducing the amount
> of space that is cleared if the journal becomes "full", and also doing
> journal checkpointing before it becomes full.  While that may reduce
> performance a small amount, it would help avoid such huge latency
> problems. I believe we have such a patch in one of the Lustre branches
> already, and while I'm not sure what kernel it is for the JBD code rarely
> changes much

The big difference between ordered and writeback is that once the slowdown 
starts, ordered goes into ~100% iowait, whereas writeback continues 100% 
user.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

Chris Mason wrote:
> On Wednesday 30 January 2008, Al Boldi wrote:
> > Jan Kara wrote:
> > > > Chris Snook wrote:
> > > > > Al Boldi wrote:
> > > > > > This RFC proposes to introduce a tunable which allows to disable
> > > > > > fsync and changes ordered into writeback writeout on a
> > > > > > per-process basis like this:
> > > > > >
> > > > > >   echo 1 > /proc/`pidof process`/softsync
> > > > >
> > > > > This is basically a kernel workaround for stupid app behavior.
> > > >
> > > > Exactly right to some extent, but don't forget the underlying
> > > > data=ordered starvation problem, which looks like a genuinely deep
> > > > problem maybe related to blockIO.
> > >
> > >   It is a problem with the way how ext3 does fsync (at least that's
> > > what we ended up with in that konqueror problem)... It has to flush
> > > the current transaction which means that app doing fsync() has to wait
> > > till all dirty data of all files on the filesystem are written (if we
> > > are in ordered mode). And that takes quite some time... There are
> > > possibilities how to avoid that but especially with freshly created
> > > files, it's tough and I don't see a way how to do it without some
> > > fundamental changes to JBD.
> >
> > Ok, but keep in mind that this starvation occurs even in the absence of
> > fsync, as the benchmarks show.
> >
> > And, a quick test of successive 1sec delayed syncs shows no hangs until
> > about 1 minute (~180mb) of db-writeout activity, when the sync abruptly
> > hangs for minutes on end, and io-wait shows almost 100%.
>
> Do you see this on older kernels as well?  The first thing we need to
> understand is if this particular stall is new.

2.6.24,22,19 and 2.4.32 show the same problem.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

Chris Mason wrote:
 On Wednesday 30 January 2008, Al Boldi wrote:
  Jan Kara wrote:
Chris Snook wrote:
 Al Boldi wrote:
  This RFC proposes to introduce a tunable which allows to disable
  fsync and changes ordered into writeback writeout on a
  per-process basis like this:
 
echo 1  /proc/`pidof process`/softsync

 This is basically a kernel workaround for stupid app behavior.
   
Exactly right to some extent, but don't forget the underlying
data=ordered starvation problem, which looks like a genuinely deep
problem maybe related to blockIO.
  
 It is a problem with the way how ext3 does fsync (at least that's
   what we ended up with in that konqueror problem)... It has to flush
   the current transaction which means that app doing fsync() has to wait
   till all dirty data of all files on the filesystem are written (if we
   are in ordered mode). And that takes quite some time... There are
   possibilities how to avoid that but especially with freshly created
   files, it's tough and I don't see a way how to do it without some
   fundamental changes to JBD.
 
  Ok, but keep in mind that this starvation occurs even in the absence of
  fsync, as the benchmarks show.
 
  And, a quick test of successive 1sec delayed syncs shows no hangs until
  about 1 minute (~180mb) of db-writeout activity, when the sync abruptly
  hangs for minutes on end, and io-wait shows almost 100%.

 Do you see this on older kernels as well?  The first thing we need to
 understand is if this particular stall is new.

2.6.24,22,19 and 2.4.32 show the same problem.


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

Andreas Dilger wrote:
 On Wednesday 30 January 2008, Al Boldi wrote:
  And, a quick test of successive 1sec delayed syncs shows no hangs until
  about 1 minute (~180mb) of db-writeout activity, when the sync abruptly
  hangs for minutes on end, and io-wait shows almost 100%.

 How large is the journal in this filesystem?  You can check via
 debugfs -R 'stat 8' /dev/XXX.

32mb.

 Is this affected by increasing
 the journal size?  You can set the journal size via mke2fs -J size=400
 at format time, or on an unmounted filesystem by running
 tune2fs -O ^has_journal /dev/XXX then tune2fs -J size=400 /dev/XXX.

Setting size=400 doesn't help, nor does size=4.

 I suspect that the stall is caused by the journal filling up, and then
 waiting while the entire journal is checkpointed back to the filesystem
 before the next transaction can start.

 It is possible to improve this behaviour in JBD by reducing the amount
 of space that is cleared if the journal becomes full, and also doing
 journal checkpointing before it becomes full.  While that may reduce
 performance a small amount, it would help avoid such huge latency
 problems. I believe we have such a patch in one of the Lustre branches
 already, and while I'm not sure what kernel it is for the JBD code rarely
 changes much

The big difference between ordered and writeback is that once the slowdown 
starts, ordered goes into ~100% iowait, whereas writeback continues 100% 
user.


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

Jan Kara wrote:
> > Chris Snook wrote:
> > > Al Boldi wrote:
> > > > This RFC proposes to introduce a tunable which allows to disable
> > > > fsync and changes ordered into writeback writeout on a per-process
> > > > basis like this:
> > > >
> > > >   echo 1 > /proc/`pidof process`/softsync
> > >
> > > This is basically a kernel workaround for stupid app behavior.
> >
> > Exactly right to some extent, but don't forget the underlying
> > data=ordered starvation problem, which looks like a genuinely deep
> > problem maybe related to blockIO.
>
>   It is a problem with the way how ext3 does fsync (at least that's what
> we ended up with in that konqueror problem)... It has to flush the
> current transaction which means that app doing fsync() has to wait till
> all dirty data of all files on the filesystem are written (if we are in
> ordered mode). And that takes quite some time... There are possibilities
> how to avoid that but especially with freshly created files, it's tough
> and I don't see a way how to do it without some fundamental changes to
> JBD.

Ok, but keep in mind that this starvation occurs even in the absence of 
fsync, as the benchmarks show.

And, a quick test of successive 1sec delayed syncs shows no hangs until about 
1 minute (~180mb) of db-writeout activity, when the sync abruptly hangs for 
minutes on end, and io-wait shows almost 100%.

Now it turns out that 'echo 3 > /proc/.../drop_caches' has no effect, but 
doing it a few more times makes the hangs go away for while, only to come 
back again and again.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

Jan Kara wrote:
  Chris Snook wrote:
   Al Boldi wrote:
This RFC proposes to introduce a tunable which allows to disable
fsync and changes ordered into writeback writeout on a per-process
basis like this:
   
  echo 1  /proc/`pidof process`/softsync
  
   This is basically a kernel workaround for stupid app behavior.
 
  Exactly right to some extent, but don't forget the underlying
  data=ordered starvation problem, which looks like a genuinely deep
  problem maybe related to blockIO.

   It is a problem with the way how ext3 does fsync (at least that's what
 we ended up with in that konqueror problem)... It has to flush the
 current transaction which means that app doing fsync() has to wait till
 all dirty data of all files on the filesystem are written (if we are in
 ordered mode). And that takes quite some time... There are possibilities
 how to avoid that but especially with freshly created files, it's tough
 and I don't see a way how to do it without some fundamental changes to
 JBD.

Ok, but keep in mind that this starvation occurs even in the absence of 
fsync, as the benchmarks show.

And, a quick test of successive 1sec delayed syncs shows no hangs until about 
1 minute (~180mb) of db-writeout activity, when the sync abruptly hangs for 
minutes on end, and io-wait shows almost 100%.

Now it turns out that 'echo 3  /proc/.../drop_caches' has no effect, but 
doing it a few more times makes the hangs go away for while, only to come 
back again and again.


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

Jan Kara wrote:
> On Sat 26-01-08 08:27:59, Al Boldi wrote:
> > Do you mean there is a locking problem?
>
>   No, but if you write to an mmaped file, then we can find out only later
> we have dirty data in pages and we call writepage() on behalf of e.g.
> pdflush().

Ok, that's a special case, which we could code for, but doesn't seem 
worthwile.  In any case, child-forks should inherit its parent mode.

> > > And in case of DB, they use direct-io
> > > anyway most of the time so they don't care about journaling mode
> > > anyway.
> >
> > Testing with sqlite3 and mysql4 shows that performance drastically
> > improves with writeback writeout.
>
>   And do you have the databases configured to use direct IO or not?

I don't think so, but these tests are only meant to expose the underlying 
problem which needs to be fixed, while this RFC proposes a useful 
workaround.

In another post Jan Kara wrote:
>   Hmm, if you're willing to test patches, then you could try a debug
> patch: http://bugzilla.kernel.org/attachment.cgi?id=14574
>   and send me the output. What kind of load do you observe problems with
> and which problems exactly?

8M-record insert into indexed db-table:
 ordered  writeback
sqlite3:  75m22s8m45s
mysql4 :  23m35s5m29s

Also, see the 'konqueror deadlocks in 2.6.22' thread.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

Jan Kara wrote:
 On Sat 26-01-08 08:27:59, Al Boldi wrote:
  Do you mean there is a locking problem?

   No, but if you write to an mmaped file, then we can find out only later
 we have dirty data in pages and we call writepage() on behalf of e.g.
 pdflush().

Ok, that's a special case, which we could code for, but doesn't seem 
worthwile.  In any case, child-forks should inherit its parent mode.

   And in case of DB, they use direct-io
   anyway most of the time so they don't care about journaling mode
   anyway.
 
  Testing with sqlite3 and mysql4 shows that performance drastically
  improves with writeback writeout.

   And do you have the databases configured to use direct IO or not?

I don't think so, but these tests are only meant to expose the underlying 
problem which needs to be fixed, while this RFC proposes a useful 
workaround.

In another post Jan Kara wrote:
   Hmm, if you're willing to test patches, then you could try a debug
 patch: http://bugzilla.kernel.org/attachment.cgi?id=14574
   and send me the output. What kind of load do you observe problems with
 and which problems exactly?

8M-record insert into indexed db-table:
 ordered  writeback
sqlite3:  75m22s8m45s
mysql4 :  23m35s5m29s

Also, see the 'konqueror deadlocks in 2.6.22' thread.


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Kernel Event Notifications (was: [RFC] Parallelize IO for e2fsck)

[Al Boldi]

KOSAKI Motohiro wrote:
> > > And from a performance point of view letting applications voluntarily
> > > free some memory is better even than starting to swap.
> >
> > Absolutely.
>
> the mem_notify patch can realize "just before starting swapping"
> notification :)
>
> to be honest, I don't know fs guys requirement.
> if lacking feature of fs guys needed, I implement it with presure if
> you tell me it.

These notifications are really useful, but it may be much wiser to pipe them 
thru some kevent-notification sub-system, instead of introducing kernel 
notifier-chain end-points left, right, and center.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

Chris Snook wrote:
> Al Boldi wrote:
> > Greetings!
> >
> > data=ordered mode has proven reliable over the years, and it does this
> > by ordering filedata flushes before metadata flushes.  But this
> > sometimes causes contention in the order of a 10x slowdown for certain
> > apps, either due to the misuse of fsync or due to inherent behaviour
> > like db's, as well as inherent starvation issues exposed by the
> > data=ordered mode.
> >
> > data=writeback mode alleviates data=order mode slowdowns, but only works
> > per-mount and is too dangerous to run as a default mode.
> >
> > This RFC proposes to introduce a tunable which allows to disable fsync
> > and changes ordered into writeback writeout on a per-process basis like
> > this:
> >
> >   echo 1 > /proc/`pidof process`/softsync
> >
> >
> > Your comments are much welcome!
>
> This is basically a kernel workaround for stupid app behavior.

Exactly right to some extent, but don't forget the underlying data=ordered 
starvation problem, which looks like a genuinely deep problem maybe related 
to blockIO.

> It
> wouldn't be the first time we've provided such an option, but we shouldn't
> do it without a very good justification.  At the very least, we need a
> test case that demonstrates the problem

See the 'konqueror deadlocks in 2.6.22' thread.

> and benchmark results that prove that this approach actually fixes it.

8M-record insert into indexed db-table:
 ordered  writeback
sqlite3:  75m22s8m45s
mysql4 :  23m35s5m29s

> I suspect we can find a cleaner fix for the problem.

I hope so, but even with a fix available addressing the data=ordered 
starvation issue, this tunable could remain useful for those apps that 
misbehave.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

Jan Kara wrote:
> > Greetings!
> >
> > data=ordered mode has proven reliable over the years, and it does this
> > by ordering filedata flushes before metadata flushes.  But this
> > sometimes causes contention in the order of a 10x slowdown for certain
> > apps, either due to the misuse of fsync or due to inherent behaviour
> > like db's, as well as inherent starvation issues exposed by the
> > data=ordered mode.
> >
> > data=writeback mode alleviates data=order mode slowdowns, but only works
> > per-mount and is too dangerous to run as a default mode.
> >
> > This RFC proposes to introduce a tunable which allows to disable fsync
> > and changes ordered into writeback writeout on a per-process basis like
> > this:
> >
> >   echo 1 > /proc/`pidof process`/softsync
>
>   I guess disabling fsync() was already commented on enough. Regarding
> switching to writeback mode on per-process basis - not easily possible
> because sometimes data is not written out by the process which stored
> them (think of mmaped file).

Do you mean there is a locking problem?

> And in case of DB, they use direct-io
> anyway most of the time so they don't care about journaling mode anyway.

Testing with sqlite3 and mysql4 shows that performance drastically improves 
with writeback writeout.

>  But as Diego wrote, there is definitely some room for improvement in
> current data=ordered mode so the difference shouldn't be as big in the
> end.

Yes, it would be nice to get to the bottom of this starvation problem, but 
even then, the proposed tunable remains useful for misbehaving apps.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

[EMAIL PROTECTED] wrote:
> On Thu, 24 Jan 2008 23:36:00 +0300, Al Boldi said:
> > This RFC proposes to introduce a tunable which allows to disable fsync
> > and changes ordered into writeback writeout on a per-process basis like
> > this:
:
:
> But if you want to give them enough rope to shoot themselves in the foot
> with, I'd suggest abusing LD_PRELOAD to replace the fsync() glibc code
> instead.  No need to clutter the kernel with rope that can be (and has
> been) done in userspace.

Ok that's possible, but as you cannot use LD_PRELOAD to deal with changing 
ordered into writeback mode, we might as well allow them to disable fsync 
here, because it is in the same use-case.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

Diego Calleja wrote:
> El Thu, 24 Jan 2008 23:36:00 +0300, Al Boldi <[EMAIL PROTECTED]> escribió:
> > Greetings!
> >
> > data=ordered mode has proven reliable over the years, and it does this
> > by ordering filedata flushes before metadata flushes.  But this
> > sometimes causes contention in the order of a 10x slowdown for certain
> > apps, either due to the misuse of fsync or due to inherent behaviour
> > like db's, as well as inherent starvation issues exposed by the
> > data=ordered mode.
>
> There's a related bug in bugzilla:
> http://bugzilla.kernel.org/show_bug.cgi?id=9546
>
> The diagnostic from Jan Kara is different though, but I think it may be
> the same problem...
>
> "One process does data-intensive load. Thus in the ordered mode the
> transaction is tiny but has tons of data buffers attached. If commit
> happens, it takes a long time to sync all the data before the commit
> can proceed... In the writeback mode, we don't wait for data buffers, in
> the journal mode amount of data to be written is really limited by the
> maximum size of a transaction and so we write by much smaller chunks
> and better latency is thus ensured."
>
>
> I'm hitting this bug too...it's surprising that there's not many people
> reporting more bugs about this, because it's really annoying.
>
>
> There's a patch by Jan Kara (that I'm including here because bugzilla
> didn't include it and took me a while to find it) which I don't know if
> it's supposed to fix the problem , but it'd be interesting to try:


Thanks a lot, but it doesn't fix it.

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

Chris Snook wrote:
 Al Boldi wrote:
  Greetings!
 
  data=ordered mode has proven reliable over the years, and it does this
  by ordering filedata flushes before metadata flushes.  But this
  sometimes causes contention in the order of a 10x slowdown for certain
  apps, either due to the misuse of fsync or due to inherent behaviour
  like db's, as well as inherent starvation issues exposed by the
  data=ordered mode.
 
  data=writeback mode alleviates data=order mode slowdowns, but only works
  per-mount and is too dangerous to run as a default mode.
 
  This RFC proposes to introduce a tunable which allows to disable fsync
  and changes ordered into writeback writeout on a per-process basis like
  this:
 
echo 1  /proc/`pidof process`/softsync
 
 
  Your comments are much welcome!

 This is basically a kernel workaround for stupid app behavior.

Exactly right to some extent, but don't forget the underlying data=ordered 
starvation problem, which looks like a genuinely deep problem maybe related 
to blockIO.

 It
 wouldn't be the first time we've provided such an option, but we shouldn't
 do it without a very good justification.  At the very least, we need a
 test case that demonstrates the problem

See the 'konqueror deadlocks in 2.6.22' thread.

 and benchmark results that prove that this approach actually fixes it.

8M-record insert into indexed db-table:
 ordered  writeback
sqlite3:  75m22s8m45s
mysql4 :  23m35s5m29s

 I suspect we can find a cleaner fix for the problem.

I hope so, but even with a fix available addressing the data=ordered 
starvation issue, this tunable could remain useful for those apps that 
misbehave.


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

Jan Kara wrote:
  Greetings!
 
  data=ordered mode has proven reliable over the years, and it does this
  by ordering filedata flushes before metadata flushes.  But this
  sometimes causes contention in the order of a 10x slowdown for certain
  apps, either due to the misuse of fsync or due to inherent behaviour
  like db's, as well as inherent starvation issues exposed by the
  data=ordered mode.
 
  data=writeback mode alleviates data=order mode slowdowns, but only works
  per-mount and is too dangerous to run as a default mode.
 
  This RFC proposes to introduce a tunable which allows to disable fsync
  and changes ordered into writeback writeout on a per-process basis like
  this:
 
echo 1  /proc/`pidof process`/softsync

   I guess disabling fsync() was already commented on enough. Regarding
 switching to writeback mode on per-process basis - not easily possible
 because sometimes data is not written out by the process which stored
 them (think of mmaped file).

Do you mean there is a locking problem?

 And in case of DB, they use direct-io
 anyway most of the time so they don't care about journaling mode anyway.

Testing with sqlite3 and mysql4 shows that performance drastically improves 
with writeback writeout.

  But as Diego wrote, there is definitely some room for improvement in
 current data=ordered mode so the difference shouldn't be as big in the
 end.

Yes, it would be nice to get to the bottom of this starvation problem, but 
even then, the proposed tunable remains useful for misbehaving apps.


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

Diego Calleja wrote:
 El Thu, 24 Jan 2008 23:36:00 +0300, Al Boldi [EMAIL PROTECTED] escribió:
  Greetings!
 
  data=ordered mode has proven reliable over the years, and it does this
  by ordering filedata flushes before metadata flushes.  But this
  sometimes causes contention in the order of a 10x slowdown for certain
  apps, either due to the misuse of fsync or due to inherent behaviour
  like db's, as well as inherent starvation issues exposed by the
  data=ordered mode.

 There's a related bug in bugzilla:
 http://bugzilla.kernel.org/show_bug.cgi?id=9546

 The diagnostic from Jan Kara is different though, but I think it may be
 the same problem...

 One process does data-intensive load. Thus in the ordered mode the
 transaction is tiny but has tons of data buffers attached. If commit
 happens, it takes a long time to sync all the data before the commit
 can proceed... In the writeback mode, we don't wait for data buffers, in
 the journal mode amount of data to be written is really limited by the
 maximum size of a transaction and so we write by much smaller chunks
 and better latency is thus ensured.


 I'm hitting this bug too...it's surprising that there's not many people
 reporting more bugs about this, because it's really annoying.


 There's a patch by Jan Kara (that I'm including here because bugzilla
 didn't include it and took me a while to find it) which I don't know if
 it's supposed to fix the problem , but it'd be interesting to try:


Thanks a lot, but it doesn't fix it.

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

Greetings!

data=ordered mode has proven reliable over the years, and it does this by 
ordering filedata flushes before metadata flushes.  But this sometimes 
causes contention in the order of a 10x slowdown for certain apps, either 
due to the misuse of fsync or due to inherent behaviour like db's, as well 
as inherent starvation issues exposed by the data=ordered mode.

data=writeback mode alleviates data=order mode slowdowns, but only works 
per-mount and is too dangerous to run as a default mode.

This RFC proposes to introduce a tunable which allows to disable fsync and 
changes ordered into writeback writeout on a per-process basis like this:

  echo 1 > /proc/`pidof process`/softsync


Your comments are much welcome!


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[RFC] ext3: per-process soft-syncing data=ordered mode

[Al Boldi]

Greetings!

data=ordered mode has proven reliable over the years, and it does this by 
ordering filedata flushes before metadata flushes.  But this sometimes 
causes contention in the order of a 10x slowdown for certain apps, either 
due to the misuse of fsync or due to inherent behaviour like db's, as well 
as inherent starvation issues exposed by the data=ordered mode.

data=writeback mode alleviates data=order mode slowdowns, but only works 
per-mount and is too dangerous to run as a default mode.

This RFC proposes to introduce a tunable which allows to disable fsync and 
changes ordered into writeback writeout on a per-process basis like this:

  echo 1  /proc/`pidof process`/softsync


Your comments are much welcome!


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: konqueror deadlocks on 2.6.22

[Al Boldi]

Chris Mason wrote:
> Running fsync in data=ordered means that all of the dirty blocks on the FS
> will get written before fsync returns.

Hm, that's strange, I expected this kind of behaviour from data=journal.

data=writeback should return immediatly, which seems it does, but 
data=ordered should only wait for metadata flush, it shouldn't wait for 
filedata flush.  Are you sure it waits for both?


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: konqueror deadlocks on 2.6.22

[Al Boldi]

Ingo Molnar wrote:
> * Oliver Pinter (Pintér Olivér) <[EMAIL PROTECTED]> wrote:
> > and then please update to CFS-v24.1
> > http://people.redhat.com/~mingo/cfs-scheduler/sched-cfs-v2.6.22.15-v24.1
> >.patch
> >
> > > Yes with CFSv20.4, as in the log.
> > >
> > > It also hangs on 2.6.23.13
>
> my feeling is that this is some sort of timing dependent race in
> konqueror/kde/qt that is exposed when a different scheduler is put in.
>
> If it disappears with CFS-v24.1 it is probably just because the timings
> will change again. Would be nice to debug this on the konqueror side and
> analyze why it fails and how. You can probably tune the timings by
> enabling SCHED_DEBUG and tweaking /proc/sys/kernel/*sched* values - in
> particular sched_latency and the granularity settings. Setting wakeup
> granularity to 0 might be one of the things that could make a
> difference.

Thanks Ingo, but Mike suggested that data=writeback may make a difference, 
which it does indeed.

So the bug seems to be related to data=ordered, although I haven't gotten any 
feedback from the ext3 gurus yet.

Seems rather critical though, as data=writeback is a dangerous mode to run.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: konqueror deadlocks on 2.6.22

[Al Boldi]

Ingo Molnar wrote:
 * Oliver Pinter (Pintér Olivér) [EMAIL PROTECTED] wrote:
  and then please update to CFS-v24.1
  http://people.redhat.com/~mingo/cfs-scheduler/sched-cfs-v2.6.22.15-v24.1
 .patch
 
   Yes with CFSv20.4, as in the log.
  
   It also hangs on 2.6.23.13

 my feeling is that this is some sort of timing dependent race in
 konqueror/kde/qt that is exposed when a different scheduler is put in.

 If it disappears with CFS-v24.1 it is probably just because the timings
 will change again. Would be nice to debug this on the konqueror side and
 analyze why it fails and how. You can probably tune the timings by
 enabling SCHED_DEBUG and tweaking /proc/sys/kernel/*sched* values - in
 particular sched_latency and the granularity settings. Setting wakeup
 granularity to 0 might be one of the things that could make a
 difference.

Thanks Ingo, but Mike suggested that data=writeback may make a difference, 
which it does indeed.

So the bug seems to be related to data=ordered, although I haven't gotten any 
feedback from the ext3 gurus yet.

Seems rather critical though, as data=writeback is a dangerous mode to run.


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: konqueror deadlocks on 2.6.22

[Al Boldi]

Chris Mason wrote:
 Running fsync in data=ordered means that all of the dirty blocks on the FS
 will get written before fsync returns.

Hm, that's strange, I expected this kind of behaviour from data=journal.

data=writeback should return immediatly, which seems it does, but 
data=ordered should only wait for metadata flush, it shouldn't wait for 
filedata flush.  Are you sure it waits for both?


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: konqueror deadlocks on 2.6.22

[Al Boldi]

Mike Galbraith wrote:
> On Sat, 2008-01-19 at 21:14 +0300, Al Boldi wrote:
> > I was just attacked by some deadlock issue involving sqlite3 and
> > konqueror. While sqlite3 continues to slowly fill a 7M-record db in
> > transaction mode, konqueror hangs for a few minutes, then continues only
> > to hang again and again.
> >
> > Looks like an fs/blockIO issue involving fsync.
> >
> > As a workaround, is there a way to make fsync soft?
>
> Do you have the fs mounted data=writeback?  A while back, I ran into
> starvation on the order of minutes with my old/full ext2 fs until
> mounting data=writeback.

You are absolutely right.  With data=writeback the hangs completely 
disappear, and sqlite3 insert performance increases 10x fold.

Now data=writeback is known to be faster than data=ordered, but a 10x fold 
increase probably points to some sync contention within the data=ordered 
logic.  Any ideas how this could be fixed?


Thanks a lot!

BTW Mike: Your server bounces my messages.

--
Al


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: konqueror deadlocks on 2.6.22

[Al Boldi]

Oliver Pinter (Pintér Olivér) wrote:
> This kernel is vanilla 2.6.22.y or with CFS?

Yes with CFSv20.4, as in the log.

It also hangs on 2.6.23.13

> On 1/19/08, Al Boldi <[EMAIL PROTECTED]> wrote:
> > I was just attacked by some deadlock issue involving sqlite3 and
> > konqueror. While sqlite3 continues to slowly fill a 7M-record db in
> > transaction mode, konqueror hangs for a few minutes, then continues only
> > to hang again and again.
> >
> > Looks like an fs/blockIO issue involving fsync.
> >
> > As a workaround, is there a way to make fsync soft?


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

konqueror deadlocks on 2.6.22

[Al Boldi]

I was just attacked by some deadlock issue involving sqlite3 and konqueror.
While sqlite3 continues to slowly fill a 7M-record db in transaction mode, 
konqueror hangs for a few minutes, then continues only to hang again and again.

Looks like an fs/blockIO issue involving fsync.

As a workaround, is there a way to make fsync soft?


Thanks!

--
Al

---
Jan 19 20:36:13 localhost kernel: SysRq : Show Blocked State
Jan 19 20:36:13 localhost kernel:   taskPC stack   pid father
Jan 19 20:36:13 localhost kernel: kjournald D c153b4c0 0   951  2
Jan 19 20:36:13 localhost kernel:c1579d70 0046 0010 c153b4c0 
c153b5fc  c1579dbc 0001 
Jan 19 20:36:13 localhost kernel:c1579d78 c03f163e c1527848 c0216d50 
0010 c1527878 0001 d7ca8dc0 
Jan 19 20:36:13 localhost kernel: c153b4c0 c012dcb0 c1579dbc 
c1579dbc 0001 d62092c0 d7ca8dc0 
Jan 19 20:36:13 localhost kernel: Call Trace:
Jan 19 20:36:13 localhost kernel:  [] io_schedule+0xe/0x20
Jan 19 20:36:13 localhost kernel:  [] get_request_wait+0x100/0x120
Jan 19 20:36:13 localhost kernel:  [] 
autoremove_wake_function+0x0/0x50
Jan 19 20:36:13 localhost kernel:  [] 
autoremove_wake_function+0x0/0x50
Jan 19 20:36:13 localhost kernel:  [] elv_merge+0xba/0x150
Jan 19 20:36:13 localhost kernel:  [] __make_request+0x6c/0x2f0
Jan 19 20:36:13 localhost kernel:  [] generic_make_request+0x13f/0x1d0
Jan 19 20:36:14 localhost kernel:  [] __slab_alloc+0x87/0xf0
Jan 19 20:36:14 localhost kernel:  [] mempool_alloc+0x2a/0xc0
Jan 19 20:36:14 localhost kernel:  [] submit_bio+0x46/0xe0
Jan 19 20:36:14 localhost kernel:  [] 
smp_apic_timer_interrupt+0x28/0x30
Jan 19 20:36:14 localhost kernel:  [] apic_timer_interrupt+0x28/0x30
Jan 19 20:36:14 localhost kernel:  [] bio_alloc_bioset+0x7f/0x160
Jan 19 20:36:14 localhost kernel:  [] end_buffer_write_sync+0x0/0x70
Jan 19 20:36:14 localhost kernel:  [] submit_bh+0xd1/0x130
Jan 19 20:36:14 localhost kernel:  [] journal_do_submit_data+0x29/0x30
Jan 19 20:36:14 localhost kernel:  [] 
journal_submit_data_buffers+0x115/0x170
Jan 19 20:36:14 localhost kernel:  [] 
journal_commit_transaction+0x1af/0xc30
Jan 19 20:36:14 localhost kernel:  [] 
autoremove_wake_function+0x0/0x50
Jan 19 20:36:14 localhost kernel:  [] 
autoremove_wake_function+0x0/0x50
Jan 19 20:36:14 localhost kernel:  [] kjournald+0x197/0x1e0
Jan 19 20:36:14 localhost kernel:  [] 
autoremove_wake_function+0x0/0x50
Jan 19 20:36:14 localhost kernel:  [] 
autoremove_wake_function+0x0/0x50
Jan 19 20:36:14 localhost kernel:  [] kjournald+0x0/0x1e0
Jan 19 20:36:14 localhost kernel:  [] kthread+0x6a/0x70
Jan 19 20:36:14 localhost kernel:  [] kthread+0x0/0x70
Jan 19 20:36:14 localhost kernel:  [] kernel_thread_helper+0x7/0x10
Jan 19 20:36:14 localhost kernel:  ===
Jan 19 20:36:14 localhost kernel: konqueror D c14b39f0 0  1922   1918
Jan 19 20:36:14 localhost kernel:dbb13e08 0082 dbb13e08 c14b39f0 
c14b3b2c c1517f40 c1517f00 0009813c 
Jan 19 20:36:14 localhost kernel:dbb13e3c c01aa5e1  c1517f50 
  c14b39f0 c012dcb0 
Jan 19 20:36:14 localhost kernel:dbb13e48 dbb13e48 c1517f50 0001 
0003  c14b39f0 c012dcb0 
Jan 19 20:36:14 localhost kernel: Call Trace:
Jan 19 20:36:14 localhost kernel:  [] log_wait_commit+0xf1/0x140
Jan 19 20:36:14 localhost kernel:  [] 
autoremove_wake_function+0x0/0x50
Jan 19 20:36:14 localhost kernel:  [] 
autoremove_wake_function+0x0/0x50
Jan 19 20:36:14 localhost kernel:  [] journal_stop+0x146/0x1c0
Jan 19 20:36:14 localhost kernel:  [] journal_start+0x93/0xc0
Jan 19 20:36:14 localhost kernel:  [] __writepage+0x0/0x30
Jan 19 20:36:14 localhost kernel:  [] journal_force_commit+0x1c/0x30
Jan 19 20:36:14 localhost kernel:  [] ext3_force_commit+0x25/0x30
Jan 19 20:36:14 localhost kernel:  [] write_inode+0x4b/0x50
Jan 19 20:36:14 localhost kernel:  [] __sync_single_inode+0x1a3/0x1d0
Jan 19 20:36:14 localhost kernel:  [] 
__writeback_single_inode+0x4e/0x1b0
Jan 19 20:36:14 localhost kernel:  [] do_writepages+0x3d/0x50
Jan 19 20:36:14 localhost kernel:  [] 
__filemap_fdatawrite_range+0x87/0x90
Jan 19 20:36:14 localhost kernel:  [] ext3_sync_file+0x98/0xe0
Jan 19 20:36:14 localhost kernel:  [] filemap_fdatawrite+0x23/0x30
Jan 19 20:36:14 localhost kernel:  [] do_fsync+0x6d/0x90
Jan 19 20:36:14 localhost kernel:  [] __do_fsync+0x27/0x50
Jan 19 20:36:14 localhost kernel:  [] syscall_call+0x7/0xb
Jan 19 20:36:14 localhost kernel:  [] svc_seq_show+0x110/0x120
Jan 19 20:36:14 localhost kernel:  ===
Jan 19 20:36:14 localhost kernel: sqlite3   D c4aa4000 0  5507   2021
Jan 19 20:36:14 localhost kernel:dd53bbf0 00200082 c0217bef c4aa4000 
c4aa413c dd53bc40  dd53bc48 
Jan 19 20:36:14 localhost kernel:dd53bbf8 c03f163e c13e0758 c017b4b7 
c03f17d5 c017b490 c89c3690 0002 
Jan 19 20:36:14 localhost kernel:c4aa4000 c017b490 c03f1882 0002 
c89c3690 0002 

konqueror deadlocks on 2.6.22

[Al Boldi]

I was just attacked by some deadlock issue involving sqlite3 and konqueror.
While sqlite3 continues to slowly fill a 7M-record db in transaction mode, 
konqueror hangs for a few minutes, then continues only to hang again and again.

Looks like an fs/blockIO issue involving fsync.

As a workaround, is there a way to make fsync soft?


Thanks!

--
Al

---
Jan 19 20:36:13 localhost kernel: SysRq : Show Blocked State
Jan 19 20:36:13 localhost kernel:   taskPC stack   pid father
Jan 19 20:36:13 localhost kernel: kjournald D c153b4c0 0   951  2
Jan 19 20:36:13 localhost kernel:c1579d70 0046 0010 c153b4c0 
c153b5fc  c1579dbc 0001 
Jan 19 20:36:13 localhost kernel:c1579d78 c03f163e c1527848 c0216d50 
0010 c1527878 0001 d7ca8dc0 
Jan 19 20:36:13 localhost kernel: c153b4c0 c012dcb0 c1579dbc 
c1579dbc 0001 d62092c0 d7ca8dc0 
Jan 19 20:36:13 localhost kernel: Call Trace:
Jan 19 20:36:13 localhost kernel:  [c03f163e] io_schedule+0xe/0x20
Jan 19 20:36:13 localhost kernel:  [c0216d50] get_request_wait+0x100/0x120
Jan 19 20:36:13 localhost kernel:  [c012dcb0] 
autoremove_wake_function+0x0/0x50
Jan 19 20:36:13 localhost kernel:  [c012dcb0] 
autoremove_wake_function+0x0/0x50
Jan 19 20:36:13 localhost kernel:  [c021383a] elv_merge+0xba/0x150
Jan 19 20:36:13 localhost kernel:  [c021779c] __make_request+0x6c/0x2f0
Jan 19 20:36:13 localhost kernel:  [c0217bef] generic_make_request+0x13f/0x1d0
Jan 19 20:36:14 localhost kernel:  [c015a847] __slab_alloc+0x87/0xf0
Jan 19 20:36:14 localhost kernel:  [c014450a] mempool_alloc+0x2a/0xc0
Jan 19 20:36:14 localhost kernel:  [c0217cc6] submit_bio+0x46/0xe0
Jan 19 20:36:14 localhost kernel:  [c0111bb8] 
smp_apic_timer_interrupt+0x28/0x30
Jan 19 20:36:14 localhost kernel:  [c01048f4] apic_timer_interrupt+0x28/0x30
Jan 19 20:36:14 localhost kernel:  [c017ebff] bio_alloc_bioset+0x7f/0x160
Jan 19 20:36:14 localhost kernel:  [c017b5e0] end_buffer_write_sync+0x0/0x70
Jan 19 20:36:14 localhost kernel:  [c017e541] submit_bh+0xd1/0x130
Jan 19 20:36:14 localhost kernel:  [c01a73c9] journal_do_submit_data+0x29/0x30
Jan 19 20:36:14 localhost kernel:  [c01a74e5] 
journal_submit_data_buffers+0x115/0x170
Jan 19 20:36:14 localhost kernel:  [c01a76ef] 
journal_commit_transaction+0x1af/0xc30
Jan 19 20:36:14 localhost kernel:  [c012dcb0] 
autoremove_wake_function+0x0/0x50
Jan 19 20:36:14 localhost kernel:  [c012dcb0] 
autoremove_wake_function+0x0/0x50
Jan 19 20:36:14 localhost kernel:  [c01a9ed7] kjournald+0x197/0x1e0
Jan 19 20:36:14 localhost kernel:  [c012dcb0] 
autoremove_wake_function+0x0/0x50
Jan 19 20:36:14 localhost kernel:  [c012dcb0] 
autoremove_wake_function+0x0/0x50
Jan 19 20:36:14 localhost kernel:  [c01a9d40] kjournald+0x0/0x1e0
Jan 19 20:36:14 localhost kernel:  [c012d89a] kthread+0x6a/0x70
Jan 19 20:36:14 localhost kernel:  [c012d830] kthread+0x0/0x70
Jan 19 20:36:14 localhost kernel:  [c0104a77] kernel_thread_helper+0x7/0x10
Jan 19 20:36:14 localhost kernel:  ===
Jan 19 20:36:14 localhost kernel: konqueror D c14b39f0 0  1922   1918
Jan 19 20:36:14 localhost kernel:dbb13e08 0082 dbb13e08 c14b39f0 
c14b3b2c c1517f40 c1517f00 0009813c 
Jan 19 20:36:14 localhost kernel:dbb13e3c c01aa5e1  c1517f50 
  c14b39f0 c012dcb0 
Jan 19 20:36:14 localhost kernel:dbb13e48 dbb13e48 c1517f50 0001 
0003  c14b39f0 c012dcb0 
Jan 19 20:36:14 localhost kernel: Call Trace:
Jan 19 20:36:14 localhost kernel:  [c01aa5e1] log_wait_commit+0xf1/0x140
Jan 19 20:36:14 localhost kernel:  [c012dcb0] 
autoremove_wake_function+0x0/0x50
Jan 19 20:36:14 localhost kernel:  [c012dcb0] 
autoremove_wake_function+0x0/0x50
Jan 19 20:36:14 localhost kernel:  [c01a6866] journal_stop+0x146/0x1c0
Jan 19 20:36:14 localhost kernel:  [c01a57d3] journal_start+0x93/0xc0
Jan 19 20:36:14 localhost kernel:  [c0147250] __writepage+0x0/0x30
Jan 19 20:36:14 localhost kernel:  [c01a68fc] journal_force_commit+0x1c/0x30
Jan 19 20:36:14 localhost kernel:  [c01a2e85] ext3_force_commit+0x25/0x30
Jan 19 20:36:14 localhost kernel:  [c017807b] write_inode+0x4b/0x50
Jan 19 20:36:14 localhost kernel:  [c0178223] __sync_single_inode+0x1a3/0x1d0
Jan 19 20:36:14 localhost kernel:  [c017829e] 
__writeback_single_inode+0x4e/0x1b0
Jan 19 20:36:14 localhost kernel:  [c01472fd] do_writepages+0x3d/0x50
Jan 19 20:36:14 localhost kernel:  [c0141207] 
__filemap_fdatawrite_range+0x87/0x90
Jan 19 20:36:14 localhost kernel:  [c01982c8] ext3_sync_file+0x98/0xe0
Jan 19 20:36:14 localhost kernel:  [c0141233] filemap_fdatawrite+0x23/0x30
Jan 19 20:36:14 localhost kernel:  [c017aced] do_fsync+0x6d/0x90
Jan 19 20:36:14 localhost kernel:  [c017ad37] __do_fsync+0x27/0x50
Jan 19 20:36:14 localhost kernel:  [c0103f1e] syscall_call+0x7/0xb
Jan 19 20:36:14 localhost kernel:  [c03f] svc_seq_show+0x110/0x120
Jan 19 20:36:14 localhost kernel:  ===
Jan 19 20:36:14 localhost kernel: 

Re: konqueror deadlocks on 2.6.22

[Al Boldi]

Oliver Pinter (Pintér Olivér) wrote:
 This kernel is vanilla 2.6.22.y or with CFS?

Yes with CFSv20.4, as in the log.

It also hangs on 2.6.23.13

 On 1/19/08, Al Boldi [EMAIL PROTECTED] wrote:
  I was just attacked by some deadlock issue involving sqlite3 and
  konqueror. While sqlite3 continues to slowly fill a 7M-record db in
  transaction mode, konqueror hangs for a few minutes, then continues only
  to hang again and again.
 
  Looks like an fs/blockIO issue involving fsync.
 
  As a workaround, is there a way to make fsync soft?


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: konqueror deadlocks on 2.6.22

[Al Boldi]

Mike Galbraith wrote:
 On Sat, 2008-01-19 at 21:14 +0300, Al Boldi wrote:
  I was just attacked by some deadlock issue involving sqlite3 and
  konqueror. While sqlite3 continues to slowly fill a 7M-record db in
  transaction mode, konqueror hangs for a few minutes, then continues only
  to hang again and again.
 
  Looks like an fs/blockIO issue involving fsync.
 
  As a workaround, is there a way to make fsync soft?

 Do you have the fs mounted data=writeback?  A while back, I ran into
 starvation on the order of minutes with my old/full ext2 fs until
 mounting data=writeback.

You are absolutely right.  With data=writeback the hangs completely 
disappear, and sqlite3 insert performance increases 10x fold.

Now data=writeback is known to be faster than data=ordered, but a 10x fold 
increase probably points to some sync contention within the data=ordered 
logic.  Any ideas how this could be fixed?


Thanks a lot!

BTW Mike: Your server bounces my messages.

--
Al


--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 0/5] USB Kconfig: Reorganize USB Kconfig Menu

[Al Boldi]

Stefan Richter wrote:
> Greg KH wrote:
> > On Sat, Jan 12, 2008 at 01:20:46PM +0300, Al Boldi wrote:
> >> Greg KH wrote:
> >>> On Sat, Jan 05, 2008 at 06:40:38PM +0300, Al Boldi wrote:
> >>>> Reorganize USB Kconfig Menu, and move USB_GADGET out into the Device
> >>>> Driver Menu. ?This helps the USB Kconfig Menu to be more
> >>>> logical/usable.
> >>>>
> >>>> Patchset against 2.6.23
> >>>
> >>> So what was the final verdict in this patch set?
> >>
> >> IMHO, it's a lot better than what we have right now, and it's split up
> >> so that you can pick and choose what you think is useful.
> >>
> >>> Can you rsend this against 2.6.24-rc7 with the requested changes (if
> >>> any) in it?
> >>
> >> The only critical change is in patch 2/5:
> >>
> >> menuconfig USB_STORAGE
> >> tristate "USB Mass Storage support"
> >> -   depends on USB && SCSI
> >> +   depends on USB && BLOCK
> >> +   select SCSI
> >>
> >>
> >> I was hoping you could take care of it, or maybe wait until 2.6.25 is
> >> out.
> >
> > Well, if you want such a change to go into 2.6.25, I need the patches
> > now :)
> >
> > So, can you respin these against 2.6.24-rc7, with the above fix, so that
> > I can apply them and test them out?
>
> May I repeat my request that nobody does "select SCSI", please?
>
> Al Boldy, would you please explain like I asked you:
>   - what is wrong with the current solution which tells the user to
> first enable SCSI to get the USB_STORAGE option,
>   - whether there are frequent end-user requests which demonstrate
> that many people currently don't realize how to enable USB_STORAGE.
>
> Please use "select" carefully, particularly *only* for options which
> enable simple compact library-like code which
>   - is small,
>   - doesn't itself depend on anything,
>   - doesn't has further configuration options in some other menu.
> None of this applies to CONFIG_SCSI.

I thought we discussed this before.  We do it here, for the same reasons 
libata does it.  In any case, the patch is optional.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 0/5] USB Kconfig: Reorganize USB Kconfig Menu

[Al Boldi]

Stefan Richter wrote:
 Greg KH wrote:
  On Sat, Jan 12, 2008 at 01:20:46PM +0300, Al Boldi wrote:
  Greg KH wrote:
  On Sat, Jan 05, 2008 at 06:40:38PM +0300, Al Boldi wrote:
  Reorganize USB Kconfig Menu, and move USB_GADGET out into the Device
  Driver Menu. ?This helps the USB Kconfig Menu to be more
  logical/usable.
 
  Patchset against 2.6.23
 
  So what was the final verdict in this patch set?
 
  IMHO, it's a lot better than what we have right now, and it's split up
  so that you can pick and choose what you think is useful.
 
  Can you rsend this against 2.6.24-rc7 with the requested changes (if
  any) in it?
 
  The only critical change is in patch 2/5:
 
  menuconfig USB_STORAGE
  tristate USB Mass Storage support
  -   depends on USB  SCSI
  +   depends on USB  BLOCK
  +   select SCSI
 
 
  I was hoping you could take care of it, or maybe wait until 2.6.25 is
  out.
 
  Well, if you want such a change to go into 2.6.25, I need the patches
  now :)
 
  So, can you respin these against 2.6.24-rc7, with the above fix, so that
  I can apply them and test them out?

 May I repeat my request that nobody does select SCSI, please?

 Al Boldy, would you please explain like I asked you:
   - what is wrong with the current solution which tells the user to
 first enable SCSI to get the USB_STORAGE option,
   - whether there are frequent end-user requests which demonstrate
 that many people currently don't realize how to enable USB_STORAGE.

 Please use select carefully, particularly *only* for options which
 enable simple compact library-like code which
   - is small,
   - doesn't itself depend on anything,
   - doesn't has further configuration options in some other menu.
 None of this applies to CONFIG_SCSI.

I thought we discussed this before.  We do it here, for the same reasons 
libata does it.  In any case, the patch is optional.


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFD] Incremental fsck

[Al Boldi]

Theodore Tso wrote:
> On Wed, Jan 09, 2008 at 02:52:14PM +0300, Al Boldi wrote:
> > Ok, but let's look at this a bit more opportunistic / optimistic.
> >
> > Even after a black-out shutdown, the corruption is pretty minimal, using
> > ext3fs at least.
>
> After a unclean shutdown, assuming you have decent hardware that
> doesn't lie about when blocks hit iron oxide, you shouldn't have any
> corruption at all.  If you have crappy hardware, then all bets are off

Maybe with barriers...

> > So let's take advantage of this fact and do an optimistic fsck, to
> > assure integrity per-dir, and assume no external corruption.  Then
> > we release this checked dir to the wild (optionally ro), and check
> > the next.  Once we find external inconsistencies we either fix it
> > unconditionally, based on some preconfigured actions, or present the
> > user with options.
>
> So what can you check?  The *only* thing you can check is whether or
> not the directory syntax looks sane, whether the inode structure looks
> sane, and whether or not the blocks reported as belong to an inode
> looks sane.

Which would make this dir/area ready for read/write access.

> What is very hard to check is whether or not the link count on the
> inode is correct.  Suppose the link count is 1, but there are actually
> two directory entries pointing at it.  Now when someone unlinks the
> file through one of the directory hard entries, the link count will go
> to zero, and the blocks will start to get reused, even though the
> inode is still accessible via another pathname.  Oops.  Data Loss.

We could buffer this, and only actually overwrite when we are completely 
finished with the fsck.

> This is why doing incremental, on-line fsck'ing is *hard*.  You're not
> going to find this while doing each directory one at a time, and if
> the filesystem is changing out from under you, it gets worse.  And
> it's not just the hard link count.  There is a similar issue with the
> block allocation bitmap.  Detecting the case where two files are
> simultaneously can't be done if you are doing it incrementally, and if
> the filesystem is changing out from under you, it's impossible, unless
> you also have the filesystem telling you every single change while it
> is happening, and you keep an insane amount of bookkeeping.

Ok, you have a point, so how about we change the implementation detail a bit, 
from external fsck to internal fsck, leveraging the internal fs bookkeeping, 
while allowing immediate but controlled read/write access.


Thanks for more thoughts!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 0/5] USB Kconfig: Reorganize USB Kconfig Menu

[Al Boldi]

Greg KH wrote:
> On Sat, Jan 12, 2008 at 01:20:46PM +0300, Al Boldi wrote:
> > Greg KH wrote:
> > > On Sat, Jan 05, 2008 at 06:40:38PM +0300, Al Boldi wrote:
> > > > Reorganize USB Kconfig Menu, and move USB_GADGET out into the Device
> > > > Driver Menu. ?This helps the USB Kconfig Menu to be more
> > > > logical/usable.
> > > >
> > > > Patchset against 2.6.23
> > >
> > > So what was the final verdict in this patch set?
> >
> > IMHO, it's a lot better than what we have right now, and it's split up
> > so that you can pick and choose what you think is useful.
> >
> > > Can you rsend this against 2.6.24-rc7 with the requested changes (if
> > > any) in it?
> >
> > The only critical change is in patch 2/5:
> >
> > menuconfig USB_STORAGE
> > tristate "USB Mass Storage support"
> > -   depends on USB && SCSI
> > +   depends on USB && BLOCK
> > +   select SCSI
> >
> >
> > I was hoping you could take care of it, or maybe wait until 2.6.25 is
> > out.
>
> Well, if you want such a change to go into 2.6.25, I need the patches
> now :)
>
> So, can you respin these against 2.6.24-rc7, with the above fix, so that
> I can apply them and test them out?

Not so easy, as I don't have rc7 available.  You know, dialup.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFD] Incremental fsck

[Al Boldi]

Theodore Tso wrote:
 On Wed, Jan 09, 2008 at 02:52:14PM +0300, Al Boldi wrote:
  Ok, but let's look at this a bit more opportunistic / optimistic.
 
  Even after a black-out shutdown, the corruption is pretty minimal, using
  ext3fs at least.

 After a unclean shutdown, assuming you have decent hardware that
 doesn't lie about when blocks hit iron oxide, you shouldn't have any
 corruption at all.  If you have crappy hardware, then all bets are off

Maybe with barriers...

  So let's take advantage of this fact and do an optimistic fsck, to
  assure integrity per-dir, and assume no external corruption.  Then
  we release this checked dir to the wild (optionally ro), and check
  the next.  Once we find external inconsistencies we either fix it
  unconditionally, based on some preconfigured actions, or present the
  user with options.

 So what can you check?  The *only* thing you can check is whether or
 not the directory syntax looks sane, whether the inode structure looks
 sane, and whether or not the blocks reported as belong to an inode
 looks sane.

Which would make this dir/area ready for read/write access.

 What is very hard to check is whether or not the link count on the
 inode is correct.  Suppose the link count is 1, but there are actually
 two directory entries pointing at it.  Now when someone unlinks the
 file through one of the directory hard entries, the link count will go
 to zero, and the blocks will start to get reused, even though the
 inode is still accessible via another pathname.  Oops.  Data Loss.

We could buffer this, and only actually overwrite when we are completely 
finished with the fsck.

 This is why doing incremental, on-line fsck'ing is *hard*.  You're not
 going to find this while doing each directory one at a time, and if
 the filesystem is changing out from under you, it gets worse.  And
 it's not just the hard link count.  There is a similar issue with the
 block allocation bitmap.  Detecting the case where two files are
 simultaneously can't be done if you are doing it incrementally, and if
 the filesystem is changing out from under you, it's impossible, unless
 you also have the filesystem telling you every single change while it
 is happening, and you keep an insane amount of bookkeeping.

Ok, you have a point, so how about we change the implementation detail a bit, 
from external fsck to internal fsck, leveraging the internal fs bookkeeping, 
while allowing immediate but controlled read/write access.


Thanks for more thoughts!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 0/5] USB Kconfig: Reorganize USB Kconfig Menu

[Al Boldi]

Greg KH wrote:
 On Sat, Jan 12, 2008 at 01:20:46PM +0300, Al Boldi wrote:
  Greg KH wrote:
   On Sat, Jan 05, 2008 at 06:40:38PM +0300, Al Boldi wrote:
Reorganize USB Kconfig Menu, and move USB_GADGET out into the Device
Driver Menu. ?This helps the USB Kconfig Menu to be more
logical/usable.
   
Patchset against 2.6.23
  
   So what was the final verdict in this patch set?
 
  IMHO, it's a lot better than what we have right now, and it's split up
  so that you can pick and choose what you think is useful.
 
   Can you rsend this against 2.6.24-rc7 with the requested changes (if
   any) in it?
 
  The only critical change is in patch 2/5:
 
  menuconfig USB_STORAGE
  tristate USB Mass Storage support
  -   depends on USB  SCSI
  +   depends on USB  BLOCK
  +   select SCSI
 
 
  I was hoping you could take care of it, or maybe wait until 2.6.25 is
  out.

 Well, if you want such a change to go into 2.6.25, I need the patches
 now :)

 So, can you respin these against 2.6.24-rc7, with the above fix, so that
 I can apply them and test them out?

Not so easy, as I don't have rc7 available.  You know, dialup.


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 0/5] USB Kconfig: Reorganize USB Kconfig Menu

[Al Boldi]

Greg KH wrote:
> On Sat, Jan 05, 2008 at 06:40:38PM +0300, Al Boldi wrote:
> > Reorganize USB Kconfig Menu, and move USB_GADGET out into the Device
> > Driver Menu. ?This helps the USB Kconfig Menu to be more logical/usable.
> >
> > Patchset against 2.6.23
>
> So what was the final verdict in this patch set?

IMHO, it's a lot better than what we have right now, and it's split up so 
that you can pick and choose what you think is useful.

> Can you rsend this against 2.6.24-rc7 with the requested changes (if
> any) in it?

The only critical change is in patch 2/5:

menuconfig USB_STORAGE
tristate "USB Mass Storage support"
-   depends on USB && SCSI
+   depends on USB && BLOCK
+   select SCSI


I was hoping you could take care of it, or maybe wait until 2.6.25 is out.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFD] Incremental fsck

[Al Boldi]

Bodo Eggert wrote:
> Al Boldi <[EMAIL PROTECTED]> wrote:
> > Even after a black-out shutdown, the corruption is pretty minimal, using
> > ext3fs at least.  So let's take advantage of this fact and do an
> > optimistic fsck, to assure integrity per-dir, and assume no external
> > corruption.  Then we release this checked dir to the wild (optionally
> > ro), and check the next. Once we find external inconsistencies we either
> > fix it unconditionally, based on some preconfigured actions, or present
> > the user with options.
>
> Maybe we can know the changes that need to be done in order to fix the
> filesystem. Let's record this information in - eh - let's call it a
> journal!

Don't mistake data=journal as an fsck replacement.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFD] Incremental fsck

[Al Boldi]

Bodo Eggert wrote:
 Al Boldi [EMAIL PROTECTED] wrote:
  Even after a black-out shutdown, the corruption is pretty minimal, using
  ext3fs at least.  So let's take advantage of this fact and do an
  optimistic fsck, to assure integrity per-dir, and assume no external
  corruption.  Then we release this checked dir to the wild (optionally
  ro), and check the next. Once we find external inconsistencies we either
  fix it unconditionally, based on some preconfigured actions, or present
  the user with options.

 Maybe we can know the changes that need to be done in order to fix the
 filesystem. Let's record this information in - eh - let's call it a
 journal!

Don't mistake data=journal as an fsck replacement.


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 0/5] USB Kconfig: Reorganize USB Kconfig Menu

[Al Boldi]

Greg KH wrote:
 On Sat, Jan 05, 2008 at 06:40:38PM +0300, Al Boldi wrote:
  Reorganize USB Kconfig Menu, and move USB_GADGET out into the Device
  Driver Menu. ?This helps the USB Kconfig Menu to be more logical/usable.
 
  Patchset against 2.6.23

 So what was the final verdict in this patch set?

IMHO, it's a lot better than what we have right now, and it's split up so 
that you can pick and choose what you think is useful.

 Can you rsend this against 2.6.24-rc7 with the requested changes (if
 any) in it?

The only critical change is in patch 2/5:

menuconfig USB_STORAGE
tristate USB Mass Storage support
-   depends on USB  SCSI
+   depends on USB  BLOCK
+   select SCSI


I was hoping you could take care of it, or maybe wait until 2.6.25 is out.


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFD] Incremental fsck

[Al Boldi]

Rik van Riel wrote:
> Al Boldi <[EMAIL PROTECTED]> wrote:
> > Ok, but let's look at this a bit more opportunistic / optimistic.
>
> You can't play fast and loose with data integrity.

Correct, but you have to be realistic...

> Besides, if we looked at things optimistically, we would conclude
> that no fsck will be needed,

And that's the reality, because people are mostly optimistic and feel 
extremely tempted to just force-mount a dirty ext3fs, instead of waiting 
hours-on-end for a complete fsck, which mostly comes back with some benign 
"inode should be zero" warning.

> ever :)

Well not ever, but most people probably fsck during scheduled shutdowns, or 
when they are forced to, due to online fs accessibility errors.

> > > http://infohost.nmt.edu/~val/review/chunkfs.pdf
>
> You will really want to read this paper, if you haven't already.

Definitely a good read, but attacking the problem from a completely different 
POV.

BTW:  Dropped some cc's due to bounces.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFD] Incremental fsck

[Al Boldi]

Rik van Riel wrote:
 Al Boldi [EMAIL PROTECTED] wrote:
  Ok, but let's look at this a bit more opportunistic / optimistic.

 You can't play fast and loose with data integrity.

Correct, but you have to be realistic...

 Besides, if we looked at things optimistically, we would conclude
 that no fsck will be needed,

And that's the reality, because people are mostly optimistic and feel 
extremely tempted to just force-mount a dirty ext3fs, instead of waiting 
hours-on-end for a complete fsck, which mostly comes back with some benign 
inode should be zero warning.

 ever :)

Well not ever, but most people probably fsck during scheduled shutdowns, or 
when they are forced to, due to online fs accessibility errors.

   http://infohost.nmt.edu/~val/review/chunkfs.pdf

 You will really want to read this paper, if you haven't already.

Definitely a good read, but attacking the problem from a completely different 
POV.

BTW:  Dropped some cc's due to bounces.


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFD] Incremental fsck

[Al Boldi]

Valerie Henson wrote:
> On Jan 8, 2008 8:40 PM, Al Boldi <[EMAIL PROTECTED]> wrote:
> > Rik van Riel wrote:
> > > Al Boldi <[EMAIL PROTECTED]> wrote:
> > > > Has there been some thought about an incremental fsck?
> > > >
> > > > You know, somehow fencing a sub-dir to do an online fsck?
> > >
> > > Search for "chunkfs"
> >
> > Sure, and there is TileFS too.
> >
> > But why wouldn't it be possible to do this on the current fs
> > infrastructure, using just a smart fsck, working incrementally on some
> > sub-dir?
>
> Several data structures are file system wide and require finding every
> allocated file and block to check that they are correct.  In
> particular, block and inode bitmaps can't be checked per subdirectory.

Ok, but let's look at this a bit more opportunistic / optimistic.

Even after a black-out shutdown, the corruption is pretty minimal, using 
ext3fs at least.  So let's take advantage of this fact and do an optimistic 
fsck, to assure integrity per-dir, and assume no external corruption.  Then 
we release this checked dir to the wild (optionally ro), and check the next.  
Once we find external inconsistencies we either fix it unconditionally, 
based on some preconfigured actions, or present the user with options.

All this could be per-dir or using some form of on-the-fly file-block-zoning.

And there probably is a lot more to it, but it should conceptually be 
possible, with more thoughts though...

> http://infohost.nmt.edu/~val/review/chunkfs.pdf


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFD] Incremental fsck

[Al Boldi]

Valerie Henson wrote:
 On Jan 8, 2008 8:40 PM, Al Boldi [EMAIL PROTECTED] wrote:
  Rik van Riel wrote:
   Al Boldi [EMAIL PROTECTED] wrote:
Has there been some thought about an incremental fsck?
   
You know, somehow fencing a sub-dir to do an online fsck?
  
   Search for chunkfs
 
  Sure, and there is TileFS too.
 
  But why wouldn't it be possible to do this on the current fs
  infrastructure, using just a smart fsck, working incrementally on some
  sub-dir?

 Several data structures are file system wide and require finding every
 allocated file and block to check that they are correct.  In
 particular, block and inode bitmaps can't be checked per subdirectory.

Ok, but let's look at this a bit more opportunistic / optimistic.

Even after a black-out shutdown, the corruption is pretty minimal, using 
ext3fs at least.  So let's take advantage of this fact and do an optimistic 
fsck, to assure integrity per-dir, and assume no external corruption.  Then 
we release this checked dir to the wild (optionally ro), and check the next.  
Once we find external inconsistencies we either fix it unconditionally, 
based on some preconfigured actions, or present the user with options.

All this could be per-dir or using some form of on-the-fly file-block-zoning.

And there probably is a lot more to it, but it should conceptually be 
possible, with more thoughts though...

 http://infohost.nmt.edu/~val/review/chunkfs.pdf


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFD] Incremental fsck

[Al Boldi]

Rik van Riel wrote:
> Al Boldi <[EMAIL PROTECTED]> wrote:
> > Has there been some thought about an incremental fsck?
> >
> > You know, somehow fencing a sub-dir to do an online fsck?
>
> Search for "chunkfs"

Sure, and there is TileFS too.

But why wouldn't it be possible to do this on the current fs infrastructure, 
using just a smart fsck, working incrementally on some sub-dir?


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[RFD] Incremental fsck

[Al Boldi]

Andi Kleen wrote:
> Theodore Tso <[EMAIL PROTECTED]> writes:
> > Now, there are good reasons for doing periodic checks every N mounts
> > and after M months.  And it has to do with PC class hardware.  (Ted's
> > aphorism: "PC class hardware is cr*p").
>
> If these reasons are good ones (some skepticism here) then the correct
> way to really handle this would be to do regular background scrubbing
> during runtime; ideally with metadata checksums so that you can actually
> detect all corruption.
>
> But since fsck is so slow and disks are so big this whole thing
> is a ticking time bomb now. e.g. it is not uncommon to require tens
> of minutes or even hours of fsck time and some server that reboots
> only every few months will eat that when it happens to reboot.
> This means you get a quite long downtime.

Has there been some thought about an incremental fsck?

You know, somehow fencing a sub-dir to do an online fsck?


Thanks for some thoughts!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[RFD] Incremental fsck

[Al Boldi]

Andi Kleen wrote:
 Theodore Tso [EMAIL PROTECTED] writes:
  Now, there are good reasons for doing periodic checks every N mounts
  and after M months.  And it has to do with PC class hardware.  (Ted's
  aphorism: PC class hardware is cr*p).

 If these reasons are good ones (some skepticism here) then the correct
 way to really handle this would be to do regular background scrubbing
 during runtime; ideally with metadata checksums so that you can actually
 detect all corruption.

 But since fsck is so slow and disks are so big this whole thing
 is a ticking time bomb now. e.g. it is not uncommon to require tens
 of minutes or even hours of fsck time and some server that reboots
 only every few months will eat that when it happens to reboot.
 This means you get a quite long downtime.

Has there been some thought about an incremental fsck?

You know, somehow fencing a sub-dir to do an online fsck?


Thanks for some thoughts!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFD] Incremental fsck

[Al Boldi]

Rik van Riel wrote:
 Al Boldi [EMAIL PROTECTED] wrote:
  Has there been some thought about an incremental fsck?
 
  You know, somehow fencing a sub-dir to do an online fsck?

 Search for chunkfs

Sure, and there is TileFS too.

But why wouldn't it be possible to do this on the current fs infrastructure, 
using just a smart fsck, working incrementally on some sub-dir?


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 2/5] USB Kconfig: Select SCSI for USB Mass Storage support

[Al Boldi]

Stefan Richter wrote:
> Al Boldi wrote:
> > Sam Ravnborg wrote:
> >> And that requires you to use select only to select symbols with
> >> no dependencies.
> >>
> >> In this case we do not know if BLOCK is enabled or not.
> >
> > Good point!  How about we solve it like this:
> >
> > menuconfig USB_STORAGE
> > tristate "USB Mass Storage support"
> > -   depends on USB && SCSI
> > +   depends on USB && BLOCK
> > +   select SCSI
>
> Still wrong. SCSI also needs HAS_DMA and SCSI_DMA.

I don't think so.  SCSI selects SCSI_DMA, it doesn't depend on it.  So it's 
safe to select SCSI here, as libata does it the same way.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 2/5] USB Kconfig: Select SCSI for USB Mass Storage support

[Al Boldi]

Stefan Richter wrote:
 Al Boldi wrote:
  Sam Ravnborg wrote:
  And that requires you to use select only to select symbols with
  no dependencies.
 
  In this case we do not know if BLOCK is enabled or not.
 
  Good point!  How about we solve it like this:
 
  menuconfig USB_STORAGE
  tristate USB Mass Storage support
  -   depends on USB  SCSI
  +   depends on USB  BLOCK
  +   select SCSI

 Still wrong. SCSI also needs HAS_DMA and SCSI_DMA.

I don't think so.  SCSI selects SCSI_DMA, it doesn't depend on it.  So it's 
safe to select SCSI here, as libata does it the same way.


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 2/5] USB Kconfig: Select SCSI for USB Mass Storage support

[Al Boldi]

Sam Ravnborg wrote:
> On Sat, Jan 05, 2008 at 11:03:30PM +0200, Adrian Bunk wrote:
> > For kconfig users, "select" is _much_ better than sending them through
> > different menus.
>
> Only if used within the current limitations of Kconfig.
> And that requires you to use select only to select symbols with
> no dependencies.
>
> In this case we do not know if BLOCK is enabled or not.

Good point!  How about we solve it like this:

menuconfig USB_STORAGE
tristate "USB Mass Storage support"
-   depends on USB && SCSI
+   depends on USB && BLOCK
+   select SCSI


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: Forcing modes in libata (was: SATA buffered read VERY slow)

[Al Boldi]

Alan Cox wrote:
> Al Boldi <[EMAIL PROTECTED]> wrote:
> > What's hindering the ability to force a mode in libata, as is possible
> > with the normal ide-driver?
>
> We want it to be correct and race free. That means we have to synchronize
> all the devices on the controller, quiesce them and recompute the speeds
> for each device then turn them all back on and resume command processing.
>
> It is quite hairy although Tejun's EH work has provided the framework for
> all of this.
>
> For now you can boot with libata.dma=1 to select DMA on disks but not CD

Great, but why isn't this in the documentation?


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 5/5] USB Kconfig: Source "drivers/usb/gadget/Kconfig" for ARCH=v850

[Al Boldi]

Source "drivers/usb/gadget/Kconfig" for ARCH=v850.


Cc: David Brownell <[EMAIL PROTECTED]>
Cc: Greg KH <[EMAIL PROTECTED]>
Cc: Andrew Morton <[EMAIL PROTECTED]>
Signed-off-by: Al Boldi <[EMAIL PROTECTED]>

---

--- 23.a/arch/v850/Kconfig
+++ 23.b/arch/v850/Kconfig
@@ -331,6 +331,8 @@ source "sound/Kconfig"
 
 source "drivers/usb/Kconfig"
 
+source "drivers/usb/gadget/Kconfig"
+
 source "arch/v850/Kconfig.debug"
 
 source "security/Kconfig"

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH 0/5] USB Kconfig: Reorganize USB Kconfig Menu

[Al Boldi]

Reorganize USB Kconfig Menu, and move USB_GADGET out into the Device Driver
Menu.  This helps the USB Kconfig Menu to be more logical/usable.

Patchset against 2.6.23

 arch/arm/Kconfig|2 ++
 arch/v850/Kconfig   |2 ++
 drivers/Kconfig |2 ++
 drivers/usb/Kconfig |   26 --
 drivers/usb/atm/Kconfig |   12 ++--
 drivers/usb/class/Kconfig   |6 +-
 drivers/usb/core/Kconfig|7 +++
 drivers/usb/gadget/Kconfig  |   10 +-
 drivers/usb/host/Kconfig|3 ++-
 drivers/usb/image/Kconfig   |7 ++-
 drivers/usb/misc/Kconfig|6 +-
 drivers/usb/serial/Kconfig  |   10 --
 drivers/usb/storage/Kconfig |   25 ++---
 13 files changed, 72 insertions(+), 46 deletions(-)



Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 4/5] USB Kconfig: Source "drivers/usb/gadget/Kconfig" for ARCH=arm

[Al Boldi]

From: David Brownell <[EMAIL PROTECTED]>

Source "drivers/usb/gadget/Kconfig" for ARCH=arm.


Cc: David Brownell <[EMAIL PROTECTED]>
Cc: Greg KH <[EMAIL PROTECTED]>
Cc: Andrew Morton <[EMAIL PROTECTED]>
Signed-off-by: Al Boldi <[EMAIL PROTECTED]>

---

--- 23.a/arch/arm/Kconfig
+++ 23.b/arch/arm/Kconfig
@@ -1049,6 +1049,8 @@ source "drivers/hid/Kconfig"
 
 source "drivers/usb/Kconfig"
 
+source "drivers/usb/gadget/Kconfig"
+
 source "drivers/mmc/Kconfig"
 
 source "drivers/rtc/Kconfig"

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 2/5] USB Kconfig: Select SCSI for USB Mass Storage support

[Al Boldi]

Select SCSI for USB Mass Storage support.


Cc: David Brownell <[EMAIL PROTECTED]>
Cc: Greg KH <[EMAIL PROTECTED]>
Cc: Andrew Morton <[EMAIL PROTECTED]>
Signed-off-by: Al Boldi <[EMAIL PROTECTED]>

---

--- 23.a/drivers/usb/storage/Kconfig
+++ 23.b/drivers/usb/storage/Kconfig
@@ -2,14 +2,10 @@
 # USB Storage driver configuration
 #
 
-comment "NOTE: USB_STORAGE needs SCSI, and 'SCSI disk support' may"
-   depends on USB
-comment "also be needed; see USB_STORAGE Help for more information"
-   depends on USB
-
 menuconfig USB_STORAGE
tristate "USB Mass Storage support"
-   depends on USB && SCSI
+   depends on USB
+   select SCSI
---help---
  Say Y here if you want to connect USB mass storage devices to your
  computer's USB port. This is the driver you need for USB
@@ -18,13 +14,17 @@ menuconfig USB_STORAGE
  similar devices. This driver may also be used for some cameras
  and card readers.
 
- This option depends on 'SCSI' support being enabled, but you
- probably also need 'SCSI device support: SCSI disk support'
+ This option selects 'SCSI' support , but you probably
+ also need 'SCSI device support: SCSI disk support'
  (BLK_DEV_SD) for most USB storage devices.
 
  To compile this driver as a module, choose M here: the
  module will be called usb-storage.
 
+if USB_STORAGE
+comment "NOTE: USB_STORAGE selects SCSI, but 'SCSI disk support' may"
+comment "also be needed; see USB_STORAGE Help for more information"
+
 config USB_STORAGE_DEBUG
bool "USB Mass Storage verbose debug"
depends on USB_STORAGE
@@ -147,6 +147,8 @@ config USB_STORAGE_KARMA
  on the resulting scsi device node returns the Karma to normal
  operation.
 
+endif # USB_STORAGE
+
 config USB_LIBUSUAL
bool "The shared table of common (or usual) storage devices"
depends on USB

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 1/5] USB Kconfig: Reorganize USB Kconfig Menu

[Al Boldi]

Reorganize USB Kconfig Menu, and move USB_GADGET out into the Device Driver
Menu.

Cc: David Brownell <[EMAIL PROTECTED]>
Cc: Greg KH <[EMAIL PROTECTED]>
Cc: Andrew Morton <[EMAIL PROTECTED]>
Signed-off-by: Al Boldi <[EMAIL PROTECTED]>

---

--- 23.a/drivers/Kconfig
+++ 23.b/drivers/Kconfig
@@ -70,6 +70,8 @@ source "drivers/hid/Kconfig"
 
 source "drivers/usb/Kconfig"
 
+source "drivers/usb/gadget/Kconfig"
+
 source "drivers/mmc/Kconfig"
 
 source "drivers/leds/Kconfig"
--- 23.a/drivers/usb/Kconfig
+++ 23.b/drivers/usb/Kconfig
@@ -2,8 +2,8 @@
 # USB device configuration
 #
 
-menuconfig USB_SUPPORT
-   bool "USB support"
+config USB_SUPPORT
+   bool
depends on HAS_IOMEM
default y
---help---
@@ -52,8 +52,8 @@ config USB_ARCH_HAS_EHCI
default PCI
 
 # ARM SA chips have a non-PCI based "OHCI-compatible" USB host interface.
-config USB
-   tristate "Support for Host-side USB"
+menuconfig USB
+   tristate "Host-side USB"
depends on USB_ARCH_HAS_HCD
---help---
  Universal Serial Bus (USB) is a specification for a serial bus
@@ -87,21 +87,16 @@ config USB
  To compile this driver as a module, choose M here: the
  module will be called usbcore.
 
+if USB
+
 source "drivers/usb/core/Kconfig"
 
 source "drivers/usb/host/Kconfig"
 
-source "drivers/usb/class/Kconfig"
-
 source "drivers/usb/storage/Kconfig"
 
-source "drivers/usb/image/Kconfig"
-
 source "drivers/usb/mon/Kconfig"
 
-comment "USB port drivers"
-   depends on USB
-
 config USB_USS720
tristate "USS720 parport driver"
depends on USB && PARPORT
@@ -133,10 +128,13 @@ config USB_USS720
 
 source "drivers/usb/serial/Kconfig"
 
-source "drivers/usb/misc/Kconfig"
-
 source "drivers/usb/atm/Kconfig"
 
-source "drivers/usb/gadget/Kconfig"
+source "drivers/usb/class/Kconfig"
+
+source "drivers/usb/image/Kconfig"
+
+source "drivers/usb/misc/Kconfig"
 
+endif # USB
 endif # USB_SUPPORT
--- 23.a/drivers/usb/serial/Kconfig
+++ 23.b/drivers/usb/serial/Kconfig
@@ -2,10 +2,7 @@
 # USB Serial device configuration
 #
 
-menu "USB Serial Converter support"
-   depends on USB!=n
-
-config USB_SERIAL
+menuconfig USB_SERIAL
tristate "USB Serial Converter support"
depends on USB
---help---
@@ -20,6 +17,8 @@ config USB_SERIAL
  To compile this driver as a module, choose M here: the
  module will be called usbserial.
 
+if USB_SERIAL
+
 config USB_SERIAL_CONSOLE
bool "USB Serial Console device support (EXPERIMENTAL)"
depends on USB_SERIAL=y && EXPERIMENTAL
@@ -581,5 +580,4 @@ config USB_EZUSB
default y
 
 
-endmenu
-
+endif # USB_SERIAL
--- 23.a/drivers/usb/core/Kconfig
+++ 23.b/drivers/usb/core/Kconfig
@@ -1,6 +1,12 @@
 #
 # USB Core configuration
 #
+menuconfig USB_CORE_MENU
+   bool "USB Core Config"
+   depends on USB
+
+if USB_CORE_MENU
+
 config USB_DEBUG
bool "USB verbose debug messages"
depends on USB
@@ -142,3 +148,4 @@ config USB_OTG_BLACKLIST_HUB
  external hubs.  OTG hosts are allowed to reduce hardware
  and software costs by not supporting external hubs.
 
+endif # USB_CORE_MENU
--- 23.a/drivers/usb/host/Kconfig
+++ 23.b/drivers/usb/host/Kconfig
@@ -1,7 +1,7 @@
 #
 # USB Host Controller Drivers
 #
-comment "USB Host Controller Drivers"
+menu "USB Host Controller Drivers"
depends on USB
 
 config USB_EHCI_HCD
@@ -248,3 +248,4 @@ config USB_R8A66597_HCD
  To compile this driver as a module, choose M here: the
  module will be called r8a66597-hcd.
 
+endmenu
--- 23.a/drivers/usb/class/Kconfig
+++ 23.b/drivers/usb/class/Kconfig
@@ -1,9 +1,12 @@
 #
 # USB Class driver configuration
 #
-comment "USB Device Class drivers"
+menuconfig USB_DEVICE_CLASS_MENU
+   bool "USB Device Class drivers"
depends on USB
 
+if USB_DEVICE_CLASS_MENU
+
 config USB_ACM
tristate "USB Modem (CDC ACM) support"
depends on USB
@@ -29,3 +32,4 @@ config USB_PRINTER
  To compile this driver as a module, choose M here: the
  module will be called usblp.
 
+endif # USB_DEVICE_CLASS_MENU
--- 23.a/drivers/usb/gadget/Kconfig
+++ 23.b/drivers/usb/gadget/Kconfig
@@ -12,10 +12,8 @@
 # With help from a special transceiver and a "Mini-AB" jack, systems with
 # both kinds of controller can also support "USB On-the-Go" (CONFIG_USB_OTG).
 #
-menu "USB Gadget Support"
-
-config USB_GADGET
-   tristate "Support for USB Gadgets"
+menuconfig USB_GADGET
+   tristate "Peripheral-side USB"

Re: [PATCH 3/5] USB Kconfig: Select ATM for USB DSL modem support

[Al Boldi]

Select ATM for USB DSL modem support.


Cc: David Brownell <[EMAIL PROTECTED]>
Cc: Greg KH <[EMAIL PROTECTED]>
Cc: Andrew Morton <[EMAIL PROTECTED]>
Signed-off-by: Al Boldi <[EMAIL PROTECTED]>

---

--- 23.a/drivers/usb/atm/Kconfig
+++ 23.b/drivers/usb/atm/Kconfig
@@ -4,7 +4,8 @@
 
 menuconfig USB_ATM
tristate "USB DSL modem support"
-   depends on USB && ATM
+   depends on USB && NET
+   select ATM
select CRC32
default n
help

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Forcing modes in libata (was: SATA buffered read VERY slow)

[Al Boldi]

Alan Cox wrote:
> hdparm supports identify to read modes on drives with libata. The one
> thing you cannot do is force modes right now.

I have some AOpen 52x/32x CD-RWriter, that advertises itself as DMA capable, 
but has a hard time sustaining such access.  The only way to make access to 
this drive safe is by reducing it to PIO-3 even on XP,  which makes libata 
somewhat unusable on this drive, especially for writing.

What's hindering the ability to force a mode in libata, as is possible with 
the normal ide-driver?


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Forcing modes in libata (was: SATA buffered read VERY slow)

[Al Boldi]

Alan Cox wrote:
 hdparm supports identify to read modes on drives with libata. The one
 thing you cannot do is force modes right now.

I have some AOpen 52x/32x CD-RWriter, that advertises itself as DMA capable, 
but has a hard time sustaining such access.  The only way to make access to 
this drive safe is by reducing it to PIO-3 even on XP,  which makes libata 
somewhat unusable on this drive, especially for writing.

What's hindering the ability to force a mode in libata, as is possible with 
the normal ide-driver?


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 5/5] USB Kconfig: Source drivers/usb/gadget/Kconfig for ARCH=v850

[Al Boldi]

Source drivers/usb/gadget/Kconfig for ARCH=v850.


Cc: David Brownell [EMAIL PROTECTED]
Cc: Greg KH [EMAIL PROTECTED]
Cc: Andrew Morton [EMAIL PROTECTED]
Signed-off-by: Al Boldi [EMAIL PROTECTED]

---

--- 23.a/arch/v850/Kconfig
+++ 23.b/arch/v850/Kconfig
@@ -331,6 +331,8 @@ source sound/Kconfig
 
 source drivers/usb/Kconfig
 
+source drivers/usb/gadget/Kconfig
+
 source arch/v850/Kconfig.debug
 
 source security/Kconfig

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 4/5] USB Kconfig: Source drivers/usb/gadget/Kconfig for ARCH=arm

[Al Boldi]

From: David Brownell [EMAIL PROTECTED]

Source drivers/usb/gadget/Kconfig for ARCH=arm.


Cc: David Brownell [EMAIL PROTECTED]
Cc: Greg KH [EMAIL PROTECTED]
Cc: Andrew Morton [EMAIL PROTECTED]
Signed-off-by: Al Boldi [EMAIL PROTECTED]

---

--- 23.a/arch/arm/Kconfig
+++ 23.b/arch/arm/Kconfig
@@ -1049,6 +1049,8 @@ source drivers/hid/Kconfig
 
 source drivers/usb/Kconfig
 
+source drivers/usb/gadget/Kconfig
+
 source drivers/mmc/Kconfig
 
 source drivers/rtc/Kconfig

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 1/5] USB Kconfig: Reorganize USB Kconfig Menu

[Al Boldi]

Reorganize USB Kconfig Menu, and move USB_GADGET out into the Device Driver
Menu.

Cc: David Brownell [EMAIL PROTECTED]
Cc: Greg KH [EMAIL PROTECTED]
Cc: Andrew Morton [EMAIL PROTECTED]
Signed-off-by: Al Boldi [EMAIL PROTECTED]

---

--- 23.a/drivers/Kconfig
+++ 23.b/drivers/Kconfig
@@ -70,6 +70,8 @@ source drivers/hid/Kconfig
 
 source drivers/usb/Kconfig
 
+source drivers/usb/gadget/Kconfig
+
 source drivers/mmc/Kconfig
 
 source drivers/leds/Kconfig
--- 23.a/drivers/usb/Kconfig
+++ 23.b/drivers/usb/Kconfig
@@ -2,8 +2,8 @@
 # USB device configuration
 #
 
-menuconfig USB_SUPPORT
-   bool USB support
+config USB_SUPPORT
+   bool
depends on HAS_IOMEM
default y
---help---
@@ -52,8 +52,8 @@ config USB_ARCH_HAS_EHCI
default PCI
 
 # ARM SA chips have a non-PCI based OHCI-compatible USB host interface.
-config USB
-   tristate Support for Host-side USB
+menuconfig USB
+   tristate Host-side USB
depends on USB_ARCH_HAS_HCD
---help---
  Universal Serial Bus (USB) is a specification for a serial bus
@@ -87,21 +87,16 @@ config USB
  To compile this driver as a module, choose M here: the
  module will be called usbcore.
 
+if USB
+
 source drivers/usb/core/Kconfig
 
 source drivers/usb/host/Kconfig
 
-source drivers/usb/class/Kconfig
-
 source drivers/usb/storage/Kconfig
 
-source drivers/usb/image/Kconfig
-
 source drivers/usb/mon/Kconfig
 
-comment USB port drivers
-   depends on USB
-
 config USB_USS720
tristate USS720 parport driver
depends on USB  PARPORT
@@ -133,10 +128,13 @@ config USB_USS720
 
 source drivers/usb/serial/Kconfig
 
-source drivers/usb/misc/Kconfig
-
 source drivers/usb/atm/Kconfig
 
-source drivers/usb/gadget/Kconfig
+source drivers/usb/class/Kconfig
+
+source drivers/usb/image/Kconfig
+
+source drivers/usb/misc/Kconfig
 
+endif # USB
 endif # USB_SUPPORT
--- 23.a/drivers/usb/serial/Kconfig
+++ 23.b/drivers/usb/serial/Kconfig
@@ -2,10 +2,7 @@
 # USB Serial device configuration
 #
 
-menu USB Serial Converter support
-   depends on USB!=n
-
-config USB_SERIAL
+menuconfig USB_SERIAL
tristate USB Serial Converter support
depends on USB
---help---
@@ -20,6 +17,8 @@ config USB_SERIAL
  To compile this driver as a module, choose M here: the
  module will be called usbserial.
 
+if USB_SERIAL
+
 config USB_SERIAL_CONSOLE
bool USB Serial Console device support (EXPERIMENTAL)
depends on USB_SERIAL=y  EXPERIMENTAL
@@ -581,5 +580,4 @@ config USB_EZUSB
default y
 
 
-endmenu
-
+endif # USB_SERIAL
--- 23.a/drivers/usb/core/Kconfig
+++ 23.b/drivers/usb/core/Kconfig
@@ -1,6 +1,12 @@
 #
 # USB Core configuration
 #
+menuconfig USB_CORE_MENU
+   bool USB Core Config
+   depends on USB
+
+if USB_CORE_MENU
+
 config USB_DEBUG
bool USB verbose debug messages
depends on USB
@@ -142,3 +148,4 @@ config USB_OTG_BLACKLIST_HUB
  external hubs.  OTG hosts are allowed to reduce hardware
  and software costs by not supporting external hubs.
 
+endif # USB_CORE_MENU
--- 23.a/drivers/usb/host/Kconfig
+++ 23.b/drivers/usb/host/Kconfig
@@ -1,7 +1,7 @@
 #
 # USB Host Controller Drivers
 #
-comment USB Host Controller Drivers
+menu USB Host Controller Drivers
depends on USB
 
 config USB_EHCI_HCD
@@ -248,3 +248,4 @@ config USB_R8A66597_HCD
  To compile this driver as a module, choose M here: the
  module will be called r8a66597-hcd.
 
+endmenu
--- 23.a/drivers/usb/class/Kconfig
+++ 23.b/drivers/usb/class/Kconfig
@@ -1,9 +1,12 @@
 #
 # USB Class driver configuration
 #
-comment USB Device Class drivers
+menuconfig USB_DEVICE_CLASS_MENU
+   bool USB Device Class drivers
depends on USB
 
+if USB_DEVICE_CLASS_MENU
+
 config USB_ACM
tristate USB Modem (CDC ACM) support
depends on USB
@@ -29,3 +32,4 @@ config USB_PRINTER
  To compile this driver as a module, choose M here: the
  module will be called usblp.
 
+endif # USB_DEVICE_CLASS_MENU
--- 23.a/drivers/usb/gadget/Kconfig
+++ 23.b/drivers/usb/gadget/Kconfig
@@ -12,10 +12,8 @@
 # With help from a special transceiver and a Mini-AB jack, systems with
 # both kinds of controller can also support USB On-the-Go (CONFIG_USB_OTG).
 #
-menu USB Gadget Support
-
-config USB_GADGET
-   tristate Support for USB Gadgets
+menuconfig USB_GADGET
+   tristate Peripheral-side USB
help
   USB is a master/slave protocol, organized with one master
   host (such as a PC) controlling up to 127 peripheral devices.
@@ -42,6 +40,8 @@ config USB_GADGET
   For more information, see http://www.linux-usb.org/gadget and
   the kernel DocBook documentation for this API.
 
+if USB_GADGET
+
 config USB_GADGET_DEBUG
boolean Debugging messages
depends on USB_GADGET  DEBUG_KERNEL  EXPERIMENTAL
@@ -522,4 +522,4 @@ config USB_MIDI_GADGET
 
 endchoice
 
-endmenu
+endif

Re: [PATCH 3/5] USB Kconfig: Select ATM for USB DSL modem support

[Al Boldi]

Select ATM for USB DSL modem support.


Cc: David Brownell [EMAIL PROTECTED]
Cc: Greg KH [EMAIL PROTECTED]
Cc: Andrew Morton [EMAIL PROTECTED]
Signed-off-by: Al Boldi [EMAIL PROTECTED]

---

--- 23.a/drivers/usb/atm/Kconfig
+++ 23.b/drivers/usb/atm/Kconfig
@@ -4,7 +4,8 @@
 
 menuconfig USB_ATM
tristate USB DSL modem support
-   depends on USB  ATM
+   depends on USB  NET
+   select ATM
select CRC32
default n
help

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 2/5] USB Kconfig: Select SCSI for USB Mass Storage support

[Al Boldi]

Select SCSI for USB Mass Storage support.


Cc: David Brownell [EMAIL PROTECTED]
Cc: Greg KH [EMAIL PROTECTED]
Cc: Andrew Morton [EMAIL PROTECTED]
Signed-off-by: Al Boldi [EMAIL PROTECTED]

---

--- 23.a/drivers/usb/storage/Kconfig
+++ 23.b/drivers/usb/storage/Kconfig
@@ -2,14 +2,10 @@
 # USB Storage driver configuration
 #
 
-comment NOTE: USB_STORAGE needs SCSI, and 'SCSI disk support' may
-   depends on USB
-comment also be needed; see USB_STORAGE Help for more information
-   depends on USB
-
 menuconfig USB_STORAGE
tristate USB Mass Storage support
-   depends on USB  SCSI
+   depends on USB
+   select SCSI
---help---
  Say Y here if you want to connect USB mass storage devices to your
  computer's USB port. This is the driver you need for USB
@@ -18,13 +14,17 @@ menuconfig USB_STORAGE
  similar devices. This driver may also be used for some cameras
  and card readers.
 
- This option depends on 'SCSI' support being enabled, but you
- probably also need 'SCSI device support: SCSI disk support'
+ This option selects 'SCSI' support , but you probably
+ also need 'SCSI device support: SCSI disk support'
  (BLK_DEV_SD) for most USB storage devices.
 
  To compile this driver as a module, choose M here: the
  module will be called usb-storage.
 
+if USB_STORAGE
+comment NOTE: USB_STORAGE selects SCSI, but 'SCSI disk support' may
+comment also be needed; see USB_STORAGE Help for more information
+
 config USB_STORAGE_DEBUG
bool USB Mass Storage verbose debug
depends on USB_STORAGE
@@ -147,6 +147,8 @@ config USB_STORAGE_KARMA
  on the resulting scsi device node returns the Karma to normal
  operation.
 
+endif # USB_STORAGE
+
 config USB_LIBUSUAL
bool The shared table of common (or usual) storage devices
depends on USB

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH 0/5] USB Kconfig: Reorganize USB Kconfig Menu

[Al Boldi]

Reorganize USB Kconfig Menu, and move USB_GADGET out into the Device Driver
Menu.  This helps the USB Kconfig Menu to be more logical/usable.

Patchset against 2.6.23

 arch/arm/Kconfig|2 ++
 arch/v850/Kconfig   |2 ++
 drivers/Kconfig |2 ++
 drivers/usb/Kconfig |   26 --
 drivers/usb/atm/Kconfig |   12 ++--
 drivers/usb/class/Kconfig   |6 +-
 drivers/usb/core/Kconfig|7 +++
 drivers/usb/gadget/Kconfig  |   10 +-
 drivers/usb/host/Kconfig|3 ++-
 drivers/usb/image/Kconfig   |7 ++-
 drivers/usb/misc/Kconfig|6 +-
 drivers/usb/serial/Kconfig  |   10 --
 drivers/usb/storage/Kconfig |   25 ++---
 13 files changed, 72 insertions(+), 46 deletions(-)



Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 2/5] USB Kconfig: Select SCSI for USB Mass Storage support

[Al Boldi]

Sam Ravnborg wrote:
 On Sat, Jan 05, 2008 at 11:03:30PM +0200, Adrian Bunk wrote:
  For kconfig users, select is _much_ better than sending them through
  different menus.

 Only if used within the current limitations of Kconfig.
 And that requires you to use select only to select symbols with
 no dependencies.

 In this case we do not know if BLOCK is enabled or not.

Good point!  How about we solve it like this:

menuconfig USB_STORAGE
tristate USB Mass Storage support
-   depends on USB  SCSI
+   depends on USB  BLOCK
+   select SCSI


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] USB Kconfig: Reorganize USB Kconfig Menu

[Al Boldi]

Adrian Bunk wrote:
> On Thu, Dec 27, 2007 at 02:18:58PM -0800, David Brownell wrote:
> > Also, looking at this in xconfig shows some oddness.  That "core"
> > submenu holds stuff that would logically be part of the toplevel
> > menu for host side USB.  While that toplevel menu has the USS720
> > driver, which seems more like a "miscellany" thing...

This one probably belongs in its own sub-menu, but as it is only one option, 
it's not considered worthwhile, so we leave it as is.

> > The comment about SCSI shouldn't show when SCSI is defined; and
> > the comment aboue "SCSI disk support" belongs with the mass storage
> > menu, not at the top level.  The way that the mass storage menu
> > is presented also hides the fact that you can get fully functional
> > mass storage support without checking *any* of the drivers there,
> > since the core already handles standard devices.
> >...
>
> The SCSI dependency should be handled through select - there's no
> reason for bothering the kconfig user wih such implementation details.

Agreed, and while we are at it, also select ATM for DSL, like this:

--- 23.a/drivers/usb/atm/Kconfig
+++ 23.b/drivers/usb/atm/Kconfig
@@ -4,7 +4,8 @@
 
 menuconfig USB_ATM
tristate "USB DSL modem support"
-   depends on USB && ATM
+   depends on USB && NET
+   select ATM
select CRC32
default n
help
--- 23.a/drivers/usb/storage/Kconfig
+++ 23.b/drivers/usb/storage/Kconfig
@@ -2,14 +2,10 @@
 # USB Storage driver configuration
 #
 
-comment "NOTE: USB_STORAGE needs SCSI, and 'SCSI disk support' may"
-   depends on USB
-comment "also be needed; see USB_STORAGE Help for more information"
-   depends on USB
-
 menuconfig USB_STORAGE
tristate "USB Mass Storage support"
-   depends on USB && SCSI
+   depends on USB
+   select SCSI
---help---
  Say Y here if you want to connect USB mass storage devices to your
  computer's USB port. This is the driver you need for USB
@@ -18,13 +14,17 @@ menuconfig USB_STORAGE
  similar devices. This driver may also be used for some cameras
  and card readers.
 
- This option depends on 'SCSI' support being enabled, but you
- probably also need 'SCSI device support: SCSI disk support'
+ This option selects 'SCSI' support , but you probably
+ also need 'SCSI device support: SCSI disk support'
  (BLK_DEV_SD) for most USB storage devices.
 
  To compile this driver as a module, choose M here: the
  module will be called usb-storage.
 
+if USB_STORAGE
+comment "NOTE: USB_STORAGE selects SCSI, but 'SCSI disk support' may"
+comment "also be needed; see USB_STORAGE Help for more information"
+
 config USB_STORAGE_DEBUG
bool "USB Mass Storage verbose debug"
depends on USB_STORAGE
@@ -147,6 +147,8 @@ config USB_STORAGE_KARMA
  on the resulting scsi device node returns the Karma to normal
  operation.
 
+endif # USB_STORAGE
+
 config USB_LIBUSUAL
bool "The shared table of common (or usual) storage devices"
depends on USB


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] USB Kconfig: Reorganize USB Kconfig Menu

[Al Boldi]

Adrian Bunk wrote:
 On Thu, Dec 27, 2007 at 02:18:58PM -0800, David Brownell wrote:
  Also, looking at this in xconfig shows some oddness.  That core
  submenu holds stuff that would logically be part of the toplevel
  menu for host side USB.  While that toplevel menu has the USS720
  driver, which seems more like a miscellany thing...

This one probably belongs in its own sub-menu, but as it is only one option, 
it's not considered worthwhile, so we leave it as is.

  The comment about SCSI shouldn't show when SCSI is defined; and
  the comment aboue SCSI disk support belongs with the mass storage
  menu, not at the top level.  The way that the mass storage menu
  is presented also hides the fact that you can get fully functional
  mass storage support without checking *any* of the drivers there,
  since the core already handles standard devices.
 ...

 The SCSI dependency should be handled through select - there's no
 reason for bothering the kconfig user wih such implementation details.

Agreed, and while we are at it, also select ATM for DSL, like this:

--- 23.a/drivers/usb/atm/Kconfig
+++ 23.b/drivers/usb/atm/Kconfig
@@ -4,7 +4,8 @@
 
 menuconfig USB_ATM
tristate USB DSL modem support
-   depends on USB  ATM
+   depends on USB  NET
+   select ATM
select CRC32
default n
help
--- 23.a/drivers/usb/storage/Kconfig
+++ 23.b/drivers/usb/storage/Kconfig
@@ -2,14 +2,10 @@
 # USB Storage driver configuration
 #
 
-comment NOTE: USB_STORAGE needs SCSI, and 'SCSI disk support' may
-   depends on USB
-comment also be needed; see USB_STORAGE Help for more information
-   depends on USB
-
 menuconfig USB_STORAGE
tristate USB Mass Storage support
-   depends on USB  SCSI
+   depends on USB
+   select SCSI
---help---
  Say Y here if you want to connect USB mass storage devices to your
  computer's USB port. This is the driver you need for USB
@@ -18,13 +14,17 @@ menuconfig USB_STORAGE
  similar devices. This driver may also be used for some cameras
  and card readers.
 
- This option depends on 'SCSI' support being enabled, but you
- probably also need 'SCSI device support: SCSI disk support'
+ This option selects 'SCSI' support , but you probably
+ also need 'SCSI device support: SCSI disk support'
  (BLK_DEV_SD) for most USB storage devices.
 
  To compile this driver as a module, choose M here: the
  module will be called usb-storage.
 
+if USB_STORAGE
+comment NOTE: USB_STORAGE selects SCSI, but 'SCSI disk support' may
+comment also be needed; see USB_STORAGE Help for more information
+
 config USB_STORAGE_DEBUG
bool USB Mass Storage verbose debug
depends on USB_STORAGE
@@ -147,6 +147,8 @@ config USB_STORAGE_KARMA
  on the resulting scsi device node returns the Karma to normal
  operation.
 
+endif # USB_STORAGE
+
 config USB_LIBUSUAL
bool The shared table of common (or usual) storage devices
depends on USB


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] USB Kconfig: Reorganize USB Kconfig Menu

[Al Boldi]

Jan Engelhardt wrote:
> On Dec 23 2007 14:49, Al Boldi wrote:
> >--- 23.a/drivers/usb/Kconfig
> >+++ 23.b/drivers/usb/Kconfig
> >@@ -2,8 +2,8 @@
> > # USB device configuration
> > #
> >
> >-menuconfig USB_SUPPORT
> >-bool "USB support"
> >+config USB_SUPPORT
> >+bool
> > depends on HAS_IOMEM
> > default y
> > ---help---
>
> With this patch, is USB_SUPPORT still needed? It is not visible (anymore),
> always is y and no other options select/depend on it (according to
> Kconfig '/' function).

Well, I left it in, because of defconfig dependencies, and HAS_IOMEM is 
needed for a few archs.

> Looks good.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH] USB Kconfig: Reorganize USB Kconfig Menu

[Al Boldi]

Reogranize USB Kconfig Menu, and move USB_GADGET out into the Device Driver
Menu.  This helps the USB Kconfig Menu to be more logical/usable.

Cc: David Brownell <[EMAIL PROTECTED]>
Cc: Greg KH <[EMAIL PROTECTED]>
Cc: Andrew Morton <[EMAIL PROTECTED]>
Signed-off-by: Al Boldi <[EMAIL PROTECTED]>

---

--- 23.a/drivers/Kconfig
+++ 23.b/drivers/Kconfig
@@ -70,6 +70,8 @@ source "drivers/hid/Kconfig"
 
 source "drivers/usb/Kconfig"
 
+source "drivers/usb/gadget/Kconfig"
+
 source "drivers/mmc/Kconfig"
 
 source "drivers/leds/Kconfig"
--- 23.a/drivers/usb/Kconfig
+++ 23.b/drivers/usb/Kconfig
@@ -2,8 +2,8 @@
 # USB device configuration
 #
 
-menuconfig USB_SUPPORT
-   bool "USB support"
+config USB_SUPPORT
+   bool
depends on HAS_IOMEM
default y
---help---
@@ -52,8 +52,8 @@ config USB_ARCH_HAS_EHCI
default PCI
 
 # ARM SA chips have a non-PCI based "OHCI-compatible" USB host interface.
-config USB
-   tristate "Support for Host-side USB"
+menuconfig USB
+   tristate "Host-side USB"
depends on USB_ARCH_HAS_HCD
---help---
  Universal Serial Bus (USB) is a specification for a serial bus
@@ -87,21 +87,16 @@ config USB
  To compile this driver as a module, choose M here: the
  module will be called usbcore.
 
+if USB
+
 source "drivers/usb/core/Kconfig"
 
 source "drivers/usb/host/Kconfig"
 
-source "drivers/usb/class/Kconfig"
-
 source "drivers/usb/storage/Kconfig"
 
-source "drivers/usb/image/Kconfig"
-
 source "drivers/usb/mon/Kconfig"
 
-comment "USB port drivers"
-   depends on USB
-
 config USB_USS720
tristate "USS720 parport driver"
depends on USB && PARPORT
@@ -133,10 +128,13 @@ config USB_USS720
 
 source "drivers/usb/serial/Kconfig"
 
-source "drivers/usb/misc/Kconfig"
-
 source "drivers/usb/atm/Kconfig"
 
-source "drivers/usb/gadget/Kconfig"
+source "drivers/usb/class/Kconfig"
+
+source "drivers/usb/image/Kconfig"
+
+source "drivers/usb/misc/Kconfig"
 
+endif # USB
 endif # USB_SUPPORT
--- 23.a/drivers/usb/serial/Kconfig
+++ 23.b/drivers/usb/serial/Kconfig
@@ -2,10 +2,7 @@
 # USB Serial device configuration
 #
 
-menu "USB Serial Converter support"
-   depends on USB!=n
-
-config USB_SERIAL
+menuconfig USB_SERIAL
tristate "USB Serial Converter support"
depends on USB
---help---
@@ -20,6 +17,8 @@ config USB_SERIAL
  To compile this driver as a module, choose M here: the
  module will be called usbserial.
 
+if USB_SERIAL
+
 config USB_SERIAL_CONSOLE
bool "USB Serial Console device support (EXPERIMENTAL)"
depends on USB_SERIAL=y && EXPERIMENTAL
@@ -581,5 +580,4 @@ config USB_EZUSB
default y
 
 
-endmenu
-
+endif # USB_SERIAL
--- 23.a/drivers/usb/core/Kconfig
+++ 23.b/drivers/usb/core/Kconfig
@@ -1,6 +1,12 @@
 #
 # USB Core configuration
 #
+menuconfig USB_CORE_MENU
+   bool "USB Core Config"
+   depends on USB
+
+if USB_CORE_MENU
+
 config USB_DEBUG
bool "USB verbose debug messages"
depends on USB
@@ -142,3 +148,4 @@ config USB_OTG_BLACKLIST_HUB
  external hubs.  OTG hosts are allowed to reduce hardware
  and software costs by not supporting external hubs.
 
+endif # USB_CORE_MENU
--- 23.a/drivers/usb/host/Kconfig
+++ 23.b/drivers/usb/host/Kconfig
@@ -1,7 +1,7 @@
 #
 # USB Host Controller Drivers
 #
-comment "USB Host Controller Drivers"
+menu "USB Host Controller Drivers"
depends on USB
 
 config USB_EHCI_HCD
@@ -248,3 +248,4 @@ config USB_R8A66597_HCD
  To compile this driver as a module, choose M here: the
  module will be called r8a66597-hcd.
 
+endmenu
--- 23.a/drivers/usb/class/Kconfig
+++ 23.b/drivers/usb/class/Kconfig
@@ -1,9 +1,12 @@
 #
 # USB Class driver configuration
 #
-comment "USB Device Class drivers"
+menuconfig USB_DEVICE_CLASS_MENU
+   bool "USB Device Class drivers"
depends on USB
 
+if USB_DEVICE_CLASS_MENU
+
 config USB_ACM
tristate "USB Modem (CDC ACM) support"
depends on USB
@@ -29,3 +32,4 @@ config USB_PRINTER
  To compile this driver as a module, choose M here: the
  module will be called usblp.
 
+endif # USB_DEVICE_CLASS_MENU
--- 23.a/drivers/usb/gadget/Kconfig
+++ 23.b/drivers/usb/gadget/Kconfig
@@ -12,10 +12,8 @@
 # With help from a special transceiver and a "Mini-AB" jack, systems with
 # both kinds of controller can also support "USB On-the-Go" (CONFIG_USB_OTG).
 #
-menu "USB Gadget Support"
-
-config USB_GADGET
-   tristate "Support for USB Gadgets"
+menuconfig USB_GADGET
+ 

[PATCH] USB Kconfig: Reorganize USB Kconfig Menu

[Al Boldi]

Reogranize USB Kconfig Menu, and move USB_GADGET out into the Device Driver
Menu.  This helps the USB Kconfig Menu to be more logical/usable.

Cc: David Brownell [EMAIL PROTECTED]
Cc: Greg KH [EMAIL PROTECTED]
Cc: Andrew Morton [EMAIL PROTECTED]
Signed-off-by: Al Boldi [EMAIL PROTECTED]

---

--- 23.a/drivers/Kconfig
+++ 23.b/drivers/Kconfig
@@ -70,6 +70,8 @@ source drivers/hid/Kconfig
 
 source drivers/usb/Kconfig
 
+source drivers/usb/gadget/Kconfig
+
 source drivers/mmc/Kconfig
 
 source drivers/leds/Kconfig
--- 23.a/drivers/usb/Kconfig
+++ 23.b/drivers/usb/Kconfig
@@ -2,8 +2,8 @@
 # USB device configuration
 #
 
-menuconfig USB_SUPPORT
-   bool USB support
+config USB_SUPPORT
+   bool
depends on HAS_IOMEM
default y
---help---
@@ -52,8 +52,8 @@ config USB_ARCH_HAS_EHCI
default PCI
 
 # ARM SA chips have a non-PCI based OHCI-compatible USB host interface.
-config USB
-   tristate Support for Host-side USB
+menuconfig USB
+   tristate Host-side USB
depends on USB_ARCH_HAS_HCD
---help---
  Universal Serial Bus (USB) is a specification for a serial bus
@@ -87,21 +87,16 @@ config USB
  To compile this driver as a module, choose M here: the
  module will be called usbcore.
 
+if USB
+
 source drivers/usb/core/Kconfig
 
 source drivers/usb/host/Kconfig
 
-source drivers/usb/class/Kconfig
-
 source drivers/usb/storage/Kconfig
 
-source drivers/usb/image/Kconfig
-
 source drivers/usb/mon/Kconfig
 
-comment USB port drivers
-   depends on USB
-
 config USB_USS720
tristate USS720 parport driver
depends on USB  PARPORT
@@ -133,10 +128,13 @@ config USB_USS720
 
 source drivers/usb/serial/Kconfig
 
-source drivers/usb/misc/Kconfig
-
 source drivers/usb/atm/Kconfig
 
-source drivers/usb/gadget/Kconfig
+source drivers/usb/class/Kconfig
+
+source drivers/usb/image/Kconfig
+
+source drivers/usb/misc/Kconfig
 
+endif # USB
 endif # USB_SUPPORT
--- 23.a/drivers/usb/serial/Kconfig
+++ 23.b/drivers/usb/serial/Kconfig
@@ -2,10 +2,7 @@
 # USB Serial device configuration
 #
 
-menu USB Serial Converter support
-   depends on USB!=n
-
-config USB_SERIAL
+menuconfig USB_SERIAL
tristate USB Serial Converter support
depends on USB
---help---
@@ -20,6 +17,8 @@ config USB_SERIAL
  To compile this driver as a module, choose M here: the
  module will be called usbserial.
 
+if USB_SERIAL
+
 config USB_SERIAL_CONSOLE
bool USB Serial Console device support (EXPERIMENTAL)
depends on USB_SERIAL=y  EXPERIMENTAL
@@ -581,5 +580,4 @@ config USB_EZUSB
default y
 
 
-endmenu
-
+endif # USB_SERIAL
--- 23.a/drivers/usb/core/Kconfig
+++ 23.b/drivers/usb/core/Kconfig
@@ -1,6 +1,12 @@
 #
 # USB Core configuration
 #
+menuconfig USB_CORE_MENU
+   bool USB Core Config
+   depends on USB
+
+if USB_CORE_MENU
+
 config USB_DEBUG
bool USB verbose debug messages
depends on USB
@@ -142,3 +148,4 @@ config USB_OTG_BLACKLIST_HUB
  external hubs.  OTG hosts are allowed to reduce hardware
  and software costs by not supporting external hubs.
 
+endif # USB_CORE_MENU
--- 23.a/drivers/usb/host/Kconfig
+++ 23.b/drivers/usb/host/Kconfig
@@ -1,7 +1,7 @@
 #
 # USB Host Controller Drivers
 #
-comment USB Host Controller Drivers
+menu USB Host Controller Drivers
depends on USB
 
 config USB_EHCI_HCD
@@ -248,3 +248,4 @@ config USB_R8A66597_HCD
  To compile this driver as a module, choose M here: the
  module will be called r8a66597-hcd.
 
+endmenu
--- 23.a/drivers/usb/class/Kconfig
+++ 23.b/drivers/usb/class/Kconfig
@@ -1,9 +1,12 @@
 #
 # USB Class driver configuration
 #
-comment USB Device Class drivers
+menuconfig USB_DEVICE_CLASS_MENU
+   bool USB Device Class drivers
depends on USB
 
+if USB_DEVICE_CLASS_MENU
+
 config USB_ACM
tristate USB Modem (CDC ACM) support
depends on USB
@@ -29,3 +32,4 @@ config USB_PRINTER
  To compile this driver as a module, choose M here: the
  module will be called usblp.
 
+endif # USB_DEVICE_CLASS_MENU
--- 23.a/drivers/usb/gadget/Kconfig
+++ 23.b/drivers/usb/gadget/Kconfig
@@ -12,10 +12,8 @@
 # With help from a special transceiver and a Mini-AB jack, systems with
 # both kinds of controller can also support USB On-the-Go (CONFIG_USB_OTG).
 #
-menu USB Gadget Support
-
-config USB_GADGET
-   tristate Support for USB Gadgets
+menuconfig USB_GADGET
+   tristate Peripheral-side USB
help
   USB is a master/slave protocol, organized with one master
   host (such as a PC) controlling up to 127 peripheral devices.
@@ -42,6 +40,8 @@ config USB_GADGET
   For more information, see http://www.linux-usb.org/gadget and
   the kernel DocBook documentation for this API.
 
+if USB_GADGET
+
 config USB_GADGET_DEBUG
boolean Debugging messages
depends on USB_GADGET  DEBUG_KERNEL  EXPERIMENTAL
@@ -522,4 +522,4

Re: [PATCH] USB Kconfig: Reorganize USB Kconfig Menu

[Al Boldi]

Jan Engelhardt wrote:
 On Dec 23 2007 14:49, Al Boldi wrote:
 --- 23.a/drivers/usb/Kconfig
 +++ 23.b/drivers/usb/Kconfig
 @@ -2,8 +2,8 @@
  # USB device configuration
  #
 
 -menuconfig USB_SUPPORT
 -bool USB support
 +config USB_SUPPORT
 +bool
  depends on HAS_IOMEM
  default y
  ---help---

 With this patch, is USB_SUPPORT still needed? It is not visible (anymore),
 always is y and no other options select/depend on it (according to
 Kconfig '/' function).

Well, I left it in, because of defconfig dependencies, and HAS_IOMEM is 
needed for a few archs.

 Looks good.


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] USB Kconfig: Declutter USB Kconfig Menu

[Al Boldi]

David Brownell wrote:
> > > Which is why my suggestion was to have them both move up a level, with
> > > host and peripheral side menus nested normally:
> > >
> > >   Device Drivers:
> > >   ...
> > >   [ ] HID devices
> > >   < > Host side USB
> > >   < > Peripheral side USB
> > >   < > MMC/SD/SDIO card support
> > >   [ ] LED support
> > >   ...
> > >
> > > That way both host and peripheral side support would have their own
> > > menu, and that pointless nesting would be reduced.
> >
> > You mean:
> > [ ] Host side USB
> > [ ] Peripheral side USB
> >
> > Right?
>
> Both of those subsystems are already set up with the main choice
> as tristate "< >//<*>".  If we're simplifying the navigation,
> I can't see a point to requiring *two* menu actions not one.

There is a point, but it's a weak one: you want it to unfold in the same menu 
as you select the option.  So, if you unfold the sub-options in a separate 
sub-menu, you better use [ ] (or just blank), as this will allow you to 
enter the sub-menu, and see the unfold take place inside the same menu.

IMHO, I prefer < >, but newbies probably prefer [ ], as it is less abstract.

Provided there is agreement, and we go with < >, it would mean a slight 
change to Host side USB, like this (note new USB Core Config sub-menu):

  --- Support for Host-side USB  
  [ ]   USB Core Config  --->   
  [ ]   USB Host Controller Drivers  --->
  ---   NOTE: USB_STORAGE needs SCSI, and 'SCSI disk support' may
  ---   also be needed; see USB_STORAGE Help for more information
  < >   USB Mass Storage support  --->  
  [ ]   The shared table of common (or usual) storage devices
  [ ]   USB Monitor   
  < >   USS720 parport driver   
  < >   USB Serial Converter support  --->
  < >   USB DSL modem support  --->
  [ ]   USB Device Class drivers  --->
  [ ]   USB Imaging devices  ---> 
  [ ]   USB Miscellaneous drivers  ---> 



Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] USB Kconfig: Declutter USB Kconfig Menu

[Al Boldi]

David Brownell wrote:
   Which is why my suggestion was to have them both move up a level, with
   host and peripheral side menus nested normally:
  
 Device Drivers:
 ...
 [ ] HID devices
   Host side USB
   Peripheral side USB
   MMC/SD/SDIO card support
 [ ] LED support
 ...
  
   That way both host and peripheral side support would have their own
   menu, and that pointless nesting would be reduced.
 
  You mean:
  [ ] Host side USB
  [ ] Peripheral side USB
 
  Right?

 Both of those subsystems are already set up with the main choice
 as tristate  /M/*.  If we're simplifying the navigation,
 I can't see a point to requiring *two* menu actions not one.

There is a point, but it's a weak one: you want it to unfold in the same menu 
as you select the option.  So, if you unfold the sub-options in a separate 
sub-menu, you better use [ ] (or just blank), as this will allow you to 
enter the sub-menu, and see the unfold take place inside the same menu.

IMHO, I prefer  , but newbies probably prefer [ ], as it is less abstract.

Provided there is agreement, and we go with  , it would mean a slight 
change to Host side USB, like this (note new USB Core Config sub-menu):

  --- Support for Host-side USB  
  [ ]   USB Core Config  ---   
  [ ]   USB Host Controller Drivers  ---
  ---   NOTE: USB_STORAGE needs SCSI, and 'SCSI disk support' may
  ---   also be needed; see USB_STORAGE Help for more information
  USB Mass Storage support  ---  
  [ ]   The shared table of common (or usual) storage devices
  [ ]   USB Monitor   
  USS720 parport driver   
  USB Serial Converter support  ---
  USB DSL modem support  ---
  [ ]   USB Device Class drivers  ---
  [ ]   USB Imaging devices  --- 
  [ ]   USB Miscellaneous drivers  --- 



Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] USB Kconfig: Declutter USB Kconfig Menu

[Al Boldi]

David Brownell wrote:
> > > The driver stacks are independent of each other, except for common
> > > data structures like what's in the  header file.  I
> > > think there's no real point, other than history, to having both sides
> > > share the same menu.
> >
> > They aren't.
>
> How can you say that, when you showed (right above!!) that they are???
> Sure, there are submenus too.  But they're clearly in the same menu.

Well, in that case even if you moved them to the upper level, then they would 
still be in the same menu, if that's what you mean.

> > USB Gadget Support is in its own sub-menu.  And strictly
> > speaking, Host-side USB should be too, but some people may feel that
> > this would be nesting it too deep, and so it unfolds in the same menu.
>
> Which is why my suggestion was to have them both move up a level, with
> host and peripheral side menus nested normally:
>
>   Device Drivers:
>   ...
>   [ ] HID devices
>   < > Host side USB
>   < > Peripheral side USB
>   < > MMC/SD/SDIO card support
>   [ ] LED support
>   ...
>
> That way both host and peripheral side support would have their own
> menu, and that pointless nesting would be reduced.

You mean:
[ ] Host side USB
[ ] Peripheral side USB

Right?

All this does is loose the USB menu, which was used to group them and to 
un-clutter the Device Drivers menu.  Note:  They would still be sharing the 
same menu, only this time it's the Device Drivers menu.

I guess it's a judgment call.  Let's see what others think.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[RFC] USB Kconfig: Declutter USB Kconfig Menu

[Al Boldi]

The current USB Kconfig menu is rather cluttered and unorganized.

Propose new layout as follows:

  l USB support 
qk
  x  Arrow keys navigate the menu.   selects submenus --->.  Highlighted 
letters  x  
  x  are hotkeys.  Pressing  includes,  excludes,  modularizes 
features.  Press x  
  x   to exit,  for Help,  for Search.  Legend: [*] built-in  [ 
]x  
  x  excluded   module  < > module capable   
 x  
  x 
lqqk
 x  
  x x  --- USB support  
   x x   
  x x  <*>   Support for Host-side USB  --->
   x x   
  x xUSB Host Controller Drivers  --->  
   x x   
  x xUSB Device Class drivers  ---> 
   x x   
  x x  ---   NOTE: USB_STORAGE needs SCSI, and 'SCSI disk support' may  
   x x   
  x x  ---   also be needed; see USB_STORAGE Help for more information  
   x x   
  x x  <*>   USB Mass Storage support  ---> 
   x x   
  x x  [ ]   The shared table of common (or usual) storage devices  
   x x   
  x x  [ ]   USB Monitor
   x x   
  x xUSB Serial Converter support  ---> 
   x x   
  x xUSB DSL modem support  --->
   x x   
  x xUSB Imaging devices  --->  
   x x   
  x xUSB Miscellaneous drivers  --->
   x x   
  x xUSB Gadget Support  --->   
   x x   
  x x   
   x x   
  x x   
   x x   
  x x   
   x x   
  x x   
   x x   
  x x   
   x x   
  x x   
   x x   
  x x   
   x x   
  x x   
   x x   
  x 
mqqj
 x  
  
tqqu
  
  x   < Exit >< Help >  
 x  
  
mqqj
  


Any comments appreciated.


Thanks!

--
Al
 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[RFC] USB Kconfig: Declutter USB Kconfig Menu

[Al Boldi]

The current USB Kconfig menu is rather cluttered and unorganized.

Propose new layout as follows:

  l USB support 
qk
  x  Arrow keys navigate the menu.  Enter selects submenus ---.  Highlighted 
letters  x  
  x  are hotkeys.  Pressing Y includes, N excludes, M modularizes 
features.  Press x  
  x  EscEsc to exit, ? for Help, / for Search.  Legend: [*] built-in  [ 
]x  
  x  excluded  M modulemodule capable   
 x  
  x 
lqqk
 x  
  x x  --- USB support  
   x x   
  x x  *   Support for Host-side USB  ---
   x x   
  x xUSB Host Controller Drivers  ---  
   x x   
  x xUSB Device Class drivers  --- 
   x x   
  x x  ---   NOTE: USB_STORAGE needs SCSI, and 'SCSI disk support' may  
   x x   
  x x  ---   also be needed; see USB_STORAGE Help for more information  
   x x   
  x x  *   USB Mass Storage support  --- 
   x x   
  x x  [ ]   The shared table of common (or usual) storage devices  
   x x   
  x x  [ ]   USB Monitor
   x x   
  x xUSB Serial Converter support  --- 
   x x   
  x xUSB DSL modem support  ---
   x x   
  x xUSB Imaging devices  ---  
   x x   
  x xUSB Miscellaneous drivers  ---
   x x   
  x xUSB Gadget Support  ---   
   x x   
  x x   
   x x   
  x x   
   x x   
  x x   
   x x   
  x x   
   x x   
  x x   
   x x   
  x x   
   x x   
  x x   
   x x   
  x x   
   x x   
  x 
mqqj
 x  
  
tqqu
  
  x   Select Exit  Help   
 x  
  
mqqj
  


Any comments appreciated.


Thanks!

--
Al
 
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC] USB Kconfig: Declutter USB Kconfig Menu

[Al Boldi]

David Brownell wrote:
   The driver stacks are independent of each other, except for common
   data structures like what's in the linux/usb/ch9.h header file.  I
   think there's no real point, other than history, to having both sides
   share the same menu.
 
  They aren't.

 How can you say that, when you showed (right above!!) that they are???
 Sure, there are submenus too.  But they're clearly in the same menu.

Well, in that case even if you moved them to the upper level, then they would 
still be in the same menu, if that's what you mean.

  USB Gadget Support is in its own sub-menu.  And strictly
  speaking, Host-side USB should be too, but some people may feel that
  this would be nesting it too deep, and so it unfolds in the same menu.

 Which is why my suggestion was to have them both move up a level, with
 host and peripheral side menus nested normally:

   Device Drivers:
   ...
   [ ] HID devices
 Host side USB
 Peripheral side USB
 MMC/SD/SDIO card support
   [ ] LED support
   ...

 That way both host and peripheral side support would have their own
 menu, and that pointless nesting would be reduced.

You mean:
[ ] Host side USB
[ ] Peripheral side USB

Right?

All this does is loose the USB menu, which was used to group them and to 
un-clutter the Device Drivers menu.  Note:  They would still be sharing the 
same menu, only this time it's the Device Drivers menu.

I guess it's a judgment call.  Let's see what others think.


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 1/2] [RFC] Simple tamper-proof device filesystem.

[Al Boldi]

Indan Zupancic wrote:
> On Mon, December 17, 2007 01:40, Tetsuo Handa wrote:
> > So, use of this filesystem alone is meaningless because
> > attackers with root privileges can do what you are saying.
> > But use of this filesystem with MAC is still valid because
> > MAC can prevent attackers with root privileges from doing what you are
> > saying.
>
> If MAC can avoid all that, then why can't it also avoid tampering with
> /dev? What security does your filesystem add at all, if it's useless
> without a MAC doing all the hard work?

I think the answer is obvious:  Tetsuo wants to add functionality that the 
MACs are missing.  So, instead of adding this functionality per MAC, he 
proposes to add it as ground work, to be combined with any MAC.

> I think you can better spend your time on read-only bind mounts.

That would be too coarse.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [patch 1/2] [RFC] Simple tamper-proof device filesystem.

[Al Boldi]

Indan Zupancic wrote:
 On Mon, December 17, 2007 01:40, Tetsuo Handa wrote:
  So, use of this filesystem alone is meaningless because
  attackers with root privileges can do what you are saying.
  But use of this filesystem with MAC is still valid because
  MAC can prevent attackers with root privileges from doing what you are
  saying.

 If MAC can avoid all that, then why can't it also avoid tampering with
 /dev? What security does your filesystem add at all, if it's useless
 without a MAC doing all the hard work?

I think the answer is obvious:  Tetsuo wants to add functionality that the 
MACs are missing.  So, instead of adding this functionality per MAC, he 
proposes to add it as ground work, to be combined with any MAC.

 I think you can better spend your time on read-only bind mounts.

That would be too coarse.


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: git guidance

[Al Boldi]

[EMAIL PROTECTED] wrote:
> On Sat, 08 Dec 2007 07:56:21 +0300, Al Boldi said:
> > It probably goes without saying, that gitfs should have some basic
> > configuration file to setup its transparent behaviour
>
> But then it's not *truly* transparent, is it?

Don't mistake transparency with some form of auto-heuristic.  Transparency 
only means that it inserts functionality without impeding your normal 
workflow.

> And that leaves another question - if you make a config file that excludes
> all the .o files - then what's backing the .o files?  Those data blocks
> need to be *someplace*.  Maybe you can do something ugly like use unionfs
> to combine your gitfs with something else to store the other files...

Or any number of other possible implementation scenarios...

> But at that point, you're probably better off just creating a properly
> designed versioning filesystem.

But gitfs is not about designing a versioning filesystem, it's about 
designing a transparent interface into git to handle an SCM use-case.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: git guidance

[Al Boldi]

[EMAIL PROTECTED] wrote:
 On Sat, 08 Dec 2007 07:56:21 +0300, Al Boldi said:
  It probably goes without saying, that gitfs should have some basic
  configuration file to setup its transparent behaviour

 But then it's not *truly* transparent, is it?

Don't mistake transparency with some form of auto-heuristic.  Transparency 
only means that it inserts functionality without impeding your normal 
workflow.

 And that leaves another question - if you make a config file that excludes
 all the .o files - then what's backing the .o files?  Those data blocks
 need to be *someplace*.  Maybe you can do something ugly like use unionfs
 to combine your gitfs with something else to store the other files...

Or any number of other possible implementation scenarios...

 But at that point, you're probably better off just creating a properly
 designed versioning filesystem.

But gitfs is not about designing a versioning filesystem, it's about 
designing a transparent interface into git to handle an SCM use-case.


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: git guidance

[Al Boldi]

[EMAIL PROTECTED] wrote:
> On Fri, 07 Dec 2007 22:04:48 +0300, Al Boldi said:
> > Because WORKFLOW C is transparent, it won't affect other workflows.  So
> > you could still use your normal WORKFLOW B in addition to WORKFLOW C,
> > gaining an additional level of version control detail at no extra cost
> > other than the git-engine scratch repository overhead.
> >
> > BTW, is git efficient enough to handle WORKFLOW C?
>
> Imagine the number of commits a 'make clean; make' will do in a kernel
> tree, as it commits all those .o files... :)

.o files???

It probably goes without saying, that gitfs should have some basic 
configuration file to setup its transparent behaviour, and which would most 
probably contain an include / exclude file-filter mask, and probably other 
basic configuration options.  But this is really secondary to the 
implementation, and the question remains whether git is efficient enough.

IOW, how big is the git commit overhead as compared to a normal copy?


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: git guidance

[Al Boldi]

Jakub Narebski wrote:
> Al Boldi <[EMAIL PROTECTED]> writes:
> > For example:
> >
> >   echo "// last comment on this file" >> /gitfs.mounted/file
> >
> > should do an implied checkpoint, and make these checkpoints immediately
> > visible under some checkpoint branch of the gitfs mounted dir.
> >
> > Note, this way the developer gets version control without even noticing,
> > and works completely transparent to any kind of application.
>
> Why not use versioning filesystem for that, for example ext3cow
> (which looks suprisingly git-like, when you take into account that
> for ext3cow history is linear and centralized, so one can use date
> or sequential number to name commits).
>
> See GitLinks page on Git Wiki, "Other links" section:
>   http://www.ext3cow.com/

Sure, Linus mentioned the cow idea before in this thread, but you would still 
need a few hacks to get some basic Version Control features.  

> Version control system is all about WORKFLOW B, where programmer
> controls when it is time to commit (and in private repository he/she
> can then rewrite history to arrive at "Perfect patch series"[*1*]);
> something that for example CVS failed at, requiring programmer to do
> a merge if upstream has any changes when trying to commit.

Because WORKFLOW C is transparent, it won't affect other workflows.  So you 
could still use your normal WORKFLOW B in addition to WORKFLOW C, gaining an 
additional level of version control detail at no extra cost other than the 
git-engine scratch repository overhead.

BTW, is git efficient enough to handle WORKFLOW C?


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: git guidance

[Al Boldi]

Andreas Ericsson wrote:
> So, to get to the bottom of this, which of the following workflows is it
> you want git to support?
>
> ### WORKFLOW A ###
> edit, edit, edit
> edit, edit, edit
> edit, edit, edit
> Oops I made a mistake and need to hop back to "current - 12".
> edit, edit, edit
> edit, edit, edit
> publish everything, similar to just tarring up your workdir and sending
> out ### END WORKFLOW A ###
>
> ### WORKFLOW B ###
> edit, edit, edit
> ok this looks good, I want to save a checkpoint here
> edit, edit, edit
> looks good again. next checkpoint
> edit, edit, edit
> oh crap, back to checkpoint 2
> edit, edit, edit
> ooh, that's better. save a checkpoint and publish those checkpoints
> ### END WORKFLOW B ###

### WORKFLOW C ###
for every save on a gitfs mounted dir, do an implied checkpoint, commit, or 
publish (should be adjustable), on its privately created on-the-fly 
repository.
### END WORKFLOW C ###

For example:

  echo "// last comment on this file" >> /gitfs.mounted/file

should do an implied checkpoint, and make these checkpoints immediately 
visible under some checkpoint branch of the gitfs mounted dir.

Note, this way the developer gets version control without even noticing, and 
works completely transparent to any kind of application.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: git guidance

[Al Boldi]

Andreas Ericsson wrote:
 So, to get to the bottom of this, which of the following workflows is it
 you want git to support?

 ### WORKFLOW A ###
 edit, edit, edit
 edit, edit, edit
 edit, edit, edit
 Oops I made a mistake and need to hop back to current - 12.
 edit, edit, edit
 edit, edit, edit
 publish everything, similar to just tarring up your workdir and sending
 out ### END WORKFLOW A ###

 ### WORKFLOW B ###
 edit, edit, edit
 ok this looks good, I want to save a checkpoint here
 edit, edit, edit
 looks good again. next checkpoint
 edit, edit, edit
 oh crap, back to checkpoint 2
 edit, edit, edit
 ooh, that's better. save a checkpoint and publish those checkpoints
 ### END WORKFLOW B ###

### WORKFLOW C ###
for every save on a gitfs mounted dir, do an implied checkpoint, commit, or 
publish (should be adjustable), on its privately created on-the-fly 
repository.
### END WORKFLOW C ###

For example:

  echo // last comment on this file  /gitfs.mounted/file

should do an implied checkpoint, and make these checkpoints immediately 
visible under some checkpoint branch of the gitfs mounted dir.

Note, this way the developer gets version control without even noticing, and 
works completely transparent to any kind of application.


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: git guidance

[Al Boldi]

[EMAIL PROTECTED] wrote:
 On Fri, 07 Dec 2007 22:04:48 +0300, Al Boldi said:
  Because WORKFLOW C is transparent, it won't affect other workflows.  So
  you could still use your normal WORKFLOW B in addition to WORKFLOW C,
  gaining an additional level of version control detail at no extra cost
  other than the git-engine scratch repository overhead.
 
  BTW, is git efficient enough to handle WORKFLOW C?

 Imagine the number of commits a 'make clean; make' will do in a kernel
 tree, as it commits all those .o files... :)

.o files???

It probably goes without saying, that gitfs should have some basic 
configuration file to setup its transparent behaviour, and which would most 
probably contain an include / exclude file-filter mask, and probably other 
basic configuration options.  But this is really secondary to the 
implementation, and the question remains whether git is efficient enough.

IOW, how big is the git commit overhead as compared to a normal copy?


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: git guidance

[Al Boldi]

Jakub Narebski wrote:
 Al Boldi [EMAIL PROTECTED] writes:
  For example:
 
echo // last comment on this file  /gitfs.mounted/file
 
  should do an implied checkpoint, and make these checkpoints immediately
  visible under some checkpoint branch of the gitfs mounted dir.
 
  Note, this way the developer gets version control without even noticing,
  and works completely transparent to any kind of application.

 Why not use versioning filesystem for that, for example ext3cow
 (which looks suprisingly git-like, when you take into account that
 for ext3cow history is linear and centralized, so one can use date
 or sequential number to name commits).

 See GitLinks page on Git Wiki, Other links section:
   http://www.ext3cow.com/

Sure, Linus mentioned the cow idea before in this thread, but you would still 
need a few hacks to get some basic Version Control features.  

 Version control system is all about WORKFLOW B, where programmer
 controls when it is time to commit (and in private repository he/she
 can then rewrite history to arrive at Perfect patch series[*1*]);
 something that for example CVS failed at, requiring programmer to do
 a merge if upstream has any changes when trying to commit.

Because WORKFLOW C is transparent, it won't affect other workflows.  So you 
could still use your normal WORKFLOW B in addition to WORKFLOW C, gaining an 
additional level of version control detail at no extra cost other than the 
git-engine scratch repository overhead.

BTW, is git efficient enough to handle WORKFLOW C?


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: git guidance

[Al Boldi]

Johannes Schindelin wrote:
> Hi,

Hi

> On Fri, 7 Dec 2007, Al Boldi wrote:
> > You need to re-read the thread.
>
> I don't know why you write that, and then say thanks.  Clearly, what you
> wrote originally, and what Andreas pointed out, were quite obvious
> indicators that git already does what you suggest.
>
> You _do_ work "transparently" (whatever you understand by that overused
> term) in the working directory, unimpeded by git.

If you go back in the thread, you may find a link to a gitfs client that 
somebody kindly posted.  This client pretty much defines the transparency 
I'm talking about.  The only problem is that it's read-only.

To make it really useful, it has to support versioning locally, disconnected 
from the server repository.  One way to implement this, could be by 
committing every update unconditionally to an on-the-fly created git 
repository private to the gitfs client.

With this transparently created private scratch repository it should then be 
possible for the same gitfs to re-expose the locally created commits, all 
without any direct user-intervention.

Later, this same scratch repository could then be managed by the normal 
git-management tools/commands to ultimately update the backend git 
repositories.

BTW:  Sorry for my previous posts that contained the wrong date; it seems 
that hibernation sometimes advances the date by a full 24h.  Has anybody 
noticed this as well?


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: git guidance

[Al Boldi]

Andreas Ericsson wrote:
> Al Boldi wrote:
> > Phillip Susi wrote:
> >> Al Boldi wrote:
> >>> IOW, git currently only implements the server-side use-case, but fails
> >>> to deliver on the client-side.  By introducing a git-client manager
> >>> that handles the transparency needs of a single user, it should be
> >>> possible to clearly isolate update semantics for both the client and
> >>> the server, each handling their specific use-case.
> >>
> >> Any talk of client or server makes no sense since git does not use a
> >> client/server model.
> >
> > Whether git uses the client/server model or not does not matter; what
> > matters is that there are two distinct use-cases at work here:  one on
> > the server/repository, and the other on the client.
>
> Git is distributed. The repository is everywhere. No server is actually
> needed. Many use one anyway since it can be convenient. It's not, however,
> necessary.

When you read server, don't read it as localized; a server can be 
distributed.  What distinguishes a server from an engine is that it has to 
handle a multi-user use-case.  How that is implemented, locally or remotely 
or distributed, is another issue.

> >> If you wish to use a centralized repository, then
> >> git can be set up to transparently push/pull to/from said repository if
> >> you wish via hooks or cron jobs.
> >
> > Again, this only handles the interface to/from the server/repository,
> > but once you pulled the sources, it leaves you without Version Control
> > on the client.
>
> No, that's CVS, SVN and other centralized scm's. With git you have perfect
> version control on each peer. That's the entire idea behind "fully
> distributed".

As explained before in this thread, replicating the git tree on the client 
still doesn't provide the required transparency.

> > By pulling the sources into a git-client manager mounted on some dir, it
> > should be possible to let the developer work naturally/transparently in
> > a readable/writeable manner, and only require his input when reverting
> > locally or committing to the server/repository.
>
> How is that different from what every SCM, including git, is doing today?
> The user needs to tell the scm when it's time to take a snapshot of the
> current state. Git is distributed though, so committing is usually not the
> same as publishing. Is that lack of a single command to commit and publish
> what's nagging you? If it's not, I completely fail to see what you're
> getting at, unless you've only ever looked at repositories without a
> worktree attached, or you think that git should work like an editor's
> "undo" functionality, which would be quite insane.

You need to re-read the thread.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: git guidance

[Al Boldi]

Phillip Susi wrote:
> Al Boldi wrote:
> > IOW, git currently only implements the server-side use-case, but fails
> > to deliver on the client-side.  By introducing a git-client manager that
> > handles the transparency needs of a single user, it should be possible
> > to clearly isolate update semantics for both the client and the server,
> > each handling their specific use-case.
>
> Any talk of client or server makes no sense since git does not use a
> client/server model.

Whether git uses the client/server model or not does not matter; what matters 
is that there are two distinct use-cases at work here:  one on the 
server/repository, and the other on the client.  

> If you wish to use a centralized repository, then
> git can be set up to transparently push/pull to/from said repository if
> you wish via hooks or cron jobs.

Again, this only handles the interface to/from the server/repository, but 
once you pulled the sources, it leaves you without Version Control on the 
client.

By pulling the sources into a git-client manager mounted on some dir, it 
should be possible to let the developer work naturally/transparently in a 
readable/writeable manner, and only require his input when reverting locally 
or committing to the server/repository.


Thanks!

--
Al

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: git guidance

[Al Boldi]

Phillip Susi wrote:
 Al Boldi wrote:
  IOW, git currently only implements the server-side use-case, but fails
  to deliver on the client-side.  By introducing a git-client manager that
  handles the transparency needs of a single user, it should be possible
  to clearly isolate update semantics for both the client and the server,
  each handling their specific use-case.

 Any talk of client or server makes no sense since git does not use a
 client/server model.

Whether git uses the client/server model or not does not matter; what matters 
is that there are two distinct use-cases at work here:  one on the 
server/repository, and the other on the client.  

 If you wish to use a centralized repository, then
 git can be set up to transparently push/pull to/from said repository if
 you wish via hooks or cron jobs.

Again, this only handles the interface to/from the server/repository, but 
once you pulled the sources, it leaves you without Version Control on the 
client.

By pulling the sources into a git-client manager mounted on some dir, it 
should be possible to let the developer work naturally/transparently in a 
readable/writeable manner, and only require his input when reverting locally 
or committing to the server/repository.


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: git guidance

[Al Boldi]

Andreas Ericsson wrote:
 Al Boldi wrote:
  Phillip Susi wrote:
  Al Boldi wrote:
  IOW, git currently only implements the server-side use-case, but fails
  to deliver on the client-side.  By introducing a git-client manager
  that handles the transparency needs of a single user, it should be
  possible to clearly isolate update semantics for both the client and
  the server, each handling their specific use-case.
 
  Any talk of client or server makes no sense since git does not use a
  client/server model.
 
  Whether git uses the client/server model or not does not matter; what
  matters is that there are two distinct use-cases at work here:  one on
  the server/repository, and the other on the client.

 Git is distributed. The repository is everywhere. No server is actually
 needed. Many use one anyway since it can be convenient. It's not, however,
 necessary.

When you read server, don't read it as localized; a server can be 
distributed.  What distinguishes a server from an engine is that it has to 
handle a multi-user use-case.  How that is implemented, locally or remotely 
or distributed, is another issue.

  If you wish to use a centralized repository, then
  git can be set up to transparently push/pull to/from said repository if
  you wish via hooks or cron jobs.
 
  Again, this only handles the interface to/from the server/repository,
  but once you pulled the sources, it leaves you without Version Control
  on the client.

 No, that's CVS, SVN and other centralized scm's. With git you have perfect
 version control on each peer. That's the entire idea behind fully
 distributed.

As explained before in this thread, replicating the git tree on the client 
still doesn't provide the required transparency.

  By pulling the sources into a git-client manager mounted on some dir, it
  should be possible to let the developer work naturally/transparently in
  a readable/writeable manner, and only require his input when reverting
  locally or committing to the server/repository.

 How is that different from what every SCM, including git, is doing today?
 The user needs to tell the scm when it's time to take a snapshot of the
 current state. Git is distributed though, so committing is usually not the
 same as publishing. Is that lack of a single command to commit and publish
 what's nagging you? If it's not, I completely fail to see what you're
 getting at, unless you've only ever looked at repositories without a
 worktree attached, or you think that git should work like an editor's
 undo functionality, which would be quite insane.

You need to re-read the thread.


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: git guidance

[Al Boldi]

Johannes Schindelin wrote:
 Hi,

Hi

 On Fri, 7 Dec 2007, Al Boldi wrote:
  You need to re-read the thread.

 I don't know why you write that, and then say thanks.  Clearly, what you
 wrote originally, and what Andreas pointed out, were quite obvious
 indicators that git already does what you suggest.

 You _do_ work transparently (whatever you understand by that overused
 term) in the working directory, unimpeded by git.

If you go back in the thread, you may find a link to a gitfs client that 
somebody kindly posted.  This client pretty much defines the transparency 
I'm talking about.  The only problem is that it's read-only.

To make it really useful, it has to support versioning locally, disconnected 
from the server repository.  One way to implement this, could be by 
committing every update unconditionally to an on-the-fly created git 
repository private to the gitfs client.

With this transparently created private scratch repository it should then be 
possible for the same gitfs to re-expose the locally created commits, all 
without any direct user-intervention.

Later, this same scratch repository could then be managed by the normal 
git-management tools/commands to ultimately update the backend git 
repositories.

BTW:  Sorry for my previous posts that contained the wrong date; it seems 
that hibernation sometimes advances the date by a full 24h.  Has anybody 
noticed this as well?


Thanks!

--
Al

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/