date:20001227


Hi,

Several days ago I got a CD with Russian filenames on it and discovered that
I'm unable to read those filenames. After some hacking I produced a patch,
which should solve this problem in the manner similar to what we have in
msdosfs module (i.e. user-provided conversion table). I have to emphasize that
it's a temporary solution until we will have iconv support in kernel.

Please somebody review attached patches.

-Maxim


Index: cd9660/cd9660_lookup.c
===
RCS file: /home/ncvs/src/sys/isofs/cd9660/cd9660_lookup.c,v
retrieving revision 1.25
diff -d -u -r1.25 cd9660_lookup.c
--- cd9660/cd9660_lookup.c  2000/10/03 04:39:50 1.25
+++ cd9660/cd9660_lookup.c  2000/12/27 10:03:04
@@ -239,7 +239,7 @@
if (namelen != 1
|| ep-name[0] != 0)
goto notfound;
-   } else if (!(res = isofncmp(name, len, ep-name, 
namelen, imp-joliet_level))) {
+   } else if (!(res = isofncmp(name, len, ep-name, 
+namelen, imp-joliet_level, imp-ctable))) {
if (isoflags  2)
ino = isodirino(ep, imp);
else
Index: cd9660/cd9660_mount.h
===
RCS file: /home/ncvs/src/sys/isofs/cd9660/cd9660_mount.h,v
retrieving revision 1.4
diff -d -u -r1.4 cd9660_mount.h
--- cd9660/cd9660_mount.h   2000/05/01 20:05:04 1.4
+++ cd9660/cd9660_mount.h   2000/12/27 10:03:04
@@ -47,6 +47,7 @@
struct  export_args export; /* network export info */
int flags;  /* mounting flags, see below */
int ssector;/* starting sector, 0 for 1st session */
+   u_char  *ctable[256];   /* Table for converting unicode filenames */
 };
 #defineISOFSMNT_NORRIP 0x0001  /* disable Rock Ridge Ext.*/
 #defineISOFSMNT_GENS   0x0002  /* enable generation numbers */
Index: cd9660/cd9660_rrip.c
===
RCS file: /home/ncvs/src/sys/isofs/cd9660/cd9660_rrip.c,v
retrieving revision 1.18
diff -d -u -r1.18 cd9660_rrip.c
--- cd9660/cd9660_rrip.c2000/05/05 09:58:17 1.18
+++ cd9660/cd9660_rrip.c2000/12/27 10:03:05
@@ -301,7 +301,7 @@
 {
isofntrans(isodir-name,isonum_711(isodir-name_len),
   ana-outbuf,ana-outlen,
-  1,isonum_711(isodir-flags)4, ana-imp-joliet_level);
+  1,isonum_711(isodir-flags)4, ana-imp-joliet_level, 
+ana-imp-ctable);
switch (*ana-outbuf) {
default:
break;
@@ -509,7 +509,7 @@
pwhead = isodir-name + isonum_711(isodir-name_len);
if (!(isonum_711(isodir-name_len)1))
pwhead++;
-   isochar(isodir-name, pwhead, ana-imp-joliet_level, c);
+   isochar(isodir-name, pwhead, ana-imp-joliet_level, c, ana-imp-ctable);
 
/* If it's not the '.' entry of the root dir obey SP field */
if (c != 0 || isonum_733(isodir-extent) != ana-imp-root_extent)
@@ -646,7 +646,7 @@
*outlen = 0;
 
isochar(isodir-name, isodir-name + isonum_711(isodir-name_len),
-   imp-joliet_level, c);
+   imp-joliet_level, c, imp-ctable);
tab = rrip_table_getname;
if (c == 0 || c == 1) {
cd9660_rrip_defname(isodir,analyze);
Index: cd9660/cd9660_util.c
===
RCS file: /home/ncvs/src/sys/isofs/cd9660/cd9660_util.c,v
retrieving revision 1.15
diff -d -u -r1.15 cd9660_util.c
--- cd9660/cd9660_util.c2000/10/29 13:56:43 1.15
+++ cd9660/cd9660_util.c2000/12/27 10:03:05
@@ -52,25 +52,28 @@
  * Return number of bytes consumed
  */
 int
-isochar(isofn, isoend, joliet_level, c)
+isochar(isofn, isoend, joliet_level, c, ctable)
   u_char *isofn;
   u_char *isoend;
   int joliet_level;
   u_char *c;
+  u_char **ctable;
 {
   *c = *isofn++;
   if (joliet_level == 0 || isofn == isoend)
   /* (00) and (01) are one byte in Joliet, too */
   return 1;
 
-  /* No Unicode support yet :-( */
+  /* Limited Unicode support yet :-( */
+  /* (requires user-supplied conversion table) */
   switch (*c) {
-  default:
-  *c = '?';
-  break;
-  case '\0':
+  case '\0':   /* ANSI */
   *c = *isofn;
   break;
+  default:
+  if ((ctable[*c] == NULL) || ((*c = ctable[*c][*isofn]) == '\0'))
+  *c = '?';
+  break;
   }
   return 2;
 }
@@ -81,12 +84,13 @@
  * Note: Version number plus ';' may be omitted.
  */
 int
-isofncmp(fn, fnlen, isofn,

Re: Unicode support in cd9660 [patch for review]


On Wed, Dec 27, 2000 at 12:05:57 +0200, Maxim Sobolev wrote:
 Please somebody review attached patches.
 + u_char  *ctable[256];   /* Table for converting unicode filenames */

You deside to use per- Unicode base conversion table, it takes much memory
and don't satisfy in any case because you miss other graphics related to
charset of OS that made CD (I mean high code table characters like
copyright, angle quotes and so on). Better variant is to use exact to/from
Unicode conversion tables, if you know exact charset of OS that made CD.
I.e. I suggest to use the method that MSDOSFS currently use which is
foreign-charset - Unicode - native charset. It takes small memory and
convert names without loss of some characters.

-- 
Andrey A. Chernov
http://ache.pp.ru/


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: Unicode support in cd9660 [patch for review]


Àíäðåé ×åðíîâ wrote:

 On Wed, Dec 27, 2000 at 12:05:57 +0200, Maxim Sobolev wrote:
  Please somebody review attached patches.
  + u_char  *ctable[256];   /* Table for converting unicode filenames */

 You deside to use per- Unicode base conversion table, it takes much memory

Not too much - only 1K per cd9660 mount point for machines with sizeof(u_char *) == 4.
This also provides opportunity to load several tables with different bases
transparently.

 and don't satisfy in any case because you miss other graphics related to
 charset of OS that made CD (I mean high code table characters like
 copyright, angle quotes and so on). Better variant is to use exact to/from
 Unicode conversion tables, if you know exact charset of OS that made CD.

I'm now sure how could I obtain charset for each of dozen+ OSes that may create a CD.

 I.e. I suggest to use the method that MSDOSFS currently use which is
 foreign-charset - Unicode - native charset. It takes small memory and
 convert names without loss of some characters.

I don't see any problems, because it's likely that usual special high code table
characters (copyright, angle quotes and so on) will be represented using Unicode
charcodes with first byte (`base') equal to 0, so they can be mapped directly into
native charset. In my implementation only Unicode characters with base !=0 are to be
translated. All less usual characters (graphics and so on) can be translated by
extending appropriate codetable to include additional translation tables with different
bases (e.g. 0x25 for graphics chars etc.).

-Maxim



To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: NFS/VM panic in current with INVARIANTS

2000-12-27 Thread Mark Murray


It looks like you guys got it! What is currently checked in (by Assar)
is working fine! :-)

M

 :--=-=-=
 :
 :Matt Dillon [EMAIL PROTECTED] writes:
 : Well, yes... that's essentially what I suggested.  You didn't say
 : anything about removing the externalized inlines, which is what you
 : do in your patch.  Looks like a fine patch to me.
 :
 :Except it didn't work.  Now here's a patch that survived building a
 :kernel and modules.  ok?
 :
 :/assar

lets see.. ok, I'm going to read the patch carefully this time..
 
This isn't quite what I had in mind.  You are still making an API
change... you are now calling zalloci() from zalloc() for non-SMP boxes.
There is no need to do that.
 
Instead what I would recommend is making zalloc() and zfree() real
procedures, putting them in vm_zone.c, and removing their inlines from
vm_zone.h.  i.e. not have *ANY* inlines at all in vm_zone.h
 
Then as real procedures in vm_zone.c these functions can have the
#ifdef SMP / related code left intact.
 
   -Matt
 
 :
 :--=-=-=
 :Content-Disposition: attachment; filename=zalloc-diff
 :
 :Index: vm_zone.c
 :===
 :RCS file: /home/ncvs/src/sys/vm/vm_zone.c,v
 :retrieving revision 1.35
 :diff -u -w -r1.35 vm_zone.c
 :--- vm_zone.c2000/12/13 10:01:00 1.35
 :+++ vm_zone.c2000/12/27 00:59:14
 :@@ -32,6 +32,59 @@
 : 
 : static MALLOC_DEFINE(M_ZONE, "ZONE", "Zone header");
 : 
 :+#define ZONE_ERROR_INVALID 0
 :+#define ZONE_ERROR_NOTFREE 1
 :+#define ZONE_ERROR_ALREADYFREE 2
 :+
 :+#define ZONE_ROUNDING   32
 :+
 :+#define ZENTRY_FREE 0x12342378
 :+/*
 :+ * void *zalloc(vm_zone_t zone) --
 :+ *  Returns an item from a specified zone.
 :+ *
 :+ * void zfree(vm_zone_t zone, void *item) --
 :+ *  Frees an item back to a specified zone.
 :+ */
 :+static __inline__ void *
 :+_zalloc(vm_zone_t z)
 :+{
 :+void *item;
 :+
 :+#ifdef INVARIANTS
 :+if (z == 0)
 :+zerror(ZONE_ERROR_INVALID);
 :+#endif
 :+
 :+if (z-zfreecnt = z-zfreemin)
 :+return _zget(z);
 :+
 :+item = z-zitems;
 :+z-zitems = ((void **) item)[0];
 :+#ifdef INVARIANTS
 :+if (((void **) item)[1] != (void *) ZENTRY_FREE)
 :+zerror(ZONE_ERROR_NOTFREE);
 :+((void **) item)[1] = 0;
 :+#endif
 :+
 :+z-zfreecnt--;
 :+z-znalloc++;
 :+return item;
 :+}
 :+
 :+static __inline__ void
 :+_zfree(vm_zone_t z, void *item)
 :+{
 :+((void **) item)[0] = z-zitems;
 :+#ifdef INVARIANTS
 :+if (((void **) item)[1] == (void *) ZENTRY_FREE)
 :+zerror(ZONE_ERROR_ALREADYFREE);
 :+((void **) item)[1] = (void *) ZENTRY_FREE;
 :+#endif
 :+z-zitems = item;
 :+z-zfreecnt++;
 :+}
 :+
 : /*
 :  * This file comprises a very simple zone allocator.  This is used
 :  * in lieu of the malloc allocator, where needed or more optimal.
 :Index: vm_zone.h
 :===
 :RCS file: /home/ncvs/src/sys/vm/vm_zone.h,v
 :retrieving revision 1.13
 :diff -u -w -r1.13 vm_zone.h
 :--- vm_zone.h1999/08/28 00:52:44 1.13
 :+++ vm_zone.h2000/12/27 00:59:14
 :@@ -43,7 +43,6 @@
 : struct vm_zone  *znext; /* list of zones for sysctl */
 : } *vm_zone_t;
 : 
 :-
 : voidzerror __P((int)) __dead2;
 : vm_zone_t   zinit __P((char *name, int size, int nentries, int flags,
 :int zalloc));
 :@@ -57,77 +56,16 @@
 :int nitems));
 : void *  _zget __P((vm_zone_t z));
 : 
 :-#define ZONE_ERROR_INVALID 0
 :-#define ZONE_ERROR_NOTFREE 1
 :-#define ZONE_ERROR_ALREADYFREE 2
 :-
 :-#define ZONE_ROUNDING   32
 :-
 :-#define ZENTRY_FREE 0x12342378
 :-/*
 :- * void *zalloc(vm_zone_t zone) --
 :- *  Returns an item from a specified zone.
 :- *
 :- * void zfree(vm_zone_t zone, void *item) --
 :- *  Frees an item back to a specified zone.
 :- */
 :-static __inline__ void *
 :-_zalloc(vm_zone_t z)
 :-{
 :-void *item;
 :-
 :-#ifdef INVARIANTS
 :-if (z == 0)
 :-zerror(ZONE_ERROR_INVALID);
 :-#endif
 :-
 :-if (z-zfreecnt = z-zfreemin)
 :-return _zget(z);
 :-
 :-item = z-zitems;
 :-z-zitems = ((void **) item)[0];
 :-#ifdef INVARIANTS
 :-if (((void **) item)[1] != (void *) ZENTRY_FREE)
 :-zerror(ZONE_ERROR_NOTFREE);
 :-((void **) item)[1] = 0;
 :-#endif
 :-
 :-z-zfreecnt--;
 :-z-znalloc++;
 :-return item;
 :-}
 :-
 :-static __inline__ void
 :-_zfree(vm_zone_t z, void *item)
 :-{
 :-((void **) item)[0] = z-zitems;
 :-#ifdef INVARIANTS
 :-if (((void **) item)[1] == (void *) ZENTRY_FREE)
 :-zerror(ZONE_ERROR_ALREADYFREE);
 :-((void **) item)[1] = (void *) ZENTRY_FREE;
 :-#endif
 :-z-zitems = item;
 :-z-zfreecnt++;
 :-}
 :-
 : static __inline__ void *
 : zalloc(vm_zone_t z)
 : {
 :-#if defined(SMP)
 :

Re: [FreeBSD-tech-jp 2988] Re: Unicode support in cd9660 [patch for review]

2000-12-27 Thread Noriyuki Soda


 On Wed, 27 Dec 2000 21:28:19 +0900,
Motomichi Matsuzaki [EMAIL PROTECTED] said:

msaki Any ideas?

There was dicussion about this issue on [EMAIL PROTECTED]
mailing list with Subject: "Unicode support in kernel" and
"code set recoding engine, V2" in October and November, 1999.

Summary of my proposal is the following:
http://mail-index.netbsd.org/tech-kern/1999/10/15/0009.html
http://mail-index.netbsd.org/tech-kern/1999/11/23/0002.html
(the former one is somewhat difficult, though)
--
soda


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: [FreeBSD-tech-jp 2988] Re: Unicode support in cd9660 [patchfor review]

2000-12-27 Thread Kenichi Okuyama


I don't know why this mail came to me, but since I was given a
chance(^^;)


 "MM" == Motomichi Matsuzaki [EMAIL PROTECTED] writes:
MM * filenames recorded on Unix filesystems (e.g. FFS, MFS) use
MM   an arbitrary codeset, for example Unicode.

Rather, let's use "codepage + codeset" information, so that we can
find the difference between Chinese "BONE" and Japanese "BONE".
WE NEED THEM TO BE DIFFERENT, YOU KNOW.

For example, save filename using 64bit per character, containing
codepage with 32bit, and codeset using UCS-4.

# We have enough diskspace and Memory to handle them, don't worry.
# And even if we didn't now, we will, within 2 years.

Do normalization for codepage against those characters that will not
be effected by codepage, so that comparison will be easier.


Many might say it's rediculous to have filename encoding different
from system call interface coding systems. But this is so only
because BUGGY UNICODE is current trend.

If we could have codeset that does not need codepage, the problem
did not occur. And the very reason why we happend to have this BUGGY
UNICODE, is because they stint bits. We should not do the same
mistakes.

So, there's only two selection.

1) Let's use Unicode for interface, and let's have large enough
   bits per characeter internally... like 256bits/character.

2) Let's create Truely Unified coding system, which not only allow
   us to describe the "currently used language", but also,
   exterminated languages like Cuneiform Characters as well.
   And use it for internally, and interface.
   ( This, also requres lot larger coding space than current.
 I think we do need 256bits anyway ).

best regards,
 
Kenichi Okuyama@Tokyo Research Lab, IBM-Japan, Co.


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: Unicode support in cd9660 [patch for review]

Motomichi Matsuzaki wrote:

At Wed, 27 Dec 2000 12:05:57 +0200,
Maxim Sobolev [EMAIL PROTECTED] wrote:
Several days ago I got a CD with Russian filenames on it and discovered that
I'm unable to read those filenames. After some hacking I produced a patch,

Vladimir Kushnir's patch will be for it.

http://www.freebsd.org/cgi/getmsg.cgi?fetch=270425+0+/usr/local/www/db/text/2000/freebsd-hackers/20001203.freebsd-hackers

and it is based on my patch:

http://triaez.kaisei.org/~mzaki/joliet/

which should solve this problem in the manner similar to what we have in
msdosfs module (i.e. user-provided conversion table). I have to emphasize that
it's a temporary solution until we will have iconv support in kernel.

*PLEASE* be careful about filename I18N.

1. Joliet extension

The Joliet extension are built on Unicode basis,
and is the "multilingual" filesystem.
We can found CDs which contain files named by all of
English, French, Russian, Chinese, and Japanese languages.
So charset conversion per mount is not sufficient.

You can specify multiple charset conversion tables for each mount point, the problem
is only to create appropriate conversion
tables (I do not have any CDs with anything than English/Russian filenames :- ).

3. Relation to userland applications

Currently, conversion table between Unicode and local charset are
widely needed and implemented, for such as the Joliet extension,
the FAT filesystem, TrueType rasterizers, WWW browsers, and so on.
We should share the tables as possible for their consintency.
So the ideal solution to code conversion are not in-kernel table
but userland shared library.
Therefore, filename code conversion should also be done in userland
as possible.

4. Rough idea of me

My preliminary idea to the filesystem I18N:

* filenames recorded on Unix filesystems (e.g. FFS, MFS) use
an arbitrary codeset, for example Unicode.

* interface between kernel and userland should use
filesystem-safe encoding, for example UTF-8.

* userland applications can convert from/to the user-requested
charsets, such as latin-2, koi8, and euc-jp, using shared library.

* the Joliet extension and UDF, which based on Unicode, need
no in-kernel conversion, in case Unix filesystems use Unicode.

* the FAT filesystem, which use both Unicode and conventional
codepages, requires in-kernel conversion in order to
write the conventional 8.3 names.

Any ideas?

Thanks for the pointing out, but I think that your proposal is too generic to be
committed any time soon (not even to mention
MFC'ing it). Moreover, as I pointed out, currently efforts to provide generic Unicode
functionality in kernel/userland are
underway, so it is likely that part of your work will be duplicated/obsoleted.

What I'm proposing here is quick'n'dirty (and limited as so) solution to allow
mounting CD's with unicode filenames on it.
This solution is targeted to be temporary until iconv-based kernel interfaces will
appear.

-Maxim

To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: [FreeBSD-tech-jp 2990] Re: Unicode support in cd9660 [patchfor review]

2000-12-27 Thread Noriyuki Soda


 On Wed, 27 Dec 2000 21:48:52 +0900 (JST),
Kenichi Okuyama [EMAIL PROTECTED] said:

okuyamak So, there's only two selection.

No. there is another one.

I.e.
3) Always use pair of "codeset" name and "codepoint" value
  for interface.

This keeps compatibility with current filesystems which use legacy
encodings like ISO-8859-1 or EUC-JP as pathnames.
My proposal follows this course.
--
soda


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: Unicode support in cd9660 [patch for review]

2000-12-27 Thread Motomichi Matsuzaki



At Wed, 27 Dec 2000 14:54:00 +0200,
Maxim Sobolev [EMAIL PROTECTED] wrote:
  The Joliet extension are built on Unicode basis,
  and is the "multilingual" filesystem.
  We can found CDs which contain files named by all of
  English, French, Russian, Chinese, and Japanese languages.
  So charset conversion per mount is not sufficient.
 You can specify multiple charset conversion tables for each mount point, the problem 
is only to create appropriate conversion
 tables (I do not have any CDs with anything than English/Russian filenames :- ).

Suppose a file which name contains multilingual characters.

Think Japanese researchers of Russian literatures.
The Microsoft Word document files about their works may have 
such complexed filenames. And Joliet can handle them.

The multiple mount point solution is insufficient to these situations.

  4. Rough idea of me
  My preliminary idea to the filesystem I18N:
 Thanks for the pointing out, but I think that your proposal is too
 generic to be committed any time soon (not even to mention MFC'ing it).

Yes, you're right. I have no more than such rough idea indeed.

 What I'm proposing here is quick'n'dirty (and limited as so) solution to allow 
mounting CD's with unicode filenames on it.
 This solution is targeted to be temporary until iconv-based kernel interfaces will 
appear.

But your solution is no effective and much harmful to multibyte users.
The "loading conversion tables on every mount points" idea is totally wrong.

-- 
Motomichi Matsuzaki [EMAIL PROTECTED] 
Dept. of Biological Sciences, Grad. School of Science, Univ. of Tokyo, Japan 



To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: Unicode support in cd9660 [patch for review]


Motomichi Matsuzaki wrote:

 At Wed, 27 Dec 2000 14:54:00 +0200,
 Maxim Sobolev [EMAIL PROTECTED] wrote:
   The Joliet extension are built on Unicode basis,
   and is the "multilingual" filesystem.
   We can found CDs which contain files named by all of
   English, French, Russian, Chinese, and Japanese languages.
   So charset conversion per mount is not sufficient.
  You can specify multiple charset conversion tables for each mount point, the 
problem is only to create appropriate conversion
  tables (I do not have any CDs with anything than English/Russian filenames :- ).

 Suppose a file which name contains multilingual characters.

 Think Japanese researchers of Russian literatures.
 The Microsoft Word document files about their works may have
 such complexed filenames. And Joliet can handle them.

Yeah, but unfortunately our fs interface can't. :(

 The multiple mount point solution is insufficient to these situations.

   4. Rough idea of me
   My preliminary idea to the filesystem I18N:
  Thanks for the pointing out, but I think that your proposal is too
  generic to be committed any time soon (not even to mention MFC'ing it).

 Yes, you're right. I have no more than such rough idea indeed.

  What I'm proposing here is quick'n'dirty (and limited as so) solution to allow 
mounting CD's with unicode filenames on it.
  This solution is targeted to be temporary until iconv-based kernel interfaces will 
appear.

 But your solution is no effective and much harmful to multibyte users.

You are not quite right. For multibyte users my solution (workaround?) is at least 
equial to the previous no-unicode case . I do
not see how it can be harmful.

-Maxim



To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: Unicode support in cd9660 [patch for review]

2000-12-27 Thread Motomichi Matsuzaki




At Wed, 27 Dec 2000 15:38:58 +0200,
Maxim Sobolev [EMAIL PROTECTED] wrote:
  But your solution is no effective and much harmful to multibyte users.
 You are not quite right. For multibyte users my solution (workaround?) is at least 
equial to the previous no-unicode case . I do
 not see how it can be harmful.

1. In just your workaround, multibyte users will take no merits.

2. Based on your direction, the size of loadable conversion table
   will immensely expand for multibyte support, or be abandoned.
   Fundamental misdesign will lead to such unfortunate situation.
   So I said your solution was harmful.

-- 
Motomichi Matsuzaki [EMAIL PROTECTED] 
Dept. of Biological Sciences, Grad. School of Science, Univ. of Tokyo, Japan 
  


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: Unicode support in cd9660 [patch for review]


Motomichi Matsuzaki wrote:

 At Wed, 27 Dec 2000 15:38:58 +0200,
 Maxim Sobolev [EMAIL PROTECTED] wrote:
   But your solution is no effective and much harmful to multibyte users.
  You are not quite right. For multibyte users my solution (workaround?) is at least 
equial to the previous no-unicode case . I do
  not see how it can be harmful.

 1. In just your workaround, multibyte users will take no merits.

 2. Based on your direction, the size of loadable conversion table
will immensely expand for multibyte support, or be abandoned.
Fundamental misdesign will lead to such unfortunate situation.
So I said your solution was harmful.

Proposed by me patches is no way an official direction of the Project and as I 
advertised are merely a workaround to allow non-English
users to read CD with native filenames until comprehensive iconv for kernel will be 
introduced. I would be glad if someone will
replace my hack with more generic solution.

-Maxim



To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: Unicode support in cd9660 [patch for review]


On Wed, Dec 27, 2000 at 12:05:57PM +0200, Maxim Sobolev scribbled:
| Several days ago I got a CD with Russian filenames on it and discovered that
| I'm unable to read those filenames. After some hacking I produced a patch,
| which should solve this problem in the manner similar to what we have in
| msdosfs module (i.e. user-provided conversion table). I have to emphasize that
| it's a temporary solution until we will have iconv support in kernel.
| 
| Please somebody review attached patches.

Please do not assume Unicode.  I18N/L10N efforts have been crying for
programmers to *not* and *never* assume *anything*.  :)

Also, this belongs more on -i18n more than anything else.  I really
do not want to generate all the traffic on -i18n alone.. :)

Have you seen ports/chinese/big5fs?  Japanese/Korean do the same thing too
You could simply make a port of this that loads KLD's.  This enables us 
to have the support without having hacks in src/sys.  I talked
to Boris at length about this.  And I think this would be the best way
to implement "temporary" hacks.

As to the progress of iconv, we should have it soon, as soon as
itojun and I work out how to import either the Citrus code
or Konstantin's code. 

If you make this a port, I say "go for it." :) If you wish to commit
to src/sys, I have strong doubts about this.  I hope you do not
mind my bluntness about this, as I really want to express the feelings.

Merry Christmas and a Happy New Year,
Michael
-- 
+--+
| [EMAIL PROTECTED] | [EMAIL PROTECTED] |
| http://peorth.iteration.net/~keichii | Yes, BSD is a conspiracy. |
+--+


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: Unicode support in cd9660 [patch for review]


On Wed, Dec 27, 2000 at 05:57:19PM +0200, Maxim Sobolev scribbled:
| Motomichi Matsuzaki wrote:
| 
|  At Wed, 27 Dec 2000 15:38:58 +0200,
|  Maxim Sobolev [EMAIL PROTECTED] wrote:
|But your solution is no effective and much harmful to multibyte users.
|   You are not quite right. For multibyte users my solution (workaround?) is at 
|least equial to the previous no-unicode case . I do
|   not see how it can be harmful.
| 
|  1. In just your workaround, multibyte users will take no merits.
| 
|  2. Based on your direction, the size of loadable conversion table
| will immensely expand for multibyte support, or be abandoned.
| Fundamental misdesign will lead to such unfortunate situation.
| So I said your solution was harmful.
| 
| Proposed by me patches is no way an official direction of the Project and as I 
|advertised are merely a workaround to allow non-English
| users to read CD with native filenames until comprehensive iconv for kernel will be 
|introduced. I would be glad if someone will
| replace my hack with more generic solution.

I think that making this "hack" a russian/xxxfs port and I think
everyone can be happy. If you want unicode FS like this, perhaps you
can have a sysutils/unicodefs.  :) 

After all, what are KLD's for but modularity? :)


-- 
+--+
| [EMAIL PROTECTED] | [EMAIL PROTECTED] |
| http://peorth.iteration.net/~keichii | Yes, BSD is a conspiracy. |
+--+


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: Cardbus woes

2000-12-27 Thread Jonathan Chen


On Thu, Dec 14, 2000 at 12:47:51PM -0800, John Baldwin wrote:
 In other news, while wandering through the cardbus code, I discovered that
 pccbb softc's have an internal mutex much to my surprise, and that they weren't
 quite being used properly AFAICT.  In the pccb0 kthread, they were
 acquired/released without Giant.  However, in the rest of the pccbb driver,
 they are acquired/released with Giant.  This leads to lock order violations 
 which could potentially deadlock the machine at some point.  Also, the pccbb0
 kthread holds the mutex across the entire card insertion and removal events,
 which is not quite right.  Mutexes should only be held for short periods of
 time.  As such, I've futzed around with the mutex operations in the pccbb
 driver some.  It may not be completely correct, but I think it is an
 improvement.  The patch for this can be found at
 http://www.FreeBSD.org/~jhb/patches/pccbb.patch.

[I just now recalled your mentioning this a while back, so here's my much
delayed response]

Originally I had write the code with spl's, and at one point decided to
convert them to mutex after finding spl's don't do anything.  However, I
don't claim to know a whole lot about mutexes or FreeBSD's implementation
of it.  Perhaps someone can point me to the Great Definitive Source?

The first thing I'm unclear about is the deal with Giant.  My understanding
is that Giant is something one would acquire if he wanted to be really
exclusive, or something like that.  John - you mentioned I was
acquiring/releasing with/without Giant - but I don't see how I'm doing
things differently in and out of the kthread.  The only place I mentioned
Giant was right before kthread_exit, and that's necessary least
kthread_exit calls panic() on me.  On your patch, you acquire Giant at the
start of the kthread - wouldn't that hold onto the lock until the thread
exits, and make other things attempting to lock Giant block?

As for the 'holding onto mutex for a long time' problem, I'm not sure if
your solution is best.  I originally put in the locks firstly because
that's how NetBSD had it, and secondly so that card removal interrupts
while card is still attaching wouldn't be lost.  Your patched version
allows for wakeup() to be called while insertion/removal is being
processed.  I've glanced through the code and didn't see anything that
requires mutual exclusivity between the kthread the the interrupt handler,
so perhaps a better solution would be to simply not use mutex at all?

Comments?

-- 
(o_ 1-2-1-2-1-2-1-2-1-2-1-2-1-2-1-2-1-2-1-2-1-2-1-2-1-2-1-2-1-2 _o)
 \\\_\Jonathan Chen  [EMAIL PROTECTED]   /_///
 )  No electrons were harmed during production of this message (
 ~


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: Unicode support in cd9660 [patch for review]

2000-12-27 Thread Konstantin Chuguev


"Michael C . Wu" wrote:

 As to the progress of iconv, we should have it soon, as soon as
 itojun and I work out how to import either the Citrus code
 or Konstantin's code.


As I could see from the CVSed Citrus code, it's a locale library rather than iconv,
and it's just got the stub calls of iconv functions. So I thought Citrus and iconv
complement each other and don't do the same. Am I wrong?

As for the iconv library itself, its userland part is complete. I'm really busy
these days despite Christmas and the New Year, but I expect to release v2.1 in 3
weeks with the following changes:

   * the two patches from ports will be incorporated;
   * a few charsets added (to provide compatibility with the libiconv port and to
 be able to use glib-1.3 with iconv - the only port still depending on
 libiconv);
   * memory and file management functions moved into a separate file; then a
 kernel-side iconv implementation can compile iconv with its own specific
 memory and file management functions (from a different file).

I tried to write a kernel module, but I don't have enough knowledge of the kernel.
If anybody would like to do it, I am ready to help.

Regards,
Konstantin.

--
  * *Konstantin Chuguev - Application Engineer
   *  *  Francis House, 112 Hills Road
 *   Cambridge CB2 1PQ, United Kingdom
 D  A  N  T  E   WWW:http://www.dante.net


N'²æìr¸zÇ§vf¢Új:+v¨· è®"¶§²æìr¸yúÞy»rêëz{bØ^nr¡ûazg¬±¨

Re: NFS/VM panic in current with INVARIANTS

2000-12-27 Thread Matt Dillon


:
:It looks like you guys got it! What is currently checked in (by Assar)
:is working fine! :-)
:
:M

Excellent news!

-Matt


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: Unicode support in cd9660 [patch for review]


On Wed, Dec 27, 2000 at 08:32:26PM +0200, Maxim Sobolev scribbled:
| "Michael C . Wu" wrote:
|  On Wed, Dec 27, 2000 at 05:57:19PM +0200, Maxim Sobolev scribbled:
|  | Motomichi Matsuzaki wrote:
|  I think that making this "hack" a russian/xxxfs port and I think
|  everyone can be happy. If you want unicode FS like this, perhaps you
|  can have a sysutils/unicodefs.  :)

| No, it's no way a ports- commit.

Thank you for the one sentence reply without giving any reasons.
As a community, I thought we are supposed to communicate.
I have expressed my reasons and ideas, and you return the favor
with an one line comment.  :)

There is nothing wrong with have a port install a KLD, vmware does it.
And ports/emulators/linux_base depends on linux.ko. 

keichii@recursive:/usr/ports$ ls japanese/msdosfs/
Makefile files/   patches.5/   pkg-descr
distinfo patches.4/   pkg-comment  pkg-plist
keichii@recursive:/usr/ports$ ls chinese/big5fs/
Makefile distinfo files/   pkg-comment  pkg-descrpkg-plist
keichii@recursive:/usr/ports$ cat chinese/big5fs/pkg-descr 
This port installs two kernel modules, cd9660.ko and
msdos.ko, which will let users read Big5 filenames on
Joliet and VFAT filesystems.

Why do you think we did this?  You have admitted that this is an
ugly hack.  Since when did we allow ugly hacks in the kernel when
we can avoid it? Japanese/Chinese chose to do this because we
do not want dirty stuff in the kernel.  We have had this FS
support since 2.2.x.  

Please understand that we do not want hacks, and we certainly do
not want to be reliant on unicode alone.  We should never ever
assume stuff in I18N, and you are doing exactly this.  Have you
any idea what this would cause?  Do you know how hard it is to remove
something after it works for a while?  We do NOT ever want to have to
do legacy support in the future, because *you* want unicode and 
you want it now.  

In addition, this would probably mean laziness in the future to 
do this support properly.  In -SMP, they are avoiding to do hacks
to make things work temporarily.  Why should I18N do otherwise?

You can commit this to work with iconv() after iconv() gets in.
I would personally cheer you on.

I apologize for any harshness in the emails, it is quite hard
to express things without facial expressions.  I think we all
care about the project a lot.  Please be assured
that I mean no offense to you.
-- 
+--+
| [EMAIL PROTECTED] | [EMAIL PROTECTED] |
| http://peorth.iteration.net/~keichii | Yes, BSD is a conspiracy. |
+--+


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: Unicode support in cd9660 [patch for review]


On Wed, Dec 27, 2000 at 13:44:11 +0200, Maxim Sobolev wrote:
 I'm now sure how could I obtain charset for each of dozen+ OSes that may create a CD.

There is not so much number, usualy only one Russian charset per OS :-)

 I don't see any problems, because it's likely that usual special high code table
 characters (copyright, angle quotes and so on) will be represented using Unicode
 charcodes with first byte (`base') equal to 0, so they can be mapped directly into
 native charset. In my implementation only Unicode characters with base !=0 are to be
 translated. All less usual characters (graphics and so on) can be translated by
 extending appropriate codetable to include additional translation tables with 
different
 bases (e.g. 0x25 for graphics chars etc.).

Well, I could live with it in case you add _whole_ windows-1251 set as
Unicode to your loadable table and provide corresponding mapping for all
matching KOI8-R characters, as MSDOSFS currently does. You can get those
tables from MSDOSFS. This is minimal basis, you can make separate table
for KOI8-U, etc. I say this because I see you try to treat KOI8-R and
KOI8-U in your patch as the same charset which is not acceptable. Also
please call those tables per local charset name and not 'cd9660'.

-- 
Andrey A. Chernov
http://ache.pp.ru/


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: Unicode support in cd9660 [patch for review]


On Wed, Dec 27, 2000 at 12:48:12 -0600, Michael C . Wu wrote:
 On Wed, Dec 27, 2000 at 08:32:26PM +0200, Maxim Sobolev scribbled:
 | "Michael C . Wu" wrote:
 |  On Wed, Dec 27, 2000 at 05:57:19PM +0200, Maxim Sobolev scribbled:
 |  | Motomichi Matsuzaki wrote:
 |  I think that making this "hack" a russian/xxxfs port and I think
 |  everyone can be happy. If you want unicode FS like this, perhaps you
 |  can have a sysutils/unicodefs.  :)
 
 | No, it's no way a ports- commit.

I agree.

 Thank you for the one sentence reply without giving any reasons.
 As a community, I thought we are supposed to communicate.
 I have expressed my reasons and ideas, and you return the favor
 with an one line comment.  :)

This is general method of telling kernel how to convert names from Unicode
to any local charset, not particulary to Russian one. Anybody feel free to
add their own single bytes charsets tables.

Yes, it is a per-FS hack, but until iconv or something like will be
integrated, some hack needed just to read CDs selling at nearby shop.

 keichii@recursive:/usr/ports$ ls japanese/msdosfs/

Japanese etc. is _very_ different and needs tons of efforts even when
implemented like similar hack, we currently talk about and limited to
single bytes charsets only.

We have almost full single bytes charset localization in the
kernel/userland, so this step is next logical extention in that direction.
I can't say something like about double bytes charsets like Japanese etc.

-- 
Andrey A. Chernov
http://ache.pp.ru/


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

HDD Problem

2000-12-27 Thread Justin W. Pauler


Not sure exactly where this goes...
so, i'm sending it to everyone :P

I've heard tell that there are problems with the VIA chipset and UDMA on 
FreeBSD. Is this true, and if so, what is the problem with?

In my system is:

FreeBSD 4.2-STABLE #0: Fri Dec  8 01:52:44 EST 2000 
CPU: Pentium III/Pentium III Xeon/Celeron (601.37-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x683  Stepping = 3
atapci0: VIA 82C686 ATA66 controller port 0xe000-0xe00f at device 7.1 on 
pci0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
ad0: 19546MB FUJITSU MPF3204AT [39714/16/63] at ata0-master UDMA66 

The drive has enough space and I have 512MB RAM. If I hit the system hard, 
lets say, rm -rf /usr/ports, the system locks up. It doesn't just stop 
responding for a little bit. It locks up tight. It kills all network 
connections, doesn't respond to the console and doesn't respond to any 
network activity. A hard reboot is the only thing that can bring it back to 
life. 

One time, someone was able to see the console when this happened, and they 
said something about a page fault. Trying to solve this, we ordered two brand 
new 256MB modules and installed them. It is still happening.

This is a production environment type of server. I've had no other problems 
with FreeBSD 4.2-S, and i'm hoping maybe this is something that has been 
fixed.

-- 

Justin W. Pauler (drnet)
E-Mail: [EMAIL PROTECTED]
WWW: http://www.jwpages.com
IRC: Undernet IRC Network, #EggDrop



To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: Unicode support in cd9660 [patch for review]



-audit trimmed, cc'ed to -i18n

On Wed, Dec 27, 2000 at 10:02:01PM +0300, áÎÄÒÅÊ þÅÒÎÏ× scribbled:
| On Wed, Dec 27, 2000 at 12:48:12 -0600, Michael C . Wu wrote:
|  On Wed, Dec 27, 2000 at 08:32:26PM +0200, Maxim Sobolev scribbled:
|  | "Michael C . Wu" wrote:
|  |  On Wed, Dec 27, 2000 at 05:57:19PM +0200, Maxim Sobolev scribbled:
|  |  | Motomichi Matsuzaki wrote:
|  |  I think that making this "hack" a russian/xxxfs port and I think
|  |  everyone can be happy. If you want unicode FS like this, perhaps you
|  |  can have a sysutils/unicodefs.  :)
|  
|  | No, it's no way a ports- commit.
| 
| I agree.

I disagree.

|  Thank you for the one sentence reply without giving any reasons.
|  As a community, I thought we are supposed to communicate.
|  I have expressed my reasons and ideas, and you return the favor
|  with an one line comment.  :)
| 
| This is general method of telling kernel how to convert names from Unicode
| to any local charset, not particulary to Russian one. Anybody feel free to
| add their own single bytes charsets tables.

READ: "single byte"  
You are breaking CJK multibyte support.  Why? Why do you want make 
software engineering mistakes?  
 
| Yes, it is a per-FS hack, but until iconv or something like will be
| integrated, some hack needed just to read CDs selling at nearby shop.
| 
|  keichii@recursive:/usr/ports$ ls japanese/msdosfs/
| 
| Japanese etc. is _very_ different and needs tons of efforts even when
| implemented like similar hack, we currently talk about and limited to
| single bytes charsets only.

Why should you ignore the multibyte stuff because it is harder?
SMPng is hard, but no one is ignoring SMP because SMP is hard to implement.
Since device drivers for cheap hardware is hard to write/fix, should
we also ignore those?  Your logic fails to evaluate to TRUE.

| We have almost full single bytes charset localization in the
| kernel/userland, so this step is next logical extention in that direction.
| I can't say something like about double bytes charsets like Japanese etc.

Right, and as much as you want single byte stuff to work, CJK is also
a large market for FreeBSD.  Where else in the world is there a full-
fledged print monthly 200page magazine for BSD but Japan?
(Sorry, but DaemonNews is not 200 pages :)  

The proposal breaks multibyte support and I do not believe it is acceptable
to do this.  You want Russian to work, and us CJK people want CJK to work.
If you were looking at this from my point of view, would you say yes?
Probably not.  

The frustration with C charset is enough, can we please not make the mistake
that PDP-based BSD's made long ago? 

-- 
+--+
| [EMAIL PROTECTED] | [EMAIL PROTECTED] |
| http://peorth.iteration.net/~keichii | Yes, BSD is a conspiracy. |
+--+


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message

Re: Unicode support in cd9660 [patch for review]