Re: [U2] Binary data corruption on copy
Louie Bergsagel skrev: What do you mean by 50 different filesystems? 50 directory / Type 19 files? On unix machines you split each disk into up to a handfull partitions called filesystems wich may be mounted ( made accessible) individually. -- mats --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
Re: [U2] Binary data corruption on copy
What do you mean by 50 different filesystems? 50 directory / Type 19 files? On 8/23/07, Ross Morrissey <[EMAIL PROTECTED]> wrote: > > At PhotoWorks circa 2003, we had approximately 500,000,000 jpegs of > various > sizes sitting in Unix directories on one IBM RS6000. > > We got around the OS limitations by storing image 12345678a.jpg as > /12/34/12345678a.jpg, and the top level directories were on 50 different > filesystems as I recall. > > All the image metadata was stored in UniVerse on a separate server. The > only software running on the image server was Apache and a couple of shell > scripts (and the embedded Tivoli HSM software - yes all the images were > backed up off site). > > I understand PhotoWorks has moved off their IBM archive in the last few > years. --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
Re: [U2] Binary data corruption on copy
At PhotoWorks circa 2003, we had approximately 500,000,000 jpegs of various sizes sitting in Unix directories on one IBM RS6000. We got around the OS limitations by storing image 12345678a.jpg as /12/34/12345678a.jpg, and the top level directories were on 50 different filesystems as I recall. All the image metadata was stored in UniVerse on a separate server. The only software running on the image server was Apache and a couple of shell scripts (and the embedded Tivoli HSM software - yes all the images were backed up off site). I understand PhotoWorks has moved off their IBM archive in the last few years. On 8/23/07, Martin Phillips <[EMAIL PROTECTED]> wrote: > > Hi Harold, > > > The simplest solution - that of simply copying the JPG files to one type > > 19 directory - is working, to my amazement. There are 171,000 mugshots > > in one directory and they are being handled correctly. > > Remember that the process of searching an operating system directory is > essentially a linear scan. This will need to examine, on average, half the > entries before finding the one you want when reading. A write may have to > scan the whole directory. > > There probably isn't much you can do about this aside from moving to a > hashed file which might be a tuning nightmare. Also, if this is Unix, each > file takes an inode so you need to keep an eye on the file system limits. > > > Martin Phillips, Ladybridge Systems Ltd > --- > u2-users mailing list > u2-users@listserver.u2ug.org > To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
Re: [U2] Binary data corruption on copy
Hi Harold, The simplest solution - that of simply copying the JPG files to one type 19 directory - is working, to my amazement. There are 171,000 mugshots in one directory and they are being handled correctly. Remember that the process of searching an operating system directory is essentially a linear scan. This will need to examine, on average, half the entries before finding the one you want when reading. A write may have to scan the whole directory. There probably isn't much you can do about this aside from moving to a hashed file which might be a tuning nightmare. Also, if this is Unix, each file takes an inode so you need to keep an eye on the file system limits. Martin Phillips, Ladybridge Systems Ltd --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Binary data corruption on copy
Thanks to all who wrote about this problem. The simplest solution -
that of simply copying the JPG files to one type 19 directory - is
working, to my amazement. There are 171,000 mugshots in one directory
and they are being handled correctly. As far as a solution goes, that's
good enough.
However, Craig's technical example regarding conversion was worth
pursuing.
I forgot to mention that there are two steps: storing the JPG record
into the file, then copying it back out to a directory. Once it's out I
can use it. Simply, the record has to go in and out of the Universe
dynamic file intact.
I mimicked the code Craig supplied then wrote a corresponding program to
copy the record out. Since the code used OCONV before storing, I guessed
ICONV would be the choice when reading form the universe dynamic file:
ASSIGN 1 TO SYSTEM(1017)
OPEN 'TARGETFILE' TO TFT ELSE STOP ;* open the uv dyn file of jpg recs
OPEN 'MYDIR' TO MYDIR ELSE STOP ;* open a type 19 file
READ JPGREC FROM TFT,'1234.JPG' ELSE STOP
OUTREC = ICONV(JPGREC,"MX0C")
WRITE OUTREC TO MYDIR,'1234.JPG'
END
This works! So either solution will work.
Harold Oaks
Sr. Analyst/Programmer
Clark County, WA
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Craig Bennett
Sent: Tuesday, August 21, 2007 5:22 PM
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] Binary data corruption on copy
Hi Harold,
if I were doing this on UV I would do the following (inelegant error
handling but its just an example :):
If 1234.jpg is in a UV type 19 file:
OPEN "SOURCEFILE" TO SFT ELSE STOP
ASSIGN 1 TO SYSTEM(1017) ;* UV Specific - disables conversion of
CRLF to @AM on reading which damages your binary data
READ JPG FROM SFT, "1234.jpg" ELSE STOP
OPEN "TARGETFILE" TO TFT ELSE STOP
* Choose one of the following options to encode your data before
storing
* 1: Hex Conversion - exactly doubles the size of the file
TDAT = OCONV(JPG, "MX0C")
* 2: Base64 conversion - slower but more space (33% size increase).
ERRCDE = ENCODE("Base64", 1, JPG, 1, TDAT, 1)
WRITE TDAT ON TFT ELSE STOP
hth,
Craig
Oaks, Harold wrote:
> Dave-
>
> Of course, unidata may be different than universe in its file
structure.
> However, I would like to try what you do.
> How do you do that hex conversion? What is the next step? For
> example, I have 1234.jpg at unix level. What specifically do you do
> to get that into your unidata file?
>
> Harold
>
> -Original Message-
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED] On Behalf Of Dave Davis
> Sent: Tuesday, August 21, 2007 10:59 AM
> To: u2-users@listserver.u2ug.org
> Subject: RE: [U2] Binary data corruption on copy
>
> That "$" trick is probably specific to pi/open.
>
> I normally convert this kind of file to hex before storing in unidata.
>
> If there's a better way, I would like to know about it.
>
> -----Original Message-----
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED] On Behalf Of Oaks, Harold
> Sent: Tuesday, August 21, 2007 1:10 PM
> To: u2-users@listserver.u2ug.org
> Subject: [U2] Binary data corruption on copy
>
> A question for the group:
>
> I have JPG format records which I wish to copy from unix level into a
> Universe file but am having trouble with it getting 'corrupted' during
> the copy. We are converting from pi/open to universe.
>
> I can do this business in pi/open by changing the name of the source
> file, prepending it with a $, then copying at TCL level into the
> pi/open file. For example, if the record at unix level is named
> 1234.jpg and is in the directory SOURCE I run this command at unix
level:
> mv 1234.jpg \$1234.jpg
> which changes the name. Now at TCL level in pi/open I can issue the
> command COPY FROM SOURCE TO IMAGE_FILE $1234.jpg After that, I can
> change the name back within the pi/open file:
> CNAME IMAGE_FILE $1234.jpg 1234.jpg
> and the copy is complete.
>
> Pi/open knows that if the record name starts with a dollar sign that
> it is binary and should not be changed. Once the copy from a unix
> directory into the pi/open dynamic file is done I can use it
> successfully.
>
> In Universe I attempt to do the same thing. The 1234.jpg name is
> changed to $1234.jpg and then copied into the Universe file
> UNIV_IMAGE_FILE, a dynamic (type 30) file.
> But it doesn't work very well - the image is corrupted, the binary
> file was messed with during the copy.
>
> I know the jpg record is intact before I start the copying. I
> wouldn't need to bother with a copy to a Universe file, in fact,
> except that we have over 130,000 images now
Re: [U2] Binary data corruption on copy
I was suggesting type 1 to use the 14 character restriction so that there wouldn't be too many files in a single directory. phil walker wrote: Use type 19 to get around the 14 character restriction. --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Binary data corruption on copy
Use type 19 to get around the 14 character restriction. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Craig Bennett Sent: Thursday, 23 August 2007 1:00 p.m. To: u2-users@listserver.u2ug.org Subject: Re: [U2] Binary data corruption on copy > The OP was worried about a Unix directory not being able to handle the > thousands of jpegs he had. > Oh. Then in UV a type 1 file would be the go (assuming filenames are longer than 14 characters or perhaps using a mapping file to convert file ids to something that would spread nicely) these are unix directories (so you need to use ASSIGN 1 TO SYSTEM(1017) when reading/writing binary data) but file names longer than 14 characters cause directories and files within them to be created). eg: Item ID XXYYY becomes: directory XX containing file YYY Item ID XX which is exactly 14 characters becomes: directory XX containing file ? If you had to store 1 million files with no more than 1000 per directory you could map your original file names to: 14 character directory prefix (between 1 and 1000) File ID within character (between 1 and 1000). You could do this sort of hashing as many times as you needed to spread your jpegs into multiple level subdirectories. Messy too look at in unix, but totally transparent to UV. Craig --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
Re: [U2] Binary data corruption on copy
The OP was worried about a Unix directory not being able to handle the thousands of jpegs he had. Oh. Then in UV a type 1 file would be the go (assuming filenames are longer than 14 characters or perhaps using a mapping file to convert file ids to something that would spread nicely) these are unix directories (so you need to use ASSIGN 1 TO SYSTEM(1017) when reading/writing binary data) but file names longer than 14 characters cause directories and files within them to be created). eg: Item ID XXYYY becomes: directory XX containing file YYY Item ID XX which is exactly 14 characters becomes: directory XX containing file ? If you had to store 1 million files with no more than 1000 per directory you could map your original file names to: 14 character directory prefix (between 1 and 1000) File ID within character (between 1 and 1000). You could do this sort of hashing as many times as you needed to spread your jpegs into multiple level subdirectories. Messy too look at in unix, but totally transparent to UV. Craig --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Binary data corruption on copy
The OP was worried about a Unix directory not being able to handle the thousands of jpegs he had. I can't offer help as to how to store binary in a hashed file (other than what some other person suggested - convert it to ascii somehow eg uuencode or whatsit64), but I think you may well be able to store these files efficiently in a directory. Look at the various filesystems available - they may be optimised to do just what you want. In the linux world, reiserfs is apparently very good at handling thousands of small files such as a mail or news spool. And I've heard of at least one project to handle directories as hashed files (I don't know any more about that, though). There might well be something out there that stores the directory listing as a btree or somesuch that's efficient. Cheers, Wol -Original Message- From: Jerry Banker [mailto:[EMAIL PROTECTED] Sent: 22 August 2007 14:23 To: u2-users@listserver.u2ug.org Subject: RE: [U2] Binary data corruption on copy Universe doesn't have the noconvert command. -Original Message- From: David A. Green [mailto:[EMAIL PROTECTED] Sent: Wednesday, August 22, 2007 8:06 AM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Binary data corruption on copy Harold, In UniData you would store your binary files in a DIR type file. When you need to read the data you would first issue the NOCONVERT ON Command to keep the auto conversion of Nulls to CHAR(128) from happening. Thanks, David A. Green DAG Consulting 480-813-1725 --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Binary data corruption on copy
NOCONVERT helps with OSBREADS/WRITES and OSREADS/WRITES, but you would still have problems reading an entire binary record from a DIR type file using the standard READ statement in UniData. You are pretty much limited to just listing Ids if stored in a dir type. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jerry Banker Sent: Wednesday, August 22, 2007 9:23 AM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Binary data corruption on copy Universe doesn't have the noconvert command. -Original Message- From: David A. Green [mailto:[EMAIL PROTECTED] Sent: Wednesday, August 22, 2007 8:06 AM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Binary data corruption on copy Harold, In UniData you would store your binary files in a DIR type file. When you need to read the data you would first issue the NOCONVERT ON Command to keep the auto conversion of Nulls to CHAR(128) from happening. Thanks, David A. Green DAG Consulting 480-813-1725 --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Binary data corruption on copy
Universe doesn't have the noconvert command. -Original Message- From: David A. Green [mailto:[EMAIL PROTECTED] Sent: Wednesday, August 22, 2007 8:06 AM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Binary data corruption on copy Harold, In UniData you would store your binary files in a DIR type file. When you need to read the data you would first issue the NOCONVERT ON Command to keep the auto conversion of Nulls to CHAR(128) from happening. Thanks, David A. Green DAG Consulting 480-813-1725 --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Binary data corruption on copy
Harold, In UniData you would store your binary files in a DIR type file. When you need to read the data you would first issue the NOCONVERT ON Command to keep the auto conversion of Nulls to CHAR(128) from happening. Thanks, David A. Green DAG Consulting 480-813-1725 --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
Re: [U2] Binary data corruption on copy
Hi Harold,
if I were doing this on UV I would do the following (inelegant error
handling but its just an example :):
If 1234.jpg is in a UV type 19 file:
OPEN "SOURCEFILE" TO SFT ELSE STOP
ASSIGN 1 TO SYSTEM(1017) ;* UV Specific - disables conversion of
CRLF to @AM on reading which damages your binary data
READ JPG FROM SFT, "1234.jpg" ELSE STOP
OPEN "TARGETFILE" TO TFT ELSE STOP
* Choose one of the following options to encode your data before storing
* 1: Hex Conversion - exactly doubles the size of the file
TDAT = OCONV(JPG, "MX0C")
* 2: Base64 conversion - slower but more space (33% size increase).
ERRCDE = ENCODE("Base64", 1, JPG, 1, TDAT, 1)
WRITE TDAT ON TFT ELSE STOP
hth,
Craig
Oaks, Harold wrote:
Dave-
Of course, unidata may be different than universe in its file structure.
However, I would like to try what you do.
How do you do that hex conversion? What is the next step? For example,
I have 1234.jpg at unix level. What specifically do you do to get that
into your unidata file?
Harold
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Dave Davis
Sent: Tuesday, August 21, 2007 10:59 AM
To: u2-users@listserver.u2ug.org
Subject: RE: [U2] Binary data corruption on copy
That "$" trick is probably specific to pi/open.
I normally convert this kind of file to hex before storing in unidata.
If there's a better way, I would like to know about it.
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Oaks, Harold
Sent: Tuesday, August 21, 2007 1:10 PM
To: u2-users@listserver.u2ug.org
Subject: [U2] Binary data corruption on copy
A question for the group:
I have JPG format records which I wish to copy from unix level into a
Universe file but am having trouble with it getting 'corrupted' during
the copy. We are converting from pi/open to universe.
I can do this business in pi/open by changing the name of the source
file, prepending it with a $, then copying at TCL level into the pi/open
file. For example, if the record at unix level is named 1234.jpg and is
in the directory SOURCE I run this command at unix level:
mv 1234.jpg \$1234.jpg
which changes the name. Now at TCL level in pi/open I can issue the
command COPY FROM SOURCE TO IMAGE_FILE $1234.jpg After that, I can
change the name back within the pi/open file:
CNAME IMAGE_FILE $1234.jpg 1234.jpg
and the copy is complete.
Pi/open knows that if the record name starts with a dollar sign that it
is binary and should not be changed. Once the copy from a unix
directory into the pi/open dynamic file is done I can use it
successfully.
In Universe I attempt to do the same thing. The 1234.jpg name is
changed to $1234.jpg and then copied into the Universe file
UNIV_IMAGE_FILE, a dynamic (type 30) file.
But it doesn't work very well - the image is corrupted, the binary file
was messed with during the copy.
I know the jpg record is intact before I start the copying. I wouldn't
need to bother with a copy to a Universe file, in fact, except that we
have over 130,000 images now and I can't believe a unix directory could
successfully handle that many records.
Any suggestions on copying so that the binary remains intact?
Thanks-
Harold
Harold D. Oaks
Sr. Analyst/Programmer
Office of the Budget and Information Systems Clark County, Washington
ph: (360) 397-6121 x4132
fax: (360) 397-2342
---
u2-users mailing list
u2-users@listserver.u2ug.org
To unsubscribe please visit http://listserver.u2ug.org/
---
u2-users mailing list
u2-users@listserver.u2ug.org
To unsubscribe please visit http://listserver.u2ug.org/
---
u2-users mailing list
u2-users@listserver.u2ug.org
To unsubscribe please visit http://listserver.u2ug.org/
--
Craig Bennett
Director
Amajuba Pty Ltd
Phone: 02 9987 0161
Mobile: 0412448781
Fax: 02 8456 5943
---
u2-users mailing list
u2-users@listserver.u2ug.org
To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Binary data corruption on copy
Dave- Of course, unidata may be different than universe in its file structure. However, I would like to try what you do. How do you do that hex conversion? What is the next step? For example, I have 1234.jpg at unix level. What specifically do you do to get that into your unidata file? Harold -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Dave Davis Sent: Tuesday, August 21, 2007 10:59 AM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Binary data corruption on copy That "$" trick is probably specific to pi/open. I normally convert this kind of file to hex before storing in unidata. If there's a better way, I would like to know about it. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Oaks, Harold Sent: Tuesday, August 21, 2007 1:10 PM To: u2-users@listserver.u2ug.org Subject: [U2] Binary data corruption on copy A question for the group: I have JPG format records which I wish to copy from unix level into a Universe file but am having trouble with it getting 'corrupted' during the copy. We are converting from pi/open to universe. I can do this business in pi/open by changing the name of the source file, prepending it with a $, then copying at TCL level into the pi/open file. For example, if the record at unix level is named 1234.jpg and is in the directory SOURCE I run this command at unix level: mv 1234.jpg \$1234.jpg which changes the name. Now at TCL level in pi/open I can issue the command COPY FROM SOURCE TO IMAGE_FILE $1234.jpg After that, I can change the name back within the pi/open file: CNAME IMAGE_FILE $1234.jpg 1234.jpg and the copy is complete. Pi/open knows that if the record name starts with a dollar sign that it is binary and should not be changed. Once the copy from a unix directory into the pi/open dynamic file is done I can use it successfully. In Universe I attempt to do the same thing. The 1234.jpg name is changed to $1234.jpg and then copied into the Universe file UNIV_IMAGE_FILE, a dynamic (type 30) file. But it doesn't work very well - the image is corrupted, the binary file was messed with during the copy. I know the jpg record is intact before I start the copying. I wouldn't need to bother with a copy to a Universe file, in fact, except that we have over 130,000 images now and I can't believe a unix directory could successfully handle that many records. Any suggestions on copying so that the binary remains intact? Thanks- Harold Harold D. Oaks Sr. Analyst/Programmer Office of the Budget and Information Systems Clark County, Washington ph: (360) 397-6121 x4132 fax: (360) 397-2342 --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
Re: [U2] Binary data corruption on copy
Harold - You need to create your Universe jpg file as a Type 19 file (a long names directory). Then you can use a Unix cp to copy the files directly from the old PI directory to the new Universe directory. Then the files will remain the same. A dynamic file is a Universe managed file and everything in it is modified as it is stored. The entire file is Unix files, not a Unix directory that will store jpg files without modifying them. Ken At 10:10 AM 8/21/2007, Oaks, Harold wrote: A question for the group: I have JPG format records which I wish to copy from unix level into a Universe file but am having trouble with it getting 'corrupted' during the copy. We are converting from pi/open to universe. Any suggestions on copying so that the binary remains intact? --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Binary data corruption on copy
Why don't you just put in a 'DIR' type file, or a number of DIR types if there will be a lot of files in there? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Dave Davis Sent: Tuesday, August 21, 2007 12:59 PM To: u2-users@listserver.u2ug.org Subject: RE: [U2] Binary data corruption on copy That "$" trick is probably specific to pi/open. I normally convert this kind of file to hex before storing in unidata. If there's a better way, I would like to know about it. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Oaks, Harold Sent: Tuesday, August 21, 2007 1:10 PM To: u2-users@listserver.u2ug.org Subject: [U2] Binary data corruption on copy A question for the group: I have JPG format records which I wish to copy from unix level into a Universe file but am having trouble with it getting 'corrupted' during the copy. We are converting from pi/open to universe. I can do this business in pi/open by changing the name of the source file, prepending it with a $, then copying at TCL level into the pi/open file. For example, if the record at unix level is named 1234.jpg and is in the directory SOURCE I run this command at unix level: mv 1234.jpg \$1234.jpg which changes the name. Now at TCL level in pi/open I can issue the command COPY FROM SOURCE TO IMAGE_FILE $1234.jpg After that, I can change the name back within the pi/open file: CNAME IMAGE_FILE $1234.jpg 1234.jpg and the copy is complete. Pi/open knows that if the record name starts with a dollar sign that it is binary and should not be changed. Once the copy from a unix directory into the pi/open dynamic file is done I can use it successfully. In Universe I attempt to do the same thing. The 1234.jpg name is changed to $1234.jpg and then copied into the Universe file UNIV_IMAGE_FILE, a dynamic (type 30) file. But it doesn't work very well - the image is corrupted, the binary file was messed with during the copy. I know the jpg record is intact before I start the copying. I wouldn't need to bother with a copy to a Universe file, in fact, except that we have over 130,000 images now and I can't believe a unix directory could successfully handle that many records. Any suggestions on copying so that the binary remains intact? Thanks- Harold Harold D. Oaks Sr. Analyst/Programmer Office of the Budget and Information Systems Clark County, Washington ph: (360) 397-6121 x4132 fax: (360) 397-2342 --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Binary data corruption on copy
That "$" trick is probably specific to pi/open. I normally convert this kind of file to hex before storing in unidata. If there's a better way, I would like to know about it. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Oaks, Harold Sent: Tuesday, August 21, 2007 1:10 PM To: u2-users@listserver.u2ug.org Subject: [U2] Binary data corruption on copy A question for the group: I have JPG format records which I wish to copy from unix level into a Universe file but am having trouble with it getting 'corrupted' during the copy. We are converting from pi/open to universe. I can do this business in pi/open by changing the name of the source file, prepending it with a $, then copying at TCL level into the pi/open file. For example, if the record at unix level is named 1234.jpg and is in the directory SOURCE I run this command at unix level: mv 1234.jpg \$1234.jpg which changes the name. Now at TCL level in pi/open I can issue the command COPY FROM SOURCE TO IMAGE_FILE $1234.jpg After that, I can change the name back within the pi/open file: CNAME IMAGE_FILE $1234.jpg 1234.jpg and the copy is complete. Pi/open knows that if the record name starts with a dollar sign that it is binary and should not be changed. Once the copy from a unix directory into the pi/open dynamic file is done I can use it successfully. In Universe I attempt to do the same thing. The 1234.jpg name is changed to $1234.jpg and then copied into the Universe file UNIV_IMAGE_FILE, a dynamic (type 30) file. But it doesn't work very well - the image is corrupted, the binary file was messed with during the copy. I know the jpg record is intact before I start the copying. I wouldn't need to bother with a copy to a Universe file, in fact, except that we have over 130,000 images now and I can't believe a unix directory could successfully handle that many records. Any suggestions on copying so that the binary remains intact? Thanks- Harold Harold D. Oaks Sr. Analyst/Programmer Office of the Budget and Information Systems Clark County, Washington ph: (360) 397-6121 x4132 fax: (360) 397-2342 --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
[U2] Binary data corruption on copy
A question for the group: I have JPG format records which I wish to copy from unix level into a Universe file but am having trouble with it getting 'corrupted' during the copy. We are converting from pi/open to universe. I can do this business in pi/open by changing the name of the source file, prepending it with a $, then copying at TCL level into the pi/open file. For example, if the record at unix level is named 1234.jpg and is in the directory SOURCE I run this command at unix level: mv 1234.jpg \$1234.jpg which changes the name. Now at TCL level in pi/open I can issue the command COPY FROM SOURCE TO IMAGE_FILE $1234.jpg After that, I can change the name back within the pi/open file: CNAME IMAGE_FILE $1234.jpg 1234.jpg and the copy is complete. Pi/open knows that if the record name starts with a dollar sign that it is binary and should not be changed. Once the copy from a unix directory into the pi/open dynamic file is done I can use it successfully. In Universe I attempt to do the same thing. The 1234.jpg name is changed to $1234.jpg and then copied into the Universe file UNIV_IMAGE_FILE, a dynamic (type 30) file. But it doesn't work very well - the image is corrupted, the binary file was messed with during the copy. I know the jpg record is intact before I start the copying. I wouldn't need to bother with a copy to a Universe file, in fact, except that we have over 130,000 images now and I can't believe a unix directory could successfully handle that many records. Any suggestions on copying so that the binary remains intact? Thanks- Harold Harold D. Oaks Sr. Analyst/Programmer Office of the Budget and Information Systems Clark County, Washington ph: (360) 397-6121 x4132 fax: (360) 397-2342 --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
