Re: corrupt zip files
On Sun, 6 May 2012 12:40:29 -0500, Bill Godfrey yak36...@yahoo.com wrote: On Sat, 5 May 2012 08:54:52 -0500, Paul Edwards wrote: Most likely the original byte was x'14' which is also quite common in this location in zip files. In code page 437 and its cousins, x'14' is the paragraph sign. In iso9959-1 and many of its cousins, x'b6' is the same paragraph sign. In 437, x'15' is the section symbol. In iso8859-1, x'a7' is the same section symbol. Your occurrence counts show there are no x'14' or x'15', but among the few characters occurring above x'7f' are x'b6' and x'a7'. I would guess that at least two separate things happened to the file at different times. First, all the high-order bits were turned off. Later, a translation was done as if the data was presumed to be in code page 437, which converted x'14' and x'15' to x'b6' and x'a7', resulting in some values having the high-order bit set. Something else happened, either at the same time or not, that changed a few other characters to other values that have the high-order bit set, for which I have no explanation. Thanks Bill. That is the best working theory I have seen to date. Unfortunately it doesn't look like I will be provided an opportunity to get to the bottom of this, because the person who sent it has already sent a proper replacement, and is not interested beyond that. BFN. Paul. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: corrupt zip files
Too many unknowns to be of much help. Can't even begin to answer your question about what software would produce a given corruption pattern; there are -so- many possibilities where a translation might occur and thus corrupt the data. Indeed, the corruption might be as massive as what you are seeing, or as small as just a single charaer. The translation may even be happening more than once. For example, I have seen Windows translate a file sent in binary mode. I had to actally name the incomming file xxx.BIN to get Windows to leave it alone. I don't know of any good way to attack this withoug going all the way back to initial file creation, understand the file attributes (to include local code page), then step through the transmission process understanding exactly what is happening at each step. -Original Message- From: IBM Mainframe Discussion List [mailto:IBM-MAIN@bama.ua.edu] On Behalf Of Paul Edwards Sent: Saturday, May 05, 2012 8:55 AM To: IBM-MAIN@bama.ua.edu Subject: corrupt zip files I have a zip file that appears to have been produced using pkzip for z/OS. However, it looks like it has been transmitted using some sort of text protocol, because the high bit has been stripped from most bytes, and some other bytes appear to have been translated. e.g. I think x'0a' in the input file has been mangled to x'b6' on the way. Does anyone know what software would do a translation like that? ..snip NOTICE: This electronic mail message and any files transmitted with it are intended exclusively for the individual or entity to which it is addressed. The message, together with any attachment, may contain confidential and/or privileged information. Any unauthorized review, use, printing, saving, copying, disclosure or distribution is strictly prohibited. If you have received this message in error, please immediately advise the sender by reply email and delete all copies. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: corrupt zip files
On Thu, 10 May 2012 14:23:10 +, Hal Merritt wrote: I don't know of any good way to attack this withoug going all the way back to initial file creation, understand the file attributes (to include local code page), then step through the transmission process understanding exactly what is happening at each step. Why are checksum utilities so rare in the Classic mainframe and Windows worlds? -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: corrupt zip files
On Sat, 5 May 2012 08:54:52 -0500, Paul Edwards wrote: I have a zip file that appears to have been produced using pkzip for z/OS. However, it looks like it has been transmitted using some sort of text protocol, because the high bit has been stripped from most bytes, and some other bytes appear to have been translated. e.g. I think x'0a' in the input file has been mangled to x'b6' on the way. Does anyone know what software would do a translation like that? I believe these characters: 0A 1060 0D 0 12 1044 14 0 15 0 1C 0 1E 0 24 1030 7F 0 are being mapped to these: 81 576 9C 361 A7 645 B6 527 BF 284 E0 249 E9 644 F2 718 Except for x'0d' which I think is just being deleted. This belief comes from counting (see below) occurrences of the various bytes in a largish (80k) file. Here is a small file that shows the problem: 00 504B0304 B6000600 08006455 22405746 PKdU@WF That x'b6' above should be x'0a' I think (that's more normal). Apparently some protocol doesn't think that x'0a' will make it through, so translates it in advance. Most likely the original byte was x'14' which is also quite common in this location in zip files. In code page 437 and its cousins, x'14' is the paragraph sign. In iso9959-1 and many of its cousins, x'b6' is the same paragraph sign. In 437, x'15' is the section symbol. In iso8859-1, x'a7' is the same section symbol. Your occurrence counts show there are no x'14' or x'15', but among the few characters occurring above x'7f' are x'b6' and x'a7'. I would guess that at least two separate things happened to the file at different times. First, all the high-order bits were turned off. Later, a translation was done as if the data was presumed to be in code page 437, which converted x'14' and x'15' to x'b6' and x'a7', resulting in some values having the high-order bit set. Something else happened, either at the same time or not, that changed a few other characters to other values that have the high-order bit set, for which I have no explanation. 10 30447A00 2802 0800 5851 0Dz...(...XQ 20 46303130 38313510 310A4330 0C447740 F010815.1.C0.Dw@ ... A0 504B0102 780BB600 06000800 64552240 PK..x...dU@ B0 57463044 7A00 2802 08003401 WF0Dz...(.4. C0 0100 5851 ..XQ D0 46303130 38316500 30016973 7976 F01081e.0.isyp.. This x'69737970' is really (once high bit is added back) x'E9F3F9F0' ie 'Z390' ie something that pkzip for z/OS inserts. E0 00014000 00050002 1600 04022800 ..@...(. F0 06005C00 08000700 0501 00070006 ..\. 000100 4B00 05000740 004B 00064462 ..K@.@Db 000110 52707073 40404040 40404040 40404040 Rpps 000120 40404040 40404040 40400A @@. Does anyone have any idea what protocol (ftp, sftp, winscp, kermit, connect:direct, http) would affect data in this manner? I've never seen mangling like that before. Thanks. Paul. 00 594 01 698 02 740 03 488 04 749 05 697 06 536 07 526 08 545 09 854 0A 1060 0B 597 0C 641 0D 0 0E 608 0F 639 10 817 11 679 12 1044 13 641 14 0 15 0 16 621 17 533 18 554 19 611 1A 517 1B 555 1C 0 1D 612 1E 0 1F 639 20 731 21 592 22 607 23 549 24 1030 25 546 26 565 27 618 28 490 29 602 2A 436 2B 684 2C 629 2D 706 2E 589 2F 667 30 547 31 815 32 670 33 530 34 569 35 598 36 570 37 619 38 723 39 572 3A 508 3B 626 3C 676 3D 626 3E 615 3F 621 40 687 41 598 42 636 43 557 44 752 45 579 46 645 47 813 48 849 49 837 4A 617 4B 563 4C 587 4D 532 4E 618 4F 705 50 538 51 541 52 553 53 573 54 467 55 416 56 653 57 681 58 727 59 599 5A 560 5B 344 5C 611 5D 293 5E 663 5F 552 60 578 61 562 62 608 63 814 64 649 65 711 66 515 67 660 68 484 69 506 6A 528 6B 627 6C 773 6D 646 6E 627 6F 599 70 602 71 657 72 636 73 620 74 521 75 516 76 732 77 631 78 596 79 715 7A 551 7B 718 7C 621 7D 606 7E 630 7F 0 80 0 81 576 82 0 83 0 84 0 85 0 86 0 87 0 88 0 89 0 8A 0 8B 0 8C 0 8D 0 8E 0 8F 0 90 0 91 0 92 0 93 0 94 0 95 0 96 0 97 0 98 0 99 0 9A 0 9B 0 9C 361 9D 0 9E 0 9F 0 A0 0 A1 0 A2 0 A3 0 A4 0 A5 0 A6 0 A7 645 A8 0 A9 0 AA 0 AB 0 AC 0 AD 0 AE 0 AF 0 B0 0 B1 0 B2 0 B3 0 B4 0 B5 0 B6 527 B7 0 B8 0 B9 0 BA 0 BB 0 BC 0 BD 0 BE 0 BF 284 C0 0 C1 0 C2 0 C3 0 C4 0 C5 0 C6 0 C7 0 C8 0 C9 0 CA 0 CB 0 CC 0 CD 0 CE 0 CF 0 D0 0 D1 0 D2 0 D3 0 D4 0 D5 0 D6 0 D7 0 D8 0 D9 0 DA 0 DB 0 DC 0 DD 0 DE 0 DF 0 E0 249 E1 0 E2 0 E3 0 E4 0 E5 0 E6 0 E7 0 E8 0 E9 644 EA 0 EB 0 EC 0 ED 0 EE 0 EF 0 F0 0 F1 0 F2 718 F3 0 F4 0 F5 0 F6 0 F7 0 F8 0 F9 0 FA 0 FB 0 FC 0 FD 0 FE 0 FF 0 -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: corrupt zip files
Having just done this, I speak from experience Almost every file transfer product will transfer either text or binary. Most have a feature that will make a choice for you. I used the make a choice for me option and wound up scrambling the three files I had transmitted. I changed the format to binary and everything worked as it should. Ken Hume IBM PD Tools Client Advocate (720)396-7776 kph...@us.ibm.com On 5/5/2012 7:54 AM, Paul Edwards wrote: I have a zip file that appears to have been produced using pkzip for z/OS. However, it looks like it has been transmitted using some sort of text protocol, because the high bit has been stripped from most bytes, and some other bytes appear to have been translated. e.g. I think x'0a' in the input file has been mangled to x'b6' on the way. Does anyone know what software would do a translation like that? I believe these characters: 0A 1060 0D 0 12 1044 14 0 15 0 1C 0 1E 0 24 1030 7F 0 are being mapped to these: 81 576 9C 361 A7 645 B6 527 BF 284 E0 249 E9 644 F2 718 Except for x'0d' which I think is just being deleted. This belief comes from counting (see below) occurrences of the various bytes in a largish (80k) file. Here is a small file that shows the problem: 00 504B0304 B6000600 08006455 22405746 PKdU@WF That x'b6' above should be x'0a' I think (that's more normal). Apparently some protocol doesn't think that x'0a' will make it through, so translates it in advance. 10 30447A00 2802 0800 5851 0Dz...(...XQ 20 46303130 38313510 310A4330 0C447740 F010815.1.C0.Dw@ ... A0 504B0102 780BB600 06000800 64552240 PK..x...dU@ B0 57463044 7A00 2802 08003401 WF0Dz...(.4. C0 0100 5851 ..XQ D0 46303130 38316500 30016973 7976 F01081e.0.isyp.. This x'69737970' is really (once high bit is added back) x'E9F3F9F0' ie 'Z390' ie something that pkzip for z/OS inserts. E0 00014000 00050002 1600 04022800 ..@...(. F0 06005C00 08000700 0501 00070006 ..\. 000100 4B00 05000740 004B 00064462 ..K@.@Db 000110 52707073 40404040 40404040 40404040 Rpps 000120 40404040 40404040 40400A @@. Does anyone have any idea what protocol (ftp, sftp, winscp, kermit, connect:direct, http) would affect data in this manner? I've never seen mangling like that before. Thanks. Paul. 00 594 01 698 02 740 03 488 04 749 05 697 06 536 07 526 08 545 09 854 0A 1060 0B 597 0C 641 0D 0 0E 608 0F 639 10 817 11 679 12 1044 13 641 14 0 15 0 16 621 17 533 18 554 19 611 1A 517 1B 555 1C 0 1D 612 1E 0 1F 639 20 731 21 592 22 607 23 549 24 1030 25 546 26 565 27 618 28 490 29 602 2A 436 2B 684 2C 629 2D 706 2E 589 2F 667 30 547 31 815 32 670 33 530 34 569 35 598 36 570 37 619 38 723 39 572 3A 508 3B 626 3C 676 3D 626 3E 615 3F 621 40 687 41 598 42 636 43 557 44 752 45 579 46 645 47 813 48 849 49 837 4A 617 4B 563 4C 587 4D 532 4E 618 4F 705 50 538 51 541 52 553 53 573 54 467 55 416 56 653 57 681 58 727 59 599 5A 560 5B 344 5C 611 5D 293 5E 663 5F 552 60 578 61 562 62 608 63 814 64 649 65 711 66 515 67 660 68 484 69 506 6A 528 6B 627 6C 773 6D 646 6E 627 6F 599 70 602 71 657 72 636 73 620 74 521 75 516 76 732 77 631 78 596 79 715 7A 551 7B 718 7C 621 7D 606 7E 630 7F 0 80 0 81 576 82 0 83 0 84 0 85 0 86 0 87 0 88 0 89 0 8A 0 8B 0 8C 0 8D 0 8E 0 8F 0 90 0 91 0 92 0 93 0 94 0 95 0 96 0 97 0 98 0 99 0 9A 0 9B 0 9C 361 9D 0 9E 0 9F 0 A0 0 A1 0 A2 0 A3 0 A4 0 A5 0 A6 0 A7 645 A8 0 A9 0 AA 0 AB 0 AC 0 AD 0 AE 0 AF 0 B0 0 B1 0 B2 0 B3 0 B4 0 B5 0 B6 527 B7 0 B8 0 B9 0 BA 0 BB 0 BC 0 BD 0 BE 0 BF 284 C0 0 C1 0 C2 0 C3 0 C4 0 C5 0 C6 0 C7 0 C8 0 C9 0 CA 0 CB 0 CC 0 CD 0 CE 0 CF 0 D0 0 D1 0 D2 0 D3 0 D4 0 D5 0 D6 0 D7 0 D8 0 D9 0 DA 0 DB 0 DC 0 DD 0 DE 0 DF 0 E0 249 E1 0 E2 0 E3 0 E4 0 E5 0 E6 0 E7 0 E8 0 E9 644 EA 0 EB 0 EC 0 ED 0 EE 0 EF 0 F0 0 F1 0 F2 718 F3 0 F4 0 F5 0 F6 0 F7 0 F8 0 F9 0 FA 0 FB 0 FC 0 FD 0 FE 0 FF 0 -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
Re: corrupt zip files
On Sat, 5 May 2012 08:54:52 -0500, Paul Edwards wrote: I have a zip file that appears to have been produced using pkzip for z/OS. However, it looks like it has been transmitted using some sort of text protocol, because the high bit has been stripped from most bytes, and some other bytes appear to have been translated. e.g. I think x'0a' in the input file has been mangled to x'b6' on the way. Does anyone know what software would do a translation like that? ... Does anyone have any idea what protocol (ftp, sftp, winscp, kermit, connect:direct, http) would affect data in this manner? I've never seen mangling like that before. You've been lucky until now. Such historical questions rapidly become unproductive. More useful, what protocol was used? What alternatives are available? If your correspondent replies with a glassy stare, Protocol? I just sent the file, ask to talk to someone who knows what he's doing. This is among the reasons IBM support urges customers to use AMATERSE. (Wouldn't it be nice if AMATERSE reported a checksum in SYSPRINT? Likewise, wouldn't it be nice if common transfer protocols reported checksums at both sending and receiving ends?) -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@bama.ua.edu with the message: INFO IBM-MAIN
