Re: [EXTERNAL] Re: Checksum or hashing
On Wed, 21 Nov 2018 19:42:33 +, Dyck, Lionel B. (RavenTek) wrote: >Here is more information on the CBTTape.Org file 900 which may provide more >information for the OP and others: > >//***FILE 900 is a set of programs which run under either old MVS > > I thought I saw in another version, but not here, an option to include RDWs in the hash. You need something like that to preclude: ABCDEF ... hashing as identical to ABC DEF -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Checksum or hashing
As others have written, I think the use of SHA-1 within an enterprise as a data integrity check is fine, so long as it meets a client's own security standards. To go further, I think our (IBM's) use of SHA-1 for the same purpose for software downloads is likewise just fine. (One must now connect to IBM's download servers for z/OS products and PTFs using SSL, and physical delivery uses R/O DVDs.) All that said, some clients are starting to ask us to use a stronger hashing algorithm. Most of them understand and agree that SHA-1 is just fine for a data integrity check. But, their security departments believe that disallowing SHA-1 is a simple overall rule that covers the security-related uses of SHA-1 even if it imposes changes on the non-security-related uses of SHA-1. It's difficult to argue with simplification logic, I must admit. We will probably have to use something stronger for software delivery, eventually. We will probably need to continue to support SHA-1 for compatibility's sake when we do. CM Poncelet wrote: FWIW SHA1 hashing is *not* secure: you should use SHA2. No idea whether there is a z/OS utility to do that: I use PGP. HTH. Chris Poncelet (retired sysprog) On 20/11/2018 13:36, Sankaranarayanan, Vignesh wrote: Hello again List! Is there any utility for z/OS that lets us create SHA1 or MD5 or some such hash/fingerprint of a dataset or USS file. The use case is to compare these hashes at source (z/OS) and destination (linux) after transferring some sizable datasets. -- John Eells IBM Poughkeepsie ee...@us.ibm.com -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Checksum or hashing
On Wed, 21 Nov 2018 04:21:10 -0600, Jantje. wrote:
>Attached is a little Java program I did years ago for exactly that purpose. It
>uses the excellent jzos for reading just about any type of dataset or file.
>
So far for attachments...
Here is the code:
package com.tme.mainframemove;
import com.ibm.jzos.FileFactory;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
public class MD5CheckSum {
public static byteíù createChecksum(String filename) throws IOException,
NoSuchAlgorithmException {
InputStream inputStream = FileFactory.newInputStream(filename);
//InputStream inputStream = new FileInputStream(filename);
byteíù buffer = new byteí1024ù;
MessageDigest complete = MessageDigest.getInstance("MD5");
int numRead;
do {
numRead = inputStream.read(buffer);
if (numRead > 0) {
complete.update(buffer, 0, numRead);
}
} while (numRead != -1);
inputStream.close();
return complete.digest();
}
public static String getMD5Checksum(String filename) throws IOException,
NoSuchAlgorithmException {
byteíù b = createChecksum(filename);
String result = "";
for (int i = 0; i < b.length; i++) {
result +=
Integer.toString((bíiù & 0xff) + 0x100, 16).substring(1);
}
return result;
}
public static void getMD5CheckSums(String parmFileName) throws IOException,
NoSuchAlgorithmException {
BufferedReader fileNames = FileFactory.newBufferedReader(parmFileName);
try {
String datasetName;
while ((datasetName = fileNames.readLine()) != null) {
System.out.print(String.format("%1#-50s", datasetName));
System.out.println(getMD5Checksum(datasetName));
}
} finally {
fileNames.close();
}
}
public static void main(String argsíù) {
try {
getMD5CheckSums(argsí0ù);
} catch (Exception e) {
e.printStackTrace();
}
}
}
--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Checksum or hashing
Gil wrote: >MitM? Key word was "accidental". Don't think MitM qualifies, does it? -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: [EXTERNAL] Re: Checksum or hashing
Here is more information on the CBTTape.Org file 900 which may provide more information for the OP and others: //***FILE 900 is a set of programs which run under either old MVS //* or z/OS to calculate MD5 checksums. Please see the //* member called @FILEMD5, which contains descriptions //* of all the MD5** pds members found in this file. //* //* support: sbgo...@cbttape.org or //* sbgo...@att.net //* //* SHA-1 and SHA-2 support added, May 2016. //* //* SHA-384 and SHA-512 support added, Aug 2016. //* //* Fixed for 8-character TSO prefixes in z/OS 2.3. //* //* SHA1SUM - Assembler source for SHA-1 generating command //* (Fix. DS1LSTAR is always 0 for a PDSE, //*so always read the dataset even if //*DS1LSTAR is 0.) //* //* SHA2SUM - Assembler source for SHA-2 generating command //* (Fix. DS1LSTAR is always 0 for a PDSE, //*so always read the dataset even if //*DS1LSTAR is 0.) //* //* SHA3SUM - Assembler source for SHA-384 generating command //* (Fix. DS1LSTAR is always 0 for a PDSE, //*so always read the dataset even if //*DS1LSTAR is 0.) //* //* SHA5SUM - Assembler source for SHA-512 generating command //* (Fix. DS1LSTAR is always 0 for a PDSE, //*so always read the dataset even if //*DS1LSTAR is 0.) //* //* MD5 - Assembler source for MD5 subprogram, which //* is a modification of the MD5 program for //* REXX. This must be linked with the MD5SUM //* TSO command. //* //* MD5A - Assembler source for same MD5 subprogram, //* with inline macros changed for IFOX00 //* assembler. Intended for MVS370 systems. //* //* MD5R - Assembler source for same MD5 subprogram, //* with 8 STCM instructions replaced with 2 //* STRV. STRV is comparatively new, from circa //* 2002. STRV is like ST but the bytes are //* stored in reverse order, as in hex 12345678 //* being stored as hex 78563412. The MVS 3.8 //* assembler IFOX00 does not support the STRV //* op code. //* //* MD5COB$ - JCL to compile and run an Enterprise Cobol //* program that calls the MD5 subroutine. //* //* MD5DATA - JCL to create test data for MD5SUM, the same //* test data used in MD5REXX //* //* MD5FORT$ - JCL to compile and run a Fortran G program //* that calls the MD5 subroutine. //* //* MD5PLI$ - JCL to compile and run an Enterprise PL/I //* program that calls the MD5 subroutine. //* //* MD5REXX - A copy of the source code for the MD5 for //* REXX by Leland Lucius, which was the base //* from which member MD5 was created. //* //* MD5SUM - assembler source for the 31-bit MD5SUM TSO //* command. It calls the MD5 subprogram. //* (Fix. DS1LSTAR is always 0
Re: Checksum or hashing
Agreed RE: "SHA-1 not secure". The security concerns with the use of SHA-1 relate to long-term hashes of security credentials like X.509 certificates. If you just want a decent checksum and aren't worried about the *very* unlikely possibility of the same hash for two different documents, then there's no issue.There's always that risk with any hash, but SHA-1 has some crypto weaknesses in being able to *engineer* such a collision. Here's an amusing article from last year. The headline is utterly misleading, and doesn't mention that it "It took nine quintillion SHA-1 computations" :-) https://www.computerworld.com/article/3173616/security/the-sha1-hash-function-is-now-completely-unsafe.html What's really sad is how many "security experts" are now running around saying that SHA-1 is unsafe for things like checksums on encrypted ssh packets. They don't know / care anything about the context, or the difference between hmac-sha1 and basic sha-1. ICSF OWH is one way to do it, but for performance critical code you can do it cheaper if you drop down and just use the CPACF instructions for SHA1, SHA2, etc. With z14s, CPACF really screams and it turns out that SHA-2 is only a little slower than SHA-1.(ICSF uses CPACF for these too, but can add overhead that is significant to some applications). ICSF OWH can also require SAF/RACF permissions, which is silly and annoying. Kirk Wolf Dovetailed Technologies http://dovetail.com PS> you can use the z/OSMF Files and Datasets REST API to retrieve a file or dataset and also get a hash. On the response is an "ETag" header which has a hash.It is opaque as far as specification, but if you look at it you can probably guess what it is. It wouldn't be good to assume that it is anything other than an opaque "checksum" string. The purpose of this in the API is pretty cool: you can request the same file again and pass it the previous ETag and you will get a response that indicates that the file has not been changed, without having to transmit it to you again. Note: This may be of little use if your purpose is to have some other software calculate and match to it, since you really don't know it is calculated. On Tue, Nov 20, 2018 at 9:06 PM Phil Smith III wrote: > CM Poncelet wrote: > > >FWIW SHA1 hashing is *not* secure: you should use SHA2. No idea whether > > >there is a z/OS utility to do that: I use PGP. HTH. > > > > FSVO "not secure". If you're just creating a fingerprint to detect > accidental damage, it's surely good enough. If you're protecting > your bank account, maybe not so much. Given that the original request was > "to compare these hashes at source (z/OS) and destination > (linux) after transferring some sizable datasets", I suspect it'll be fine. > > > > In any case, ICSF can do it: > > > https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.csfb400/owh.htm > > > > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN > -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Checksum or hashing
On Tue, 20 Nov 2018 13:36:56 +, Sankaranarayanan, Vignesh wrote: >Is there any utility for z/OS that lets us create SHA1 or MD5 or some such >hash/fingerprint of a dataset or USS file. >The use case is to compare these hashes at source (z/OS) and destination >(linux) after transferring some sizable datasets. Not sure if there is a utility… Attached is a little Java program I did years ago for exactly that purpose. It uses the excellent jzos for reading just about any type of dataset or file. Should be easy enough to amend it to use some stronger hash than the MD5 I used way back when... Cheers, Jantje. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: [EXTERNAL] Re: Checksum or hashing
A genius, gentleman, and a scholar. Thank you so much! – Vignesh Mainframe Infrastructure -Original Message- From: IBM Mainframe Discussion List On Behalf Of Dyck, Lionel B. (RavenTek) Sent: 20 November 2018 16:30 To: IBM-MAIN@LISTSERV.UA.EDU Subject: Re: [EXTERNAL] Re: Checksum or hashing I didn't see the original posting but if you're looking for tools to do a checksum on a z/OS data set check out file 900 on the CBTTAPE.ORG site: From the file: //***FILE 900 is a set of programs which run under either old MVS //* or z/OS to calculate MD5 checksums. Please see the //* member called @FILEMD5, which contains descriptions //* of all the MD5** pds members found in this file. //* //* support: sbgo...@cbttape.org or //* sbgo...@att.net //* //* SHA-1 and SHA-2 support added, May 2016. //* //* SHA-384 and SHA-512 support added, Aug 2016. -- Lionel B. Dyck (Contractor) < Mainframe Systems Programmer – RavenTek Solution Partners -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN MARKSANDSPENCER.COM Unless otherwise stated above: Marks and Spencer plc Registered Office: Waterside House 35 North Wharf Road London W2 1NW Registered No. 214436 in England and Wales. Telephone (020) 7935 4422 Facsimile (020) 7487 2670 www.marksandspencer.com Please note that electronic mail may be monitored. This e-mail is confidential. If you received it by mistake, please let us know and then delete it from your system; you should not copy, disclose, or distribute its contents to anyone nor act in reliance on this e-mail, as this is prohibited and may be unlawful. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Checksum or hashing
On Tue, 20 Nov 2018 22:06:10 -0500, Phil Smith III wrote: > >FSVO "not secure". If you're just creating a fingerprint to detect accidental >damage, it's surely good enough. If you're protecting >your bank account, maybe not so much. Given that the original request was "to >compare these hashes at source (z/OS) and destination >(linux) after transferring some sizable datasets", I suspect it'll be fine. > MitM? >In any case, ICSF can do it: >https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.csfb400/owh.htm > I notice that CBTTape 900 has an option to include RDWs in the hash. This might be necessary to detect splitting or joining of records. It's simplest if the OP's data can be treated as a simple binary stream. If text, ASCII<->EBCDIC (here we go again) must be verified, and inserted newlines handled. -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Checksum or hashing
CM Poncelet wrote: >FWIW SHA1 hashing is *not* secure: you should use SHA2. No idea whether >there is a z/OS utility to do that: I use PGP. HTH. FSVO "not secure". If you're just creating a fingerprint to detect accidental damage, it's surely good enough. If you're protecting your bank account, maybe not so much. Given that the original request was "to compare these hashes at source (z/OS) and destination (linux) after transferring some sizable datasets", I suspect it'll be fine. In any case, ICSF can do it: https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.csfb400/owh.htm -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Checksum or hashing
FWIW SHA1 hashing is *not* secure: you should use SHA2. No idea whether there is a z/OS utility to do that: I use PGP. HTH. Chris Poncelet (retired sysprog) On 20/11/2018 13:36, Sankaranarayanan, Vignesh wrote: > Hello again List! > > Is there any utility for z/OS that lets us create SHA1 or MD5 or some such > hash/fingerprint of a dataset or USS file. > The use case is to compare these hashes at source (z/OS) and destination > (linux) after transferring some sizable datasets. > > Thanks in advance! > > - Vignesh > Mainframe Infrastructure > > > MARKSANDSPENCER.COM > > Unless otherwise stated above: > Marks and Spencer plc > Registered Office: > Waterside House > 35 North Wharf Road > London > W2 1NW > > Registered No. 214436 in England and Wales. > > Telephone (020) 7935 4422 > Facsimile (020) 7487 2670 > > www.marksandspencer.com > > Please note that electronic mail may be monitored. > > This e-mail is confidential. If you received it by mistake, please let us > know and then delete it from your system; you should not copy, disclose, or > distribute its contents to anyone nor act in reliance on this e-mail, as this > is prohibited and may be unlawful. > > -- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN > . > -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: [EXTERNAL] Re: Checksum or hashing
I didn't see the original posting but if you're looking for tools to do a checksum on a z/OS data set check out file 900 on the CBTTAPE.ORG site: From the file: //***FILE 900 is a set of programs which run under either old MVS //* or z/OS to calculate MD5 checksums. Please see the //* member called @FILEMD5, which contains descriptions //* of all the MD5** pds members found in this file. //* //* support: sbgo...@cbttape.org or //* sbgo...@att.net //* //* SHA-1 and SHA-2 support added, May 2016. //* //* SHA-384 and SHA-512 support added, Aug 2016. -- Lionel B. Dyck (Contractor) < Mainframe Systems Programmer – RavenTek Solution Partners -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Checksum or hashing
On Tue, 20 Nov 2018 09:09:00 -0500, John Eells wrote: >There is a Java class you can use (if you have Java) and an ICSF service >you can use (requires ICSF setup). > o SAMPLIB has Rexx samples that may be useful o SMP/E uses a Java interface. Turning on tracing (in the CLIENT data set, IIRC) may show how it does it. o MD5 is deprecated as too weak for security. It's probably adequate for finding accidental transmission errors, as is good old /bin/cksum. o Linux has sha1sum, sha224sum(1), sha256sum(1), sha384sum(1), and sha512sum(1). o You can extract a (weak) checksum in place with z/OS: cp -B "//'data.set.name'" /dev/fd/2 | cksum >Sankaranarayanan, Vignesh wrote: >> >> Is there any utility for z/OS that lets us create SHA1 or MD5 or some such >> hash/fingerprint of a dataset or USS file. >> The use case is to compare these hashes at source (z/OS) and destination >> (linux) after transferring some sizable datasets. -- gil -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Checksum or hashing
There is a Java class you can use (if you have Java) and an ICSF service you can use (requires ICSF setup). Sankaranarayanan, Vignesh wrote: Hello again List! Is there any utility for z/OS that lets us create SHA1 or MD5 or some such hash/fingerprint of a dataset or USS file. The use case is to compare these hashes at source (z/OS) and destination (linux) after transferring some sizable datasets. Thanks in advance! -- John Eells IBM Poughkeepsie ee...@us.ibm.com -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Checksum or hashing
Hello again List! Is there any utility for z/OS that lets us create SHA1 or MD5 or some such hash/fingerprint of a dataset or USS file. The use case is to compare these hashes at source (z/OS) and destination (linux) after transferring some sizable datasets. Thanks in advance! - Vignesh Mainframe Infrastructure MARKSANDSPENCER.COM Unless otherwise stated above: Marks and Spencer plc Registered Office: Waterside House 35 North Wharf Road London W2 1NW Registered No. 214436 in England and Wales. Telephone (020) 7935 4422 Facsimile (020) 7487 2670 www.marksandspencer.com Please note that electronic mail may be monitored. This e-mail is confidential. If you received it by mistake, please let us know and then delete it from your system; you should not copy, disclose, or distribute its contents to anyone nor act in reliance on this e-mail, as this is prohibited and may be unlawful. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
