I've seen a lot of syntax errors in AI-generated Rexx as well. You need to provide rules, and it doesn't always follow them. I'm planning a lint for Rexx specifically for claude to use internally.
-- Shmuel (Seymour J.) Metz http://mason.gmu.edu/~smetz3 עַם יִשְׂרָאֵל חַי נֵ֣צַח יִשְׂרָאֵ֔ל לֹ֥א יְשַׁקֵּ֖ר ________________________________________ From: IBM Mainframe Discussion List <[email protected]> on behalf of Brian Westerman <[email protected]> Sent: Wednesday, June 24, 2026 12:34 AM To: [email protected] <[email protected]> Subject: Re: AI on the mainframe External Message: Use Caution I have found that when it comes to writing assembler code, that it generates more errors than useful code. However if you have a small routine or sub-routine that you would like to know a "better" way to do something, then it isn't bad at all. It may not actually assemble, but the concepts are fairly solid. Wherever it gets it's base for generating the code appears to be deep into what the actual instructions do, but it has a real problem trying to put things together. For instance, if you wanted to create a routine to manipulate a field, that is something that it can build code for, but if you wanted to do something with that field after it was manipulated, it gets lost very quickly. I'm not worried about it putting me out of the assembler coding business any time soon. :) Brian On Wed, 24 Jun 2026 10:27:09 +1000, Andrew Rowley <[email protected]> wrote: >On 24/06/2026 1:34 am, Phil Smith III wrote: >> Yes, it does depend on what you mean by "AI", and I'm afraid I was unclear. >> >> I really meant two things: >> 1) "Is anyone actually exploiting Telum?" and >> 2) "Is anyone doing any LLM stuff?" > >I think current LLMs are beyond what can reasonably run on the >mainframe. Abilities and expectations require too much computing power. > >But I have been experimenting with LLMs for mainframe related stuff. > >- I tried to get it to write some assembler. It generated about 150 >lines of convincing assembler, but that was about 130 lines more than I >thought I needed and didn't actually assemble so I ended up writing it >by hand. However, it was quite good at answering questions about HOW to >do things (not always correct! but often opening new lines of thought), >finding examples and explaining what examples were doing, so from that >point of view it was useful. > >- It is very good at analyzing Java and C code and finding problems or >pointing out potential problems. > >- It's good at Java so if I point it at my Java SMF reporting samples >and say "write me a new SMF report showing ..." it generally does a good job > >- It occasionally does annoying things like using em dash in comments, >which meant that git checkouts on z/OS always showed as modified due to >UTF8-EBCDIC transalation problem > >- More complex code definitely requires someone to review it. A couple >of common problems: > >1) Swallowing errors. It writes code to handle a problem and continue >when the problem should result in the program failing. > >2) Every so often code seems to reach critical mass and explode. It >generates hundreds of lines of code for simple functions. You have to go >in and prune and simplify, otherwise it ends up unmaintainable. > >I'm sure that it could do better if I chose different models or used >better prompts and rules. Prompts often feel a bit like a >non-deterministic programming language. > >Overall LLMs are incredible, but they seem like a productivity tool >rather than something that should work unsupervised. > >-- >Andrew Rowley >Black Hill Software > >---------------------------------------------------------------------- >For IBM-MAIN subscribe / signoff / archive access instructions, >send email to [email protected] with the message: INFO IBM-MAIN ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO IBM-MAIN ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO IBM-MAIN
