Re: Problem with amtape "hanging" when forked from Java
On Tue, Mar 08, 2022 at 18:27:47 -0500, Robert Heller wrote: > For some unfathomably reason amtape "hangs" when forked from a Java program. > > I've written a Java program that goes through vaulted tapes and forks amtape > (using Runtime.getRuntime().exec(()), and when a non-existant tape label is > asked for, amtape "hangs". I cannot figure out why or how to get amtape to > just exit with an error (which I can then handle). Robert, did you ever resolve the problem you were having with amtape hanging? Nathan Nathan Stratton Treadway - [email protected] - Mid-Atlantic region Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239 Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
RE: Problem with amtape "hanging" when forked from Java
We've had that problem here as well with amtape. It is hard to reproduce. If it helps... I've recreated only once *and* it was in perl's DESTRUCTORs that it ran. At least that's what I thought I could see via gdb. If someone can recreate this consistently I'd really be glad to try and fix it. As it is ... we will work soon on a solution to the tape locks such that a new lock can detect that all old ones are stale or useless and remove them. That is our current planned workaround for amtape. -- CH > -Original Message- > From: [email protected] [email protected]> On Behalf Of Exuvo > Sent: Thursday, March 10, 2022 8:22 PM > To: Robert Heller > Cc: Nathan Stratton Treadway ; ? amanda users > > Subject: Re: Problem with amtape "hanging" when forked from Java > > Also don't use stream.readAllBytes, it will block until the stream end is > reached > ie the process exits and closes the stream. Which completely defeats the > purpose of the waitFor. > > I suggest using something like: > > Process p = Runtime.getRuntime().exec(cmd); long start = > System.currentTimeMillis(); InputStream out = p.getInputStream(); InputStream > err = p.getErrorStream(); > > while (p.isAlive() && System.currentTimeMillis() - start < 60 * 1000) { >if(out != null) { > int avail = out.available(); > out.skip(avail); // or a read if you want the data >} >if(err != null) { > int avail = err.available(); > err.skip(avail); // or a read if you want the data >} >Thread.sleep(1); > } > > Anton "exuvo" Olsson > [email protected] > > On 2022-03-11 04:05, Exuvo wrote: > > An educated guess from me would be that the particular error message you > are waiting for is not actually written to stderror but normal output stream > which you never read. Specifically this part of java.lang.Process > documentation > "Because some native platforms only provide limited buffer size for standard > input and output streams, failure to promptly write the input stream or read > the > output stream of the subprocess may cause the subprocess to block, or even > deadlock. ". > > > > Anton "exuvo" Olsson > >[email protected] > > > > On 2022-03-10 15:55, Robert Heller wrote: > >> At Wed, 9 Mar 2022 22:50:29 -0500 (EST) Robert Heller > wrote: > >> > >>> At Wed, 9 Mar 2022 19:11:40 -0500 Nathan Stratton Treadway > wrote: > >>> > >>>> On Tue, Mar 08, 2022 at 18:27:47 -0500, Robert Heller wrote: > >>>>> I've written a Java program that goes through vaulted tapes and > >>>>> forks amtape (using Runtime.getRuntime().exec(()), and when a > >>>>> non-existant tape label is asked for, amtape "hangs". I cannot > >>>>> figure out why or how to get amtape to just exit with an error (which I > can then handle). > >>>> Does it still hang if you pass an argument "-ointeractivity=''" > >>>> when you exec amtape? > >>> Did not try that. > >> OK, it still hangs. > >> > >> Here is the Java fragment: > >> > >> public class FlushOldVaults extends BackupVault { > >> private static final String AMTAPE = "/usr/sbin/amtape"; > >> private static final String AMTAPEOPT1 = "-otpchanger=vault_changer"; > >> private static final String AMTAPEOPT2 = "-ointeractivity="; > >> private String amtape(String tape) throws Exception { > >> String cmd[] = new String[6]; > >> cmd[0] = AMTAPE; > >> cmd[1] = AMTAPEOPT1; > >> cmd[2] = AMTAPEOPT2; > >> cmd[3] = configuration.AMCONFIG(); > >> cmd[4] = "label"; > >> cmd[5] = tape; > >> System.err.printf("*** FlushOldVaults.amtape(): tape is > >> '%s'\n",tape); > >> Process p = Runtime.getRuntime().exec(cmd); > >> InputStream err = p.getErrorStream(); > >> if (err != null) err.readAllBytes(); > >> if (!p.waitFor(60L, java.util.concurrent.TimeUnit.SECONDS)) { > >> System.err.printf("*** FlushOldVaults.amtape(): process > >> timeout\n"); > >> String kill[] = new String[2]; > >> kill[0] = "/bin/kill"; > >> Long j = new Long(p.pid()); > >> kill[1] = j.toString(); > >> Proces
Re: Problem with amtape "hanging" when forked from Java
Also don't use stream.readAllBytes, it will block until the stream end is
reached ie the process exits and closes the stream. Which completely defeats
the purpose of the waitFor.
I suggest using something like:
Process p = Runtime.getRuntime().exec(cmd);
long start = System.currentTimeMillis();
InputStream out = p.getInputStream();
InputStream err = p.getErrorStream();
while (p.isAlive() && System.currentTimeMillis() - start < 60 * 1000) {
if(out != null) {
int avail = out.available();
out.skip(avail); // or a read if you want the data
}
if(err != null) {
int avail = err.available();
err.skip(avail); // or a read if you want the data
}
Thread.sleep(1);
}
Anton "exuvo" Olsson
[email protected]
On 2022-03-11 04:05, Exuvo wrote:
An educated guess from me would be that the particular error message you are waiting for
is not actually written to stderror but normal output stream which you never read.
Specifically this part of java.lang.Process documentation "Because some native
platforms only provide limited buffer size for standard input and output streams, failure
to promptly write the input stream or read the output stream of the subprocess may cause
the subprocess to block, or even deadlock. ".
Anton "exuvo" Olsson
[email protected]
On 2022-03-10 15:55, Robert Heller wrote:
At Wed, 9 Mar 2022 22:50:29 -0500 (EST) Robert Heller
wrote:
At Wed, 9 Mar 2022 19:11:40 -0500 Nathan Stratton Treadway
wrote:
On Tue, Mar 08, 2022 at 18:27:47 -0500, Robert Heller wrote:
I've written a Java program that goes through vaulted tapes and forks amtape
(using Runtime.getRuntime().exec(()), and when a non-existant tape label is
asked for, amtape "hangs". I cannot figure out why or how to get amtape to
just exit with an error (which I can then handle).
Does it still hang if you pass an argument "-ointeractivity=''" when you
exec amtape?
Did not try that.
OK, it still hangs.
Here is the Java fragment:
public class FlushOldVaults extends BackupVault {
private static final String AMTAPE = "/usr/sbin/amtape";
private static final String AMTAPEOPT1 = "-otpchanger=vault_changer";
private static final String AMTAPEOPT2 = "-ointeractivity=";
private String amtape(String tape) throws Exception {
String cmd[] = new String[6];
cmd[0] = AMTAPE;
cmd[1] = AMTAPEOPT1;
cmd[2] = AMTAPEOPT2;
cmd[3] = configuration.AMCONFIG();
cmd[4] = "label";
cmd[5] = tape;
System.err.printf("*** FlushOldVaults.amtape(): tape is '%s'\n",tape);
Process p = Runtime.getRuntime().exec(cmd);
InputStream err = p.getErrorStream();
if (err != null) err.readAllBytes();
if (!p.waitFor(60L, java.util.concurrent.TimeUnit.SECONDS)) {
System.err.printf("*** FlushOldVaults.amtape(): process
timeout\n");
String kill[] = new String[2];
kill[0] = "/bin/kill";
Long j = new Long(p.pid());
kill[1] = j.toString();
Process killproc = Runtime.getRuntime().exec(kill);
killproc.waitFor();
}
int status = p.waitFor();
System.err.printf("*** FlushOldVaults.amtape(): status = %d\n",status);
if (status != 0) throw new Exception(AMTAPE+" "+AMTAPEOPT1+" "+AMTAPEOPT2+"
"+configuration.AMCONFIG()+" label "+tape+": failed");
String tpchanger = amgetconf("CHANGER:vault_changer:tpchanger");
int colon = tpchanger.indexOf(':');
if (colon < 0) {
return tpchanger+"/data/";
} else {
return tpchanger.substring(colon+1)+"/data/";
}
}
}
If it does still hang, what do "lsof -p" and "strace -p" show on the
amtape process while it's stuck?
Nathan
Nathan Stratton Treadway - [email protected] - Mid-Atlantic region
Ray Ontko & Co. - Software consulting services - http://www.ontko.com/
GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239
Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
Re: Problem with amtape "hanging" when forked from Java
An educated guess from me would be that the particular error message you are waiting for is not actually written to stderror but normal output stream which you never read. Specifically this part of java.lang.Process documentation "Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, or even deadlock. ". Anton "exuvo" Olsson [email protected] On 2022-03-10 15:55, Robert Heller wrote: At Wed, 9 Mar 2022 22:50:29 -0500 (EST) Robert Heller wrote: At Wed, 9 Mar 2022 19:11:40 -0500 Nathan Stratton Treadway wrote: On Tue, Mar 08, 2022 at 18:27:47 -0500, Robert Heller wrote: I've written a Java program that goes through vaulted tapes and forks amtape (using Runtime.getRuntime().exec(()), and when a non-existant tape label is asked for, amtape "hangs". I cannot figure out why or how to get amtape to just exit with an error (which I can then handle). Does it still hang if you pass an argument "-ointeractivity=''" when you exec amtape? Did not try that. OK, it still hangs. Here is the Java fragment: public class FlushOldVaults extends BackupVault { private static final String AMTAPE = "/usr/sbin/amtape"; private static final String AMTAPEOPT1 = "-otpchanger=vault_changer"; private static final String AMTAPEOPT2 = "-ointeractivity="; private String amtape(String tape) throws Exception { String cmd[] = new String[6]; cmd[0] = AMTAPE; cmd[1] = AMTAPEOPT1; cmd[2] = AMTAPEOPT2; cmd[3] = configuration.AMCONFIG(); cmd[4] = "label"; cmd[5] = tape; System.err.printf("*** FlushOldVaults.amtape(): tape is '%s'\n",tape); Process p = Runtime.getRuntime().exec(cmd); InputStream err = p.getErrorStream(); if (err != null) err.readAllBytes(); if (!p.waitFor(60L, java.util.concurrent.TimeUnit.SECONDS)) { System.err.printf("*** FlushOldVaults.amtape(): process timeout\n"); String kill[] = new String[2]; kill[0] = "/bin/kill"; Long j = new Long(p.pid()); kill[1] = j.toString(); Process killproc = Runtime.getRuntime().exec(kill); killproc.waitFor(); } int status = p.waitFor(); System.err.printf("*** FlushOldVaults.amtape(): status = %d\n",status); if (status != 0) throw new Exception(AMTAPE+" "+AMTAPEOPT1+" "+AMTAPEOPT2+" "+configuration.AMCONFIG()+" label "+tape+": failed"); String tpchanger = amgetconf("CHANGER:vault_changer:tpchanger"); int colon = tpchanger.indexOf(':'); if (colon < 0) { return tpchanger+"/data/"; } else { return tpchanger.substring(colon+1)+"/data/"; } } } If it does still hang, what do "lsof -p" and "strace -p" show on the amtape process while it's stuck? Nathan Nathan Stratton Treadway - [email protected] - Mid-Atlantic region Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239 Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
Re: Problem with amtape "hanging" when forked from Java
On Thu, Mar 10, 2022 at 17:08:36 -0500, Robert Heller wrote: > (I have no interactivity configuration in any of my > configurations files, so it is presumably defaulting to being empty.) (See below...) > Here is the diff: > > *** amtape-java.debug 2022-03-10 16:14:52.556321620 -0500 > --- amtape-shell.debug2022-03-10 16:11:33.357521853 -0500 > *** > *** 1,87 > ! Thu Mar 10 16:13:15.068034312 2022: pid 19101: thd-0x55f1f8259c00: amtape: > pid 19101 ruid 34 euid 34 version 3.5.1: start at Thu Mar 10 16:13:15 2022 > ! Thu Mar 10 16:13:15.068101557 2022: pid 19101: thd-0x55f1f8259c00: amtape: > Arguments: -otpchanger=vault_changer -ointeractivity= deepsoft-normal label > examplevault > ! Thu Mar 10 16:13:15.068391863 2022: pid 19101: thd-0x55f1f8259c00: amtape: > config_overrides: tpchanger vault_changer > ! Thu Mar 10 16:13:15.068403765 2022: pid 19101: thd-0x55f1f8259c00: amtape: > config_overrides: interactivity > ! Thu Mar 10 16:13:15.068513641 2022: pid 19101: thd-0x55f1f8259c00: amtape: > reading config file /etc/amanda/deepsoft-normal/amanda.conf > ! Thu Mar 10 16:13:15.068536891 2022: pid 19101: thd-0x55f1f8259c00: amtape: > reading config file /etc/amanda/deepsoft-common/amanda.conf I had in mind your paging through the two files in separate windows side-by-side, eyeballing for differences in program activity. For "diff" to be much use you really need to strip off the front part of each line (datetime, pid, and thd), using something like: $ sed -e's/.*00: amtape:/amtape:/' amtape-java.debug > amtape-java.debug_clean $ sed -e's/.*00: amtape:/amtape:/' amtape_shell.debug > amtape_shell.debug_clean (and then running the diff on the _clean versions). However, looking through the listing you posted, I did notice something: > --- 1,104 > ! Thu Mar 10 16:09:04.258210052 2022: pid 18689: thd-0x55c92c51d000: amtape: > pid 18689 ruid 34 euid 34 version 3.5.1: start at Thu Mar 10 16:09:04 2022 > ! Thu Mar 10 16:09:04.258287566 2022: pid 18689: thd-0x55c92c51d000: amtape: > Arguments: -otpchanger=vault_changer deepsoft-normal label examplevault > ! Thu Mar 10 16:09:04.258592995 2022: pid 18689: thd-0x55c92c51d000: amtape: > config_overrides: tpchanger vault_changer > ! Thu Mar 10 16:09:04.258713649 2022: pid 18689: thd-0x55c92c51d000: amtape: > reading config file /etc/amanda/deepsoft-normal/amanda.conf > ! Thu Mar 10 16:09:04.258739621 2022: pid 18689: thd-0x55c92c51d000: amtape: > reading config file /etc/amanda/deepsoft-common/amanda.conf > ! Thu Mar 10 16:09:04.312355119 2022: pid 18689: thd-0x55c92c51d000: amtape: > pid 18689 ruid 34 euid 34 version 3.5.1: rename at Thu Mar 10 16:09:04 2022 > ! Thu Mar 10 16:09:04.322326242 2022: pid 18689: thd-0x55c92c51d000: amtape: > Disabling Amanda::Interactivity::stdin because STDIN is not readable So it does appear that some Interactivity based on the stdin.pm module was used by that particular invocation of amtape I'm still pretty confused about the exact behavior you are seeing (i.e. why it's hanging in the Java context, and why the attempt to pass in an empty -ointeractivity option doesn't seem to make any difference, etc.)... ... but since the stdin plugin is deprecated, it seems worth investigating where that's getting configured in the first place... I see from the above messages that amanda is reading both deepsoft-normal/amanda.conf and deepsoft-common/amanda.conf . (If it's not clear from those files where the default interactivity is being set, it might help for you to post the output of "amadmin deepsoft-normal config"...) Nathan Nathan Stratton Treadway - [email protected] - Mid-Atlantic region Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239 Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
Re: Problem with amtape "hanging" when forked from Java
At Thu, 10 Mar 2022 15:07:14 -0500 Nathan Stratton Treadway wrote: > > On Thu, Mar 10, 2022 at 14:03:07 -0500, Robert Heller wrote: > > It prints an error message and returns an error status: > > > > backup@newserver:~$ amtape -otpchanger=vault_changer -ointeractivity='' > > wendellfreelibrary label wendellfreelibrary-vault-030 > > ERROR: Source Volume 'wendellfreelibrary-vault-030' not found > > > > (and does not hang) > > Hmmm. > > What happens from the command line if you leave off the -ointeractivity > parameter? It works just fine. (I have no interactivity configuration in any of my configurations files, so it is presumably defaulting to being empty.) > > The strange thing is that the strace does seem to show a repeated > looping over all the vtape slots, presumably continually searching for > the specfied label but I didn't notice any activity that is > obviously interactivity-related in between the loops. So it's not clear > why amtape is looping within the Java context but immeidately > terminating with an error message in the shell/tclsh contexts. > > > > I'm *guessing* that the Java Process created by exec() wants to deal with > > the > > printout, but can't. > > > > In the strace you posted, amtape appeared to be looping through the > slots multiple times without attempting to write that "ERROR" message > anywhere, so off hand I would guess amtape is getting put into a > different search mode, rather than a problem with the Java side not > being able to accept the output. I'm not sure what would trigger that > different mode, though. > > I suppose my next suggestion would be to compare carefully the amtape > .debug files between a run from the command line and a run from within > Java -- any difference between the two could be a hint > Here is the diff: *** amtape-java.debug 2022-03-10 16:14:52.556321620 -0500 --- amtape-shell.debug 2022-03-10 16:11:33.357521853 -0500 *** *** 1,87 ! Thu Mar 10 16:13:15.068034312 2022: pid 19101: thd-0x55f1f8259c00: amtape: pid 19101 ruid 34 euid 34 version 3.5.1: start at Thu Mar 10 16:13:15 2022 ! Thu Mar 10 16:13:15.068101557 2022: pid 19101: thd-0x55f1f8259c00: amtape: Arguments: -otpchanger=vault_changer -ointeractivity= deepsoft-normal label examplevault ! Thu Mar 10 16:13:15.068391863 2022: pid 19101: thd-0x55f1f8259c00: amtape: config_overrides: tpchanger vault_changer ! Thu Mar 10 16:13:15.068403765 2022: pid 19101: thd-0x55f1f8259c00: amtape: config_overrides: interactivity ! Thu Mar 10 16:13:15.068513641 2022: pid 19101: thd-0x55f1f8259c00: amtape: reading config file /etc/amanda/deepsoft-normal/amanda.conf ! Thu Mar 10 16:13:15.068536891 2022: pid 19101: thd-0x55f1f8259c00: amtape: reading config file /etc/amanda/deepsoft-common/amanda.conf ! Thu Mar 10 16:13:15.104145340 2022: pid 19101: thd-0x55f1f8259c00: amtape: pid 19101 ruid 34 euid 34 version 3.5.1: rename at Thu Mar 10 16:13:15 2022 ! Thu Mar 10 16:13:15.127817939 2022: pid 19101: thd-0x55f1f8259c00: amtape: chg-disk: Dir /backupdisk/deepsoft_vault/slots ! Thu Mar 10 16:13:15.127851840 2022: pid 19101: thd-0x55f1f8259c00: amtape: chg-disk: Using statefile '/backupdisk/deepsoft_vault/slots/state' ! Thu Mar 10 16:13:15.128193583 2022: pid 19101: thd-0x55f1f8259c00: amtape: find_volume labeled 'examplevault' ! Thu Mar 10 16:13:15.135960182 2022: pid 19101: thd-0x55f1f8259c00: amtape: parse_inventory: load slot 141 ! Thu Mar 10 16:13:15.136117147 2022: pid 19101: thd-0x55f1f8259c00: amtape: /usr/lib/x86_64-linux-gnu/amanda/perl/Amanda/Recovery/Scan.pm:420:info:120 slot 141 ! Thu Mar 10 16:13:15.136850450 2022: pid 19101: thd-0x55f1f8259c00: amtape: dir_name: /backupdisk/deepsoft_vault/slots/slot141/ ! Thu Mar 10 16:13:15.137187946 2022: pid 19101: thd-0x55f1f8259c00: amtape: Device file:/backupdisk/deepsoft_vault/slots/slot141 error = 'File 0 not found' ! Thu Mar 10 16:13:15.137208586 2022: pid 19101: thd-0x55f1f8259c00: amtape: Device file:/backupdisk/deepsoft_vault/slots/slot141 setting status flag(s): DEVICE_STATUS_VOLUME_UNLABELED ! Thu Mar 10 16:13:15.137639552 2022: pid 19101: thd-0x55f1f8259c00: amtape: /usr/lib/x86_64-linux-gnu/amanda/perl/Amanda/Recovery/Scan.pm:471:error:122 File 0 not found ! Thu Mar 10 16:13:15.137942996 2022: pid 19101: thd-0x55f1f8259c00: amtape: new Amanda::Changer::Error: type='fatal', message='File 0 not found' ! Thu Mar 10 16:13:15.145936026 2022: pid 19101: thd-0x55f1f8259c00: amtape: parse_inventory: load slot 142 ! Thu Mar 10 16:13:15.145988390 2022: pid 19101: thd-0x55f1f8259c00: amtape: /usr/lib/x86_64-linux-gnu/amanda/perl/Amanda/Recovery/Scan.pm:420:info:120 slot 142 ! Thu Mar 10 16:13:15.146610802 2022: pid 19101: thd-0x55f1f8259c00: amtape: dir_name: /backupdisk/deepsoft_vault/slots/slot142/ ! Thu Mar 10 16:13:15.146858924 2022: pid 19101: thd-0x55f1f825
Re: Problem with amtape "hanging" when forked from Java
On Thu, Mar 10, 2022 at 14:03:07 -0500, Robert Heller wrote: > It prints an error message and returns an error status: > > backup@newserver:~$ amtape -otpchanger=vault_changer -ointeractivity='' > wendellfreelibrary label wendellfreelibrary-vault-030 > ERROR: Source Volume 'wendellfreelibrary-vault-030' not found > > (and does not hang) Hmmm. What happens from the command line if you leave off the -ointeractivity parameter? The strange thing is that the strace does seem to show a repeated looping over all the vtape slots, presumably continually searching for the specfied label but I didn't notice any activity that is obviously interactivity-related in between the loops. So it's not clear why amtape is looping within the Java context but immeidately terminating with an error message in the shell/tclsh contexts. > I'm *guessing* that the Java Process created by exec() wants to deal with the > > printout, but can't. > In the strace you posted, amtape appeared to be looping through the slots multiple times without attempting to write that "ERROR" message anywhere, so off hand I would guess amtape is getting put into a different search mode, rather than a problem with the Java side not being able to accept the output. I'm not sure what would trigger that different mode, though. I suppose my next suggestion would be to compare carefully the amtape .debug files between a run from the command line and a run from within Java -- any difference between the two could be a hint Nathan Nathan Stratton Treadway - [email protected] - Mid-Atlantic region Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239 Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
Re: Problem with amtape "hanging" when forked from Java
At Thu, 10 Mar 2022 13:11:04 -0500 Nathan Stratton Treadway
wrote:
>
> On Wed, Mar 09, 2022 at 22:50:29 -0500, Robert Heller wrote:
> > At Wed, 9 Mar 2022 23:50:45 +0100 Exuvo wrote:
> >
> > >
> > > Could you give the exact command line you give when it hangs?
> >
> > /usr/sbin/amtape -otpchanger=vault_changer wendellfreelibrary label
> > wendellfreelibrary_vault-030
> >
>
> What does this command say/do when run from the command line (for a tape
> that causes a hang in the context of your Java program)?
It prints an error message and returns an error status:
backup@newserver:~$ amtape -otpchanger=vault_changer -ointeractivity=''
wendellfreelibrary label wendellfreelibrary-vault-030
ERROR: Source Volume 'wendellfreelibrary-vault-030' not found
(and does not hang)
It actually does the same when forked from a Tcl program:
backup@newserver:~$ tclsh
% catch {exec amtape -otpchanger=vault_changer -ointeractivity=
wendellfreelibrary label wendellfreelibrary-vault-030} result
1
% set result
ERROR: Source Volume 'wendellfreelibrary-vault-030' not found
%
(I think the "''" for interactivity is only needed from the shell -- from both
Tcl and Java, those quote marks need to be ommited or else amtape is looking
for an interactivity named "''", rather then an null interactivity.)
I'm *guessing* that the Java Process created by exec() wants to deal with the
printout, but can't.
>
>
> Nathan
>
>
> Nathan Stratton Treadway - [email protected] - Mid-Atlantic region
> Ray Ontko & Co. - Software consulting services - http://www.ontko.com/
> GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239
> Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
>
>
>
--
Robert Heller -- Cell: 413-658-7953 GV: 978-633-5364
Deepwoods Software-- Custom Software Services
http://www.deepsoft.com/ -- Linux Administration Services
[email protected] -- Webhosting Services
Re: Problem with amtape "hanging" when forked from Java
On Thu, Mar 10, 2022 at 13:02:24 -0500, Robert Heller wrote:
> At Thu, 10 Mar 2022 12:46:43 -0500 Nathan Stratton Treadway
> wrote:
>
> >
> > On Thu, Mar 10, 2022 at 09:55:30 -0500, Robert Heller wrote:
> > > Here is the Java fragment:
> > >
> > > public class FlushOldVaults extends BackupVault {
> > > private static final String AMTAPE = "/usr/sbin/amtape";
> > > private static final String AMTAPEOPT1 = "-otpchanger=vault_changer";
> > > private static final String AMTAPEOPT2 = "-ointeractivity=";
> >
> > You would probably be able to confirm this by looking in the amanda
> > log/debug files for the amtape process (i.e.
> > /var/log/amanda/server//amtape.*.debug on Ubuntu) , but I'm pretty
> > sure that you do actually need the empty argument in order to disable
> > the interactivity, something like
> > private static final String AMTAPEOPT2 = "-ointeractivity=''";
>
> It does not like the empty argument, amtape throws a error status and the
> Java subprocess returns a failure status. It is "happy" with what I have,
> except it hangs on the broken tape.
Okay, sounds like the argument parsing is different in the Java .exec()
context than on a shell command line. The important question is whether
or not the interactivity is actually disabled perhaps the amtape .debug
file gives some confirmation?
> The timeout never times out. The amtape process goes into Sleep state and
> the
> Java program just hangs.
In that case hopefully strace -p/lsof -p on the amtape process (or any
other Amanda processes that amtape spawns) can give some hint as to what
it's waiting for
Nathan
Nathan Stratton Treadway - [email protected] - Mid-Atlantic region
Ray Ontko & Co. - Software consulting services - http://www.ontko.com/
GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239
Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
Re: Problem with amtape "hanging" when forked from Java
On Wed, Mar 09, 2022 at 22:50:29 -0500, Robert Heller wrote: > At Wed, 9 Mar 2022 23:50:45 +0100 Exuvo wrote: > > > > > Could you give the exact command line you give when it hangs? > > /usr/sbin/amtape -otpchanger=vault_changer wendellfreelibrary label > wendellfreelibrary_vault-030 > What does this command say/do when run from the command line (for a tape that causes a hang in the context of your Java program)? Nathan Nathan Stratton Treadway - [email protected] - Mid-Atlantic region Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239 Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
Re: Problem with amtape "hanging" when forked from Java
At Thu, 10 Mar 2022 12:46:43 -0500 Nathan Stratton Treadway
wrote:
>
> On Thu, Mar 10, 2022 at 09:55:30 -0500, Robert Heller wrote:
> > Here is the Java fragment:
> >
> > public class FlushOldVaults extends BackupVault {
> > private static final String AMTAPE = "/usr/sbin/amtape";
> > private static final String AMTAPEOPT1 = "-otpchanger=vault_changer";
> > private static final String AMTAPEOPT2 = "-ointeractivity=";
>
> You would probably be able to confirm this by looking in the amanda
> log/debug files for the amtape process (i.e.
> /var/log/amanda/server//amtape.*.debug on Ubuntu) , but I'm pretty
> sure that you do actually need the empty argument in order to disable
> the interactivity, something like
> private static final String AMTAPEOPT2 = "-ointeractivity=''";
It does not like the empty argument, amtape throws a error status and the
Java subprocess returns a failure status. It is "happy" with what I have,
except it hangs on the broken tape.
>
> (I am not particularly certain that interactivity is your specific
> problem, but it seemed a plausible explaination and one that that was
> fairly easy to test out...)
>
>
> > if (!p.waitFor(60L, java.util.concurrent.TimeUnit.SECONDS)) {
> > System.err.printf("*** FlushOldVaults.amtape(): process
> > timeout\n");
> > String kill[] = new String[2];
> > kill[0] = "/bin/kill";
> > Long j = new Long(p.pid());
> > kill[1] = j.toString();
> > Process killproc = Runtime.getRuntime().exec(kill);
> > killproc.waitFor();
>
> Note that in order to use the strace debugging, you'll probably need to
> disable this timeout+kill logic -- otherwise the amtape process won't
> hang out long enough for you to figure out what it's trying to do when
> when it's "stuck"
The timeout never times out. The amtape process goes into Sleep state and the
Java program just hangs.
>
>
> Nathan
>
>
>
>
>
> Nathan Stratton Treadway - [email protected] - Mid-Atlantic region
> Ray Ontko & Co. - Software consulting services - http://www.ontko.com/
> GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239
> Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
>
>
>
--
Robert Heller -- Cell: 413-658-7953 GV: 978-633-5364
Deepwoods Software-- Custom Software Services
http://www.deepsoft.com/ -- Linux Administration Services
[email protected] -- Webhosting Services
Re: Problem with amtape "hanging" when forked from Java
On Thu, Mar 10, 2022 at 09:55:30 -0500, Robert Heller wrote:
> Here is the Java fragment:
>
> public class FlushOldVaults extends BackupVault {
> private static final String AMTAPE = "/usr/sbin/amtape";
> private static final String AMTAPEOPT1 = "-otpchanger=vault_changer";
> private static final String AMTAPEOPT2 = "-ointeractivity=";
You would probably be able to confirm this by looking in the amanda
log/debug files for the amtape process (i.e.
/var/log/amanda/server//amtape.*.debug on Ubuntu) , but I'm pretty
sure that you do actually need the empty argument in order to disable
the interactivity, something like
private static final String AMTAPEOPT2 = "-ointeractivity=''";
(I am not particularly certain that interactivity is your specific
problem, but it seemed a plausible explaination and one that that was
fairly easy to test out...)
> if (!p.waitFor(60L, java.util.concurrent.TimeUnit.SECONDS)) {
> System.err.printf("*** FlushOldVaults.amtape(): process
> timeout\n");
> String kill[] = new String[2];
> kill[0] = "/bin/kill";
> Long j = new Long(p.pid());
> kill[1] = j.toString();
> Process killproc = Runtime.getRuntime().exec(kill);
> killproc.waitFor();
Note that in order to use the strace debugging, you'll probably need to
disable this timeout+kill logic -- otherwise the amtape process won't
hang out long enough for you to figure out what it's trying to do when
when it's "stuck"
Nathan
Nathan Stratton Treadway - [email protected] - Mid-Atlantic region
Ray Ontko & Co. - Software consulting services - http://www.ontko.com/
GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239
Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
Re: Problem with amtape "hanging" when forked from Java
At Wed, 9 Mar 2022 22:50:29 -0500 (EST) Robert Heller
wrote:
>
> At Wed, 9 Mar 2022 19:11:40 -0500 Nathan Stratton Treadway
> wrote:
>
> >
> > On Tue, Mar 08, 2022 at 18:27:47 -0500, Robert Heller wrote:
> > >
> > > I've written a Java program that goes through vaulted tapes and forks
> > > amtape
> > > (using Runtime.getRuntime().exec(()), and when a non-existant tape label
> > > is
> > > asked for, amtape "hangs". I cannot figure out why or how to get amtape
> > > to
> > > just exit with an error (which I can then handle).
> >
> > Does it still hang if you pass an argument "-ointeractivity=''" when you
> > exec amtape?
>
> Did not try that.
OK, it still hangs.
Here is the Java fragment:
public class FlushOldVaults extends BackupVault {
private static final String AMTAPE = "/usr/sbin/amtape";
private static final String AMTAPEOPT1 = "-otpchanger=vault_changer";
private static final String AMTAPEOPT2 = "-ointeractivity=";
private String amtape(String tape) throws Exception {
String cmd[] = new String[6];
cmd[0] = AMTAPE;
cmd[1] = AMTAPEOPT1;
cmd[2] = AMTAPEOPT2;
cmd[3] = configuration.AMCONFIG();
cmd[4] = "label";
cmd[5] = tape;
System.err.printf("*** FlushOldVaults.amtape(): tape is '%s'\n",tape);
Process p = Runtime.getRuntime().exec(cmd);
InputStream err = p.getErrorStream();
if (err != null) err.readAllBytes();
if (!p.waitFor(60L, java.util.concurrent.TimeUnit.SECONDS)) {
System.err.printf("*** FlushOldVaults.amtape(): process timeout\n");
String kill[] = new String[2];
kill[0] = "/bin/kill";
Long j = new Long(p.pid());
kill[1] = j.toString();
Process killproc = Runtime.getRuntime().exec(kill);
killproc.waitFor();
}
int status = p.waitFor();
System.err.printf("*** FlushOldVaults.amtape(): status = %d\n",status);
if (status != 0) throw new Exception(AMTAPE+" "+AMTAPEOPT1+"
"+AMTAPEOPT2+" "+configuration.AMCONFIG()+" label "+tape+": failed");
String tpchanger = amgetconf("CHANGER:vault_changer:tpchanger");
int colon = tpchanger.indexOf(':');
if (colon < 0) {
return tpchanger+"/data/";
} else {
return tpchanger.substring(colon+1)+"/data/";
}
}
}
>
> >
> > If it does still hang, what do "lsof -p" and "strace -p" show on the
> > amtape process while it's stuck?
> >
> > Nathan
> >
> >
> > Nathan Stratton Treadway - [email protected] - Mid-Atlantic region
> > Ray Ontko & Co. - Software consulting services - http://www.ontko.com/
> > GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239
> > Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
> >
> >
> >
>
--
Robert Heller -- Cell: 413-658-7953 GV: 978-633-5364
Deepwoods Software-- Custom Software Services
http://www.deepsoft.com/ -- Linux Administration Services
[email protected] -- Webhosting Services
Re: Problem with amtape "hanging" when forked from Java
At Wed, 9 Mar 2022 19:11:40 -0500 Nathan Stratton Treadway wrote: > > On Tue, Mar 08, 2022 at 18:27:47 -0500, Robert Heller wrote: > > > > I've written a Java program that goes through vaulted tapes and forks > > amtape > > (using Runtime.getRuntime().exec(()), and when a non-existant tape label is > > asked for, amtape "hangs". I cannot figure out why or how to get amtape to > > just exit with an error (which I can then handle). > > Does it still hang if you pass an argument "-ointeractivity=''" when you > exec amtape? Did not try that. > > If it does still hang, what do "lsof -p" and "strace -p" show on the > amtape process while it's stuck? > > Nathan > > > Nathan Stratton Treadway - [email protected] - Mid-Atlantic region > Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ > GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239 > Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239 > > > -- Robert Heller -- Cell: 413-658-7953 GV: 978-633-5364 Deepwoods Software-- Custom Software Services http://www.deepsoft.com/ -- Linux Administration Services [email protected] -- Webhosting Services
Re: Problem with amtape "hanging" when forked from Java
At Wed, 9 Mar 2022 23:50:45 +0100 Exuvo wrote: > > Could you give the exact command line you give when it hangs? /usr/sbin/amtape -otpchanger=vault_changer wendellfreelibrary label wendellfreelibrary_vault-030 That partitular tape is missing, but still existed in a separate database (long story). > > Anton "exuvo" Olsson > [email protected] > > On 2022-03-09 00:27, Robert Heller wrote: > > For some unfathomably reason amtape "hangs" when forked from a Java program. > > > > I've written a Java program that goes through vaulted tapes and forks amtape > > (using Runtime.getRuntime().exec(()), and when a non-existant tape label is > > asked for, amtape "hangs". I cannot figure out why or how to get amtape to > > just exit with an error (which I can then handle). > > > > This is with Amanda 3.5.1 on a Ubuntu 18.04 system. > > > > (The Java version is OpenJDK 11.0.13, in case that matters.) > > > > > > -- Robert Heller -- Cell: 413-658-7953 GV: 978-633-5364 Deepwoods Software-- Custom Software Services http://www.deepsoft.com/ -- Linux Administration Services [email protected] -- Webhosting Services
Re: Problem with amtape "hanging" when forked from Java
On Tue, Mar 08, 2022 at 18:27:47 -0500, Robert Heller wrote: > > I've written a Java program that goes through vaulted tapes and forks amtape > (using Runtime.getRuntime().exec(()), and when a non-existant tape label is > asked for, amtape "hangs". I cannot figure out why or how to get amtape to > just exit with an error (which I can then handle). Does it still hang if you pass an argument "-ointeractivity=''" when you exec amtape? If it does still hang, what do "lsof -p" and "strace -p" show on the amtape process while it's stuck? Nathan Nathan Stratton Treadway - [email protected] - Mid-Atlantic region Ray Ontko & Co. - Software consulting services - http://www.ontko.com/ GPG Key: http://www.ontko.com/~nathanst/gpg_key.txt ID: 1023D/ECFB6239 Key fingerprint = 6AD8 485E 20B9 5C71 231C 0C32 15F3 ADCD ECFB 6239
Re: Problem with amtape "hanging" when forked from Java
Could you give the exact command line you give when it hangs? Anton "exuvo" Olsson [email protected] On 2022-03-09 00:27, Robert Heller wrote: For some unfathomably reason amtape "hangs" when forked from a Java program. I've written a Java program that goes through vaulted tapes and forks amtape (using Runtime.getRuntime().exec(()), and when a non-existant tape label is asked for, amtape "hangs". I cannot figure out why or how to get amtape to just exit with an error (which I can then handle). This is with Amanda 3.5.1 on a Ubuntu 18.04 system. (The Java version is OpenJDK 11.0.13, in case that matters.)
