Re: [Bacula-users] Trying to restore, but can't

2018-06-13 Thread Ryan Butler 
I did some looking into the index thing and found that I had some extra indices 
on the File table that apparently were crippling performance.

These two pages led me in the right direction:

http://www.bacula.org/7.0.x-manuals/en/main/Catalog_Maintenance.html#SECTION004392000

http://wiki.bacula.org/doku.php?id=faq#restore_takes_a_long_time_to_retrieve_sql_results_from_mysql_catalog

Once I dropped the extra indices, the "Building directory tree" step took about 
1 minute (even with 3 million files, even with only 16GB RAM).

-ryan

From: Josh Fisher 
Sent: Wednesday, June 13, 2018 3:59 AM
To: Ryan Butler ; bacula-users@lists.sourceforge.net
Subject: Re: [Bacula-users] Trying to restore, but can't


On 6/12/2018 12:35 PM, Ryan Butler wrote:
Hello all,

After a failed upgrade to one of our organization's web apps, I'm trying to 
restore the web app directory from Bacula. I went into bconsole, restore, 
option 3 (enter jobid), and have been sitting on "Building directory tree for 
JobId(s) 56810 ..." for the past 12 hours. The MySQL process is using 100% CPU 
on one core, but other than that, I have no indication that anything is 
happening. And this is something that I need to figure out why it is happening, 
but what I really need right now is to just get the web app restored so users 
don't start getting angry.

100% CPU on one core with MySQL is often due to a missing index.



I really just need to restore a single directory (and all of its files and 
subdirectories), so I've been looking at option 11 (enter a list of directories 
to restore for jobid), and it looks like I'd have to manually specify all the 
subdirectories, which is doable. However, would that option also need to "Build 
directory tree", and therefore I'd be back at the original problem?

Well, option 11 should use a different query than option 3, so if the problem 
is a missing index, then option 11 may well work better than option 3.



We're on version 7.0.5, which I know is old, and I have plans to update it 
after this disaster.

I suggest moving the catalog to postgres on SSD storage, while you're at it.



Here's the details on that particular Job:

JobFiles: 2,927,789
JobBytes: 54,769,194,913 (54 GB)

Any help is appreciated!!!

Ryan Butler
Systems Administrator

[/Users/ryan_butler/Library/Containers/com.microsoft.Outlook/Data/Library/Caches/Signatures/signature_1633926894]<https://www.inflexion.org/>





--

Check out the vibrant tech community on one of the world's most

engaging tech sites, Slashdot.org! http://sdm.link/slashdot




___

Bacula-users mailing list

Bacula-users@lists.sourceforge.net<mailto:Bacula-users@lists.sourceforge.net>

https://lists.sourceforge.net/lists/listinfo/bacula-users

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Trying to restore, but can't

2018-06-13 Thread Dimitri Maziuk via Bacula-users 

On 6/13/2018 5:59 AM, Josh Fisher wrote:


I suggest moving the catalog to postgres on SSD storage, while you're at it.


Mine's on spinning rust and it still works OK given enough RAM.

I think another issue here is the expectations: we ran a one-CPU 
(relatively) low-RAM test server for a while and saw that in testing. So 
when we started using bacula in production we already knew restores 
could take a while.


(And ditch mysql: allegedly it can be tuned up for tolerable 
performance, but why bother when there's postgres.)


Dima

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Trying to restore, but can't

2018-06-13 Thread Josh Fisher 


On 6/12/2018 12:35 PM, Ryan Butler wrote:


Hello all,

After a failed upgrade to one of our organization’s web apps, I’m 
trying to restore the web app directory from Bacula. I went into 
bconsole, restore, option 3 (enter jobid), and have been sitting on 
“Building directory tree for JobId(s) 56810 …” for the past 12 hours. 
The MySQL process is using 100% CPU on one core, but other than that, 
I have no indication that anything is happening. And this is something 
that I need to figure out why it is happening, but what I really need 
right now is to just get the web app restored so users don’t start 
getting angry.




100% CPU on one core with MySQL is often due to a missing index.

I really just need to restore a single directory (and all of its files 
and subdirectories), so I’ve been looking at option 11 (enter a list 
of directories to restore for jobid), and it looks like I’d have to 
manually specify all the subdirectories, which is doable. However, 
would that option also need to “Build directory tree”, and therefore 
I’d be back at the original problem?




Well, option 11 should use a different query than option 3, so if the 
problem is a missing index, then option 11 may well work better than 
option 3.


We’re on version 7.0.5, which I know is old, and I have plans to 
update it after this disaster.




I suggest moving the catalog to postgres on SSD storage, while you're at it.


Here’s the details on that particular Job:

JobFiles: 2,927,789

JobBytes: 54,769,194,913 (54 GB)

Any help is appreciated!!!

*Ryan Butler*

Systems Administrator

/Users/ryan_butler/Library/Containers/com.microsoft.Outlook/Data/Library/Caches/Signatures/signature_1633926894 





--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Trying to restore, but can't

2018-06-12 Thread Ryan Butler 
It's attempting to restore directly from a single FULL backup job.

I know that once it finishes generating the directory listing that I can mark 
the directory I want and then it will go ahead and restore everything in it 
including subfolders. However, option 11, from the online manual says "Item 11 
allows you to enter a list of JobIds from which you can select directories to 
be restored. You may then enter a full path to a directory name.. All 
files in those directories will be restored, but if the directory contains 
subdirectories, nothing will be restored in the subdirectory unless you 
explicitly enter its name."

Either way, this is fine, I just really need to know if using option 11 has it 
generating a directory listing the same as the option 3 that I am using, 
because this is the part that it's taking forever on, and I'm really just 
looking for a way to get this directory restored as quickly as possible. Once 
I've gotten the directory restored, I'm going to look into improving restore 
times. (This is a Bacula system that I inherited from a previous sysadmin)

The server that Bacula is running on has 12 cores and 16GB of RAM. This 
directory listing generation has so far been running for 28 hours. Is there any 
hope of this finishing or are there any suggestions?

Thanks!

-ryan

-Original Message-
From: Phil Stracchino  
Sent: Tuesday, June 12, 2018 3:25 PM
To: bacula-users@lists.sourceforge.net
Subject: Re: [Bacula-users] Trying to restore, but can't

On 06/12/18 12:35, Ryan Butler wrote:

> I really just need to restore a single directory (and all of its files 
> and subdirectories), so I've been looking at option 11 (enter a list 
> of directories to restore for jobid), and it looks like I'd have to 
> manually specify all the subdirectories, which is doable. However, 
> would that option also need to "Build directory tree", and therefore 
> I'd be back at the original problem?

If you tell it to restore a directory, everything under that directory will 
automatically be included.


> Here's the details on that particular Job:
> 
> JobFiles: 2,927,789
> 
> JobBytes: 54,769,194,913 (54 GB)


How many incrementals/differentials since your last FULL?  That's almost three 
million files in that Job.  If it's got to scan a large number of
jobs ...   yeah, that could take a while.


--
  Phil Stracchino
  Babylon Communications
  ph...@caerllewys.net
  p...@co.ordinate.org
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958

--
Check out the vibrant tech community on one of the world's most engaging tech 
sites, Slashdot.org! http://sdm.link/slashdot 
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Trying to restore, but can't

2018-06-12 Thread Dimitri Maziuk via Bacula-users 
On 06/12/2018 11:35 AM, Ryan Butler wrote:

> We're on version 7.0.5, which I know is old, and I have plans to update it 
> after this disaster.
> 
> Here's the details on that particular Job:
> 
> JobFiles: 2,927,789

Version has nothing to do with it. There are several SQL joins involved
in getting from job to file, plus as Phil said: through all incrementals
to the last full. You simply need a database and hardware that can do
that on 3M files in a tolerable timeframe.

My production machines have 16-24 cores and 64-256GB RAM. And postgres.

-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu



signature.asc
Description: OpenPGP digital signature
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Trying to restore, but can't

2018-06-12 Thread Phil Stracchino 
On 06/12/18 12:35, Ryan Butler wrote:

> I really just need to restore a single directory (and all of its files
> and subdirectories), so I’ve been looking at option 11 (enter a list of
> directories to restore for jobid), and it looks like I’d have to
> manually specify all the subdirectories, which is doable. However, would
> that option also need to “Build directory tree”, and therefore I’d be
> back at the original problem?

If you tell it to restore a directory, everything under that directory
will automatically be included.


> Here’s the details on that particular Job:
> 
> JobFiles: 2,927,789
> 
> JobBytes: 54,769,194,913 (54 GB)


How many incrementals/differentials since your last FULL?  That's almost
three million files in that Job.  If it's got to scan a large number of
jobs ...   yeah, that could take a while.


-- 
  Phil Stracchino
  Babylon Communications
  ph...@caerllewys.net
  p...@co.ordinate.org
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] Trying to restore, but can't

2018-06-12 Thread Ryan Butler 
Hello all,

After a failed upgrade to one of our organization's web apps, I'm trying to 
restore the web app directory from Bacula. I went into bconsole, restore, 
option 3 (enter jobid), and have been sitting on "Building directory tree for 
JobId(s) 56810 ..." for the past 12 hours. The MySQL process is using 100% CPU 
on one core, but other than that, I have no indication that anything is 
happening. And this is something that I need to figure out why it is happening, 
but what I really need right now is to just get the web app restored so users 
don't start getting angry.

I really just need to restore a single directory (and all of its files and 
subdirectories), so I've been looking at option 11 (enter a list of directories 
to restore for jobid), and it looks like I'd have to manually specify all the 
subdirectories, which is doable. However, would that option also need to "Build 
directory tree", and therefore I'd be back at the original problem?

We're on version 7.0.5, which I know is old, and I have plans to update it 
after this disaster.

Here's the details on that particular Job:

JobFiles: 2,927,789
JobBytes: 54,769,194,913 (54 GB)

Any help is appreciated!!!

Ryan Butler
Systems Administrator

[/Users/ryan_butler/Library/Containers/com.microsoft.Outlook/Data/Library/Caches/Signatures/signature_1633926894]

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users