Re: [PVE-User] Ceph Cache Tiering

2016-10-10 Thread Lindsay Mathieson 

On 10/10/2016 8:19 PM, Brian :: wrote:

I think with clusters with VM type workload and at the scale that
proxmox users tend to build < 20 OSD servers that cache tier is adding
layer of complexity that isn't going to payback. If you want decent
IOPS / throughput at this scale with Ceph no spinning rust allowed
anywhere:)


I think you're right, for all the talk of small scale deployments and 
commodity hardware, ceph on the small business scale is a poor 
price/performance ratio. To get decent performance out of it you have to 
spend big on terrabyte SSD's, battery backed disk controllers etc.


Its a shame, the flexibility of it is really good and rock solid in the 
9.x range if you don't push it :) Its ability to quickly heal only dirty 
data is outsanding. I just tested a simple 3 node setup backed by our 
ZFS pools and was only getting 50MB/s seq writes. If I added SSD 
journals that probably would have got to the 100MB/s level, which would 
have been good enough, but there is one deal breaker for us and thats 
snapshots - they are incredibly slow to restore, 45 minutes for one that 
was only a few minutes old. It gets worse the more writes you add. qcow2 
snapshots on gluster only take a couple of minutes, and we use snapshots 
a lot for testing development and support.




Additionally i think it is error prone.
I ran into the problem, that a ssd stuck because it was full causing the 
complete storage to stuck.


It does seem to be very much a work in progress :)

--
Lindsay Mathieson

___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Ceph Cache Tiering

2016-10-10 Thread Eneko Lacunza 

Hi,

El 10/10/16 a las 13:46, Lindsay Mathieson escribió:

On 10/10/2016 8:19 PM, Brian :: wrote:

I think with clusters with VM type workload and at the scale that
proxmox users tend to build < 20 OSD servers that cache tier is adding
layer of complexity that isn't going to payback. If you want decent
IOPS / throughput at this scale with Ceph no spinning rust allowed
anywhere:)


I think you're right, for all the talk of small scale deployments and 
commodity hardware, ceph on the small business scale is a poor 
price/performance ratio. To get decent performance out of it you have 
to spend big on terrabyte SSD's, battery backed disk controllers etc.
We have multiple small scale ceph clusters in SMB, all the clients and 
ourselves very happy so far :)


Its a shame, the flexibility of it is really good and rock solid in 
the 9.x range if you don't push it :) Its ability to quickly heal only 
dirty data is outsanding. I just tested a simple 3 node setup backed 
by our ZFS pools and was only getting 50MB/s seq writes. If I added 
SSD journals that probably would have got to the 100MB/s level, which 
would have been good enough, but there is one deal breaker for us and 
thats snapshots - they are incredibly slow to restore, 45 minutes for 
one that was only a few minutes old. It gets worse the more writes you 
add. qcow2 snapshots on gluster only take a couple of minutes, and we 
use snapshots a lot for testing development and support.
But this is nonsense, ZFS backed Ceph?! You're supposed to give full 
disks to ceph, so that performance increases as you add more disks! :) 
Can't comment on snapshots, we don't use them.



Additionally i think it is error prone.
I ran into the problem, that a ssd stuck because it was full causing 
the complete storage to stuck.

It does seem to be very much a work in progress :)

I think cache pools are in better shape in 10.2.x, but we haven't used them.

Cheers
Eneko


--
Zuzendari Teknikoa / Director Técnico
Binovo IT Human Project, S.L.
Telf. 943493611
  943324914
Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa)
www.binovo.es

___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Ceph Cache Tiering

2016-10-10 Thread Lindsay Mathieson 

On 10/10/2016 10:22 PM, Eneko Lacunza wrote:
But this is nonsense, ZFS backed Ceph?! You're supposed to give full 
disks to ceph, so that performance increases as you add more disks


I've tried it both ways, the performance is much the same. ZFS also 
increases in performance the more disks you throw it, which is passed 
onto ceph.



+Compression

+Auto Bit rot detection and repair

+A lot of flexibility

--
Lindsay Mathieson

___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Ceph Cache Tiering

2016-10-10 Thread Adam Thompson 
The default PVE setup puts an XFS filesystem onto each "full disk" assigned to 
CEPH.  CEPH does **not** write directly to raw devices, so the choice of 
filesystem is largely irrelevant.
Granted, ZFS is a "heavier" filesystem than XFS, but it's no better or worse 
than running CEPH on XFS on Hardware RAID, which I've done elsewhere.
CEPH gives you the ability to not need software or hardware RAID.
ZFS gives you the ability to not need hardware RAID.
Layering them - assuming you have enough memory and CPU cycles - can be very 
beneficial.
Neither CEPH nor XFS does deduplication or compression, which ZFS does.  
Depending on what kind of CPU you have, turning on compression can dramatically 
*speed up* I/O.  Depending on how much RAM you have, turning on deduplication 
can dramatically decrease disk space used.
Although, TBH, at that point I'd just do what I have running in production 
right now: a reasonably-powerful SPARC64 NFS fileserver, and run QCOW2 files 
over NFS.  Performs better than CEPH did on 1Gbps infrastructure.
-Adam

> -Original Message-
> From: pve-user [mailto:pve-user-boun...@pve.proxmox.com] On
> Behalf Of Lindsay Mathieson
> Sent: October 10, 2016 09:21
> To: pve-user@pve.proxmox.com
> Subject: Re: [PVE-User] Ceph Cache Tiering
> 
> On 10/10/2016 10:22 PM, Eneko Lacunza wrote:
> > But this is nonsense, ZFS backed Ceph?! You're supposed to give full
> > disks to ceph, so that performance increases as you add more disks
> 
> I've tried it both ways, the performance is much the same. ZFS also
> increases in performance the more disks you throw it, which is passed
> onto ceph.
> 
> 
> +Compression
> 
> +Auto Bit rot detection and repair
> 
> +A lot of flexibility
> 
> --
> Lindsay Mathieson
> 
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Ceph Cache Tiering

2016-10-10 Thread Lindsay Mathieson 

On 11/10/2016 2:05 AM, Eneko Lacunza wrote:
The choice of filesystem is not "largely irrelevant"; filesystems are 
quite complex and the choice is relevant. With ZFS, you're in unknown 
territory AFAIK as it is not regularly tested in ceph development; I 
think only ext4 and XFS are regularly tested.

ext4 is now no longer supported and recommended against



--
Lindsay Mathieson

___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Ceph Cache Tiering

2016-10-10 Thread Yannis Milios 
>>...but there is one deal breaker for us and thats snapshots - they are
incredibly >> slow to restore.

You can try to clone instead of rolling back an image to the snapshot. It's
much faster and the recommended method by official Ceph documentation.

http://docs.ceph.com/docs/jewel/rbd/rbd-snapshot/#rollback-snapshot
___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Ceph Cache Tiering

2016-10-10 Thread Lindsay Mathieson 

On 11/10/2016 6:28 AM, Yannis Milios wrote:

You can try to clone instead of rolling back an image to the snapshot. It's
much faster and the recommended method by official Ceph documentation.


Not integrated with Proxmox/

--
Lindsay Mathieson

___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] ceph - slow rbd ls -l

2016-10-10 Thread Thiago Damas 
2016-10-10 20:47 GMT-03:00 Lindsay Mathieson :

> On 11/10/2016 7:59 AM, Thiago Damas wrote:
>
>> I'm experiencing some timeouts when creating new disks/VMs, using a ceph
>> storage.
>>Is there some way to reduce the long listing of rbd ls, ie "rbd ls -l"?
>>
>
> Hows your "ceph -s" look?
>
>
> Are the logs showing any particular OSD's as being slow to respond?
>
> --
> Lindsay Mathieson
>
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>


Look:

~# ceph -s
cluster 71b07854-5d50-46e7-a130-2230712cd3aa
 health HEALTH_OK
 monmap e9: 5 mons at
{0=192.168.33.AAA:6789/0,1=192.168.33.BBB:6789/0,2=192.168.33.CCC:6789/0,3=192.168.33.DDD:6789/0,4=192.168.33.EEE:6789/0}
election epoch 294, quorum 0,1,2,3,4 0,1,2,3,4
 osdmap e3466: 40 osds: 40 up, 40 in
  pgmap v15080315: 2048 pgs, 1 pools, 7910 GB data, 1995 kobjects
23467 GB used, 122 TB / 145 TB avail
2048 active+clean
  client io 36159 kB/s rd, 11745 kB/s wr, 1385 op/s

~# ceph osd perf
osd fs_commit_latency(ms) fs_apply_latency(ms)
  0 1   10
  1 1   13
  2 07
  3 1   12
  4 06
  5 07
  6 05
  7 19
  8 1   10
  9 19
 10 1   11
 11 16
 12 1   13
 13 0   22
 14 27
 15 18
 16 18
 17 08
 18 19
 19 17
 20 3   13
 21 2   30
 22 3   16
 23 1   10
 24 18
 25 08
 26 17
 27 18
 28 17
 29 06
 30 1   13
 31 19
 32 1   11
 33 18
 34 17
 35 1   10
 36 1   13
 37 16
 38 19
 39 2   12

~# time rbd ls -l
...lines and lines...
real 0m37.351s
user 0m0.552s
sys 0m0.232s


  Thiago
___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] P2V Windows2003 Server (AD)

2016-10-10 Thread HK 590 Dicky 梁家棋 資科 

oh.. so useful, thx a lot

Best Regards,



資訊科技部 系統管理員 梁家棋
Dicky Leung
System Administrator
IT Department
電話: (852) 24428990
傳真: (852) 24757095
電郵: 5...@hshcl.com
網址: www.hshcl.com
DISCLAIMER:-
http://www.hshcl.com/disclaimer.htm 

Jean R. Franco 於 07/10/2016 19:31 寫道:

Hi,

I used the self image to do it:
http://pve.proxmox.com/wiki/Migration_of_servers_to_Proxmox_VE#Physical_.28running.29_server_to_Proxmox_VE_.28KVM.29_using_SelfImage

Works all the time.

Best regards,

- Mensagem original -
De: "Lindsay Mathieson" 
Para: "PVE User List" 
Enviadas: Sexta-feira, 7 de outubro de 2016 6:59:04
Assunto: Re: [PVE-User] P2V Windows2003 Server (AD)

On 7/10/2016 7:43 PM, HK 590 Dicky 梁家棋 資科 wrote:

any suggestion to migrate physical windows 2003 server to Proxmox ?

I've done via windows full backup and restore in the past.


Use bog std hardware in the VM - IDE Drive, Intel network adaptor etc.



___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Ceph Cache Tiering

2016-10-10 Thread Eneko Lacunza 

Hi,

The choice of filesystem is not "largely irrelevant"; filesystems are 
quite complex and the choice is relevant. With ZFS, you're in unknown 
territory AFAIK as it is not regularly tested in ceph development; I 
think only ext4 and XFS are regularly tested. And there are known 
limits/problems with ext4 for example, and seems that also apply to 
zfsonlinux (I think ext4 has lower limit yet):

https://github.com/zfsonlinux/zfs/issues/4913

Also no word about ZFS in recommendations:
http://docs.ceph.com/docs/jewel/rados/configuration/filesystem-recommendations/

It can be done? Yes - Lindsay is doing it successfully.

Is it advisable? - I don't think so. :-)

Anyway it seems they're getting rid of the filesystem in the near future 
with bluestore ;)


Cheers
Eneko

El 10/10/16 a las 16:29, Adam Thompson escribió:

The default PVE setup puts an XFS filesystem onto each "full disk" assigned to 
CEPH.  CEPH does **not** write directly to raw devices, so the choice of filesystem is 
largely irrelevant.
Granted, ZFS is a "heavier" filesystem than XFS, but it's no better or worse 
than running CEPH on XFS on Hardware RAID, which I've done elsewhere.
CEPH gives you the ability to not need software or hardware RAID.
ZFS gives you the ability to not need hardware RAID.
Layering them - assuming you have enough memory and CPU cycles - can be very 
beneficial.
Neither CEPH nor XFS does deduplication or compression, which ZFS does.  
Depending on what kind of CPU you have, turning on compression can dramatically 
*speed up* I/O.  Depending on how much RAM you have, turning on deduplication 
can dramatically decrease disk space used.
Although, TBH, at that point I'd just do what I have running in production 
right now: a reasonably-powerful SPARC64 NFS fileserver, and run QCOW2 files 
over NFS.  Performs better than CEPH did on 1Gbps infrastructure.
-Adam


-Original Message-
From: pve-user [mailto:pve-user-boun...@pve.proxmox.com] On
Behalf Of Lindsay Mathieson
Sent: October 10, 2016 09:21
To: pve-user@pve.proxmox.com
Subject: Re: [PVE-User] Ceph Cache Tiering

On 10/10/2016 10:22 PM, Eneko Lacunza wrote:

But this is nonsense, ZFS backed Ceph?! You're supposed to give full
disks to ceph, so that performance increases as you add more disks

I've tried it both ways, the performance is much the same. ZFS also
increases in performance the more disks you throw it, which is passed
onto ceph.


+Compression

+Auto Bit rot detection and repair

+A lot of flexibility

--
Lindsay Mathieson

___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user



--
Zuzendari Teknikoa / Director Técnico
Binovo IT Human Project, S.L.
Telf. 943493611
  943324914
Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa)
www.binovo.es

___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Ceph Cache Tiering

2016-10-10 Thread Kautz, Valentin 
I can only acknowledge this. Speed went down as the cache tier went full. Even 
with superfast PCIe SSDs.

Additionally i think it is error prone.
I ran into the problem, that a ssd stuck because it was full causing the 
complete storage to stuck.
There is a parameter for the maximum size of the cache, but it does not prevent 
a single ssd in the cache-tier to be filled completely - at least i didn't find 
any method despite of the balance factor for the osds, but that didn't help.
I my setup there were 3 SSDs for caching.
One of them went full, even when the sum of the used storage was smaller than 
the maximum cache size.

Kind Regards

Valentin

Von: pve-user [pve-user-boun...@pve.proxmox.com] im Auftrag von 
Lindsay Mathieson [lindsay.mathie...@gmail.com]
Gesendet: Sonntag, 9. Oktober 2016 00:21
An: PVE User List
Betreff: Re: [PVE-User] Ceph Cache Tiering

On 9/10/2016 7:45 AM, Lindsay Mathieson wrote:
> cache tiering was limited and a poor fit for VM Hosting, generally the
> performance was with it

"was *worse* with it"


:)

--
Lindsay Mathieson

___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] ceph - slow rbd ls -l

2016-10-10 Thread Lindsay Mathieson 

On 11/10/2016 7:59 AM, Thiago Damas wrote:

I'm experiencing some timeouts when creating new disks/VMs, using a ceph
storage.
   Is there some way to reduce the long listing of rbd ls, ie "rbd ls -l"?


Hows your "ceph -s" look?


Are the logs showing any particular OSD's as being slow to respond?

--
Lindsay Mathieson

___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Ceph Cache Tiering

2016-10-10 Thread Brian :: 
Hi Lindsay

I think with clusters with VM type workload and at the scale that
proxmox users tend to build < 20 OSD servers that cache tier is adding
layer of complexity that isn't going to payback. If you want decent
IOPS / throughput at this scale with Ceph no spinning rust allowed
anywhere  :)

Regards


On Sat, Oct 8, 2016 at 11:21 PM, Lindsay Mathieson
 wrote:
> On 9/10/2016 7:45 AM, Lindsay Mathieson wrote:
>>
>> cache tiering was limited and a poor fit for VM Hosting, generally the
>> performance was with it
>
>
> "was *worse* with it"
>
>
> :)
>
>
> --
> Lindsay Mathieson
>
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user