Re: [Gluster-users] Volume hacked

[Arman Khalatyan]

check out the syslogs of iptables logs  on ip address access during that
time.
maybe you should move in the future to the centralised logging independent
of vm infrastructure

Am 07.08.2017 2:20 nachm. schrieb :

> > It really depends on the application if locks are used. Most (Linux)
> > applications will use advisory locks. This means that locking is only
> > effective when all participating applications use and honour the locks.
> > If one application uses (advisory) locks, and an other application now,
> > well, then all bets are off.
> >
> > It is also possible to delete files that are in active use. The contens
> > will still be served by the filesystem, but there is no accessible
> > filename anymore. If the VMs using those files are still running, there
> > might be a way to create a new filename for the data. If the VMs have
> > been stopped, and the file-descriptior has been closed, the data will be
> > gone :-/
> >
>
> Oh the data was gone long before I stopped the VM, every binary was
> doing I/O errors when accessed, only whatever was in ram (ssh ..) when
> the disk got suppressed was still working.
>
> I'm a bit surpised they could be deleted, but I imagine qemu through
> libgfapi doesn't really access the file as a whole, maybe just the part
> it needs when it needs it. In any case the gluster logs show clearly
> file descriptor errors from 8h47 AM UTC, which seems to match our first
> monitoring alerts. I assume that's when the deletion happened.
>
> Now I just need to figure out what they used to access the volume, I
> hope it's just NFS since that's the only thing I can think of.
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Volume hacked

[lemonnierk]

> It really depends on the application if locks are used. Most (Linux)
> applications will use advisory locks. This means that locking is only
> effective when all participating applications use and honour the locks.
> If one application uses (advisory) locks, and an other application now,
> well, then all bets are off.
> 
> It is also possible to delete files that are in active use. The contens
> will still be served by the filesystem, but there is no accessible
> filename anymore. If the VMs using those files are still running, there
> might be a way to create a new filename for the data. If the VMs have
> been stopped, and the file-descriptior has been closed, the data will be
> gone :-/
>

Oh the data was gone long before I stopped the VM, every binary was
doing I/O errors when accessed, only whatever was in ram (ssh ..) when
the disk got suppressed was still working.

I'm a bit surpised they could be deleted, but I imagine qemu through
libgfapi doesn't really access the file as a whole, maybe just the part
it needs when it needs it. In any case the gluster logs show clearly
file descriptor errors from 8h47 AM UTC, which seems to match our first
monitoring alerts. I assume that's when the deletion happened.

Now I just need to figure out what they used to access the volume, I
hope it's just NFS since that's the only thing I can think of.


signature.asc
Description: Digital signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Volume hacked

[Niels de Vos]

On Sun, Aug 06, 2017 at 08:54:33PM +0100, lemonni...@ulrar.net wrote:
> Thinking about it, is it even normal they managed to delete the VM disks?
> Shoudn't they have gotten "file in use" errors ? Or does libgfapi not
> lock the access files ?

It really depends on the application if locks are used. Most (Linux)
applications will use advisory locks. This means that locking is only
effective when all participating applications use and honour the locks.
If one application uses (advisory) locks, and an other application now,
well, then all bets are off.

It is also possible to delete files that are in active use. The contens
will still be served by the filesystem, but there is no accessible
filename anymore. If the VMs using those files are still running, there
might be a way to create a new filename for the data. If the VMs have
been stopped, and the file-descriptior has been closed, the data will be
gone :-/

Niels


> 
> 
> On Sun, Aug 06, 2017 at 03:57:06PM +0100, lemonni...@ulrar.net wrote:
> > Hi,
> > 
> > This morning one of our cluster was hacked, all the VM disks were
> > deleted and a file README.txt was left with inside just
> > "http://virtualisan.net/contactus.php :D"
> > 
> > I don't speak the language but with google translete it looks like it's
> > just a webdev company or something like that, a bit surprised ..
> > In any case, we'd really like to know how that happened.
> > 
> > I realised NFS is accessible by anyone (sigh), is there a way to check
> > if that is what they used ? I tried reading the nfs.log but it's not
> > really clear if someone used it or not. What do I need to look for in
> > there to see if someone mounted the volume ?
> > There are stuff in the log on one of the bricks (only one), 
> > and as we aren't using NFS for that volume that in itself seems
> > suspicious.
> > 
> > Thanks
> 
> 
> 
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://lists.gluster.org/mailman/listinfo/gluster-users
> 



> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Volume hacked

[Amar Tumballi]

On Mon, Aug 7, 2017 at 2:17 PM,  wrote:

> On Mon, Aug 07, 2017 at 10:40:08AM +0200, Arman Khalatyan wrote:
> > Interesting problem...
> > Did you considered an insider job?( comes to mind http://verelox.com
> >  recent troubles)
>
> I would be really really surprised, we are only 5 / 6 with access and as
> far as I know no one has a problem with the company.
> The last person to leave did so last year, and we revoked everything (I
> hope). And I can't think of a reason they'd leave the website of a
> hungarian company in there, we contacted them and they think it's one
> of their ex-employee trying to cause them problems.
> I think we were just unlucky, but I'd really love to confirm how they
> did it
>
>
For any filesystem access through GlusterFS, a successful handshake at the
server-side is mandatory.

You should have the log of the clients connected to these server machines
in brick logs (mostly at /var/log/glusterfs/bricks/*.log), check them for
any external IP.

Gluster doesn't provide any extra protection right now, other than what is
provided by POSIX standard (ie, user access control). So, if user is 'root'
in his machine, and there is no_root_squash option, then technically he can
delete all the files in the volume, if he can mount the volume. The major
'authentication' control provided are by IP based authentications.

At this time, if your volume didn't had more granular control on
'auth.allow' options, then we can check the log and try to understand which
client caused this.

Regards,
Amar


>
> > On Mon, Aug 7, 2017 at 3:30 AM, W Kern  wrote:
> >
> > >
> > >
> > > On 8/6/2017 4:57 PM, lemonni...@ulrar.net wrote:
> > >
> > >
> > > Gluster already uses a vlan, the problem is that there is no easy way
> > > that I know of to tell gluster not to listen on an interface, and I
> > > can't not have a public IP on the server. I really wish ther was a
> > > simple "listen only on this IP/interface" option for this
> > >
> > >
> > > What about this?
> > >
> > > transport.socket.bind-address
> > >
> > > I know the were some BZs on it with earlier Gluster Versions, so I
> assume its still there now.
> > >
> > > -bill
> > >
> > >
> > >
> > >
> > > ___
> > > Gluster-users mailing list
> > > Gluster-users@gluster.org
> > > http://lists.gluster.org/mailman/listinfo/gluster-users
> > >
>
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://lists.gluster.org/mailman/listinfo/gluster-users
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>



-- 
Amar Tumballi (amarts)
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Volume hacked

[lemonnierk]

On Mon, Aug 07, 2017 at 10:40:08AM +0200, Arman Khalatyan wrote:
> Interesting problem...
> Did you considered an insider job?( comes to mind http://verelox.com
>  recent troubles)

I would be really really surprised, we are only 5 / 6 with access and as
far as I know no one has a problem with the company.
The last person to leave did so last year, and we revoked everything (I
hope). And I can't think of a reason they'd leave the website of a
hungarian company in there, we contacted them and they think it's one
of their ex-employee trying to cause them problems.
I think we were just unlucky, but I'd really love to confirm how they
did it

> 
> On Mon, Aug 7, 2017 at 3:30 AM, W Kern  wrote:
> 
> >
> >
> > On 8/6/2017 4:57 PM, lemonni...@ulrar.net wrote:
> >
> >
> > Gluster already uses a vlan, the problem is that there is no easy way
> > that I know of to tell gluster not to listen on an interface, and I
> > can't not have a public IP on the server. I really wish ther was a
> > simple "listen only on this IP/interface" option for this
> >
> >
> > What about this?
> >
> > transport.socket.bind-address
> >
> > I know the were some BZs on it with earlier Gluster Versions, so I assume 
> > its still there now.
> >
> > -bill
> >
> >
> >
> >
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://lists.gluster.org/mailman/listinfo/gluster-users
> >

> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users



signature.asc
Description: Digital signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Volume hacked

[Arman Khalatyan]

Interesting problem...
Did you considered an insider job?( comes to mind http://verelox.com
 recent troubles)

On Mon, Aug 7, 2017 at 3:30 AM, W Kern  wrote:

>
>
> On 8/6/2017 4:57 PM, lemonni...@ulrar.net wrote:
>
>
> Gluster already uses a vlan, the problem is that there is no easy way
> that I know of to tell gluster not to listen on an interface, and I
> can't not have a public IP on the server. I really wish ther was a
> simple "listen only on this IP/interface" option for this
>
>
> What about this?
>
> transport.socket.bind-address
>
> I know the were some BZs on it with earlier Gluster Versions, so I assume its 
> still there now.
>
> -bill
>
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Volume hacked

[W Kern]

On 8/6/2017 4:57 PM, lemonni...@ulrar.net wrote:


Gluster already uses a vlan, the problem is that there is no easy way
that I know of to tell gluster not to listen on an interface, and I
can't not have a public IP on the server. I really wish ther was a
simple "listen only on this IP/interface" option for this


What about this?

transport.socket.bind-address

I know the were some BZs on it with earlier Gluster Versions, so I assume its 
still there now.

-bill


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Volume hacked

[lemonnierk]

> You should add VLANS, and/or overlay networks and/or Mac Address 
> filtering/locking/security which raises the bar quite a bit for hackers. 
> Perhaps your provider can help you with that.
> 

Gluster already uses a vlan, the problem is that there is no easy way
that I know of to tell gluster not to listen on an interface, and I
can't not have a public IP on the server. I really wish ther was a
simple "listen only on this IP/interface" option for this

> Then there is the Gluster Auth stuff, which is cert based as I recall. 
> Unfortunately, I don't have any experience with it as we have relied on 
> unique seperate physical networks for our clusters.
> Hackers (and us) can't even get to our Gluster boxes except via IP/KVM 
> or the client itself.
> 

Well never used it, but I never thought I needed that since the vlan
gluster uses is private so outside users can't reach it. Didn't realise
NFS works with access to any one node since we don't use it.

> 
> Well if you aren't using it, then turn NFS off. I think NFS is turned 
> off by default in the new versions anyway in favor of NFS-Ganesha.

Yeah, we are still on 3.7 for now, I haven't taken the time to test
newer versions yet. Since 3.7.15 does everything we need pretty well,
not really felt the need for that.

> 
> But the original question remains, did they get into just the Gluster 
> boxes or are they in the Client already?
> 
> Unless they rooted the boxes and cleaned the logs, there should be some 
> traces of activity in the various system and gluster logs. The various 
> root kit checker programs may find something (chkrootkit)
> 

Well it's one and the same, gluster is installed on the proxmox servers
so the VM are just using localhost as their disk storage. So either they
got into the volume itself (NFS or some other way I haven't thought of),
or they got root on the hypervisors but in that case why f*ck up with
the volume instead of everything else.
Since everything else looks okay, I think they just had access to the
volume, and the only way I can think of is NFS. But I don't see anything
really suspicious in nfs.log, it seems to me like only normal glusterd
restart logs

I'll be sure to scan for rootkits tomorrow just in case, but I assume
they would have re-wiped everything if they still had access.
Googling the link they left I found a forum where some guy got his hard
drive wiped in a similar manner on his router a few days ago, it looks
like someone having fun wiping unsecured NAS .. What a great way to
spend your free time :(



signature.asc
Description: Digital signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Volume hacked

[wk]

On 8/6/2017 1:09 PM, lemonni...@ulrar.net wrote:



Are your gluster nodes physically isolated on their own network/switch?

Nope, impossible to do for us


ok, yes, that makes it much harder to secure.

You should add VLANS, and/or overlay networks and/or Mac Address 
filtering/locking/security which raises the bar quite a bit for hackers. 
Perhaps your provider can help you with that.


Then there is the Gluster Auth stuff, which is cert based as I recall. 
Unfortunately, I don't have any experience with it as we have relied on 
unique seperate physical networks for our clusters.
Hackers (and us) can't even get to our Gluster boxes except via IP/KVM 
or the client itself.


I'm now curious as to what you find and am thinking we should be looking 
at the Gluster Auth protocols as well.




In other words can an outsider access them directly without having to
compromise a NFS client machine first?


Yes, but we don't have any NFS client, only libgfapi.
I added a bunch of iptables rules to prevent that from happening, if
they did use NFS which I am unsure of. If they used something else to
access the volume though, who knows .. It hasn't been re-hacked since so
that's a good sign.


Well if you aren't using it, then turn NFS off. I think NFS is turned 
off by default in the new versions anyway in favor of NFS-Ganesha.


But the original question remains, did they get into just the Gluster 
boxes or are they in the Client already?


Unless they rooted the boxes and cleaned the logs, there should be some 
traces of activity in the various system and gluster logs. The various 
root kit checker programs may find something (chkrootkit)


-bill

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Volume hacked

[lemonnierk]

On Sun, Aug 06, 2017 at 01:01:56PM -0700, wk wrote:
> I'm not sure what you mean by saying "NFS is available by anyone"?
> 
> Are your gluster nodes physically isolated on their own network/switch?

Nope, impossible to do for us

> 
> In other words can an outsider access them directly without having to 
> compromise a NFS client machine first?
> 

Yes, but we don't have any NFS client, only libgfapi.
I added a bunch of iptables rules to prevent that from happening, if
they did use NFS which I am unsure of. If they used something else to
access the volume though, who knows .. It hasn't been re-hacked since so
that's a good sign.

> -bill
> 
> 
> On 8/6/2017 7:57 AM, lemonni...@ulrar.net wrote:
> > Hi,
> >
> > This morning one of our cluster was hacked, all the VM disks were
> > deleted and a file README.txt was left with inside just
> > "http://virtualisan.net/contactus.php :D"
> >
> > I don't speak the language but with google translete it looks like it's
> > just a webdev company or something like that, a bit surprised ..
> > In any case, we'd really like to know how that happened.
> >
> > I realised NFS is accessible by anyone (sigh), is there a way to check
> > if that is what they used ? I tried reading the nfs.log but it's not
> > really clear if someone used it or not. What do I need to look for in
> > there to see if someone mounted the volume ?
> > There are stuff in the log on one of the bricks (only one),
> > and as we aren't using NFS for that volume that in itself seems
> > suspicious.
> >
> > Thanks
> >
> >
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://lists.gluster.org/mailman/listinfo/gluster-users
> 

> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users



signature.asc
Description: Digital signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Volume hacked

[wk]

I'm not sure what you mean by saying "NFS is available by anyone"?

Are your gluster nodes physically isolated on their own network/switch?

In other words can an outsider access them directly without having to 
compromise a NFS client machine first?


-bill


On 8/6/2017 7:57 AM, lemonni...@ulrar.net wrote:

Hi,

This morning one of our cluster was hacked, all the VM disks were
deleted and a file README.txt was left with inside just
"http://virtualisan.net/contactus.php :D"

I don't speak the language but with google translete it looks like it's
just a webdev company or something like that, a bit surprised ..
In any case, we'd really like to know how that happened.

I realised NFS is accessible by anyone (sigh), is there a way to check
if that is what they used ? I tried reading the nfs.log but it's not
really clear if someone used it or not. What do I need to look for in
there to see if someone mounted the volume ?
There are stuff in the log on one of the bricks (only one),
and as we aren't using NFS for that volume that in itself seems
suspicious.

Thanks


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Volume hacked

[lemonnierk]

Thinking about it, is it even normal they managed to delete the VM disks?
Shoudn't they have gotten "file in use" errors ? Or does libgfapi not
lock the access files ?


On Sun, Aug 06, 2017 at 03:57:06PM +0100, lemonni...@ulrar.net wrote:
> Hi,
> 
> This morning one of our cluster was hacked, all the VM disks were
> deleted and a file README.txt was left with inside just
> "http://virtualisan.net/contactus.php :D"
> 
> I don't speak the language but with google translete it looks like it's
> just a webdev company or something like that, a bit surprised ..
> In any case, we'd really like to know how that happened.
> 
> I realised NFS is accessible by anyone (sigh), is there a way to check
> if that is what they used ? I tried reading the nfs.log but it's not
> really clear if someone used it or not. What do I need to look for in
> there to see if someone mounted the volume ?
> There are stuff in the log on one of the bricks (only one), 
> and as we aren't using NFS for that volume that in itself seems
> suspicious.
> 
> Thanks



> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users



signature.asc
Description: Digital signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users