Re: [Ganglia-developers] patches for: [Sec] Gmetadserver BoFandnetwork overload + [Feature] multiple requestsper connoninteractive port
On Mon, Jan 19, 2009 at 5:44 AM, Carlo Marcelo Arenas Belon
wrote:
> agree, but that is to be done in the context of getting "multi-patch"
> committed and backported, but not in fixing this buffer overflow in the
> interactive port, which is what BUG223 is about.
ok, guess I'll start a different thread about this later on once we're
worked out #223
> from what I check while trying
> some fuzzing we have still a problem (probably introduced with the
> buffer overflow patch) when the request is too long (over 2048 bytes) as
> shown by :
>
> $ echo "/`python -c \"print \\"%s/%s/%s\\" % ('a'*1700,'b'*300,'c'*48)\"`" |
> netcat 127.0.0.1 8652
what problem are you seeing? trunk (r1950) does not reflect what we're
talking about as it includes my original return 1 if element is not
found which leads to the truncated xml output. Reverting to 1233 and
applying the latest patch from #223 works fine for me and I get back
the entire tree as there's no a*1700 grid.
--
"Behind every great man there's a great backpack" - B.
--
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
___
Ganglia-developers mailing list
Ganglia-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] patches for: [Sec] Gmetadserver BoFandnetwork overload + [Feature] multiple requestsper connoninteractive port
On Sun, Jan 18, 2009 at 11:54:14PM +0800, Spike Spiegel wrote:
> On Sun, Jan 18, 2009 at 10:27 PM, Carlo Marcelo Arenas Belon
> wrote:
> > http://bugzilla.ganglia.info/cgi-bin/bugzilla/attachment.cgi?id=188&action=view
> > that should apply cleanly to 3.1.1, 3.0.7 and 2.5.7
>
> Looks fine to me, altho I'd argue that without the other changes
> supporting the multi-patch becomes a risk for reasons I already
> explained.
since the "multi-patch" is not yet committed it is not an issue yet
and considering that the buffer overflow will need to be backported up
to 2.5.7 (which will never had any features added as it is already out of
maintenance), decoupling the changes this way is recommended.
> I'd feel much better if we agreed on a solution to
> propagate errors back to the client and implement that alongside with
> not returning the entire tree.
agree, but that is to be done in the context of getting "multi-patch"
committed and backported, but not in fixing this buffer overflow in the
interactive port, which is what BUG223 is about.
> >> Is that what you meant when you said "banging to resulting binary"?
> >
> > partially; scripts would be able to help only after the testing
> > parameters had been defined, and at least for this test might be limited
> > by the fact that the interactive port is mainly used by the web
> > frontend.
>
> I'm not sure I follow. Especially in this case, but also true in
> general, as far as testing gmetad goes you "only" need to make sure
> that:
> 1) it can pull data
> 2) it can store data
> 3) it can serve data
3a) non interactively (usually through TCP/8651)
3b) interactively (usually through TCP/8652)
4) it can summarize data (this requires at least another gmetad)
> with a gmond running and that you can point gmetad at you need 3
> scripts to test the above:
> 1) one script to generate udp traffic to send to gmond
gmetric could help here
> 2) one script to read the data off gmetad
> 3) one script to read the data off of the rrd files
>
> the script that generates the synthetic data can be completely random
> as long as it has a list of valid elements and range values, which you
> also use to test for errors by futzing with those parameters. The same
> auto generated data should be in output to the second script. Given
> that you control sampling and polling rate I see this quite feasible
> and a good integration test. Load testing would be different as with
> lots of rapid reads and writes it would be harder to ensure what
> you're supposed to be seeing.
>
> am I missing something obvious?
for this problem you need only the scripts from 3b, but you first need to
define which cases you are looking for and that are considered valid so
that a script can help validate them and from what I check while trying
some fuzzing we have still a problem (probably introduced with the
buffer overflow patch) when the request is too long (over 2048 bytes) as
shown by :
$ echo "/`python -c \"print \\"%s/%s/%s\\" % ('a'*1700,'b'*300,'c'*48)\"`" |
netcat 127.0.0.1 8652
Carlo
--
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
___
Ganglia-developers mailing list
Ganglia-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] patches for: [Sec] Gmetadserver BoFandnetwork overload + [Feature] multiple requestsper connoninteractive port
On Sun, Jan 18, 2009 at 10:27 PM, Carlo Marcelo Arenas Belon wrote: > http://bugzilla.ganglia.info/cgi-bin/bugzilla/attachment.cgi?id=188&action=view > that should apply cleanly to 3.1.1, 3.0.7 and 2.5.7 Looks fine to me, altho I'd argue that without the other changes supporting the multi-patch becomes a risk for reasons I already explained. I'd feel much better if we agreed on a solution to propagate errors back to the client and implement that alongside with not returning the entire tree. >> Is that what you meant when you said "banging to resulting binary"? > > partially; scripts would be able to help only after the testing > parameters had been defined, and at least for this test might be limited > by the fact that the interactive port is mainly used by the web > frontend. I'm not sure I follow. Especially in this case, but also true in general, as far as testing gmetad goes you "only" need to make sure that: 1) it can pull data 2) it can store data 3) it can serve data with a gmond running and that you can point gmetad at you need 3 scripts to test the above: 1) one script to generate udp traffic to send to gmond 2) one script to read the data off gmetad 3) one script to read the data off of the rrd files the script that generates the synthetic data can be completely random as long as it has a list of valid elements and range values, which you also use to test for errors by futzing with those parameters. The same auto generated data should be in output to the second script. Given that you control sampling and polling rate I see this quite feasible and a good integration test. Load testing would be different as with lots of rapid reads and writes it would be harder to ensure what you're supposed to be seeing. am I missing something obvious? -- "Behind every great man there's a great backpack" - B. -- This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] patches for: [Sec] Gmetadserver BoFandnetwork overload + [Feature] multiple requestsper connoninteractive port
On Sun, Jan 18, 2009 at 09:53:32PM +0800, Spike Spiegel wrote: > On Sun, Jan 18, 2009 at 7:35 PM, Carlo Marcelo Arenas Belon > wrote: > >> other than that looks good to me. > > > > could you check the "simplified" one?, this problem was introduced in > > 2003 and therefore affects all versions of ganglia since then (including > > 2.5.7 which is not supported anymore and that will need to be patched by > > the users of it which include Debian/Ubuntu, Novell/OpenSuSE and > > probably others). > > apologies but I lost you there, what do you mean with the "simplified" one? http://bugzilla.ganglia.info/cgi-bin/bugzilla/attachment.cgi?id=188&action=view that should apply cleanly to 3.1.1, 3.0.7 and 2.5.7 > Is that what you meant when you said "banging to resulting binary"? partially; scripts would be able to help only after the testing parameters had been defined, and at least for this test might be limited by the fact that the interactive port is mainly used by the web frontend. Carlo -- This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] patches for: [Sec] Gmetad server BoF and network overload + [Feature] multiple requests per conn on interactive port
On Sun, Jan 18, 2009 at 9:08 PM, Carlo Marcelo Arenas Belon wrote: > the proposed solution will result in a truncated XML which then will fail to > be parsed in the client and in an obscure error like "unable to write > XML tree info". > > agree that returning the whole tree isn't the best way to signal a > syntax error, but returning a truncated XML will be more difficult to > handle in the client side as depending on the implementation used it > will fail to even load with an exception. > > because the connection to the client is getting severed when it is > malformed it will also show strange errors like "unable to write root > preamble (DTD, etc)" or "Connection reset by peer" in the client. That is true, but afaics not a problem introduced by this patch. In the original version if by any reason process_path failed you'd log the msg, close the fd and go back to sleep, hence sending back an invalid xml since the DTD + root_report_start ran already but root_report_end won't. I agree that some kind of problem with process_path itself is a different case and less frequent than a typo, tho. Anyway, I'm absolutely for improving error propagation to the client, what do you suggest? Return an empty but valid tree? Or should an explicit error message be sent? In which case would it be possible to have an ELEMENT for that? thanks -- "Behind every great man there's a great backpack" - B. -- This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] patches for: [Sec] Gmetadserver BoFandnetwork overload + [Feature] multiple requestsper connoninteractive port
On Sun, Jan 18, 2009 at 7:35 PM, Carlo Marcelo Arenas Belon wrote: >> other than that looks good to me. > > could you check the "simplified" one?, this problem was introduced in > 2003 and therefore affects all versions of ganglia since then (including > 2.5.7 which is not supported anymore and that will need to be patched by > the users of it which include Debian/Ubuntu, Novell/OpenSuSE and > probably others). apologies but I lost you there, what do you mean with the "simplified" one? >> Two things: >> 1) How has this been tested? I did some myself and got to wonder how >> you guys did it, do you have any standardized approach? > > sadly there is no test suite associated with ganglia code and therefore > there is no "standardized" approach other than applying the patch and > banging the resulting binary to see if it works reliably. alright, I was thinking of a couple scripts to generate traffic and then do the queries, I think Jesse mentioned something like that on irc based on gmetric. I believe something like that would be useful, and either python of perl could be enough to write something threaded to generate enough load for testing I guess. Is that what you meant when you said "banging to resulting binary"? -- "Behind every great man there's a great backpack" - B. -- This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] patches for: [Sec] Gmetad server BoF and network overload + [Feature] multiple requests per conn on interactive port
On Tue, Jan 13, 2009 at 11:41:19PM +0800, Spike Spiegel wrote:
>
> === DoS attacks
> 1) Given REQUESTLEN=2048, and 3 characters to be the minimum to craft a valid
> and nonexistent path "/x", with the above feature implemented it would be
> possible to trigger 2048/3 calls to process_path which would possibly lead to
> CPU overload.
this is not handled by any of the provided patches but since processing
is aborted as soon as the path is considered invalid the depth of the
path is not relevant for CPU or bandwith utilization
> 2) extension to 1) - as it is ganglia returns the entire tree if an element is
> not found. with large trees 2048/3 requests could easily result in several GBs
> of data being transferred. Related to this if you look at gmetad/server.c
> lines
> 601:606 you'll see this:
> err_msg("Got a malformed path request from %s",
> remote_ip);
> /* Send them the entire tree to discourage attacks. */
> strcpy(request, "/");
> which leads to the same scenario as above.
the amount of data returned is not dependent on the depth of the path
because it will always be the full XML tree (once).
> What I propose is that for both cases, malformed request and non existent
> items, we log an error and bail out. This would solve 2) and most of 1) making
> the call possibly exist far quicker.
the proposed solution will result in a truncated XML which then will fail to
be parsed in the client and in an obscure error like "unable to write
XML tree info".
agree that returning the whole tree isn't the best way to signal a
syntax error, but returning a truncated XML will be more difficult to
handle in the client side as depending on the implementation used it
will fail to even load with an exception.
because the connection to the client is getting severed when it is
malformed it will also show strange errors like "unable to write root
preamble (DTD, etc)" or "Connection reset by peer" in the client.
Carlo
--
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
___
Ganglia-developers mailing list
Ganglia-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-developers
Re: [Ganglia-developers] patches for: [Sec] Gmetadserver BoFandnetwork overload + [Feature] multiple requestsper connoninteractive port
On Sun, Jan 18, 2009 at 11:22:27AM +0800, Spike Spiegel wrote: > > the comment should be removed since the +1 is there: > > + /* +1 not needed as q-p is already accounting for that */ > + element = malloc(len + 1); Committed revision 1950 > other than that looks good to me. could you check the "simplified" one?, this problem was introduced in 2003 and therefore affects all versions of ganglia since then (including 2.5.7 which is not supported anymore and that will need to be patched by the users of it which include Debian/Ubuntu, Novell/OpenSuSE and probably others). > Two things: > 1) How has this been tested? I did some myself and got to wonder how > you guys did it, do you have any standardized approach? sadly there is no test suite associated with ganglia code and therefore there is no "standardized" approach other than applying the patch and banging the resulting binary to see if it works reliably. > 2) you mention backports to 3.1 and then move on to 3.1.2, what about > 3.0? Some of us (quite a few?) are still running 3.0 and afaik kostas > already applied the patch to that branch and ran some tests (and so > did I - and server.c hasn't changed for a long time so it should be > indeed a safe operation) it will be included in 3.0.8 as well. Carlo -- This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword ___ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
