Re: [Offtopic] Line fitting [was Re: Numpy outlier removal]

2013-01-11 Thread Alan Spence
On 09 Jan 2013, at 00:02:11 Steven D'Aprano st...@pearwood.info wrote: The point I keep making, that everybody seems to be ignoring, is that eyeballing a line of best fit is subjective, unreliable and impossible to verify. How could I check that the line you say is the best fit actually

Re: [Offtopic] Line fitting [was Re: Numpy outlier removal]

2013-01-08 Thread Terry Reedy
On 1/7/2013 8:23 PM, Steven D'Aprano wrote: On Mon, 07 Jan 2013 22:32:54 +, Oscar Benjamin wrote: An example: Earlier today I was looking at some experimental data. A simple model of the process underlying the experiment suggests that two variables x and y will vary in direct proportion to

Re: [Offtopic] Line fitting [was Re: Numpy outlier removal]

2013-01-08 Thread Oscar Benjamin
On 8 January 2013 01:23, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: On Mon, 07 Jan 2013 22:32:54 +, Oscar Benjamin wrote: [...] I also think it would be highly foolish to go so far with refusing to eyeball data that you would accept the output of some regression

Re: [Offtopic] Line fitting [was Re: Numpy outlier removal]

2013-01-08 Thread Robert Kern
On 08/01/2013 06:35, Chris Angelico wrote: On Tue, Jan 8, 2013 at 1:06 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: given that weather patterns have been known to follow cycles at least that long. That is not a given. Weather patterns don't last for thirty years. Perhaps

Re: [Offtopic] Line fitting [was Re: Numpy outlier removal]

2013-01-08 Thread Maarten
On Tuesday, January 8, 2013 10:07:08 AM UTC+1, Terry Reedy wrote: With the line constrained to go through 0,0, a line eyeballed with a clear ruler could easily be better than either regression line, as a human will tend to minimize the deviations *perpendicular to the line*, which is the

Re: [Offtopic] Line fitting [was Re: Numpy outlier removal]

2013-01-08 Thread Chris Angelico
On Wed, Jan 9, 2013 at 2:55 AM, Robert Kern robert.k...@gmail.com wrote: On 08/01/2013 06:35, Chris Angelico wrote: ... it looks quite significant to show a line going from the bottom of the graph to the top, but sounds a lot less noteworthy when you see it as a half-degree increase on about

Re: [Offtopic] Line fitting [was Re: Numpy outlier removal]

2013-01-08 Thread Robert Kern
On 08/01/2013 20:14, Chris Angelico wrote: On Wed, Jan 9, 2013 at 2:55 AM, Robert Kern robert.k...@gmail.com wrote: On 08/01/2013 06:35, Chris Angelico wrote: ... it looks quite significant to show a line going from the bottom of the graph to the top, but sounds a lot less noteworthy when you

Re: [Offtopic] Line fitting [was Re: Numpy outlier removal]

2013-01-08 Thread Steven D'Aprano
On Tue, 08 Jan 2013 04:07:08 -0500, Terry Reedy wrote: But that is not fitting a line by eye, which is what I am talking about. With the line constrained to go through 0,0 a line eyeballed with a clear ruler could easily be better than either regression line, as a human will tend to

Re: [Offtopic] Line fitting [was Re: Numpy outlier removal]

2013-01-08 Thread Jason Friedman
Statistical analysis is a huge science. So is lying. And I'm not sure most people can pick one from the other. Chris, your sentence causes me to think of Mr. Twain's sentence, or at least the one he popularized: http://www.twainquotes.com/Statistics.html. --

Re: [Offtopic] Line fitting [was Re: Numpy outlier removal]

2013-01-08 Thread Jason Friedman
Statistical analysis is a huge science. So is lying. And I'm not sure most people can pick one from the other. Chris, your sentence causes me to think of Mr. Twain's sentence, or at least the one he popularized: http://www.twainquotes.com/Statistics.html. --

Re: [Offtopic] Line fitting [was Re: Numpy outlier removal]

2013-01-08 Thread Steven D'Aprano
On Wed, 09 Jan 2013 07:14:51 +1100, Chris Angelico wrote: Three types of lies. Oh, surely more than that. White lies. Regular or garden variety lies. Malicious lies. Accidental or innocent lies. FUD -- fear, uncertainty, doubt. Half-truths. Lying by omission. Exaggeration and

[Offtopic] Line fitting [was Re: Numpy outlier removal]

2013-01-07 Thread Steven D'Aprano
On Mon, 07 Jan 2013 15:20:57 +, Oscar Benjamin wrote: There are sometimes good reasons to get a line of best fit by eye. In particular if your data contains clusters that are hard to separate, sometimes it's useful to just pick out roughly where you think a line through a subset of the

Re: [Offtopic] Line fitting [was Re: Numpy outlier removal]

2013-01-07 Thread Chris Angelico
On Tue, Jan 8, 2013 at 4:58 AM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: Anyone can fool themselves into placing a line through a subset of non- linear data. Or, sadly more often, *deliberately* cherry picking fake clusters in order to fool others. Here is a real world

Re: [Offtopic] Line fitting [was Re: Numpy outlier removal]

2013-01-07 Thread Oscar Benjamin
On 7 January 2013 17:58, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: On Mon, 07 Jan 2013 15:20:57 +, Oscar Benjamin wrote: There are sometimes good reasons to get a line of best fit by eye. In particular if your data contains clusters that are hard to separate, sometimes

Re: [Offtopic] Line fitting [was Re: Numpy outlier removal]

2013-01-07 Thread Steven D'Aprano
On Mon, 07 Jan 2013 22:32:54 +, Oscar Benjamin wrote: An example: Earlier today I was looking at some experimental data. A simple model of the process underlying the experiment suggests that two variables x and y will vary in direct proportion to one another and the data broadly reflects

Re: [Offtopic] Line fitting [was Re: Numpy outlier removal]

2013-01-07 Thread Steven D'Aprano
On Tue, 08 Jan 2013 06:43:46 +1100, Chris Angelico wrote: On Tue, Jan 8, 2013 at 4:58 AM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: Anyone can fool themselves into placing a line through a subset of non- linear data. Or, sadly more often, *deliberately* cherry picking fake

Re: [Offtopic] Line fitting [was Re: Numpy outlier removal]

2013-01-07 Thread Chris Angelico
On Tue, Jan 8, 2013 at 1:06 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: given that weather patterns have been known to follow cycles at least that long. That is not a given. Weather patterns don't last for thirty years. Perhaps you are talking about climate patterns? Yes,