Re: redundant decision tree model

Hello, a small recap for who is interested. There was already a ticket covering the case that I failed to find when I checked. As a result the other one has been correctly marked as duplicate: https://issues.apache.org/jira/browse/SPARK-3159 I have created a PR for this that you can check here

Re: redundant decision tree model

Thanks for your feedback Sean, I agree with you. I have logged a JIRA case (https://issues.apache.org/jira/browse/SPARK-23409), I will take a look at the code more in detail and see if I come up with a PR to handle this. On 13 February 2018 at 12:00, Sean Owen wrote: > I

Re: redundant decision tree model

I think the simple pruning you have in mind was just never implemented. That sort of pruning wouldn't help much if the nodes maintained a distribution over classes, as those are rarely identical, but, they just maintain a single class prediction. After training, I see no value in keeping those

Re: redundant decision tree model

Hello Nick, thanks for the pointer, that's interesting. However, there seems to be a major difference with what I was discussing. The JIRA issue relates to overfitting and consideration on information gain, while what I propose is a much simpler "syntactic" pruning. Consider a fragment of the

Re: redundant decision tree model

There is a long outstanding JIRA issue about it: https://issues.apache.org/jira/browse/SPARK-3155. It is probably still a useful feature to have for trees but the priority is not that high since it may not be that useful for the tree ensemble models. On Tue, 13 Feb 2018 at 11:52 Alessandro