aosagie opened a new pull request #23813: [SPARK-26721][ML] Remove per tree feature importance normalization for gbt classifier/regressor URL: https://github.com/apache/spark/pull/23813 ## What changes were proposed in this pull request? It was discovered that scikit learn was miscalculating GBT feature importances due to mistakenly normalizing each individual tree. This fixes the issue in SparkML (which appears to have followed what scikit learn did). See: https://github.com/scikit-learn/scikit-learn/pull/11176 ## How was this patch tested? Manually tested by running the following script: ```python import pandas from sklearn.datasets import fetch_california_housing from pyspark.ml.feature import VectorAssembler from pyspark.ml.regression import GBTRegressor, RandomForestRegressor california = fetch_california_housing() pandas_df = pandas.DataFrame(california.data, columns=california.feature_names) pandas_df["label"] = pandas.Series(california.target) df = spark.createDataFrame(pandas_df) train, test = df.randomSplit([.75, .25], seed=0) train2 = VectorAssembler(inputCols=california.feature_names, outputCol="features").transform(train) gbt = GBTRegressor(seed=0, lossType="absolute", maxDepth=3) gbt_model = gbt.fit(train2) print(sorted(zip(california.feature_names, gbt_model.featureImportances), key=lambda tup: -tup[1])) #Before Change: [('Longitude', 0.2581418258949404), ('Latitude', 0.2558924988641387), ('MedInc', 0.24361394155329505), ('AveOccup', 0.11946847304433225), ('HouseAge', 0.07752951696478831), ('AveBedrms', 0.02594190009061629), ('AveRooms', 0.01941184358788898), ('Population', 0.0)] #After Change: [('MedInc', 0.40777614558392367), ('Longitude', 0.20928977828611595), ('Latitude', 0.18723522315674387), ('AveOccup', 0.12284687396572314), ('HouseAge', 0.04361683830030022), ('AveRooms', 0.020705699971547562), ('AveBedrms', 0.008529440735645566), ('Population', 0.0)] rf = RandomForestRegressor(seed=0) rf_model = rf.fit(train2) print(sorted(zip(california.feature_names, rf_model.featureImportances), key=lambda tup: -tup[1])) #Before Change: [('MedInc', 0.5960043801299608), ('AveOccup', 0.11802695085456516), ('Latitude', 0.10557829783042827), ('AveRooms', 0.08226198073251881), ('Longitude', 0.05014360511503636), ('HouseAge', 0.036513356705612766), ('AveBedrms', 0.01075420375328864), ('Population', 0.0007172248785891252)] #After Change: [('MedInc', 0.5960043801299608), ('AveOccup', 0.11802695085456516), ('Latitude', 0.10557829783042827), ('AveRooms', 0.08226198073251881), ('Longitude', 0.05014360511503636), ('HouseAge', 0.036513356705612766), ('AveBedrms', 0.01075420375328864), ('Population', 0.0007172248785891252)] ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
