[GitHub] spark pull request #14233: [SPARK-16490] [Examples] added a python example f...
Github user setjet closed the pull request at: https://github.com/apache/spark/pull/14233 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14233: [SPARK-16490] [Examples] added a python example f...
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/14233#discussion_r83560385 --- Diff: examples/src/main/python/mllib/chisq_selector_example.py --- @@ -0,0 +1,54 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from __future__ import print_function +from pyspark import SparkContext + +import numpy as np + +# $example on$ +from pyspark.mllib.regression import LabeledPoint +from pyspark.mllib.feature import ChiSqSelector +from pyspark.mllib.util import MLUtils +# $example off$ + +if __name__ == "__main__": +sc = SparkContext(appName="ChiSqSelectorExample") + +# $example on$ +# Load and parse the data file into an RDD of LabeledPoint. +data = MLUtils.loadLibSVMFile(sc, 'data/mllib/sample_libsvm_data.txt') + +# Discretize data in 16 equal bins since ChiSqSelector requires categorical features +def distributeOverBins(lp): +return np.array(map(lambda x: x % 16, lp.features.toArray())) + +# Even though features are doubles, the ChiSqSelector treats each unique value as a category +discretizedData = data.map(lambda lp: LabeledPoint(lp.label, distributeOverBins(lp))) + +# Create ChiSqSelector that will select top 50 of 692 features +selector = ChiSqSelector(numTopFeatures=50) + +# Create ChiSqSelector model (selecting features) +transformer = selector.fit(discretizedData) + +# Filter the top 50 features from each feature vector +filteredData = transformer.transform(discretizedData.map(lambda lp: lp.features)) --- End diff -- Right sorry I assumed this was ml probably because thats where we are doing the new feature work, but documenting the old mllib makes sense. Even though the PyDoc says "Applies transformation on a vector", looking at the JavaVectorTransformer base class it does take RDDs of vectors, so we shouldn't need to map it inside - indeed the PyDoc param also says `:param vector: Vector or RDD of Vector to be transformed.` Have you tried passing in an RDD of Vectors? If that doesn't work its certainly a bug we should fix, --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14233: [SPARK-16490] [Examples] added a python example f...
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/14233#discussion_r83560259 --- Diff: examples/src/main/python/mllib/chisq_selector_example.py --- @@ -0,0 +1,56 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from __future__ import print_function +from pyspark import SparkContext + +import numpy as np + +# $example on$ +from pyspark.mllib.regression import LabeledPoint +from pyspark.mllib.feature import ChiSqSelector +from pyspark.mllib.util import MLUtils +# $example off$ + +if __name__ == "__main__": +sc = SparkContext(appName="ChiSqSelectorExample") + +# $example on$ +# Load and parse the data file into an RDD of LabeledPoint. +data = MLUtils.loadLibSVMFile(sc, 'data/mllib/sample_libsvm_data.txt') --- End diff -- Oh right, sorry I was looking at ml, I think its fine to leave as is then since they are all the same (and no need to align ml / mllib examples together if they haven't been previously) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14233: [SPARK-16490] [Examples] added a python example f...
Github user rubenjanssen commented on a diff in the pull request: https://github.com/apache/spark/pull/14233#discussion_r83558453 --- Diff: examples/src/main/python/mllib/chisq_selector_example.py --- @@ -0,0 +1,54 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from __future__ import print_function +from pyspark import SparkContext + +import numpy as np + +# $example on$ +from pyspark.mllib.regression import LabeledPoint +from pyspark.mllib.feature import ChiSqSelector +from pyspark.mllib.util import MLUtils +# $example off$ + +if __name__ == "__main__": +sc = SparkContext(appName="ChiSqSelectorExample") + +# $example on$ +# Load and parse the data file into an RDD of LabeledPoint. +data = MLUtils.loadLibSVMFile(sc, 'data/mllib/sample_libsvm_data.txt') + +# Discretize data in 16 equal bins since ChiSqSelector requires categorical features +def distributeOverBins(lp): +return np.array(map(lambda x: x % 16, lp.features.toArray())) + +# Even though features are doubles, the ChiSqSelector treats each unique value as a category +discretizedData = data.map(lambda lp: LabeledPoint(lp.label, distributeOverBins(lp))) + +# Create ChiSqSelector that will select top 50 of 692 features +selector = ChiSqSelector(numTopFeatures=50) + +# Create ChiSqSelector model (selecting features) +transformer = selector.fit(discretizedData) + +# Filter the top 50 features from each feature vector +filteredData = transformer.transform(discretizedData.map(lambda lp: lp.features)) --- End diff -- That is the case in ml, but not in mllib unfortunately. As said before, the best way that would be aligned with the other examples in mllib doesn't seem to be possible as its not supported at the moment. Given that this is an issue specific to mllib and not to ml, and that ml is preferred over mllib, makes me wonder if its worth the effort to attempt resolving this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14233: [SPARK-16490] [Examples] added a python example f...
Github user sethah commented on a diff in the pull request: https://github.com/apache/spark/pull/14233#discussion_r83558440 --- Diff: examples/src/main/python/mllib/chisq_selector_example.py --- @@ -0,0 +1,56 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from __future__ import print_function +from pyspark import SparkContext + +import numpy as np + +# $example on$ +from pyspark.mllib.regression import LabeledPoint +from pyspark.mllib.feature import ChiSqSelector +from pyspark.mllib.util import MLUtils +# $example off$ + +if __name__ == "__main__": +sc = SparkContext(appName="ChiSqSelectorExample") + +# $example on$ +# Load and parse the data file into an RDD of LabeledPoint. +data = MLUtils.loadLibSVMFile(sc, 'data/mllib/sample_libsvm_data.txt') --- End diff -- I think it's fine to leave them as they are so long as we're properly demonstrating how to use the selector here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14233: [SPARK-16490] [Examples] added a python example f...
Github user rubenjanssen commented on a diff in the pull request: https://github.com/apache/spark/pull/14233#discussion_r83557977 --- Diff: examples/src/main/python/mllib/chisq_selector_example.py --- @@ -0,0 +1,56 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from __future__ import print_function +from pyspark import SparkContext + +import numpy as np + +# $example on$ +from pyspark.mllib.regression import LabeledPoint +from pyspark.mllib.feature import ChiSqSelector +from pyspark.mllib.util import MLUtils +# $example off$ + +if __name__ == "__main__": +sc = SparkContext(appName="ChiSqSelectorExample") + +# $example on$ +# Load and parse the data file into an RDD of LabeledPoint. +data = MLUtils.loadLibSVMFile(sc, 'data/mllib/sample_libsvm_data.txt') --- End diff -- Those examples are actually in ml, not mllib. This is in line with the examples in Mllib for both java and scala. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14233: [SPARK-16490] [Examples] added a python example f...
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/14233#discussion_r8355 --- Diff: examples/src/main/python/mllib/chisq_selector_example.py --- @@ -0,0 +1,54 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from __future__ import print_function +from pyspark import SparkContext + +import numpy as np + +# $example on$ +from pyspark.mllib.regression import LabeledPoint +from pyspark.mllib.feature import ChiSqSelector +from pyspark.mllib.util import MLUtils +# $example off$ + +if __name__ == "__main__": +sc = SparkContext(appName="ChiSqSelectorExample") + +# $example on$ +# Load and parse the data file into an RDD of LabeledPoint. +data = MLUtils.loadLibSVMFile(sc, 'data/mllib/sample_libsvm_data.txt') + +# Discretize data in 16 equal bins since ChiSqSelector requires categorical features +def distributeOverBins(lp): +return np.array(map(lambda x: x % 16, lp.features.toArray())) + +# Even though features are doubles, the ChiSqSelector treats each unique value as a category +discretizedData = data.map(lambda lp: LabeledPoint(lp.label, distributeOverBins(lp))) + +# Create ChiSqSelector that will select top 50 of 692 features +selector = ChiSqSelector(numTopFeatures=50) + +# Create ChiSqSelector model (selecting features) +transformer = selector.fit(discretizedData) + +# Filter the top 50 features from each feature vector +filteredData = transformer.transform(discretizedData.map(lambda lp: lp.features)) --- End diff -- The transform function for `ChiSqSelectorModel` in Python can directly take in a Dataset/DataFrame - (see ChiSqSelectorModel in http://spark.apache.org/docs/latest/api/python/pyspark.ml.html#module-pyspark.ml.feature and the doctests of `ChiSqSelector`). That being said it takes a column of vectors rather than a column of labeled points (the doctests illustrate this as does the scala example in `ChiSqSelectorExample.scala`). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14233: [SPARK-16490] [Examples] added a python example f...
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/14233#discussion_r83556749 --- Diff: examples/src/main/python/mllib/chisq_selector_example.py --- @@ -0,0 +1,56 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from __future__ import print_function +from pyspark import SparkContext + +import numpy as np + +# $example on$ +from pyspark.mllib.regression import LabeledPoint +from pyspark.mllib.feature import ChiSqSelector +from pyspark.mllib.util import MLUtils +# $example off$ + +if __name__ == "__main__": +sc = SparkContext(appName="ChiSqSelectorExample") + +# $example on$ +# Load and parse the data file into an RDD of LabeledPoint. +data = MLUtils.loadLibSVMFile(sc, 'data/mllib/sample_libsvm_data.txt') --- End diff -- It might make more sense (for consistency) to use the same inputs as in `ChiSqSelectorExample.scala` and `JavaChiSqSelectorExample.java` where by its just a few rows created in the example its self rather than loaded from disk. This way you can skip the discretization step and have the example focus on just ChiSqSelector's use. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14233: [SPARK-16490] [Examples] added a python example f...
Github user rubenjanssen commented on a diff in the pull request: https://github.com/apache/spark/pull/14233#discussion_r83552931 --- Diff: examples/src/main/python/mllib/chisq_selector_example.py --- @@ -0,0 +1,54 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from __future__ import print_function +from pyspark import SparkContext + +import numpy as np + +# $example on$ +from pyspark.mllib.regression import LabeledPoint +from pyspark.mllib.feature import ChiSqSelector +from pyspark.mllib.util import MLUtils +# $example off$ + +if __name__ == "__main__": +sc = SparkContext(appName="ChiSqSelectorExample") + +# $example on$ +# Load and parse the data file into an RDD of LabeledPoint. +data = MLUtils.loadLibSVMFile(sc, 'data/mllib/sample_libsvm_data.txt') + +# Discretize data in 16 equal bins since ChiSqSelector requires categorical features +def distributeOverBins(lp): +return np.array(map(lambda x: x % 16, lp.features.toArray())) + +# Even though features are doubles, the ChiSqSelector treats each unique value as a category +discretizedData = data.map(lambda lp: LabeledPoint(lp.label, distributeOverBins(lp))) + +# Create ChiSqSelector that will select top 50 of 692 features +selector = ChiSqSelector(numTopFeatures=50) + +# Create ChiSqSelector model (selecting features) +transformer = selector.fit(discretizedData) + +# Filter the top 50 features from each feature vector +filteredData = transformer.transform(discretizedData.map(lambda lp: lp.features)) --- End diff -- Could you please elaborate how we can do that in a nice way? filteredData = discretizedData.map(lambda lp: LabeledPoint(lp.label, transformer.transform(lp.features))) is not possible as it will throw a nested RDD exception. This is unfortunately not supported in python right now: (in feature.py) 'Note: In Python, transform cannot currently be used within an RDD transformation or action. Call transform directly on the RDD instead. ' --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14233: [SPARK-16490] [Examples] added a python example f...
Github user sethah commented on a diff in the pull request: https://github.com/apache/spark/pull/14233#discussion_r81648607 --- Diff: examples/src/main/python/mllib/chisq_selector_example.py --- @@ -0,0 +1,54 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from __future__ import print_function +from pyspark import SparkContext + +import numpy as np + +# $example on$ +from pyspark.mllib.regression import LabeledPoint +from pyspark.mllib.feature import ChiSqSelector +from pyspark.mllib.util import MLUtils +# $example off$ + +if __name__ == "__main__": +sc = SparkContext(appName="ChiSqSelectorExample") + +# $example on$ +# Load and parse the data file into an RDD of LabeledPoint. +data = MLUtils.loadLibSVMFile(sc, 'data/mllib/sample_libsvm_data.txt') + +# Discretize data in 16 equal bins since ChiSqSelector requires categorical features +def distributeOverBins(lp): +return np.array(map(lambda x: x % 16, lp.features.toArray())) --- End diff -- also, this can be `np.floor(lp.features.toArray() / 16)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14233: [SPARK-16490] [Examples] added a python example f...
Github user sethah commented on a diff in the pull request: https://github.com/apache/spark/pull/14233#discussion_r81652253 --- Diff: examples/src/main/python/mllib/chisq_selector_example.py --- @@ -0,0 +1,54 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from __future__ import print_function +from pyspark import SparkContext + +import numpy as np + +# $example on$ +from pyspark.mllib.regression import LabeledPoint +from pyspark.mllib.feature import ChiSqSelector +from pyspark.mllib.util import MLUtils +# $example off$ + +if __name__ == "__main__": +sc = SparkContext(appName="ChiSqSelectorExample") + +# $example on$ +# Load and parse the data file into an RDD of LabeledPoint. +data = MLUtils.loadLibSVMFile(sc, 'data/mllib/sample_libsvm_data.txt') + +# Discretize data in 16 equal bins since ChiSqSelector requires categorical features +def distributeOverBins(lp): +return np.array(map(lambda x: x % 16, lp.features.toArray())) + +# Even though features are doubles, the ChiSqSelector treats each unique value as a category +discretizedData = data.map(lambda lp: LabeledPoint(lp.label, distributeOverBins(lp))) + +# Create ChiSqSelector that will select top 50 of 692 features +selector = ChiSqSelector(numTopFeatures=50) + +# Create ChiSqSelector model (selecting features) +transformer = selector.fit(discretizedData) + +# Filter the top 50 features from each feature vector +filteredData = transformer.transform(discretizedData.map(lambda lp: lp.features)) --- End diff -- let's make this an RDD of `LabeledPoint` to match Scala. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14233: [SPARK-16490] [Examples] added a python example f...
Github user sethah commented on a diff in the pull request: https://github.com/apache/spark/pull/14233#discussion_r81647984 --- Diff: examples/src/main/python/mllib/chisq_selector_example.py --- @@ -0,0 +1,54 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from __future__ import print_function +from pyspark import SparkContext + +import numpy as np + +# $example on$ +from pyspark.mllib.regression import LabeledPoint +from pyspark.mllib.feature import ChiSqSelector +from pyspark.mllib.util import MLUtils +# $example off$ + +if __name__ == "__main__": +sc = SparkContext(appName="ChiSqSelectorExample") + +# $example on$ +# Load and parse the data file into an RDD of LabeledPoint. +data = MLUtils.loadLibSVMFile(sc, 'data/mllib/sample_libsvm_data.txt') + +# Discretize data in 16 equal bins since ChiSqSelector requires categorical features +def distributeOverBins(lp): +return np.array(map(lambda x: x % 16, lp.features.toArray())) --- End diff -- `x % 16` is not the same as `np.floor(x // 16)` or `(x / 16).floor` in scala. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
