Re: How to discretize Continuous Variable with Spark DataFrames

2016-01-25 Thread Joshua TAYLOR
s porpoises > > _http://www.inside-r.org/packages/cran/arules/docs/discretize > > R code for example : > > ### equal frequency > table(discretize(data$some_column, "frequency", categories=10)) > > > #k-means > table(discretize(data$some_column, "cl

Trouble dropping columns from a DataFrame that has other columns with dots in their names

2016-01-25 Thread Joshua TAYLOR
le.java:45) at SparkExample.main(SparkExample.java:54) Case 1 +---+---+ |a_b|a.c| +---+---+ | t| 2| | u| 4| +---+---+ Case 2 +---+ |a.c| +---+ | 2| | 4| +---+ Case 3 +---+ |a.c| +---+ | 2| | 4| +---+ Thanks in advance, Joshua -- Joshua Taylor, http://www.cs.rpi.edu/~tayloj/ - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org

Re: Trouble dropping columns from a DataFrame that has other columns with dots in their names

2016-01-25 Thread Joshua TAYLOR
, Michael Armbrust <mich...@databricks.com> wrote: > Looks like you found a bug. I've filed them here: > > SPARK-12987 - Drop fails when columns contain dots > SPARK-12988 - Can't drop columns that contain dots > > On Fri, Jan 22, 2016 at 3:18 PM, Joshua TAYLOR <jos

Trouble dropping columns from a DataFrame that has other columns with dots in their names

2016-01-24 Thread Joshua TAYLOR
51) at org.apache.spark.sql.DataFrame.drop(DataFrame.scala:1286) at SparkExample.lambda$0(SparkExample.java:45) at SparkExample.main(SparkExample.java:54) Case 1 +---+---+ |a_b|a.c| +---+---+ | t| 2| | u| 4| +---+---+ Case 2 +---+ |a.c| +---+ | 2| | 4| +---+ Case 3 +---+ |a.c| +---+ | 2| | 4| +---+ Thanks in advance, Joshua -- Joshua Taylor, http://www.cs.rpi.edu/~tayloj/

Trouble dropping columns from a DataFrame that has other columns with dots in their names

2016-01-22 Thread Joshua TAYLOR
ql$DataFrame$$withPlan(DataFrame.scala:2165) at org.apache.spark.sql.DataFrame.select(DataFrame.scala:751) at org.apache.spark.sql.DataFrame.drop(DataFrame.scala:1286) at SparkExample.lambda$0(SparkExample.java:45) at SparkExample.main(SparkExample.java:54) Case 1 +---+---+ |a_b|a.c| +---+---+ | t| 2| | u| 4| +---+---+ Case 2 +---+ |a.c| +---+ | 2| | 4| +---+ Case 3 +---+ |a.c| +---+ | 2| | 4| +---+ Thanks in advance, Joshua -- Joshua Taylor, http://www.cs.rpi.edu/~tayloj/

Trouble dropping columns from a DataFrame that has other columns with dots in their names

2016-01-22 Thread Joshua TAYLOR
51) at org.apache.spark.sql.DataFrame.drop(DataFrame.scala:1286) at SparkExample.lambda$0(SparkExample.java:45) at SparkExample.main(SparkExample.java:54) Case 1 +---+---+ |a_b|a.c| +---+---+ | t| 2| | u| 4| +---+---+ Case 2 +---+ |a.c| +---+ | 2| | 4| +---+ Case 3 +---+ |a.c| +---+ | 2| | 4| +---+ Thanks in advance, Joshua -- Joshua Taylor, http://www.cs.rpi.edu/~tayloj/