unsubscribe

[glen]

unsubscribe




--

Jacky Wang

Re: Spark MLlib:Collaborative Filtering

[glen]

Hash it to int




On 2016-08-24 16:28 , Devi P.V Wrote:


Hi all,
I am newbie in collaborative filtering.I want to implement collaborative 
filtering algorithm(need to find top 10 recommended products) using Spark and 
Scala.I have a rating dataset where userID & ProductID are String type.

UserID   ProductID Rating
b3a68043-c1  p1-160ff5fDS-f74   1
b3a68043-c2  p5-160ff5fDS-f74   1
b3a68043-c0  p9-160ff5fDS-f74   1


I tried ALS algorithm using spark MLlib.But it support rating userID & 
productID only Integer type.How can I solve this problem?


Thanks In Advance

how to debug spark app?

[glen]

Any tool like gdb? Which support break point at some line or some function?

Ratings in mllib.recommendation

[glen]

Re: how to set database in DataFrame.saveAsTable?

[Glen]

Any example code?

In pyspark:
sqlContex.sql("use mytable")
my_df.saveAsTable("tmp_spark_debug", mode="overwrite")

1. The code above seems not register the table in hive. I have to create
table from hdfs in hive, it reports some format error: rcformat and parquet.
2. Rerun the saveAsTable using  mode="overwrite" in saveAsTable, it reports
the table already exists.
3. Sometimes it creates a directory in  hive/warehouse/tmp_spark_debug, not
in hive/warehouse/mytable/tmp_spark_debug.


My goal is simple:
df.saveAsTable('blablabla')  // create a hive table in some database, then
it can be visited by hive.

I tried lots of time, it seems there are lots of bug in pyspark. Or my
mehtod is wrong?

2016-02-21 10:04 GMT+08:00 gen tang <gen.tan...@gmail.com>:

> Hi,
>
> You can use
> sqlContext.sql("use ")
> before use dataframe.saveAsTable
>
> Hope it could be helpful
>
> Cheers
> Gen
>
>
> On Sun, Feb 21, 2016 at 9:55 AM, Glen <cng...@gmail.com> wrote:
>
>> For dataframe in spark, so the table can be visited by hive.
>>
>> --
>> Jacky Wang
>>
>
>


-- 
Jacky Wang

how to set database in DataFrame.saveAsTable?

[Glen]

For dataframe in spark, so the table can be visited by hive.

-- 
Jacky Wang

java.io.IOException: failure to login

[glen]

.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: javax.security.auth.login.LoginException: Can't find user name
at
org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:197)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:784)
at
javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
at javax.security.auth.login.LoginContext$5.run(LoginContext.java:721)
at javax.security.auth.login.LoginContext$5.run(LoginContext.java:719)
at java.security.AccessController.doPrivileged(Native Method)
at
javax.security.auth.login.LoginContext.invokeCreatorPriv(LoginContext.java:718)
at javax.security.auth.login.LoginContext.login(LoginContext.java:591)
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:771)
... 38 more
The versions of the various dependencies are shown below

hadoop-client 2.6
apache spark 1.4.0 / 1.4.1
junit 4.12
easy mock 3.31
power mock 1.6.2
I've tried this with various versions of Spark. The above test passes with
the following versions of Spark

1.1.1
1.2.2
It starts failing from Spark 1.3.0 onwards.

Any ideas what I need to change to get this to work?


Thanks

Glen



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/java-io-IOException-failure-to-login-tp24026.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org