Re: How to implement top() and filter() on object List for JavaRDD
_sorted_users.first() >>>> >>>> 2015-07-07 16:54 GMT+02:00 Hafsa Asif [via Apache Spark User List] <[hidden >>>> email] <http:///user/SendEmail.jtp?type=node&node=23696&i=2>>: >>>> >>>>> Rusty, >>>>> >>>>> I am very thankful for your help. Actually, I am facing difficulty in >>>>> objects. My plan is that, I have an object list containing list of User >>>>> objects. After parallelizing it through spark context, I apply comparator >>>>> on user.getUserName(). As usernames are sorted, their related user object >>>>> are sorted according to user names. >>>>> In the end when I apply top, I get the whole object of user . >>>>> >>>>> Some like this: >>>>> >>>>> public static Comparator UserComparator >>>>> = new Comparator() { >>>>> >>>>> public int compare(User usr1, User usr2) { >>>>> String userName1 = usr1.getUserName().toUpperCase(); >>>>> String userName2 = usr1.getUserName().toUpperCase(); >>>>> >>>>> //ascending order >>>>> return userName1.compareTo(userName2); >>>>> >>>>> //descending order >>>>> //return fruitName2.compareTo(fruitName1); >>>>> } >>>>> >>>>> }; >>>>> >>>>> JavaRDD rdd = context.parallelize(usersList); >>>>> >>>>> 2015-07-07 16:05 GMT+02:00 rusty [via Apache Spark User List] <[hidden >>>>> email] <http:///user/SendEmail.jtp?type=node&node=23688&i=0>>: >>>>> >>>>>> JavaRDD lines2 = ctx.parallelize(Arrays.asList("3", "6", "2", >>>>>> "5", "8", "6", "7")); >>>>>> List top =lines2.top(7, new >>>>>> CustomComaprator()); >>>>>> for (String integer : top) { >>>>>> System.out.println(integer); >>>>>> } >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> class CustomComaprator implements Serializable, Comparator { >>>>>> /** >>>>>> * >>>>>> */ >>>>>> public CustomComaprator() { >>>>>> // TODO Auto-generated constructor stub >>>>>> >>>>>> } >>>>>> >>>>>> private static final long serialVersionUID = >>>>>> 2004092520677431781L; >>>>>> >>>>>> @Override >>>>>> public int compare(T o11, T o12) { >>>>>> int o1 = Integer.parseInt(String.valueOf(o11)); >>>>>> int o2 = Integer.parseInt(String.valueOf(o12)); >>>>>> >>>>>> return o1 > o2 ? 1 : o1 == o2 ? 0 : -1; >>>>>> } >>>>>> >>>>>> } >>>>>> >>>>>> >>>>>> -- >>>>>> If you reply to this email, your message will be added to the >>>>>> discussion below: >>>>>> >>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-implement-top-and-filter-on-object-List-for-JavaRDD-tp23669p23684.html >>>>>> To unsubscribe from How to implement top() and filter() on object >>>>>> List for JavaRDD, click here. >>>>>> NAML >>>>>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>>>>> >>>>> >>>>> >>>>> >>>>> ---------- >>>>> If you reply to this email, your message will be added to the >>>>> discussion below: >>>>> >>>>> http://apache
Re: How to implement top() and filter() on object List for JavaRDD
String userName2 = usr1.getUserName().toUpperCase(); >>>> >>>> //ascending order >>>> return userName1.compareTo(userName2); >>>> >>>> //descending order >>>> //return fruitName2.compareTo(fruitName1); >>>> } >>>> >>>> }; >>>> >>>> JavaRDD rdd = context.parallelize(usersList); >>>> >>>> 2015-07-07 16:05 GMT+02:00 rusty [via Apache Spark User List] <[hidden >>>> email] <http:///user/SendEmail.jtp?type=node&node=23688&i=0>>: >>>> >>>>> JavaRDD lines2 = ctx.parallelize(Arrays.asList("3", "6", "2", >>>>> "5", "8", "6", "7")); >>>>> List top =lines2.top(7, new >>>>> CustomComaprator()); >>>>> for (String integer : top) { >>>>> System.out.println(integer); >>>>> } >>>>> >>>>> >>>>> >>>>> >>>>> class CustomComaprator implements Serializable, Comparator { >>>>> /** >>>>> * >>>>> */ >>>>> public CustomComaprator() { >>>>> // TODO Auto-generated constructor stub >>>>> >>>>> } >>>>> >>>>> private static final long serialVersionUID = >>>>> 2004092520677431781L; >>>>> >>>>> @Override >>>>> public int compare(T o11, T o12) { >>>>> int o1 = Integer.parseInt(String.valueOf(o11)); >>>>> int o2 = Integer.parseInt(String.valueOf(o12)); >>>>> >>>>> return o1 > o2 ? 1 : o1 == o2 ? 0 : -1; >>>>> } >>>>> >>>>> } >>>>> >>>>> >>>>> -- >>>>> If you reply to this email, your message will be added to the >>>>> discussion below: >>>>> >>>>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-implement-top-and-filter-on-object-List-for-JavaRDD-tp23669p23684.html >>>>> To unsubscribe from How to implement top() and filter() on object >>>>> List for JavaRDD, click here. >>>>> NAML >>>>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>>>> >>>> >>>> >>>> >>>> -- >>>> If you reply to this email, your message will be added to the >>>> discussion below: >>>> >>>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-implement-top-and-filter-on-object-List-for-JavaRDD-tp23669p23688.html >>>> To unsubscribe from How to implement top() and filter() on object List >>>> for JavaRDD, click here. >>>> NAML >>>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>>> >>> >>> >>> >>> -- >>> Bien cordialement, >>> >>> >>> *-* >>> *Brahim OUBRIK* >>> *Elève-Ingénieur en Informatique à Polytech Paris Sud.* >>> *Cell +33 0651137644 <%2B33%200651137644>* >>> >>> >>> *---------------------------------* >>> >> >> >> >> -- >> Bien cordialement, >> >> >> *-* >> *Brahim OUBRIK* >> *Elève-Ingénieur en Informatique à Polytech Paris Sud.* >> *Cell +33 0651137644 <%2B33%200651137644>* >> >> >> *-* >> > > > > -- > Bien cordialement, > > > *-* > *Brahim OUBRIK* > *Elève-Ingénieur en Informatique à Polytech Paris Sud.* > *Cell +33 0651137644 <%2B33%200651137644>* > > > *-* > > > -- > If you reply to this email, your message will be added to the discussion > below: > > http://apache-spark-user-list.1001560.n3.nabble.com/How-to-implement-top-and-filter-on-object-List-for-JavaRDD-tp23669p23696.html > To unsubscribe from How to implement top() and filter() on object List > for JavaRDD, click here > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=23669&code=aGFmc2EuYXNpZkBtYXRjaGluZ3V1LmNvbXwyMzY2OXwtMTA0ODgyNjY3NA==> > . > NAML > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-implement-top-and-filter-on-object-List-for-JavaRDD-tp23669p23698.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: How to implement top() and filter() on object List for JavaRDD
Sorry ignore my last reply. Rusty, I am very thankful for your help. Actually, I am facing difficulty in objects. My plan is that, I have an object list containing list of User objects. After parallelizing it through spark context, I apply comparator on user.getUserName(). As usernames are sorted, their related user object are sorted according to user names. In the end when I apply top, I get the whole object of user . Some like this: public static Comparator UserComparator = new Comparator() { public int compare(User usr1, User usr2) { String userName1 = usr1.getUserName().toUpperCase(); String userName2 = usr1.getUserName().toUpperCase(); //ascending order return userName1.compareTo(userName2); //descending order //return fruitName2.compareTo(fruitName1); } }; JavaRDD rdd = context.parallelize(usersList); List top =rdd.top(1, UserComparator);// but it is giving me serialization issues. Can you guide me that how can I sort out this issue? I also applied your code, it runs fine but I want it with object (You know Object gives me tough time). Best Hafsa 2015-07-07 16:54 GMT+02:00 Hafsa Asif : > Rusty, > > I am very thankful for your help. Actually, I am facing difficulty in > objects. My plan is that, I have an object list containing list of User > objects. After parallelizing it through spark context, I apply comparator > on user.getUserName(). As usernames are sorted, their related user object > are sorted according to user names. > In the end when I apply top, I get the whole object of user . > > Some like this: > > public static Comparator UserComparator > = new Comparator() { > > public int compare(User usr1, User usr2) { > String userName1 = usr1.getUserName().toUpperCase(); > String userName2 = usr1.getUserName().toUpperCase(); > > //ascending order > return userName1.compareTo(userName2); > > //descending order > //return fruitName2.compareTo(fruitName1); > } > > }; > > JavaRDD rdd = context.parallelize(usersList); > > 2015-07-07 16:05 GMT+02:00 rusty [via Apache Spark User List] <[hidden > email] <http:///user/SendEmail.jtp?type=node&node=23688&i=0>>: > >> JavaRDD lines2 = ctx.parallelize(Arrays.asList("3", "6", "2", >> "5", "8", "6", "7")); >> List top =lines2.top(7, new >> CustomComaprator()); >> for (String integer : top) { >> System.out.println(integer); >> } >> >> >> >> >> class CustomComaprator implements Serializable, Comparator { >> /** >> * >> */ >> public CustomComaprator() { >> // TODO Auto-generated constructor stub >> >> } >> >> private static final long serialVersionUID = >> 2004092520677431781L; >> >> @Override >> public int compare(T o11, T o12) { >> int o1 = Integer.parseInt(String.valueOf(o11)); >> int o2 = Integer.parseInt(String.valueOf(o12)); >> >> return o1 > o2 ? 1 : o1 == o2 ? 0 : -1; >> } >> >> } >> >> >> -- >> If you reply to this email, your message will be added to the >> discussion below: >> >> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-implement-top-and-filter-on-object-List-for-JavaRDD-tp23669p23684.html >> To unsubscribe from How to implement top() and filter() on object List >> for JavaRDD, click here. >> NAML >> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >> > > > -- > View this message in context: Re: How to implement top() and filter() on > object List for JavaRDD > <http://apache-spark-user-list.1001560.n3.nabble.com/How-to-implement-top-and-filter-on-object-List-for-JavaRDD-tp23669p23688.html> > > Sent from the Apache Spark User List mailing list archive > <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com. >
Re: How to implement top() and filter() on object List for JavaRDD
Rusty, I am very thankful for your help. Actually, I am facing difficulty in objects. My plan is that, I have an object list containing list of User objects. After parallelizing it through spark context, I apply comparator on user.getUserName(). As usernames are sorted, their related user object are sorted according to user names. In the end when I apply top, I get the whole object of user . Some like this: public static Comparator UserComparator = new Comparator() { public int compare(User usr1, User usr2) { String userName1 = usr1.getUserName().toUpperCase(); String userName2 = usr1.getUserName().toUpperCase(); //ascending order return userName1.compareTo(userName2); //descending order //return fruitName2.compareTo(fruitName1); } }; JavaRDD rdd = context.parallelize(usersList); 2015-07-07 16:05 GMT+02:00 rusty [via Apache Spark User List] < ml-node+s1001560n23684...@n3.nabble.com>: > JavaRDD lines2 = ctx.parallelize(Arrays.asList("3", "6", "2", > "5", "8", "6", "7")); > List top =lines2.top(7, new > CustomComaprator()); > for (String integer : top) { > System.out.println(integer); > } > > > > > class CustomComaprator implements Serializable, Comparator { > /** > * > */ > public CustomComaprator() { > // TODO Auto-generated constructor stub > > } > > private static final long serialVersionUID = 2004092520677431781L; > > @Override > public int compare(T o11, T o12) { > int o1 = Integer.parseInt(String.valueOf(o11)); > int o2 = Integer.parseInt(String.valueOf(o12)); > > return o1 > o2 ? 1 : o1 == o2 ? 0 : -1; > } > > } > > > ------------------ > If you reply to this email, your message will be added to the discussion > below: > > http://apache-spark-user-list.1001560.n3.nabble.com/How-to-implement-top-and-filter-on-object-List-for-JavaRDD-tp23669p23684.html > To unsubscribe from How to implement top() and filter() on object List > for JavaRDD, click here > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=23669&code=aGFmc2EuYXNpZkBtYXRjaGluZ3V1LmNvbXwyMzY2OXwtMTA0ODgyNjY3NA==> > . > NAML > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-implement-top-and-filter-on-object-List-for-JavaRDD-tp23669p23688.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: How to implement top() and filter() on object List for JavaRDD
Here's a simplified example: SparkConf conf = new SparkConf().setAppName( "Sigmoid").setMaster("local"); JavaSparkContext sc = new JavaSparkContext(conf); List user = new ArrayList(); user.add("Jack"); user.add("Jill"); user.add("Jack"); user.add("Bob"); JavaRDD userRDD = sc.parallelize(user); //Now Lets filter all Jacks! JavaRDD jackRDD = userRDD *.filter(new Function() {* *public Boolean call(String v1) throws Exception {* *return v1.equals("Jack");* *}* *}*); //Lets print all jacks! for (String s : jackRDD.collect()) { System.out.println(s); } Thanks Best Regards On Tue, Jul 7, 2015 at 5:39 PM, Hafsa Asif wrote: > I have also tried this stupid code snippet, only thinking that it may even > compile code > Function1 FILTER_USER = new AbstractFunction1 >() > { > public Object apply(User user){ > return user; > } > }; > > > FILTER_USER is fine but cannot be applied to the following two options but > no results: > User[] filterUsr = > (User[])rdd.rdd().retag(User.class).filter(FILTER_USER); > > User userFilter = (User) rdd.rdd().filter(FILTER_USER); > > Giving issue: Inconertable types > I really need proper code related to this issue. > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/How-to-implement-top-and-filter-on-object-List-for-JavaRDD-tp23669p23677.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Re: How to implement top() and filter() on object List for JavaRDD
I would suggest you take alook to DataFrames. Also, I do not think you should implement comparators for user class as a whole, rather you should get the attribute to sort/compar on and delete sorting to data type of inherent attribute. Eg. sorting can be done by name and if so, it should be string sorting. In that case, you can create another KeyValue RDD whr key is the "chosen" attribute and value is user object and do a sortByKey. On Tue, Jul 7, 2015 at 10:09 PM, Hafsa Asif wrote: > I have also tried this stupid code snippet, only thinking that it may even > compile code > Function1 FILTER_USER = new AbstractFunction1 >() > { > public Object apply(User user){ > return user; > } > }; > > > FILTER_USER is fine but cannot be applied to the following two options but > no results: > User[] filterUsr = > (User[])rdd.rdd().retag(User.class).filter(FILTER_USER); > > User userFilter = (User) rdd.rdd().filter(FILTER_USER); > > Giving issue: Inconertable types > I really need proper code related to this issue. > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/How-to-implement-top-and-filter-on-object-List-for-JavaRDD-tp23669p23677.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > -- Best Regards, Ayan Guha
Re: How to implement top() and filter() on object List for JavaRDD
I have also tried this stupid code snippet, only thinking that it may even compile code Function1 FILTER_USER = new AbstractFunction1() { public Object apply(User user){ return user; } }; FILTER_USER is fine but cannot be applied to the following two options but no results: User[] filterUsr = (User[])rdd.rdd().retag(User.class).filter(FILTER_USER); User userFilter = (User) rdd.rdd().filter(FILTER_USER); Giving issue: Inconertable types I really need proper code related to this issue. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-implement-top-and-filter-on-object-List-for-JavaRDD-tp23669p23677.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: How to implement top() and filter() on object List for JavaRDD
Thank u for your quick response. But, I tried this and get the error as shown in pic error.jpg <http://apache-spark-user-list.1001560.n3.nabble.com/file/n23676/error.jpg> -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-implement-top-and-filter-on-object-List-for-JavaRDD-tp23669p23676.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
How to implement top() and filter() on object List for JavaRDD
Hi, I have an object list of Users and I want to implement top() and filter() methods on the object list. Let me explain you the whole scenario: 1. I have User object list named "usersList". I fill it during record set. User user = new User(); user.setUserName(record.getValue("username").toString()); user.setPassword(record.getValue("password").toString()); usersList.add(user); 2. I successfully implement first() and take() methods like this, they are giving results: JavaRDD rdd = context.parallelize(usersList); /*Getting First User*/ User firstUsr = (User) rdd.rdd().first(); System.out.println("First User: "+firstUsr.getUserName()); /*Getting Take 2 Users*/ User[] takeUsr = (User[])rdd.rdd().retag(User.class).take(2); for(int ctusr=0 ; ctusr ord). I donot understand how to implement Ordering. I tried two possibilities: (i) User[] topUsr = (User[])rdd.rdd().retag(User.class).top(1, null); (ii) User[] topUsr = (User[])rdd.rdd().retag(User.class).top(1, new Ordering() { @Override public Some tryCompare(User user, User t1) { return null; } @Override public int compare(User usr1, User usr2) { usr1 = new User(); usr2 = new User(); for(int u=0 ; u < usersList.size(); u++){ usr1 = (User)usersList.get(u); usr2 = (User)usersList.get(u); } String userName1 = usr1.getUserName(); String userName2 = usr2.getUserName(); System.out.println("userName1: "+userName1); //ascending order return userName1.compareTo(userName2); //descending order //return userName2.compareTo(userName1); } @Override public boolean lteq(User user, User t1) { return false; } @Override public boolean gteq(User user, User t1) { return false; } @Override public boolean lt(User user, User t1) { return false; } @Override public boolean gt(User user, User t1) { return false; } @Override public boolean equiv(User user, User t1) { return false; } @Override public User max(User user, User t1) { return null; } @Override public User min(User user, User t1) { return null; } @Override public Ordering reverse() { return null; } @Override public Ordering on(Function1 function1) { return null; } @Override public Ops mkOrderingOps(User user) { return null; } } ); BOTH ARE GIVING ME NULLPOINTER EXCEPTION. Kindly guide me how to properly implement it? 4. I also want to implement the filter() in same way. User[] filterUsr = (User[])rdd.rdd().retag(User.class).filter(Function1); but could not write Function1 for it. I suppose that filter out the user whose name is "Bob". How it can be possible by using object list and User objects? Can you give me code sample for Function1? I really appreciate your help and guidance. Thanks, Hafsa -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-implement-top-and-filter-on-object-List-for-JavaRDD-tp23669.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org