No TypeTag Available for String

2017-05-30 Thread krishmah 
I am currently using Spark 2.0.1 with Scala 2.11.8. However same code works
with Scala 2.10.6. Please advise if I am missing something

import org.apache.spark.sql.functions.udf

val getFileName = udf{z:String => z.takeRight(z.length
-z.lastIndexOf("/")-1)}

and this gives me following error messages

No Type Tag Available for String and

not enough arguments for method udf: (implicit evidence$2:
reflect.runtime.universe.TypeTag[String], implicit evidence$3:
reflect.runtime.universe.TypeTag[String])org.apache.spark.sql.expressions.UserDefinedFunction.
Unspecified value parameters evidence$2, evidence$3.



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/No-TypeTag-Available-for-String-tp28720.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Shuffle Service - Connection Inactive - Creating new one

2016-06-01 Thread krishmah 
 I am seeing this in my logs. It appears to reopen connection to Shuffle
Service. Whenever this happens, I am seeing the partition to take longer
time to complete. I was running a job with 1000 partitions. About 600
partitions, it was completing in less than 20 mins a partition and after
that I started seeing each partition taking about 1.5 hours. When I was
checking the logs to see what was going on, I was seeing lot of these
connections getting created. Any thing that I am doing wrongly which is
creating this issue?

16/05/31 04:49:47 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-102.us-west-2.compute.internal/24.1.1.102:7337,
creating a new one.
16/05/31 04:50:27 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-3-136.us-west-2.compute.internal/24.1.3.136:7337,
creating a new one.
16/05/31 04:57:59 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-101.us-west-2.compute.internal/24.1.1.101:7337,
creating a new one.
16/05/31 04:58:15 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-3-136.us-west-2.compute.internal/24.1.3.136:7337,
creating a new one.
16/05/31 04:58:26 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-102.us-west-2.compute.internal/24.1.1.102:7337,
creating a new one.
16/05/31 05:02:33 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-101.us-west-2.compute.internal/24.1.1.101:7337,
creating a new one.
16/05/31 05:02:39 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-3-136.us-west-2.compute.internal/24.1.3.136:7337,
creating a new one.
16/05/31 05:06:19 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-3-136.us-west-2.compute.internal/24.1.3.136:7337,
creating a new one.
16/05/31 05:06:26 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-102.us-west-2.compute.internal/24.1.1.102:7337,
creating a new one.
16/05/31 05:09:45 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-102.us-west-2.compute.internal/24.1.1.102:7337,
creating a new one.
16/05/31 05:12:27 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-101.us-west-2.compute.internal/24.1.1.101:7337,
creating a new one.
16/05/31 05:17:10 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-3-136.us-west-2.compute.internal/24.1.3.136:7337,
creating a new one.
16/05/31 05:19:20 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-102.us-west-2.compute.internal/24.1.1.102:7337,
creating a new one.
16/05/31 05:19:20 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-101.us-west-2.compute.internal/24.1.1.101:7337,
creating a new one.
16/05/31 05:21:34 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-102.us-west-2.compute.internal/24.1.1.102:7337,
creating a new one.
16/05/31 05:30:43 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-102.us-west-2.compute.internal/24.1.1.102:7337,
creating a new one.
16/05/31 05:32:57 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-3-136.us-west-2.compute.internal/24.1.3.136:7337,
creating a new one.
16/05/31 05:33:01 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-101.us-west-2.compute.internal/24.1.1.101:7337,
creating a new one.
16/05/31 05:35:08 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-102.us-west-2.compute.internal/24.1.1.102:7337,
creating a new one.
16/05/31 05:38:06 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-3-136.us-west-2.compute.internal/24.1.3.136:7337,
creating a new one.
16/05/31 05:38:53 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-102.us-west-2.compute.internal/24.1.1.102:7337,
creating a new one.
16/05/31 05:40:23 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-3-136.us-west-2.compute.internal/24.1.3.136:7337,
creating a new one.
16/05/31 05:41:47 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-1-101.us-west-2.compute.internal/24.1.1.101:7337,
creating a new one.
16/05/31 05:43:40 INFO client.TransportClientFactory: Found inactive
connection to ip-24-1-3-136.us-west-2.



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Shuffle-Service-Connection-Inactive-Creating-new-one-tp27061.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org