The reason is likely because first() is entirely executed on the driver node in the same process, while collect() needs to connect with worker nodes.
Usually the first time you run an action, most of the JVM code are not optimized, and the classloader also needs to load a lot of things on the fly. Having to connect with other processes via RPC can slow the first execution down in collect. That said, if you run this a few times (in the same driver program) and it is still much slower, you should look into other factors such as network congestion, cpu/memory load on workers, etc. BTW - this is a dev list for Spark development itself (not Spark application development). Questions like this probably go better in the user list in the future. On Fri, Jan 24, 2014 at 6:34 PM, Chen Jin <[email protected]> wrote: > Hi Tathagata, > > Thanks for the detailed explanation, I thought so too.However, > currently I only have one text partition which contains two lines. > Each line is like tens of characters in total. Why is there such a big > difference between first() and collect()? Instead 2x, I have around > 30x difference. > > > On Fri, Jan 24, 2014 at 6:25 PM, Tathagata Das > <[email protected]> wrote: > > RDD.first() doesnt have to scan the whole partition. It gets only the > first > > item and returns it. > > RDD.collect() has to scan the whole partition, collect all of it and send > > all of it back (serialization + deserialization costs, etc.) > > > > TD > > > > > > On Fri, Jan 24, 2014 at 5:55 PM, Chen Jin <[email protected]> wrote: > > > >> Hi All, > >> > >> I have some metadata saved as a single partition on HDFS (a few > >> hundred bytes) and when I want to get the content of the data: > >> > >> JavaRDD blob = sc.textFile(); > >> List<String> lines = blob.collect(); > >> > >> However, collect takes probably more than 3 seconds at least but > >> first() only take 0.1 second, > >> > >> Could you advise on what's the best practice to read small files using > >> spark. > >> > >> -chen > >> > >> > >> On Fri, Jan 24, 2014 at 3:23 PM, Kapil Malik <[email protected]> wrote: > >> > Hi Andrew, > >> > > >> > > >> > > >> > Here's the exception I get while trying to build an OSGi bundle using > >> maven SCR plugin - > >> > > >> > > >> > > >> > [ERROR] Failed to execute goal > >> org.apache.felix:maven-scr-plugin:1.9.0:scr > (generate-scr-scrdescriptor) on > >> project repo-spark: Execution generate-scr-scrdescriptor of goal > >> org.apache.felix:maven-scr-plugin:1.9.0:scr failed: Invalid signature > file > >> digest for Manifest main attributes -> [Help 1] > >> > > >> > org.apache.maven.lifecycle.LifecycleExecutionException: Failed to > >> execute goal org.apache.felix:maven-scr-plugin:1.9.0:scr > >> (generate-scr-scrdescriptor) on project repo-spark: Execution > >> generate-scr-scrdescriptor of goal > >> org.apache.felix:maven-scr-plugin:1.9.0:scr failed: Invalid signature > file > >> digest for Manifest main attributes > >> > > >> > at > >> > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:225) > >> > > >> > ... > >> > > >> > Caused by: org.apache.maven.plugin.PluginExecutionException: Execution > >> generate-scr-scrdescriptor of goal > >> org.apache.felix:maven-scr-plugin:1.9.0:scr failed: Invalid signature > file > >> digest for Manifest main attributes > >> > > >> > at > >> > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:110) > >> > > >> > at > >> > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) > >> > > >> > ... 19 more > >> > > >> > Caused by: java.lang.SecurityException: Invalid signature file digest > >> for Manifest main attributes > >> > > >> > at > >> > sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:240) > >> > > >> > ... > >> > > >> > > >> > > >> > > >> > > >> > Also, from eclipse, if I build a simple main program. Then, I can > create > >> an executable JAR in 3 ways - > >> > > >> > a. Extract required libraries into generated JAR ( individual > >> classes inside my JAR) > >> > > >> > On running main program on this JAR – > >> > > >> > Exception in thread "main" > com.typesafe.config.ConfigException$Missing: > >> No configuration setting found for key > 'akka.remote.log-received-messages' > >> > > >> > at > >> com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:126) > >> > > >> > at > >> com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:146) > >> > > >> > > >> > > >> > b. Package required libraries into generated JAR (all JARs inside > >> my JAR) > >> > > >> > On running main program on this JAR – > >> > > >> > Exception in thread "main" java.lang.reflect.InvocationTargetException > >> > > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> > > >> > at > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > >> > > >> > at > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> > > >> > at java.lang.reflect.Method.invoke(Method.java:616) > >> > > >> > at > >> > org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58) > >> > > >> > Caused by: java.lang.Exception: Could not find resource path for Web > UI: > >> org/apache/spark/ui/static > >> > > >> > at > >> org.apache.spark.ui.JettyUtils$.createStaticHandler(JettyUtils.scala:89) > >> > > >> > at org.apache.spark.ui.SparkUI.<init>(SparkUI.scala:40) > >> > > >> > at > org.apache.spark.SparkContext.<init>(SparkContext.scala:122) > >> > > >> > at > >> > org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:67) > >> > > >> > > >> > > >> > c. Copy required libraries into a sub-folder next to generated > JAR > >> > > >> > This works well ! But problem is that it’s not portable to a java > server. > >> > > >> > > >> > > >> > Regards, > >> > > >> > > >> > > >> > Kapil Malik | [email protected] > >> > > >> > > >> > > >> > -----Original Message----- > >> > From: Andrew Ash [mailto:[email protected]] > >> > Sent: 25 January 2014 04:08 > >> > To: [email protected] > >> > Cc: [email protected] > >> > Subject: Re: Running spark driver inside a servlet > >> > > >> > > >> > > >> > Can you paste the exception you're seeing? > >> > > >> > > >> > > >> > Sent from my mobile phone > >> > > >> > On Jan 24, 2014 2:36 PM, "Kapil Malik" <[email protected]<mailto: > >> [email protected]>> wrote: > >> > > >> > > >> > > >> >> Hi all, > >> > > >> >> > >> > > >> >> > >> > > >> >> > >> > > >> >> Is it possible to create a Spark Context (i.e. the driver program) > >> > > >> >> from a servlet deployed on some application server ? > >> > > >> >> > >> > > >> >> I am able to run spark Java driver successfully via maven / > standalone > >> > > >> >> (after specifying the classpath), but when I bundle spark libraries > in > >> > > >> >> a JAR, along with my servlet (using maven shade plugin), it gives me > >> > > >> >> security exception. Any suggestions? > >> > > >> >> > >> > > >> >> > >> > > >> >> > >> > > >> >> Thanks and regards, > >> > > >> >> > >> > > >> >> > >> > > >> >> > >> > > >> >> Kapil Malik > >> > > >> >> > >> > > >> >> > >> > > >> >> > >> >
