Re: Long Running Spark Streaming getting slower
Hi Mich, batch interval is 10 seconds, and I don't use sliding window. Typical message count per batch is ~100k. -- John Simon On Fri, Jun 10, 2016 at 10:31 AM, Mich Talebzadeh <mich.talebza...@gmail.com > wrote: > Hi John, > > I did not notice anything unusual in your env variables. > > However, what are the batch interval, the windowsLength and SlindingWindow > interval. > > Also how many messages are sent by Kafka in a typical batch interval? > > HTH > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > > On 10 June 2016 at 18:21, john.simon <john.si...@tapjoy.com> wrote: > >> Hi all, >> >> I'm running Spark Streaming with Kafka Direct Stream, but after >> running a couple of days, the batch processing time almost doubles. >> I didn't find any slowdown on JVM GC logs, but I did find that Spark >> broadcast variable reading time increasing. >> Initially it takes less than 10ms, but after 3 days it takes more than >> 60ms. It's really puzzling since I don't use broadcast variables at >> all. >> >> My application needs to run 24/7, so I hope there's something I'm >> missing to correct this behavior. >> >> FYI, we're running on AWS EMR with Spark version 1.6.1, in YARN client >> mode. >> Attached spark application environment settings file. >> >> -- >> John Simon >> >> *environment.txt* (7K) Download Attachment >> <http://apache-spark-user-list.1001560.n3.nabble.com/attachment/27138/0/environment.txt> >> >> -- >> View this message in context: Long Running Spark Streaming getting slower >> <http://apache-spark-user-list.1001560.n3.nabble.com/Long-Running-Spark-Streaming-getting-slower-tp27138.html> >> Sent from the Apache Spark User List mailing list archive >> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com. >> > >
Re: Spark Streaming getting slower
Sorry, forgot to mention that I don't use broadcast variables. That's why I'm puzzled here. -- John Simon On Thu, Jun 9, 2016 at 11:09 AM, John Simon <john.si...@tapjoy.com> wrote: > Hi, > > I'm running Spark Streaming with Kafka Direct Stream, batch interval > is 10 seconds. > After running about 72 hours, the batch processing time almost doubles. > I didn't find anything wrong on JVM GC logs, but I did find that > broadcast variable reading time increasing, like this: > > initially: > > ``` > 16/06/08 18:17:02 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 0 took 223 ms > 16/06/08 18:17:08 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 2 took 73 ms > 16/06/08 18:17:10 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 3 took 13 ms > 16/06/08 18:17:11 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 4 took 9 ms > 16/06/08 18:17:12 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 5 took 8 ms > 16/06/08 18:17:14 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 6 took 13 ms > 16/06/08 18:17:15 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 8 took 7 ms > 16/06/08 18:17:16 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 10 took 8 ms > 16/06/08 18:17:20 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 12 took 6 ms > 16/06/08 18:17:20 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 13 took 7 ms > ``` > > after 23 hours: > > ``` > 16/06/09 17:23:43 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 282938 took 18 ms > 16/06/09 17:23:43 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 282939 took 20 ms > 16/06/09 17:23:43 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 282940 took 20 ms > 16/06/09 17:23:43 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 282941 took 14 ms > 16/06/09 17:23:44 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 282942 took 17 ms > 16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 282943 took 18 ms > 16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 282944 took 14 ms > 16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 282945 took 18 ms > 16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 282946 took 14 ms > 16/06/09 17:23:45 INFO broadcast.TorrentBroadcast: Reading broadcast > variable 282947 took 18 ms > ``` > > FYI, we're running on AWS EMR with Spark version 1.6.1, in YARN client mode. > > application environment follows: > > ``` > Java Home /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre > Java Version 1.7.0_101 (Oracle Corporation) > Scala Version version 2.10.5 > > spark.app.id application_1463430559850_0056 > spark.cleaner.ttl 60 > spark.default.parallelism 96 > spark.driver.appUIAddress http://172.16.4.168:4041 > spark.driver.extraClassPath > /etc/hadoop/conf:/usr/lib/hadoop-lzo/lib/*:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/* > spark.driver.extraJavaOptions > -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties > -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 > -XX:MaxHeapFreeRatio=70 -XX:+CMSClassUnloadingEnabled > -XX:MaxPermSize=512M -XX:OnOutOfMemoryError='kill -9 %p' > spark.driver.extraLibraryPath > /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native > spark.driver.host 172.16.4.168 > spark.driver.port 42142 > spark.dynamicAllocation.enabled false > spark.eventLog.compress true > spark.eventLog.dir hdfs:///var/log/spark/apps > spark.eventLog.enabled false > spark.executor.cores 8 > spark.executor.extraClassPath > /etc/hadoop/conf:/usr/lib/hadoop-lzo/lib/*:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/* > spark.executor.extraJavaOptions -verbose:gc -XX:+PrintGCDetails > -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC > -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 > -XX:+CMSClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p' > spark.executor.extraLibraryPath > /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native > spark.executor.id driver > spark.executor.instances 24 > spark.executor.memory 6G > spark.externalBlockStore.folderName spark-9ede6685-a7f9-4d65-bfb4-0920e8c5ba25 > spark.history.fs.cleaner.enabled true > spark.history.fs.cleaner.maxAge 2d > spark.history.fs.logDirectory hdfs:///var/log/spark/apps > spark.history.ui.port 18080 > spark.master yarn-client > spark.org.apache.hadoop.yarn.server.webproxy.amfilte
Spark Streaming getting slower
spark.yarn.historyServer.address ip-172-16-4-168.ec2.internal:18080 SPARK_SUBMIT true SPARK_YARN_MODE true awt.toolkit sun.awt.X11.XToolkit file.encoding UTF-8 file.encoding.pkg sun.io file.separator / java.awt.graphicsenv sun.awt.X11GraphicsEnvironment java.awt.printerjob sun.print.PSPrinterJob java.class.version 51.0 java.endorsed.dirs /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/endorsed java.ext.dirs /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/ext:/usr/java/packages/lib/ext java.home /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre java.io.tmpdir /tmp java.library.path /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib java.runtime.name OpenJDK Runtime Environment java.runtime.version 1.7.0_101-mockbuild_2016_04_26_18_10-b00 java.specification.name Java Platform API Specification java.specification.vendor Oracle Corporation java.specification.version 1.7 java.vendor Oracle Corporation java.vendor.url http://java.oracle.com/ java.vendor.url.bug http://bugreport.sun.com/bugreport/ java.version 1.7.0_101 java.vm.info mixed mode java.vm.name OpenJDK 64-Bit Server VM java.vm.specification.name Java Virtual Machine Specification java.vm.specification.vendor Oracle Corporation java.vm.specification.version 1.7 java.vm.vendor Oracle Corporation java.vm.version 24.95-b01 line.separator log4j.configuration file:///etc/spark/conf/log4j.properties os.arch amd64 os.name Linux os.version 4.4.5-15.26.amzn1.x86_64 path.separator : sun.arch.data.model 64 sun.boot.class.path /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/resources.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/rt.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/jsse.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/jce.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/charsets.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/rhino.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/jfr.jar:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/classes sun.boot.library.path /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101.x86_64/jre/lib/amd64 sun.cpu.endian little sun.cpu.isalist sun.io.unicode.encoding UnicodeLittle sun.java.launcher SUN_STANDARD sun.jnu.encoding UTF-8 sun.management.compiler HotSpot 64-Bit Tiered Compilers sun.nio.ch.bugLevel sun.os.patch.level unknown user.country US user.dir /home/hadoop user.home /home/hadoop user.language en user.name hadoop user.timezone UTC ``` -- John Simon - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
