Sreetej Lakkam created SPARK-15843: -------------------------------------- Summary: Spark RAM issue Key: SPARK-15843 URL: https://issues.apache.org/jira/browse/SPARK-15843 Project: Spark Issue Type: Question Components: Spark Shell Affects Versions: 1.6.1 Environment: RASPBIAN JESSIE Full desktop image based on Debian Jessie Reporter: Sreetej Lakkam
Trying to run Spark 1.6.1 on Hadoop 2.7.1 over Raspberry Pi - 3. On submitting spark-shell sudo /opt/spark-1.6.1-bin-hadoop2.6/bin/spark-shell produces an error Java HotSpot(TM) Client VM warning: INFO: os::commit_memory(0x49540000, 715915264, 0) failed; error='Cannot allocate memory' (errno=12) # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 715915264 bytes for committing reserved memory. # An error report file with more information is saved as: # /home/pi/hs_err_pid2179.log Find the log file hs_err_pid2179.log # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 715915264 bytes for committing reserved memory. # Possible reasons: # The system is out of physical RAM or swap space # In 32 bit mode, the process size limit was hit # Possible solutions: # Reduce memory load on the system # Increase physical memory or swap space # Check if swap backing store is full # Use 64 bit Java on a 64 bit OS # Decrease Java heap size (-Xmx/-Xms) # Decrease number of Java threads # Decrease Java thread stack sizes (-Xss) # Set larger code cache with -XX:ReservedCodeCacheSize= # This output file may be truncated or incomplete. # # Out of Memory Error (os_linux.cpp:2627), pid=2179, tid=1983399008 # # JRE version: (8.0_65-b17) (build ) # Java VM: Java HotSpot(TM) Client VM (25.65-b01 mixed mode, sharing linux-arm ) # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # --------------- T H R E A D --------------- Current thread (0x76207400): JavaThread "Unknown thread" [_thread_in_vm, id=2200, stack(0x76335000,0x76385000)] Stack: [0x76335000,0x76385000] --------------- P R O C E S S --------------- Java Threads: ( => current thread ) Other Threads: =>0x76207400 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=2200, stack(0x76335000,0x76385000)] VM state:not at safepoint (not fully initialized) VM Mutex/Monitor currently owned by a thread: None GC Heap History (0 events): No events Deoptimization events (0 events): No events Internal exceptions (0 events): No events Events (0 events): No events Dynamic libraries: 00008000-00009000 r-xp 00000000 b3:02 22725 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/bin/java 00010000-00011000 rw-p 00000000 b3:02 22725 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/bin/java 003e5000-00406000 rw-p 00000000 00:00 0 [heap] 33dff000-33eaa000 rw-p 00000000 00:00 0 33eaa000-33fff000 ---p 00000000 00:00 0 33fff000-49540000 rw-p 00000000 00:00 0 740c3000-740c4000 rw-p 00000000 00:00 0 740c4000-74143000 ---p 00000000 00:00 0 74143000-7416b000 rwxp 00000000 00:00 0 7416b000-76143000 ---p 00000000 00:00 0 76143000-7615a000 r-xp 00000000 b3:02 22812 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libzip.so 7615a000-76161000 ---p 00017000 b3:02 22812 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libzip.so 76161000-76162000 rw-p 00016000 b3:02 22812 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libzip.so 76162000-7616d000 r-xp 00000000 b3:02 3196 /lib/arm-linux-gnueabihf/libnss_files-2.19.so 7616d000-7617c000 ---p 0000b000 b3:02 3196 /lib/arm-linux-gnueabihf/libnss_files-2.19.so 7617c000-7617d000 r--p 0000a000 b3:02 3196 /lib/arm-linux-gnueabihf/libnss_files-2.19.so 7617d000-7617e000 rw-p 0000b000 b3:02 3196 /lib/arm-linux-gnueabihf/libnss_files-2.19.so 7617e000-76187000 r-xp 00000000 b3:02 3204 /lib/arm-linux-gnueabihf/libnss_nis-2.19.so 76187000-76196000 ---p 00009000 b3:02 3204 /lib/arm-linux-gnueabihf/libnss_nis-2.19.so 76196000-76197000 r--p 00008000 b3:02 3204 /lib/arm-linux-gnueabihf/libnss_nis-2.19.so 76197000-76198000 rw-p 00009000 b3:02 3204 /lib/arm-linux-gnueabihf/libnss_nis-2.19.so 76198000-761a9000 r-xp 00000000 b3:02 3193 /lib/arm-linux-gnueabihf/libnsl-2.19.so 761a9000-761b8000 ---p 00011000 b3:02 3193 /lib/arm-linux-gnueabihf/libnsl-2.19.so 761b8000-761b9000 r--p 00010000 b3:02 3193 /lib/arm-linux-gnueabihf/libnsl-2.19.so 761b9000-761ba000 rw-p 00011000 b3:02 3193 /lib/arm-linux-gnueabihf/libnsl-2.19.so 761ba000-761bc000 rw-p 00000000 00:00 0 761bc000-761c3000 r-xp 00000000 b3:02 3194 /lib/arm-linux-gnueabihf/libnss_compat-2.19.so 761c3000-761d2000 ---p 00007000 b3:02 3194 /lib/arm-linux-gnueabihf/libnss_compat-2.19.so 761d2000-761d3000 r--p 00006000 b3:02 3194 /lib/arm-linux-gnueabihf/libnss_compat-2.19.so 761d3000-761d4000 rw-p 00007000 b3:02 3194 /lib/arm-linux-gnueabihf/libnss_compat-2.19.so 761d4000-761f7000 r-xp 00000000 b3:02 22787 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libjava.so 761f7000-761ff000 ---p 00023000 b3:02 22787 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libjava.so 761ff000-76200000 rw-p 00023000 b3:02 22787 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libjava.so 76200000-76221000 rw-p 00000000 00:00 0 76221000-76300000 ---p 00000000 00:00 0 7631e000-76324000 r-xp 00000000 b3:02 3223 /lib/arm-linux-gnueabihf/librt-2.19.so 76324000-76333000 ---p 00006000 b3:02 3223 /lib/arm-linux-gnueabihf/librt-2.19.so 76333000-76334000 r--p 00005000 b3:02 3223 /lib/arm-linux-gnueabihf/librt-2.19.so 76334000-76335000 rw-p 00006000 b3:02 3223 /lib/arm-linux-gnueabihf/librt-2.19.so 76335000-76338000 ---p 00000000 00:00 0 76338000-76385000 rwxp 00000000 00:00 0 [stack:2200] 76385000-763ee000 r-xp 00000000 b3:02 3185 /lib/arm-linux-gnueabihf/libm-2.19.so 763ee000-763fe000 ---p 00069000 b3:02 3185 /lib/arm-linux-gnueabihf/libm-2.19.so 763fe000-763ff000 r--p 00069000 b3:02 3185 /lib/arm-linux-gnueabihf/libm-2.19.so 763ff000-76400000 rw-p 0006a000 b3:02 3185 /lib/arm-linux-gnueabihf/libm-2.19.so 76400000-76983000 r-xp 00000000 b3:02 22814 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/client/libjvm.so 76983000-7698b000 ---p 00583000 b3:02 22814 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/client/libjvm.so 7698b000-769b7000 rw-p 00583000 b3:02 22814 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/client/libjvm.so 769b7000-76dda000 rw-p 00000000 00:00 0 76dda000-76df7000 r-xp 00000000 b3:02 3170 /lib/arm-linux-gnueabihf/libgcc_s.so.1 76df7000-76e06000 ---p 0001d000 b3:02 3170 /lib/arm-linux-gnueabihf/libgcc_s.so.1 76e06000-76e07000 rw-p 0001c000 b3:02 3170 /lib/arm-linux-gnueabihf/libgcc_s.so.1 76e07000-76f32000 r-xp 00000000 b3:02 3155 /lib/arm-linux-gnueabihf/libc-2.19.so 76f32000-76f42000 ---p 0012b000 b3:02 3155 /lib/arm-linux-gnueabihf/libc-2.19.so 76f42000-76f44000 r--p 0012b000 b3:02 3155 /lib/arm-linux-gnueabihf/libc-2.19.so 76f44000-76f45000 rw-p 0012d000 b3:02 3155 /lib/arm-linux-gnueabihf/libc-2.19.so 76f45000-76f48000 rw-p 00000000 00:00 0 76f48000-76f4a000 r-xp 00000000 b3:02 3164 /lib/arm-linux-gnueabihf/libdl-2.19.so 76f4a000-76f59000 ---p 00002000 b3:02 3164 /lib/arm-linux-gnueabihf/libdl-2.19.so 76f59000-76f5a000 r--p 00001000 b3:02 3164 /lib/arm-linux-gnueabihf/libdl-2.19.so 76f5a000-76f5b000 rw-p 00002000 b3:02 3164 /lib/arm-linux-gnueabihf/libdl-2.19.so 76f5b000-76f6f000 r-xp 00000000 b3:02 22815 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/jli/libjli.so 76f6f000-76f77000 ---p 00014000 b3:02 22815 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/jli/libjli.so 76f77000-76f78000 rw-p 00014000 b3:02 22815 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/jli/libjli.so 76f78000-76f8c000 r-xp 00000000 b3:02 3220 /lib/arm-linux-gnueabihf/libpthread-2.19.so 76f8c000-76f9c000 ---p 00014000 b3:02 3220 /lib/arm-linux-gnueabihf/libpthread-2.19.so 76f9c000-76f9d000 r--p 00014000 b3:02 3220 /lib/arm-linux-gnueabihf/libpthread-2.19.so 76f9d000-76f9e000 rw-p 00015000 b3:02 3220 /lib/arm-linux-gnueabihf/libpthread-2.19.so 76f9e000-76fa0000 rw-p 00000000 00:00 0 76fa1000-76faa000 r-xp 00000000 b3:02 22811 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libverify.so 76faa000-76fb2000 ---p 00009000 b3:02 22811 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libverify.so 76fb2000-76fb3000 rw-p 00009000 b3:02 22811 /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/lib/arm/libverify.so 76fb3000-76fb8000 r-xp 00000000 b3:02 20453 /usr/lib/arm-linux-gnueabihf/libarmmem.so 76fb8000-76fc7000 ---p 00005000 b3:02 20453 /usr/lib/arm-linux-gnueabihf/libarmmem.so 76fc7000-76fc8000 rw-p 00004000 b3:02 20453 /usr/lib/arm-linux-gnueabihf/libarmmem.so 76fc8000-76fe8000 r-xp 00000000 b3:02 3119 /lib/arm-linux-gnueabihf/ld-2.19.so 76fe9000-76ff1000 rw-s 00000000 b3:02 409006 /tmp/hsperfdata_root/2179 76ff1000-76ff2000 r--p 00000000 00:00 0 76ff2000-76ff5000 rw-p 00000000 00:00 0 76ff5000-76ff7000 rw-p 00000000 00:00 0 76ff7000-76ff8000 r--p 0001f000 b3:02 3119 /lib/arm-linux-gnueabihf/ld-2.19.so 76ff8000-76ff9000 rw-p 00020000 b3:02 3119 /lib/arm-linux-gnueabihf/ld-2.19.so 7ec8f000-7ecb0000 rwxp 00000000 00:00 0 [stack] 7ed85000-7ed86000 r-xp 00000000 00:00 0 [sigpage] 7ed86000-7ed87000 r--p 00000000 00:00 0 [vvar] 7ed87000-7ed88000 r-xp 00000000 00:00 0 [vdso] ffff0000-ffff1000 r-xp 00000000 00:00 0 [vectors] VM Arguments: jvm_args: -Dscala.usejavacp=true -Xms1g -Xmx1g java_command: org.apache.spark.deploy.SparkSubmit --class org.apache.spark.repl.Main --name Spark shell spark-shell java_class_path (initial): /opt/spark-1.6.1-bin-hadoop2.6/conf/:/opt/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar:/opt/spark-1.6.1-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar:/opt/spark-1.6.1-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar:/opt/spark-1.6.1-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar Launcher Type: SUN_STANDARD Environment Variables: JAVA_HOME=/usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin USERNAME=root SHELL=/bin/bash DISPLAY=:0.0 Signal Handlers: SIGSEGV: [libjvm.so+0x506638], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO SIGBUS: [libjvm.so+0x506638], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO SIGFPE: [libjvm.so+0x3fd4ec], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO SIGPIPE: [libjvm.so+0x3fd4ec], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO SIGXFSZ: [libjvm.so+0x3fd4ec], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO SIGILL: [libjvm.so+0x3fd4ec], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none SIGUSR2: [libjvm.so+0x3fd650], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO SIGHUP: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none SIGINT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none SIGTERM: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none SIGQUIT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none --------------- S Y S T E M --------------- OS:PRETTY_NAME="Raspbian GNU/Linux 8 (jessie)" NAME="Raspbian GNU/Linux" VERSION_ID="8" VERSION="8 (jessie)" ID=raspbian ID_LIKE=debian HOME_URL="http://www.raspbian.org/" SUPPORT_URL="http://www.raspbian.org/RaspbianForums" BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs" uname:Linux 4.1.18-v7+ #846 SMP Thu Feb 25 14:22:53 GMT 2016 armv7l libc:glibc 2.19 NPTL 2.19 rlimit: STACK 8192k, CORE 0k, NPROC 7719, NOFILE 65536, AS infinity load average:1.36 0.61 0.23 /proc/meminfo: MemTotal: 996776 kB MemFree: 339060 kB MemAvailable: 531380 kB Buffers: 19752 kB Cached: 197208 kB SwapCached: 0 kB Active: 500964 kB Inactive: 119008 kB Active(anon): 403400 kB Inactive(anon): 7856 kB Active(file): 97564 kB Inactive(file): 111152 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 102396 kB SwapFree: 102396 kB Dirty: 156 kB Writeback: 0 kB AnonPages: 402992 kB Mapped: 77148 kB Shmem: 8316 kB Slab: 17608 kB SReclaimable: 8168 kB SUnreclaim: 9440 kB KernelStack: 4304 kB PageTables: 3788 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 600784 kB Committed_AS: 1410392 kB VmallocTotal: 1056768 kB VmallocUsed: 5176 kB VmallocChunk: 806504 kB CmaTotal: 8192 kB CmaFree: 3732 kB CPU:total 4 (ARMv7), vfp /proc/cpuinfo: processor : 0 model name : ARMv7 Processor rev 4 (v7l) BogoMIPS : 76.80 Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 CPU implementer : 0x41 CPU architecture: 7 CPU variant : 0x0 CPU part : 0xd03 CPU revision : 4 processor : 1 model name : ARMv7 Processor rev 4 (v7l) BogoMIPS : 76.80 Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 CPU implementer : 0x41 CPU architecture: 7 CPU variant : 0x0 CPU part : 0xd03 CPU revision : 4 processor : 2 model name : ARMv7 Processor rev 4 (v7l) BogoMIPS : 76.80 Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 CPU implementer : 0x41 CPU architecture: 7 CPU variant : 0x0 CPU part : 0xd03 CPU revision : 4 processor : 3 model name : ARMv7 Processor rev 4 (v7l) BogoMIPS : 76.80 Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32 CPU implementer : 0x41 CPU architecture: 7 CPU variant : 0x0 CPU part : 0xd03 CPU revision : 4 Hardware : BCM2709 Revision : a02082 Serial : 0000000010852207 Memory: 4k page, physical 996776k(339060k free), swap 102396k(102396k free) vm_info: Java HotSpot(TM) Client VM (25.65-b01) for linux-arm-vfp-hflt JRE (1.8.0_65-b17), built on Oct 6 2015 16:19:04 by "java_re" with gcc 4.7.2 20120910 (prerelease) time: Wed Jun 8 22:49:10 2016 elapsed time: 0 seconds (0d 0h 0m 0s) Could you please help me resolve the issue?? -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org