Thank you Billy and Manoj.
I already tuned up Xmx to 18gb [2] although Kylin still fails on startup.
It tries to build cube that I requested earlier. So now I can't even open
up Kylin UI.
It dumps .hprof file so I ran that through Eclipse Memory Analyzer and it
shows
only 1.5 Gb of heap was used..
[image: Inline image 1]
Any other idea how to fix or work around this issue?
kylin.out last lines before Kylin dies:
INFO: Starting ProtocolHandler ["ajp-bio-9009"]
> Jan 09, 2018 8:40:22 AM org.apache.catalina.startup.Catalina start
> INFO: Server startup in 12451 ms
> java.lang.OutOfMemoryError: Requested array size exceeds VM limit
> Dumping heap to java_pid24321.hprof ...
> Heap dump file created [1890369302 bytes in 3.112 secs]
> #
> # java.lang.OutOfMemoryError: Requested array size exceeds VM limit
> # -XX:OnOutOfMemoryError="kill -9 %p"
> # Executing /bin/sh -c "kill -9 24321"...
kylin.log last lines before Kylin dies:
2018-01-09 08:40:23,680 DEBUG [pool-7-thread-1] cachesync.Broadcaster:141 :
> Servers in the cluster: [pc1udatahad03:7070]
> 2018-01-09 08:40:23,747 DEBUG [pool-7-thread-1] cachesync.Broadcaster:148
> : Announcing new broadcast event: BroadcastEvent{entity=cube, event=update,
> cacheKey=mv2cube}
> 2018-01-09 08:40:24,644 DEBUG [http-bio-7070-exec-1]
> cachesync.Broadcaster:256 : BroadcastingUPDATE, cube, mv2cube
> 2018-01-09 08:40:24,647 INFO [http-bio-7070-exec-1] cube.CubeManager:796
> : Reloaded cube mv2cube being CUBE[name=mv2cube] having 1 segments
> 2018-01-09 08:40:24,656 DEBUG [http-bio-7070-exec-1]
> cachesync.Broadcaster:256 : BroadcastingUPDATE, project_data, Discover
> 2018-01-09 08:40:24,657 INFO [http-bio-7070-exec-1]
> service.CacheService:143 : removeOLAPDataSource is called for project
> Discover
> 2018-01-09 08:40:24,657 INFO [http-bio-7070-exec-1]
> service.CacheService:125 : cleaning cache for project Discover (currently
> remove all entries)
> 2018-01-09 08:40:24,661 DEBUG [http-bio-7070-exec-1]
> cachesync.Broadcaster:290 : Done broadcastingUPDATE, project_data, Discover
> 2018-01-09 08:40:24,664 DEBUG [http-bio-7070-exec-1]
> cachesync.Broadcaster:290 : Done broadcastingUPDATE, cube, mv2cube
[2]
export KYLIN_JVM_SETTINGS="*-Xms16g -Xmx18g* -XX:MaxPermSize=256m
> -XX:NewSize=3g -XX:MaxNewSize=3g -XX:SurvivorRatio=4
> -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled
> -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
> -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -verbose:gc
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> -Xloggc:$KYLIN_HOME/logs/kylin.gc.$$ -XX:+UseGCLogFileRotation
> -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=64M"
--
Ruslan Dautkhanov
On Tue, Jan 9, 2018 at 3:42 AM, Billy Liu <[email protected]> wrote:
> This document http://kylin.apache.org/docs21/install/advance_settings.html
> introduces how to allocate more memory for Kylin, could you have a try?
>
> 2018-01-09 15:42 GMT+08:00 Kumar, Manoj H <[email protected]>:
>
>> Whats JVM size given for Kylin? Try with 16 GB Memory mentioned in
>> setenv.sh file. You need to use max. JVM for loading billions of data.
>>
>>
>>
>> Regards,
>>
>> Manoj
>>
>>
>>
>> *From:* Ruslan Dautkhanov [[email protected]]
>> *Sent:* Tuesday, January 09, 2018 12:59 PM
>> *To:* [email protected]
>> *Subject:* Re: cachesync.Broadcaster:172 : error running wiping
>> java.lang.IllegalArgumentException
>>
>>
>>
>> Hi Billy,
>>
>>
>>
>> Thank you - that was it. Although it was just hiding real problem - with
>> jvm memory [1].
>>
>>
>>
>> I have requested to build a cube on 12b records and can't now make Kylin
>> back to stable state.
>>
>> Kylin crashes on startup.
>>
>> I have already tried to tune up Xmx and such but still no luck [2].
>>
>>
>>
>> [1]
>>
>>
>>
>> INFO: Starting ProtocolHandler ["ajp-bio-9009"]
>> Jan 09, 2018 12:09:49 AM org.apache.catalina.startup.Catalina start
>> INFO: Server startup in 12762 ms
>> java.lang.OutOfMemoryError: Requested array size exceeds VM limit
>> Dumping heap to java_pid60827.hprof ...
>> Heap dump file created [1731330948 bytes in 2.963 secs]
>> #
>> # java.lang.OutOfMemoryError: Requested array size exceeds VM limit
>> # -XX:OnOutOfMemoryError="kill -9 %p"
>> # Executing /bin/sh -c "kill -9 60827"...
>> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
>> MaxPermSize=256M; support was removed in 8.0
>> Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is
>> deprecated and will likely be removed in a future release
>> Error occurred during initialization of VM
>> java.lang.OutOfMemoryError: unable to create new native thread
>> at java.lang.Thread.start0(Native Method)
>> at java.lang.Thread.start(Thread.java:717)
>> at java.lang.ref.Reference.<clinit>(Reference.java:232)
>>
>>
>>
>>
>>
>> [2]
>>
>>
>>
>> export KYLIN_JVM_SETTINGS="-Xms2g -*Xmx16g* -Xss2g -XX:MaxPermSize=256M
>> -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps
>> -Xloggc:$KYLIN_HOME/logs/kylin.gc.$$ -XX:+UseGCLogFileRotation
>> -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=64M"
>>
>>
>>
>>
>>
>>
>>
>> --
>> Ruslan Dautkhanov
>>
>>
>>
>> On Mon, Jan 8, 2018 at 8:29 PM, Billy Liu <[email protected]> wrote:
>>
>> Hello Ruslan,
>>
>>
>>
>> Could you check the "*kylin.server.cluster-servers". There should no
>> "http://"; as prefix. *
>>
>>
>>
>> 2018-01-09 0:59 GMT+08:00 Ruslan Dautkhanov <[email protected]>:
>>
>> Below error crashes Kylin.
>>
>> Attempts to start Kylin back results in the same exception.
>>
>> This started happening after a new cube build was scheduled..
>>
>> Is this a known bug and any ideas how to work around this ?
>>
>>
>>
>>
>>
>> 2018-01-08 09:54:06,427 INFO [Scheduler 1698178565 Job
>> b595302c-35d4-45fd-a895-6766747eeae7-81] cube.CubeManager:358 : Updating
>> cube instance 'mv2cube'
>>
>>
>>
>> 2018-01-08 09:54:06,430 WARN [Scheduler 1698178565 Job
>> b595302c-35d4-45fd-a895-6766747eeae7-81] model.Segments:421 : NEW
>> segment start does not fit/connect with other segments:
>> mv2cube[20100101000000_20180101000000]
>>
>>
>>
>> 2018-01-08 09:54:06,430 WARN [Scheduler 1698178565 Job
>> b595302c-35d4-45fd-a895-6766747eeae7-81] model.Segments:423 : NEW
>> segment end does not fit/connect with other segments:
>> mv2cube[20100101000000_20180101000000]
>>
>>
>>
>>
>> 2018-01-08 09:54:06,454 INFO [Scheduler 1698178565 Job
>> b595302c-35d4-45fd-a895-6766747eeae7-81] cli.DictionaryGeneratorCLI:57 :
>> Building dictionary for MV_PROD.CONVERTED_FACT.INDIVID
>> 2018-01-08 09:54:06,454 DEBUG [pool-7-thread-1] cachesync.Broadcaster:141
>> : Servers in the cluster: [http://pc1udatahad03:7070]
>>
>>
>>
>>
>> 2018-01-08 09:54:06,466 ERROR [pool-7-thread-1]
>> *cachesync.Broadcaster:172 : error running wiping
>> java.lang.IllegalArgumentException*: URI: http://pc1udatahad03:7070 --
>> does not match pattern (?:([^:]+)[:]([^@]+)[@])?([^:]+)(?:[:](\d+))?
>> at org.apache.kylin.common.restclient.RestClient.<init>(RestCli
>> ent.java:91)
>> at org.apache.kylin.metadata.cachesync.Broadcaster$1.run(Broadc
>> aster.java:144)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1149)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>>
>>
>>
>>
>>
>> This message is confidential and subject to terms at: http://
>> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
>> privilege, viruses and monitoring of electronic messages. If you are not
>> the intended recipient, please delete this message and notify the sender
>> immediately. Any unauthorized use is strictly prohibited.
>>
>
>
