[jira] [Updated] (KYLIN-3872) Kylin web UI http 404 error
[
https://issues.apache.org/jira/browse/KYLIN-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mimimiracle updated KYLIN-3872:
---
Description:
kylin server is started but the web cannot be access
{code:java}
[root@server1 logs]# netstat -anp| grep 7070
tcp 0 0 0.0.0.0:7070 0.0.0.0:* LISTEN 11487/java
{code}
below is some of log records
{code:java}
2019-03-13 18:28:18,697 WARN [localhost-startStop-1]
scan.StandardJarScanner:146 : Failed to scan
[file:/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_ja_JP.jar]
from classloader hierarchy
java.io.FileNotFoundException:
/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_ja_JP.jar
(No such file or directory)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:225)
at java.util.zip.ZipFile.(ZipFile.java:155)
at java.util.jar.JarFile.(JarFile.java:166)
at java.util.jar.JarFile.(JarFile.java:130)
at
org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
at
org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:65)
at
org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
at
org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:374)
at
org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:309)
at
org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:266)
at
org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:229)
at
org.apache.catalina.startup.ContextConfig.processJarsForWebFragments(ContextConfig.java:1888)
at
org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1116)
at
org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:765)
at
org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:299)
at
org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5154)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
java.io.FileNotFoundException:
/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_pt_BR.jar
(No such file or directory)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:225)
at java.util.zip.ZipFile.(ZipFile.java:155)
at java.util.jar.JarFile.(JarFile.java:166)
at java.util.jar.JarFile.(JarFile.java:130)
at
org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
at
org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:65)
at
org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
at
org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:374)
at
org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:309)
at
org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:266)
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
at
org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
at
org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
at
org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
at
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4792)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5256)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at
[jira] [Comment Edited] (KYLIN-3872) Kylin web UI http 404 error
[
https://issues.apache.org/jira/browse/KYLIN-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792270#comment-16792270
]
mimimiracle edited comment on KYLIN-3872 at 3/14/19 3:10 AM:
-
Hi Chao Long
I use this
{panel}
Binary for Hadoop 3 download:
apache-kylin-2.6.1-bin-cdh60.tar.gz
{panel}
the log changed, May be this cased issue:ClassNotFoundException:
org.apache.commons.configuration.ConfigurationException
was (Author: mimimiracle):
Hi Chao Long
I use this
{panel}
Binary for Hadoop 3 download:
apache-kylin-2.6.1-bin-cdh60.tar.gz
{panel}
> Kylin web UI http 404 error
>
>
> Key: KYLIN-3872
> URL: https://issues.apache.org/jira/browse/KYLIN-3872
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.6.1
> Environment: CDH6.1
> Hadoop3.0.0+cdh6.1.0
> HDFS 3.0.0+cdh6.1.0
> YARN 3.0.0+cdh6.1.0
> HBase 2.1.0+cdh6.1.0
> Hive 2.1.1+cdh6.1.0
> spark 2.4.0+cdh6.1.0
> ZooKeeper 3.4.5+cdh6.1.0
>Reporter: mimimiracle
>Priority: Blocker
> Labels: Kylin, tomcat
>
> kylin server is started but the web cannot be access
> {code:java}
> [root@server1 logs]# netstat -anp| grep 7070
> tcp 0 0 0.0.0.0:7070 0.0.0.0:* LISTEN 11487/java
> {code}
> below is some of log records
> {code:java}
> 2019-03-13 18:28:18,697 WARN [localhost-startStop-1]
> scan.StandardJarScanner:146 : Failed to scan
> [file:/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_ja_JP.jar]
> from classloader hierarchy
> java.io.FileNotFoundException:
> /opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_ja_JP.jar
> (No such file or directory)
> at java.util.zip.ZipFile.open(Native Method)
> at java.util.zip.ZipFile.(ZipFile.java:225)
> at java.util.zip.ZipFile.(ZipFile.java:155)
> at java.util.jar.JarFile.(JarFile.java:166)
> at java.util.jar.JarFile.(JarFile.java:130)
> at
> org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
> at
> org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:65)
> at
> org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:374)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:309)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:266)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:229)
> at
> org.apache.catalina.startup.ContextConfig.processJarsForWebFragments(ContextConfig.java:1888)
> at
> org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1116)
> at
> org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:765)
> at
> org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:299)
> at
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
> at
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5154)
> at
> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
> at
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
> at
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
> java.io.FileNotFoundException:
> /opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_pt_BR.jar
> (No such file or directory)
> at java.util.zip.ZipFile.open(Native Method)
> at java.util.zip.ZipFile.(ZipFile.java:225)
> at java.util.zip.ZipFile.(ZipFile.java:155)
> at java.util.jar.JarFile.(JarFile.java:166)
> at java.util.jar.JarFile.(JarFile.java:130)
> at
> org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
> at
> org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:65)
> at
> org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:374)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:309)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:266)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
> at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
[jira] [Updated] (KYLIN-3872) Kylin web UI http 404 error
[
https://issues.apache.org/jira/browse/KYLIN-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mimimiracle updated KYLIN-3872:
---
Description:
kylin server is started but the web cannot be access
{code:java}
[root@server1 logs]# netstat -anp| grep 7070
tcp 0 0 0.0.0.0:7070 0.0.0.0:* LISTEN 11487/java
{code}
below is some of log records
{code:java}
2019-03-13 18:28:18,697 WARN [localhost-startStop-1]
scan.StandardJarScanner:146 : Failed to scan
[file:/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_ja_JP.jar]
from classloader hierarchy
java.io.FileNotFoundException:
/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_ja_JP.jar
(No such file or directory)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:225)
at java.util.zip.ZipFile.(ZipFile.java:155)
at java.util.jar.JarFile.(JarFile.java:166)
at java.util.jar.JarFile.(JarFile.java:130)
at
org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
at
org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:65)
at
org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
at
org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:374)
at
org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:309)
at
org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:266)
at
org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:229)
at
org.apache.catalina.startup.ContextConfig.processJarsForWebFragments(ContextConfig.java:1888)
at
org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1116)
at
org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:765)
at
org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:299)
at
org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5154)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
java.io.FileNotFoundException:
/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_pt_BR.jar
(No such file or directory)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:225)
at java.util.zip.ZipFile.(ZipFile.java:155)
at java.util.jar.JarFile.(JarFile.java:166)
at java.util.jar.JarFile.(JarFile.java:130)
at
org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
at
org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:65)
at
org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
at
org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:374)
at
org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:309)
at
org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:266)
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
at
org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
at
org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
at
org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
at
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4792)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5256)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at
[jira] [Updated] (KYLIN-3872) Kylin web UI http 404 error
[
https://issues.apache.org/jira/browse/KYLIN-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mimimiracle updated KYLIN-3872:
---
Description:
kylin server is started but the web cannot be access
{code:java}
[root@server1 logs]# netstat -anp| grep 7070
tcp 0 0 0.0.0.0:7070 0.0.0.0:* LISTEN 11487/java
{code}
below is some of log records
{code:java}
2019-03-13 18:28:18,697 WARN [localhost-startStop-1]
scan.StandardJarScanner:146 : Failed to scan
[file:/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_ja_JP.jar]
from classloader hierarchy
java.io.FileNotFoundException:
/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_ja_JP.jar
(No such file or directory)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:225)
at java.util.zip.ZipFile.(ZipFile.java:155)
at java.util.jar.JarFile.(JarFile.java:166)
at java.util.jar.JarFile.(JarFile.java:130)
at
org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
at
org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:65)
at
org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
at
org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:374)
at
org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:309)
at
org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:266)
at
org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:229)
at
org.apache.catalina.startup.ContextConfig.processJarsForWebFragments(ContextConfig.java:1888)
at
org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1116)
at
org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:765)
at
org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:299)
at
org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5154)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
java.io.FileNotFoundException:
/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_pt_BR.jar
(No such file or directory)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:225)
at java.util.zip.ZipFile.(ZipFile.java:155)
at java.util.jar.JarFile.(JarFile.java:166)
at java.util.jar.JarFile.(JarFile.java:130)
at
org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
at
org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:65)
at
org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
at
org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:374)
at
org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:309)
at
org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:266)
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
at
org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
at
org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
at
org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
at
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4792)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5256)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at
[jira] [Created] (KYLIN-3873) Improve SparkFactDistinct performance
Chao Long created KYLIN-3873: Summary: Improve SparkFactDistinct performance Key: KYLIN-3873 URL: https://issues.apache.org/jira/browse/KYLIN-3873 Project: Kylin Issue Type: Improvement Reporter: Chao Long Assignee: Chao Long Fix For: v2.6.2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3872) Kylin web UI http 404 error
[
https://issues.apache.org/jira/browse/KYLIN-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792270#comment-16792270
]
mimimiracle commented on KYLIN-3872:
Hi Chao Long
I use this
{panel}
Binary for Hadoop 3 download:
apache-kylin-2.6.1-bin-cdh60.tar.gz
{panel}
> Kylin web UI http 404 error
>
>
> Key: KYLIN-3872
> URL: https://issues.apache.org/jira/browse/KYLIN-3872
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.6.1
> Environment: CDH6.1
> Hadoop3.0.0+cdh6.1.0
> HDFS 3.0.0+cdh6.1.0
> YARN 3.0.0+cdh6.1.0
> HBase 2.1.0+cdh6.1.0
> Hive 2.1.1+cdh6.1.0
> spark 2.4.0+cdh6.1.0
> ZooKeeper 3.4.5+cdh6.1.0
>Reporter: mimimiracle
>Priority: Blocker
> Labels: Kylin, tomcat
>
> kylin server is started but the web cannot be access
> {code:java}
> [root@server1 logs]# netstat -anp| grep 7070
> tcp 0 0 0.0.0.0:7070 0.0.0.0:* LISTEN 11487/java
> {code}
> below is some of log records
> {code:java}
> 2019-03-13 18:28:18,697 WARN [localhost-startStop-1]
> scan.StandardJarScanner:146 : Failed to scan
> [file:/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_ja_JP.jar]
> from classloader hierarchy
> java.io.FileNotFoundException:
> /opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_ja_JP.jar
> (No such file or directory)
> at java.util.zip.ZipFile.open(Native Method)
> at java.util.zip.ZipFile.(ZipFile.java:225)
> at java.util.zip.ZipFile.(ZipFile.java:155)
> at java.util.jar.JarFile.(JarFile.java:166)
> at java.util.jar.JarFile.(JarFile.java:130)
> at
> org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
> at
> org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:65)
> at
> org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:374)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:309)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:266)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:229)
> at
> org.apache.catalina.startup.ContextConfig.processJarsForWebFragments(ContextConfig.java:1888)
> at
> org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1116)
> at
> org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:765)
> at
> org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:299)
> at
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
> at
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5154)
> at
> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
> at
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
> at
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
> java.io.FileNotFoundException:
> /opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_pt_BR.jar
> (No such file or directory)
> at java.util.zip.ZipFile.open(Native Method)
> at java.util.zip.ZipFile.(ZipFile.java:225)
> at java.util.zip.ZipFile.(ZipFile.java:155)
> at java.util.jar.JarFile.(JarFile.java:166)
> at java.util.jar.JarFile.(JarFile.java:130)
> at
> org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
> at
> org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:65)
> at
> org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:374)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:309)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:266)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
> at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
> at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
> at
>
[GitHub] [kylin] codecov-io edited a comment on issue #507: KYLIN-3857 Disable quote when needed
codecov-io edited a comment on issue #507: KYLIN-3857 Disable quote when needed URL: https://github.com/apache/kylin/pull/507#issuecomment-471365737 # [Codecov](https://codecov.io/gh/apache/kylin/pull/507?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@fe43a3e`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `35%`. [](https://codecov.io/gh/apache/kylin/pull/507?src=pr=tree) ```diff @@Coverage Diff@@ ## master #507 +/- ## = Coverage ? 24.44% Complexity? 5025 = Files ? 1160 Lines ?70407 Branches ?10015 = Hits ?17212 Misses?51457 Partials ? 1738 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/507?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../java/org/apache/kylin/common/KylinConfigBase.java](https://codecov.io/gh/apache/kylin/pull/507/diff?src=pr=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi9LeWxpbkNvbmZpZ0Jhc2UuamF2YQ==) | `14.21% <0%> (ø)` | `42 <0> (?)` | | | [...kylin/source/adhocquery/HivePushDownConverter.java](https://codecov.io/gh/apache/kylin/pull/507/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vc291cmNlL2FkaG9jcXVlcnkvSGl2ZVB1c2hEb3duQ29udmVydGVyLmphdmE=) | `65.51% <0%> (ø)` | `16 <0> (?)` | | | [...pache/kylin/source/hive/GarbageCollectionStep.java](https://codecov.io/gh/apache/kylin/pull/507/diff?src=pr=tree#diff-c291cmNlLWhpdmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3NvdXJjZS9oaXZlL0dhcmJhZ2VDb2xsZWN0aW9uU3RlcC5qYXZh) | `0% <0%> (ø)` | `0 <0> (?)` | | | [...java/org/apache/kylin/metadata/model/TableRef.java](https://codecov.io/gh/apache/kylin/pull/507/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vbWV0YWRhdGEvbW9kZWwvVGFibGVSZWYuamF2YQ==) | `50% <0%> (ø)` | `11 <0> (?)` | | | [...ain/java/org/apache/kylin/job/JoinedFlatTable.java](https://codecov.io/gh/apache/kylin/pull/507/diff?src=pr=tree#diff-Y29yZS1qb2Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2pvYi9Kb2luZWRGbGF0VGFibGUuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (?)` | | | [...pache/kylin/source/datagen/ModelDataGenerator.java](https://codecov.io/gh/apache/kylin/pull/507/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vc291cmNlL2RhdGFnZW4vTW9kZWxEYXRhR2VuZXJhdG9yLmphdmE=) | `89.3% <100%> (ø)` | `53 <1> (?)` | | | [...a/org/apache/kylin/common/util/HiveCmdBuilder.java](https://codecov.io/gh/apache/kylin/pull/507/diff?src=pr=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi91dGlsL0hpdmVDbWRCdWlsZGVyLmphdmE=) | `82.19% <100%> (ø)` | `12 <0> (?)` | | | [...in/sdk/datasource/framework/conv/SqlConverter.java](https://codecov.io/gh/apache/kylin/pull/507/diff?src=pr=tree#diff-ZGF0YXNvdXJjZS1zZGsvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Nkay9kYXRhc291cmNlL2ZyYW1ld29yay9jb252L1NxbENvbnZlcnRlci5qYXZh) | `90.62% <100%> (ø)` | `7 <3> (?)` | | | [.../apache/kylin/job/util/FlatTableSqlQuoteUtils.java](https://codecov.io/gh/apache/kylin/pull/507/diff?src=pr=tree#diff-Y29yZS1qb2Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2pvYi91dGlsL0ZsYXRUYWJsZVNxbFF1b3RlVXRpbHMuamF2YQ==) | `21.81% <37.5%> (ø)` | `10 <2> (?)` | | | [...va/org/apache/kylin/source/hive/HiveInputBase.java](https://codecov.io/gh/apache/kylin/pull/507/diff?src=pr=tree#diff-c291cmNlLWhpdmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3NvdXJjZS9oaXZlL0hpdmVJbnB1dEJhc2UuamF2YQ==) | `13.17% <62.5%> (ø)` | `4 <0> (?)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/507?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/507?src=pr=footer). Last update [fe43a3e...4011eff](https://codecov.io/gh/apache/kylin/pull/507?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at:
[GitHub] [kylin] hit-lacus commented on a change in pull request #507: KYLIN-3857 Disable quote when needed
hit-lacus commented on a change in pull request #507: KYLIN-3857 Disable quote
when needed
URL: https://github.com/apache/kylin/pull/507#discussion_r265395327
##
File path:
core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
##
@@ -982,6 +982,14 @@ public int getDefaultDecimalScale() {
}
}
+public boolean enableHiveDdlQuote(){
+return
Boolean.parseBoolean(getOptional("kylin.source.hive.enable.quote", TRUE));
Review comment:
I have fix this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (KYLIN-3862) Check the binary packages
[ https://issues.apache.org/jira/browse/KYLIN-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792233#comment-16792233 ] ASF subversion and git services commented on KYLIN-3862: Commit 2bdd4e1263b705b21f7276d1007935bf8bdb4ebd in kylin's branch refs/heads/2.6.x from Shao Feng Shi [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=2bdd4e1 ] KYLIN-3862 add LICENSE and NOTICE to binary package > Check the binary packages > - > > Key: KYLIN-3862 > URL: https://issues.apache.org/jira/browse/KYLIN-3862 > Project: Kylin > Issue Type: Task >Reporter: Shaofeng SHI >Priority: Major > > As to the approval of binary packages: > > It's not possible in general to check the exact contents of a binary, however > there are some checks that should be made: > - sigs and hashes must be OK > - the package must contain the correct NOTICE and LICENSE files for the > included content > - the package must not contain any content not derived from the source. > - in the case of bundled binaries, reviewers must check that all contents are > represented in the LICENSE (and NOTICE file if required). > The bundle must not contain any files that are prohibited from distribution > (category X). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3862) Check the binary packages
[ https://issues.apache.org/jira/browse/KYLIN-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792236#comment-16792236 ] ASF subversion and git services commented on KYLIN-3862: Commit acfb865fe339b3ee3cc3dd84ad01d0cc1df8942d in kylin's branch refs/heads/2.5.x from Shao Feng Shi [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=acfb865 ] KYLIN-3862 add LICENSE and NOTICE to binary package > Check the binary packages > - > > Key: KYLIN-3862 > URL: https://issues.apache.org/jira/browse/KYLIN-3862 > Project: Kylin > Issue Type: Task >Reporter: Shaofeng SHI >Priority: Major > > As to the approval of binary packages: > > It's not possible in general to check the exact contents of a binary, however > there are some checks that should be made: > - sigs and hashes must be OK > - the package must contain the correct NOTICE and LICENSE files for the > included content > - the package must not contain any content not derived from the source. > - in the case of bundled binaries, reviewers must check that all contents are > represented in the LICENSE (and NOTICE file if required). > The bundle must not contain any files that are prohibited from distribution > (category X). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3571) Not build Spark in Kylin's binary package
[ https://issues.apache.org/jira/browse/KYLIN-3571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792235#comment-16792235 ] ASF subversion and git services commented on KYLIN-3571: Commit 2e4c53aeaaa61a5b0f90f2f7f4e006afbaea727a in kylin's branch refs/heads/2.5.x from chao long [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=2e4c53a ] KYLIN-3571 Not build Spark in Kylin's binary package > Not build Spark in Kylin's binary package > - > > Key: KYLIN-3571 > URL: https://issues.apache.org/jira/browse/KYLIN-3571 > Project: Kylin > Issue Type: Improvement > Components: Environment >Reporter: Shaofeng SHI >Assignee: Chao Long >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3868) Implement Flink batch cubing job builder
[ https://issues.apache.org/jira/browse/KYLIN-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792221#comment-16792221 ] ASF subversion and git services commented on KYLIN-3868: Commit 3e5eda7acddbb9fa6bb22b5251cbffd553826742 in kylin's branch refs/heads/engine-flink from yanghua [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=3e5eda7 ] KYLIN-3868 Implement Flink batch cubing job builder > Implement Flink batch cubing job builder > > > Key: KYLIN-3868 > URL: https://issues.apache.org/jira/browse/KYLIN-3868 > Project: Kylin > Issue Type: Sub-task > Components: Flink Engine >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > > The builder used to config the cube build steps. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3868) Implement Flink batch cubing job builder
[ https://issues.apache.org/jira/browse/KYLIN-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792220#comment-16792220 ] ASF GitHub Bot commented on KYLIN-3868: --- shaofengshi commented on pull request #514: KYLIN-3868 Implement Flink batch cubing job builder URL: https://github.com/apache/kylin/pull/514 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement Flink batch cubing job builder > > > Key: KYLIN-3868 > URL: https://issues.apache.org/jira/browse/KYLIN-3868 > Project: Kylin > Issue Type: Sub-task > Components: Flink Engine >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > > The builder used to config the cube build steps. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] shaofengshi merged pull request #514: KYLIN-3868 Implement Flink batch cubing job builder
shaofengshi merged pull request #514: KYLIN-3868 Implement Flink batch cubing job builder URL: https://github.com/apache/kylin/pull/514 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] codecov-io commented on issue #514: KYLIN-3868 Implement Flink batch cubing job builder
codecov-io commented on issue #514: KYLIN-3868 Implement Flink batch cubing job builder URL: https://github.com/apache/kylin/pull/514#issuecomment-472498731 # [Codecov](https://codecov.io/gh/apache/kylin/pull/514?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`engine-flink@d4f8eb2`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `0%`. [](https://codecov.io/gh/apache/kylin/pull/514?src=pr=tree) ```diff @@ Coverage Diff @@ ## engine-flink #514 +/- ## === Coverage? 24.27% Complexity ? 5029 === Files ? 1166 Lines ?70903 Branches?10078 === Hits?17215 Misses ?51948 Partials? 1740 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/514?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...org/apache/kylin/engine/flink/FlinkExecutable.java](https://codecov.io/gh/apache/kylin/pull/514/diff?src=pr=tree#diff-ZW5naW5lLWZsaW5rL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9lbmdpbmUvZmxpbmsvRmxpbmtFeGVjdXRhYmxlLmphdmE=) | `0% <0%> (ø)` | `0 <0> (?)` | | | [...ylin/engine/flink/FlinkBatchCubingJobBuilder2.java](https://codecov.io/gh/apache/kylin/pull/514/diff?src=pr=tree#diff-ZW5naW5lLWZsaW5rL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9lbmdpbmUvZmxpbmsvRmxpbmtCYXRjaEN1YmluZ0pvYkJ1aWxkZXIyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (?)` | | | [...e/kylin/engine/flink/FlinkOnYarnConfigMapping.java](https://codecov.io/gh/apache/kylin/pull/514/diff?src=pr=tree#diff-ZW5naW5lLWZsaW5rL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9lbmdpbmUvZmxpbmsvRmxpbmtPbllhcm5Db25maWdNYXBwaW5nLmphdmE=) | `0% <0%> (ø)` | `0 <0> (?)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/514?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/514?src=pr=footer). Last update [d4f8eb2...af1af4b](https://codecov.io/gh/apache/kylin/pull/514?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3868) Implement Flink batch cubing job builder
[ https://issues.apache.org/jira/browse/KYLIN-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791799#comment-16791799 ] ASF GitHub Bot commented on KYLIN-3868: --- yanghua commented on pull request #514: KYLIN-3868 Implement Flink batch cubing job builder URL: https://github.com/apache/kylin/pull/514 JIRA ref : https://issues.apache.org/jira/browse/KYLIN-3868 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement Flink batch cubing job builder > > > Key: KYLIN-3868 > URL: https://issues.apache.org/jira/browse/KYLIN-3868 > Project: Kylin > Issue Type: Sub-task > Components: Flink Engine >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > > The builder used to config the cube build steps. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] asfgit commented on issue #514: KYLIN-3868 Implement Flink batch cubing job builder
asfgit commented on issue #514: KYLIN-3868 Implement Flink batch cubing job builder URL: https://github.com/apache/kylin/pull/514#issuecomment-472463994 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] yanghua opened a new pull request #514: KYLIN-3868 Implement Flink batch cubing job builder
yanghua opened a new pull request #514: KYLIN-3868 Implement Flink batch cubing job builder URL: https://github.com/apache/kylin/pull/514 JIRA ref : https://issues.apache.org/jira/browse/KYLIN-3868 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3862) Check the binary packages
[ https://issues.apache.org/jira/browse/KYLIN-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791788#comment-16791788 ] ASF subversion and git services commented on KYLIN-3862: Commit fe43a3eba2716a8a534bb7916fbc425b22849157 in kylin's branch refs/heads/master from Shao Feng Shi [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=fe43a3e ] KYLIN-3862 add LICENSE and NOTICE to binary package > Check the binary packages > - > > Key: KYLIN-3862 > URL: https://issues.apache.org/jira/browse/KYLIN-3862 > Project: Kylin > Issue Type: Task >Reporter: Shaofeng SHI >Priority: Major > > As to the approval of binary packages: > > It's not possible in general to check the exact contents of a binary, however > there are some checks that should be made: > - sigs and hashes must be OK > - the package must contain the correct NOTICE and LICENSE files for the > included content > - the package must not contain any content not derived from the source. > - in the case of bundled binaries, reviewers must check that all contents are > represented in the LICENSE (and NOTICE file if required). > The bundle must not contain any files that are prohibited from distribution > (category X). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3824) Spark - Extract Fact Table Distinct Columns step causes java.lang.OutOfMemoryError: Java heap space
[ https://issues.apache.org/jira/browse/KYLIN-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3824. --- > Spark - Extract Fact Table Distinct Columns step causes > java.lang.OutOfMemoryError: Java heap space > --- > > Key: KYLIN-3824 > URL: https://issues.apache.org/jira/browse/KYLIN-3824 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Affects Versions: v2.5.0, v2.6.0, v2.5.1, v2.5.2 > Environment: CentOS 7 > 3 workers and 1 master. > 4 cpu, 16GB RAM each >Reporter: Alexander >Assignee: Alexander >Priority: Major > Fix For: v2.6.1 > > Attachments: KYLIN-3824.master.001.patch > > > Try to build huge cube on weak envirment. > Environment: > Cluster with 3 nodes. > Max AM container size - 5GB. > > kylin_intermediate table ~500 files of size started from 4kb up to 300mb. > > When spark job executor take file larger than ~70MB on step > mapPartitionsToPair (194) it got exception: > 2019-02-21 20:29:40 ERROR SparkUncaughtExceptionHandler:91 - [Container in > shutdown] Uncaught exception in thread Thread[Executor task launch worker for > task 1,5,main] > java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOfRange(Arrays.java:3664) > at java.lang.String.(String.java:207) > at java.lang.String.substring(String.java:1969) > at java.lang.String.split(String.java:2353) > at java.lang.String.split(String.java:2422) > at org.apache.kylin.engine.spark.SparkUtil$1.call(SparkUtil.java:164) > at org.apache.kylin.engine.spark.SparkUtil$1.call(SparkUtil.java:160) > at > org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1.apply(JavaPairRDD.scala:1040) > at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) > at scala.collection.convert.Wrappers$IteratorWrapper.next(Wrappers.scala:31) > at com.google.common.collect.Lists.newArrayList(Lists.java:145) > at > org.apache.kylin.engine.spark.SparkFactDistinct$FlatOutputFucntion.call(SparkFactDistinct.java:313) > at > org.apache.kylin.engine.spark.SparkFactDistinct$FlatOutputFucntion.call(SparkFactDistinct.java:239) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:186) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:186) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:801) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:801) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:49) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) > at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) > at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) > at org.apache.spark.scheduler.Task.run(Task.scala:109) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3824) Spark - Extract Fact Table Distinct Columns step causes java.lang.OutOfMemoryError: Java heap space
[ https://issues.apache.org/jira/browse/KYLIN-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3824: Fix Version/s: (was: v3.0.0) v2.6.1 > Spark - Extract Fact Table Distinct Columns step causes > java.lang.OutOfMemoryError: Java heap space > --- > > Key: KYLIN-3824 > URL: https://issues.apache.org/jira/browse/KYLIN-3824 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Affects Versions: v2.5.0, v2.6.0, v2.5.1, v2.5.2 > Environment: CentOS 7 > 3 workers and 1 master. > 4 cpu, 16GB RAM each >Reporter: Alexander >Assignee: Alexander >Priority: Major > Fix For: v2.6.1 > > Attachments: KYLIN-3824.master.001.patch > > > Try to build huge cube on weak envirment. > Environment: > Cluster with 3 nodes. > Max AM container size - 5GB. > > kylin_intermediate table ~500 files of size started from 4kb up to 300mb. > > When spark job executor take file larger than ~70MB on step > mapPartitionsToPair (194) it got exception: > 2019-02-21 20:29:40 ERROR SparkUncaughtExceptionHandler:91 - [Container in > shutdown] Uncaught exception in thread Thread[Executor task launch worker for > task 1,5,main] > java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOfRange(Arrays.java:3664) > at java.lang.String.(String.java:207) > at java.lang.String.substring(String.java:1969) > at java.lang.String.split(String.java:2353) > at java.lang.String.split(String.java:2422) > at org.apache.kylin.engine.spark.SparkUtil$1.call(SparkUtil.java:164) > at org.apache.kylin.engine.spark.SparkUtil$1.call(SparkUtil.java:160) > at > org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1.apply(JavaPairRDD.scala:1040) > at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) > at scala.collection.convert.Wrappers$IteratorWrapper.next(Wrappers.scala:31) > at com.google.common.collect.Lists.newArrayList(Lists.java:145) > at > org.apache.kylin.engine.spark.SparkFactDistinct$FlatOutputFucntion.call(SparkFactDistinct.java:313) > at > org.apache.kylin.engine.spark.SparkFactDistinct$FlatOutputFucntion.call(SparkFactDistinct.java:239) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:186) > at > org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:186) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:801) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:801) > at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:49) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) > at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) > at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) > at org.apache.spark.scheduler.Task.run(Task.scala:109) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3872) Kylin web UI http 404 error
[
https://issues.apache.org/jira/browse/KYLIN-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791664#comment-16791664
]
Chao Long edited comment on KYLIN-3872 at 3/13/19 12:50 PM:
Hi [~mimimiracle], did you use kylin binary package for cdh6?
was (Author: wayne0101):
Hi [~mimimiracle], did you use the binary package for cdh6?
> Kylin web UI http 404 error
>
>
> Key: KYLIN-3872
> URL: https://issues.apache.org/jira/browse/KYLIN-3872
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.6.1
> Environment: CDH6.1
> Hadoop3.0.0+cdh6.1.0
> HDFS 3.0.0+cdh6.1.0
> YARN 3.0.0+cdh6.1.0
> HBase 2.1.0+cdh6.1.0
> Hive 2.1.1+cdh6.1.0
> spark 2.4.0+cdh6.1.0
> ZooKeeper 3.4.5+cdh6.1.0
>Reporter: mimimiracle
>Priority: Blocker
> Labels: Kylin, tomcat
>
> kylin server is started but the web cannot be access
> {code:java}
> [root@server1 logs]# netstat -anp| grep 7070
> tcp 0 0 0.0.0.0:7070 0.0.0.0:* LISTEN 11487/java
> {code}
> below is some of log records
> {code:java}
> 2019-03-13 18:28:18,697 WARN [localhost-startStop-1]
> scan.StandardJarScanner:146 : Failed to scan
> [file:/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_ja_JP.jar]
> from classloader hierarchy
> java.io.FileNotFoundException:
> /opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_ja_JP.jar
> (No such file or directory)
> at java.util.zip.ZipFile.open(Native Method)
> at java.util.zip.ZipFile.(ZipFile.java:225)
> at java.util.zip.ZipFile.(ZipFile.java:155)
> at java.util.jar.JarFile.(JarFile.java:166)
> at java.util.jar.JarFile.(JarFile.java:130)
> at
> org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
> at
> org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:65)
> at
> org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:374)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:309)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:266)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:229)
> at
> org.apache.catalina.startup.ContextConfig.processJarsForWebFragments(ContextConfig.java:1888)
> at
> org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1116)
> at
> org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:765)
> at
> org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:299)
> at
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
> at
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5154)
> at
> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
> at
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
> at
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
> java.io.FileNotFoundException:
> /opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_pt_BR.jar
> (No such file or directory)
> at java.util.zip.ZipFile.open(Native Method)
> at java.util.zip.ZipFile.(ZipFile.java:225)
> at java.util.zip.ZipFile.(ZipFile.java:155)
> at java.util.jar.JarFile.(JarFile.java:166)
> at java.util.jar.JarFile.(JarFile.java:130)
> at
> org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
> at
> org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:65)
> at
> org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:374)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:309)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:266)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
> at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
> at
>
[jira] [Commented] (KYLIN-3872) Kylin web UI http 404 error
[
https://issues.apache.org/jira/browse/KYLIN-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791664#comment-16791664
]
Chao Long commented on KYLIN-3872:
--
Hi [~mimimiracle], did you use the binary package for cdh6?
> Kylin web UI http 404 error
>
>
> Key: KYLIN-3872
> URL: https://issues.apache.org/jira/browse/KYLIN-3872
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.6.1
> Environment: CDH6.1
> Hadoop3.0.0+cdh6.1.0
> HDFS 3.0.0+cdh6.1.0
> YARN 3.0.0+cdh6.1.0
> HBase 2.1.0+cdh6.1.0
> Hive 2.1.1+cdh6.1.0
> spark 2.4.0+cdh6.1.0
> ZooKeeper 3.4.5+cdh6.1.0
>Reporter: mimimiracle
>Priority: Blocker
> Labels: Kylin, tomcat
>
> kylin server is started but the web cannot be access
> {code:java}
> [root@server1 logs]# netstat -anp| grep 7070
> tcp 0 0 0.0.0.0:7070 0.0.0.0:* LISTEN 11487/java
> {code}
> below is some of log records
> {code:java}
> 2019-03-13 18:28:18,697 WARN [localhost-startStop-1]
> scan.StandardJarScanner:146 : Failed to scan
> [file:/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_ja_JP.jar]
> from classloader hierarchy
> java.io.FileNotFoundException:
> /opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_ja_JP.jar
> (No such file or directory)
> at java.util.zip.ZipFile.open(Native Method)
> at java.util.zip.ZipFile.(ZipFile.java:225)
> at java.util.zip.ZipFile.(ZipFile.java:155)
> at java.util.jar.JarFile.(JarFile.java:166)
> at java.util.jar.JarFile.(JarFile.java:130)
> at
> org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
> at
> org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:65)
> at
> org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:374)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:309)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:266)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:229)
> at
> org.apache.catalina.startup.ContextConfig.processJarsForWebFragments(ContextConfig.java:1888)
> at
> org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1116)
> at
> org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:765)
> at
> org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:299)
> at
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
> at
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5154)
> at
> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
> at
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
> at
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
> java.io.FileNotFoundException:
> /opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_pt_BR.jar
> (No such file or directory)
> at java.util.zip.ZipFile.open(Native Method)
> at java.util.zip.ZipFile.(ZipFile.java:225)
> at java.util.zip.ZipFile.(ZipFile.java:155)
> at java.util.jar.JarFile.(JarFile.java:166)
> at java.util.jar.JarFile.(JarFile.java:130)
> at
> org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
> at
> org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:65)
> at
> org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:374)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:309)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:266)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
> at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
> at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
> at
>
[jira] [Commented] (KYLIN-3866) Whether to set mapreduce.application.classpath is determined by the user
[ https://issues.apache.org/jira/browse/KYLIN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791612#comment-16791612 ] Guangxu Cheng commented on KYLIN-3866: -- Add parameter "kylin.engine.mr.use-local-classpath" and set the default value to true, mainly to keep the compatibility of versions. For old users, upgrading kylin to a higher version has no effect on them. For new users, this feature can be turned off with this parameter if it is not needed. If the changes can be listed in the upgrade document to remind the old user.I personally think it is okay to remove this logic. WDYT? Thanks > Whether to set mapreduce.application.classpath is determined by the user > > > Key: KYLIN-3866 > URL: https://issues.apache.org/jira/browse/KYLIN-3866 > Project: Kylin > Issue Type: Bug >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Attachments: KYLIN-3866.master.001.patch > > > In the kylin environment, if there is no configuration parameter > "mapreduce.application.classpath", then kylin will use the command "mapred > classpath" to get the local classpath and set it to > "mapreduce.application.classpath", However, it will cause the following two > problems: > 1. The local classpath may not be the same as the MapReduce cluster. > 2. There is no need to configure the parameter > "mapreduce.application.classpath" on the client side in some cluster. > So, it may be a better choice for users to decide whether they need to > configure this parameter or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3872) Kylin web UI http 404 error
mimimiracle created KYLIN-3872:
--
Summary: Kylin web UI http 404 error
Key: KYLIN-3872
URL: https://issues.apache.org/jira/browse/KYLIN-3872
Project: Kylin
Issue Type: Bug
Affects Versions: v2.6.1
Environment: CDH6.1
Hadoop 3.0.0+cdh6.1.0
HDFS3.0.0+cdh6.1.0
YARN3.0.0+cdh6.1.0
HBase 2.1.0+cdh6.1.0
Hive 2.1.1+cdh6.1.0
spark 2.4.0+cdh6.1.0
ZooKeeper 3.4.5+cdh6.1.0
Reporter: mimimiracle
kylin server is started but the web cannot be access
{code:java}
[root@server1 logs]# netstat -anp| grep 7070
tcp 0 0 0.0.0.0:7070 0.0.0.0:* LISTEN 11487/java
{code}
below is some of log records
{code:java}
2019-03-13 18:28:18,697 WARN [localhost-startStop-1]
scan.StandardJarScanner:146 : Failed to scan
[file:/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_ja_JP.jar]
from classloader hierarchy
java.io.FileNotFoundException:
/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_ja_JP.jar
(No such file or directory)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:225)
at java.util.zip.ZipFile.(ZipFile.java:155)
at java.util.jar.JarFile.(JarFile.java:166)
at java.util.jar.JarFile.(JarFile.java:130)
at
org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
at
org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:65)
at
org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
at
org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:374)
at
org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:309)
at
org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:266)
at
org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:229)
at
org.apache.catalina.startup.ContextConfig.processJarsForWebFragments(ContextConfig.java:1888)
at
org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1116)
at
org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:765)
at
org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:299)
at
org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5154)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
java.io.FileNotFoundException:
/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_pt_BR.jar
(No such file or directory)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:225)
at java.util.zip.ZipFile.(ZipFile.java:155)
at java.util.jar.JarFile.(JarFile.java:166)
at java.util.jar.JarFile.(JarFile.java:130)
at
org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
at
org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:65)
at
org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
at
org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:374)
at
org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:309)
at
org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:266)
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
at
org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
at
org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
at
[jira] [Resolved] (KYLIN-3830) Wrong result when 'SELECT SUM(dim1)' without set a relative metric of dim1.
[ https://issues.apache.org/jira/browse/KYLIN-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-3830. - Resolution: Fixed > Wrong result when 'SELECT SUM(dim1)' without set a relative metric of dim1. > --- > > Key: KYLIN-3830 > URL: https://issues.apache.org/jira/browse/KYLIN-3830 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.5.2 >Reporter: Yuzhang QIU >Assignee: Yuzhang QIU >Priority: Major > Fix For: v2.6.2 > > > Hi, dear team: > I design an cube1 based on table table1 with dim1, dim2, dim3 and only > one metric count(1), and 'SELECT SUM(dim1) FROM table1 group by dim2', Kylin > process this SQL and return some result1. It seems ok. But as we know, Kylin > don't store the detail data, the dimensions' members have been encoded and > stored in Hbase as rowkey(cause I don't set any metric with an column). So, > is the result1 right? > Then, I clone cube1 to cube2, and set a metric SUM(dim1). the same SQL has > been passed to kylin and got result2. It's different from result1 at the > aggregation field. I also pass same SQL to hive and got result3, it's same > with result2. > Yes, I turn off the pushdown. > I think there are some problems. > I can't upload some picture of results for secret policy, sorry for that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3866) Whether to set mapreduce.application.classpath is determined by the user
[ https://issues.apache.org/jira/browse/KYLIN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI reassigned KYLIN-3866: --- Assignee: Guangxu Cheng > Whether to set mapreduce.application.classpath is determined by the user > > > Key: KYLIN-3866 > URL: https://issues.apache.org/jira/browse/KYLIN-3866 > Project: Kylin > Issue Type: Bug >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Attachments: KYLIN-3866.master.001.patch > > > In the kylin environment, if there is no configuration parameter > "mapreduce.application.classpath", then kylin will use the command "mapred > classpath" to get the local classpath and set it to > "mapreduce.application.classpath", However, it will cause the following two > problems: > 1. The local classpath may not be the same as the MapReduce cluster. > 2. There is no need to configure the parameter > "mapreduce.application.classpath" on the client side in some cluster. > So, it may be a better choice for users to decide whether they need to > configure this parameter or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3866) Whether to set mapreduce.application.classpath is determined by the user
[ https://issues.apache.org/jira/browse/KYLIN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791575#comment-16791575 ] Shaofeng SHI commented on KYLIN-3866: - Now I know the problem, thank you for the information! The change will be good; I see you add a config parameter to keep the old behavior, is that parameter is necessary? I mean, if the "mapreduce.application.classpath" is absent, can we just let it go (totally not get the classpath from local configuration)? > Whether to set mapreduce.application.classpath is determined by the user > > > Key: KYLIN-3866 > URL: https://issues.apache.org/jira/browse/KYLIN-3866 > Project: Kylin > Issue Type: Bug >Reporter: Guangxu Cheng >Priority: Major > Attachments: KYLIN-3866.master.001.patch > > > In the kylin environment, if there is no configuration parameter > "mapreduce.application.classpath", then kylin will use the command "mapred > classpath" to get the local classpath and set it to > "mapreduce.application.classpath", However, it will cause the following two > problems: > 1. The local classpath may not be the same as the MapReduce cluster. > 2. There is no need to configure the parameter > "mapreduce.application.classpath" on the client side in some cluster. > So, it may be a better choice for users to decide whether they need to > configure this parameter or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3871) Kylin inside Cloudera CDH Quickstart Sandbox
[ https://issues.apache.org/jira/browse/KYLIN-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791552#comment-16791552 ] Shaofeng SHI commented on KYLIN-3871: - To run the integration test, Hortonworks sandbox 2.4 is the only choice I think. We didn't do that on CDH sandbox. > Kylin inside Cloudera CDH Quickstart Sandbox > > > Key: KYLIN-3871 > URL: https://issues.apache.org/jira/browse/KYLIN-3871 > Project: Kylin > Issue Type: Test > Components: Job Engine, Real-time Streaming >Affects Versions: v2.6.0 > Environment: Cloudera Quickstart Docker image: > - OS: centos6 > - memory: 13GB > - disk: 20GB > - java: 1.8 > - maven: 3.5.3 >Reporter: Yanwen Lin >Priority: Blocker > > When doing integration test, I met the following error. I know this is > related to Java version error and the reason is that Kylin use java1.8 while > Cloudera use java1.7. So I manually installed java 1.8 and set JAVA_HOME to > point to Spark2.x. (I also type spark-submit --version to check this). But > the bug did not go away. I guess during the process of Spark job, some > command may change the Java version back to java1.7 (not sure). Is there > anyway to force it not change back to Java1.7 or any workaround? > I have successfully finished maven installing and unit tests. > *Branch:* > realtime-streaming > *Executed command with problem:* > mvn verify -fae -Dhdp.version=2.4.0.0-169 -P sandbox > *Error stack:* > Exception in thread "main" java.lang.UnsupportedClassVersionError: > org/apache/spark/network/util/ByteUnit : Unsupported major.minor version 52.0 > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:800) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at org.apache.spark.deploy.history.config$.(config.scala:44) > at org.apache.spark.deploy.history.config$.(config.scala) > at org.apache.spark.SparkConf$.(SparkConf.scala:635) > at org.apache.spark.SparkConf$.(SparkConf.scala) > at org.apache.spark.SparkConf.set(SparkConf.scala:94) > at > org.apache.spark.SparkConf$$anonfun$loadFromSystemProperties$3.apply(SparkConf.scala:76) > at > org.apache.spark.SparkConf$$anonfun$loadFromSystemProperties$3.apply(SparkConf.scala:75) > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) > at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221) > at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428) > at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428) > at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428) > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) > at org.apache.spark.SparkConf.loadFromSystemProperties(SparkConf.scala:75) > at org.apache.spark.SparkConf.(SparkConf.scala:70) > at org.apache.spark.SparkConf.(SparkConf.scala:57) > at > org.apache.spark.deploy.yarn.ApplicationMaster.(ApplicationMaster.scala:62) > at > org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:838) > at > org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:869) > at > org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3867) Enable JDBC to use key store & trust store for https connection
[
https://issues.apache.org/jira/browse/KYLIN-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791529#comment-16791529
]
ASF subversion and git services commented on KYLIN-3867:
Commit 699d650b076d6309f1087ee3c95705c253616075 in kylin's branch
refs/heads/master from kyotoYaho
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=699d650 ]
KYLIN-3867 leverage jdbc specified properties to enable JDBC to use key store &
trust store for https connection
> Enable JDBC to use key store & trust store for https connection
> ---
>
> Key: KYLIN-3867
> URL: https://issues.apache.org/jira/browse/KYLIN-3867
> Project: Kylin
> Issue Type: Improvement
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v3.0.0
>
>
> In some cases, JDBC clients need to pass its client cert & key to the
> downstream for authentication. Then the JDBC needs to create a connection
> with key store & trust store.
> {code}
> import java.io.BufferedReader;
> import java.io.InputStream;
> import java.io.InputStreamReader;
> import java.net.URL;
> import javax.net.ssl.HttpsURLConnection;
> import javax.net.ssl.SSLSocketFactory;
>
> public class SSLContextTest5 {
>
> public static void main( String[] args )
> throws Exception
> {
> System.setProperty("javax.net.ssl.keyStoreType", "jks");
> System.setProperty("javax.net.ssl.trustStoreType", "jks");
> System.setProperty("javax.net.ssl.keyStore",
> "/export/home/miweng/test2/mystore.jks");
> System.setProperty("javax.net.ssl.trustStore",
> "/export/home/miweng/test2/truststore.jks");
> System.setProperty("javax.net.ssl.keyStorePassword", "changeit");
> System.setProperty("javax.net.ssl.trustStorePassword", "changeit");
>
> //System.setProperty("https.protocols", "TLSv1.2");
>
> try {
> URL url = new URL( "https://manage-proxy.vip.xxx.com/ws/spf;);
> HttpsURLConnection con = (HttpsURLConnection) url.openConnection();
>
> SSLSocketFactory sslsocketfactory = (SSLSocketFactory)
> SSLSocketFactory.getDefault();
> con.setSSLSocketFactory(sslsocketfactory);
> InputStream inputstream = con.getInputStream();
> InputStreamReader inputstreamreader = new
> InputStreamReader(inputstream);
> BufferedReader bufferedreader = new BufferedReader(inputstreamreader);
>
> String string = null;
> while ((string = bufferedreader.readLine()) != null) {
> System.out.println("Received " + string);
> }
> } catch (Exception e) { e.printStackTrace(); }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3867) Enable JDBC to use key store & trust store for https connection
[
https://issues.apache.org/jira/browse/KYLIN-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791528#comment-16791528
]
ASF subversion and git services commented on KYLIN-3867:
Commit 6a80d36ac8ae3f7caf5ce630434cefa94366f0db in kylin's branch
refs/heads/master from kyotoYaho
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=6a80d36 ]
KYLIN-3867 leverage system properties to enable JDBC to use key store & trust
store for https connection
> Enable JDBC to use key store & trust store for https connection
> ---
>
> Key: KYLIN-3867
> URL: https://issues.apache.org/jira/browse/KYLIN-3867
> Project: Kylin
> Issue Type: Improvement
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v3.0.0
>
>
> In some cases, JDBC clients need to pass its client cert & key to the
> downstream for authentication. Then the JDBC needs to create a connection
> with key store & trust store.
> {code}
> import java.io.BufferedReader;
> import java.io.InputStream;
> import java.io.InputStreamReader;
> import java.net.URL;
> import javax.net.ssl.HttpsURLConnection;
> import javax.net.ssl.SSLSocketFactory;
>
> public class SSLContextTest5 {
>
> public static void main( String[] args )
> throws Exception
> {
> System.setProperty("javax.net.ssl.keyStoreType", "jks");
> System.setProperty("javax.net.ssl.trustStoreType", "jks");
> System.setProperty("javax.net.ssl.keyStore",
> "/export/home/miweng/test2/mystore.jks");
> System.setProperty("javax.net.ssl.trustStore",
> "/export/home/miweng/test2/truststore.jks");
> System.setProperty("javax.net.ssl.keyStorePassword", "changeit");
> System.setProperty("javax.net.ssl.trustStorePassword", "changeit");
>
> //System.setProperty("https.protocols", "TLSv1.2");
>
> try {
> URL url = new URL( "https://manage-proxy.vip.xxx.com/ws/spf;);
> HttpsURLConnection con = (HttpsURLConnection) url.openConnection();
>
> SSLSocketFactory sslsocketfactory = (SSLSocketFactory)
> SSLSocketFactory.getDefault();
> con.setSSLSocketFactory(sslsocketfactory);
> InputStream inputstream = con.getInputStream();
> InputStreamReader inputstreamreader = new
> InputStreamReader(inputstream);
> BufferedReader bufferedreader = new BufferedReader(inputstreamreader);
>
> String string = null;
> while ((string = bufferedreader.readLine()) != null) {
> System.out.println("Received " + string);
> }
> } catch (Exception e) { e.printStackTrace(); }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3867) Enable JDBC to use key store & trust store for https connection
[
https://issues.apache.org/jira/browse/KYLIN-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791531#comment-16791531
]
ASF GitHub Bot commented on KYLIN-3867:
---
shaofengshi commented on pull request #510: KYLIN-3867 Enable JDBC to use key
store & trust store for https connection
URL: https://github.com/apache/kylin/pull/510
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Enable JDBC to use key store & trust store for https connection
> ---
>
> Key: KYLIN-3867
> URL: https://issues.apache.org/jira/browse/KYLIN-3867
> Project: Kylin
> Issue Type: Improvement
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v3.0.0
>
>
> In some cases, JDBC clients need to pass its client cert & key to the
> downstream for authentication. Then the JDBC needs to create a connection
> with key store & trust store.
> {code}
> import java.io.BufferedReader;
> import java.io.InputStream;
> import java.io.InputStreamReader;
> import java.net.URL;
> import javax.net.ssl.HttpsURLConnection;
> import javax.net.ssl.SSLSocketFactory;
>
> public class SSLContextTest5 {
>
> public static void main( String[] args )
> throws Exception
> {
> System.setProperty("javax.net.ssl.keyStoreType", "jks");
> System.setProperty("javax.net.ssl.trustStoreType", "jks");
> System.setProperty("javax.net.ssl.keyStore",
> "/export/home/miweng/test2/mystore.jks");
> System.setProperty("javax.net.ssl.trustStore",
> "/export/home/miweng/test2/truststore.jks");
> System.setProperty("javax.net.ssl.keyStorePassword", "changeit");
> System.setProperty("javax.net.ssl.trustStorePassword", "changeit");
>
> //System.setProperty("https.protocols", "TLSv1.2");
>
> try {
> URL url = new URL( "https://manage-proxy.vip.xxx.com/ws/spf;);
> HttpsURLConnection con = (HttpsURLConnection) url.openConnection();
>
> SSLSocketFactory sslsocketfactory = (SSLSocketFactory)
> SSLSocketFactory.getDefault();
> con.setSSLSocketFactory(sslsocketfactory);
> InputStream inputstream = con.getInputStream();
> InputStreamReader inputstreamreader = new
> InputStreamReader(inputstream);
> BufferedReader bufferedreader = new BufferedReader(inputstreamreader);
>
> String string = null;
> while ((string = bufferedreader.readLine()) != null) {
> System.out.println("Received " + string);
> }
> } catch (Exception e) { e.printStackTrace(); }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [kylin] shaofengshi merged pull request #510: KYLIN-3867 Enable JDBC to use key store & trust store for https connection
shaofengshi merged pull request #510: KYLIN-3867 Enable JDBC to use key store & trust store for https connection URL: https://github.com/apache/kylin/pull/510 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3865) Centralize the zookeeper related info
[ https://issues.apache.org/jira/browse/KYLIN-3865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791501#comment-16791501 ] ASF GitHub Bot commented on KYLIN-3865: --- shaofengshi commented on pull request #512: KYLIN-3865 Centralize the zookeeper related info URL: https://github.com/apache/kylin/pull/512 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Centralize the zookeeper related info > - > > Key: KYLIN-3865 > URL: https://issues.apache.org/jira/browse/KYLIN-3865 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.0.0 > > > Currently zookeeper related infos, like zk path, curator client, are > scattered many places, like > ZookeeperDistributedLock, DistributedScheduler, CuratorScheduler. It's better > to centralize them. There'll be several benefits: > * Avoid creating multiple curator clients > * Easy to manage the zk path by leveraging zkChRoot -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3871) Kylin inside Cloudera CDH Quickstart Sandbox
Yanwen Lin created KYLIN-3871: - Summary: Kylin inside Cloudera CDH Quickstart Sandbox Key: KYLIN-3871 URL: https://issues.apache.org/jira/browse/KYLIN-3871 Project: Kylin Issue Type: Test Components: Job Engine Affects Versions: v2.6.0 Environment: Cloudera Quickstart Docker image: - OS: centos6 - memory: 13GB - disk: 20GB - java: 1.8 - maven: 3.5.3 Reporter: Yanwen Lin When doing integration test, I met the following error. I know this is related to Java version error and the reason is that Kylin use java1.8 while Cloudera use java1.7. So I manually installed java 1.8 and set JAVA_HOME to point to Spark2.x. (I also type spark-submit --version to check this). But the bug did not go away. I guess during the process of Spark job, some command may change the Java version back to java1.7 (not sure). Is there anyway to force it not change back to Java1.7 or any workaround? I have successfully finished maven installing and unit tests. *Branch:* realtime-streaming *Executed command with problem:* mvn verify -fae -Dhdp.version=2.4.0.0-169 -P sandbox *Error stack:* Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/spark/network/util/ByteUnit : Unsupported major.minor version 52.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at org.apache.spark.deploy.history.config$.(config.scala:44) at org.apache.spark.deploy.history.config$.(config.scala) at org.apache.spark.SparkConf$.(SparkConf.scala:635) at org.apache.spark.SparkConf$.(SparkConf.scala) at org.apache.spark.SparkConf.set(SparkConf.scala:94) at org.apache.spark.SparkConf$$anonfun$loadFromSystemProperties$3.apply(SparkConf.scala:76) at org.apache.spark.SparkConf$$anonfun$loadFromSystemProperties$3.apply(SparkConf.scala:75) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221) at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428) at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428) at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) at org.apache.spark.SparkConf.loadFromSystemProperties(SparkConf.scala:75) at org.apache.spark.SparkConf.(SparkConf.scala:70) at org.apache.spark.SparkConf.(SparkConf.scala:57) at org.apache.spark.deploy.yarn.ApplicationMaster.(ApplicationMaster.scala:62) at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:838) at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:869) at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3865) Centralize the zookeeper related info
[ https://issues.apache.org/jira/browse/KYLIN-3865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791502#comment-16791502 ] ASF subversion and git services commented on KYLIN-3865: Commit a6aa9c812aa7cf93d97fcd8eb554a8c289d1f20d in kylin's branch refs/heads/master from kyotoYaho [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=a6aa9c8 ] KYLIN-3865 Centralize the zookeeper related info > Centralize the zookeeper related info > - > > Key: KYLIN-3865 > URL: https://issues.apache.org/jira/browse/KYLIN-3865 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.0.0 > > > Currently zookeeper related infos, like zk path, curator client, are > scattered many places, like > ZookeeperDistributedLock, DistributedScheduler, CuratorScheduler. It's better > to centralize them. There'll be several benefits: > * Avoid creating multiple curator clients > * Easy to manage the zk path by leveraging zkChRoot -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] shaofengshi merged pull request #512: KYLIN-3865 Centralize the zookeeper related info
shaofengshi merged pull request #512: KYLIN-3865 Centralize the zookeeper related info URL: https://github.com/apache/kylin/pull/512 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (KYLIN-3871) Kylin inside Cloudera CDH Quickstart Sandbox
[ https://issues.apache.org/jira/browse/KYLIN-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanwen Lin updated KYLIN-3871: -- Component/s: Real-time Streaming > Kylin inside Cloudera CDH Quickstart Sandbox > > > Key: KYLIN-3871 > URL: https://issues.apache.org/jira/browse/KYLIN-3871 > Project: Kylin > Issue Type: Test > Components: Job Engine, Real-time Streaming >Affects Versions: v2.6.0 > Environment: Cloudera Quickstart Docker image: > - OS: centos6 > - memory: 13GB > - disk: 20GB > - java: 1.8 > - maven: 3.5.3 >Reporter: Yanwen Lin >Priority: Blocker > > When doing integration test, I met the following error. I know this is > related to Java version error and the reason is that Kylin use java1.8 while > Cloudera use java1.7. So I manually installed java 1.8 and set JAVA_HOME to > point to Spark2.x. (I also type spark-submit --version to check this). But > the bug did not go away. I guess during the process of Spark job, some > command may change the Java version back to java1.7 (not sure). Is there > anyway to force it not change back to Java1.7 or any workaround? > I have successfully finished maven installing and unit tests. > *Branch:* > realtime-streaming > *Executed command with problem:* > mvn verify -fae -Dhdp.version=2.4.0.0-169 -P sandbox > *Error stack:* > Exception in thread "main" java.lang.UnsupportedClassVersionError: > org/apache/spark/network/util/ByteUnit : Unsupported major.minor version 52.0 > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:800) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at org.apache.spark.deploy.history.config$.(config.scala:44) > at org.apache.spark.deploy.history.config$.(config.scala) > at org.apache.spark.SparkConf$.(SparkConf.scala:635) > at org.apache.spark.SparkConf$.(SparkConf.scala) > at org.apache.spark.SparkConf.set(SparkConf.scala:94) > at > org.apache.spark.SparkConf$$anonfun$loadFromSystemProperties$3.apply(SparkConf.scala:76) > at > org.apache.spark.SparkConf$$anonfun$loadFromSystemProperties$3.apply(SparkConf.scala:75) > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) > at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221) > at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428) > at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428) > at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428) > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) > at org.apache.spark.SparkConf.loadFromSystemProperties(SparkConf.scala:75) > at org.apache.spark.SparkConf.(SparkConf.scala:70) > at org.apache.spark.SparkConf.(SparkConf.scala:57) > at > org.apache.spark.deploy.yarn.ApplicationMaster.(ApplicationMaster.scala:62) > at > org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:838) > at > org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:869) > at > org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3866) Whether to set mapreduce.application.classpath is determined by the user
[
https://issues.apache.org/jira/browse/KYLIN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791409#comment-16791409
]
Guangxu Cheng commented on KYLIN-3866:
--
{quote}While, I still not understand the requirment: in which case the
"mapreduce.application.classpath" is absent, and still not allow to use local
cmd to get the classpath? Usually Kylin is installed in a Hadoop client node,
which need the MR/Hive/HBase be well configured, the classpath configuration
should be correct for all Hadoop node. Could you please elaborate the problem
you encountered, and how common it is? Thank you!
{quote}
[~Shaofengshi] Thanks for your review.
1. "mapreduce.application.classpath" is not a required parameter. If the
parameter is not configured, the default value will be used. see:
[https://github.com/apache/hadoop/blob/5d8c8cd568ae54c1be366e786c738b63bd53e271/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java#L219]
In our cluster, we use the default value and do not need the user to configure
the parameter on the client side.
2. We are not deploying kylin on the nodes of the MR cluster because the MR
cluster is maintained by other departments within Tencent and we do not have
permission to deploy other applications on the nodes of the MR cluster. We also
don't know the installation path of the MR cluster.
In Tencent, Kylin is deployed separately, Hadoop/HBase/Hive client is deployed
on the kylin node, but the path of deploying Hadoop client may not be the same
as the MR cluster. So, the path obtained by the command "mapred classpath" is
only the path of Hadoop client not the path of the MR cluster deployment.
Now, kylin forces the user to configure this parameter. If it is not
configured, the path obtained by using the command "mapred classpath" (which
may be wrong value) will be assigned to the parameter, which I think it is
unreasonable.
So, it may be more reasonable for the user to decide whether to use the
parameter.
> Whether to set mapreduce.application.classpath is determined by the user
>
>
> Key: KYLIN-3866
> URL: https://issues.apache.org/jira/browse/KYLIN-3866
> Project: Kylin
> Issue Type: Bug
>Reporter: Guangxu Cheng
>Priority: Major
> Attachments: KYLIN-3866.master.001.patch
>
>
> In the kylin environment, if there is no configuration parameter
> "mapreduce.application.classpath", then kylin will use the command "mapred
> classpath" to get the local classpath and set it to
> "mapreduce.application.classpath", However, it will cause the following two
> problems:
> 1. The local classpath may not be the same as the MapReduce cluster.
> 2. There is no need to configure the parameter
> "mapreduce.application.classpath" on the client side in some cluster.
> So, it may be a better choice for users to decide whether they need to
> configure this parameter or not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
