[jira] [Commented] (KYLIN-1770) Can't use PreparedStatement with "between and" expression
[
https://issues.apache.org/jira/browse/KYLIN-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367244#comment-15367244
]
Yiming Liu commented on KYLIN-1770:
---
Has filed a Calcite JIRA for this issue:
https://issues.apache.org/jira/browse/CALCITE-1310
> Can't use PreparedStatement with "between and" expression
> -
>
> Key: KYLIN-1770
> URL: https://issues.apache.org/jira/browse/KYLIN-1770
> Project: Kylin
> Issue Type: Bug
> Components: Driver - JDBC
>Affects Versions: v1.5.1, v1.5.2
>Reporter: Dayue Gao
>
> Sample code to reproduce:
> {code:java}
> final String sql = "select count(*) from kylin_sales where
> LSTG_SITE_ID between ? and ?";
> try (PreparedStatement stmt = conn.prepareStatement(sql)) {
> stmt.setInt(1, 0);
> stmt.setInt(2, 5);
> try (ResultSet rs = stmt.executeQuery()) {
> printResultSet(rs);
> }
> }
> {code}
> Exception stack trace from server log:
> {noformat}
> java.sql.SQLException: Error while preparing statement [select count(*) from
> kylin_sales where LSTG_SITE_ID between ? and ?]
> at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
> at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:203)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:184)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:85)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:153)
> at
> org.apache.kylin.rest.service.QueryService.execute(QueryService.java:353)
> at
> org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:274)
> at
> org.apache.kylin.rest.service.QueryService.query(QueryService.java:120)
> at
> org.apache.kylin.rest.service.QueryService$$FastClassByCGLIB$$4957273f.invoke()
> at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at
> org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:618)
> at
> org.apache.kylin.rest.service.QueryService$$EnhancerByCGLIB$$8610374f.query()
> at
> org.apache.kylin.rest.controller.QueryController.doQueryWithCache(QueryController.java:192)
> at
> org.apache.kylin.rest.controller.QueryController.prepareQuery(QueryController.java:101)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:213)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:126)
> at
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:578)
> at
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
> at
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
> at
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
> at
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
> at
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at
>
[jira] [Commented] (KYLIN-1855) Should exclude those joins in whose related lookup tables no dimensions are used in cube
[ https://issues.apache.org/jira/browse/KYLIN-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367243#comment-15367243 ] Zhong Yanghong commented on KYLIN-1855: --- Users may just use the inner join. And the joints in the sql shown in GUI for creating flat table is based on model rather than cube. It's inconsistent with the joins checking which is based on cube rather than model. It's a little confusing. > Should exclude those joins in whose related lookup tables no dimensions are > used in cube > > > Key: KYLIN-1855 > URL: https://issues.apache.org/jira/browse/KYLIN-1855 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > A cube is based on a model in which a star schema is defined. In some cases, > the cube utilizes only a few lookup tables rather than all. In this case, > when creating the sql for the flat table, those lookup tables should not be > included. Otherwise, it will confuse users when query. If users do query > according to the definition of the flat table, error of no realization will > occur due to lack of the related join. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1855) Should exclude those joins in whose related lookup tables no dimensions are used in cube
[ https://issues.apache.org/jira/browse/KYLIN-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367234#comment-15367234 ] Shaofeng SHI commented on KYLIN-1855: - If a lookup table wasn't included in a cube, why it need the SQL have to include it? Can you give an example? Thanks. > Should exclude those joins in whose related lookup tables no dimensions are > used in cube > > > Key: KYLIN-1855 > URL: https://issues.apache.org/jira/browse/KYLIN-1855 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > A cube is based on a model in which a star schema is defined. In some cases, > the cube utilizes only a few lookup tables rather than all. In this case, > when creating the sql for the flat table, those lookup tables should not be > included. Otherwise, it will confuse users when query. If users do query > according to the definition of the flat table, error of no realization will > occur due to lack of the related join. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1830) Put KYLIN_JVM_SETTINGS to kylin.properties
[ https://issues.apache.org/jira/browse/KYLIN-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367146#comment-15367146 ] Richard Calaba commented on KYLIN-1830: --- Fair enough - I had no idea so I did mine property parsing to support commented parameters , paramteres in quotes , spliting only after 1st '=' sign, atd ... if all this is supported by the bin/get-properties.sh then fine -> it will be even faster as mine code scans the property every time the export_property_override is used instead of export parameter. It was designed this way to make the old setenv.sh and the new one very similiar (export_property_override instead of export =. Also mine implementation supports specifying default value in the script so it doesn't have to be overriden in kylin.properties file. > Put KYLIN_JVM_SETTINGS to kylin.properties > -- > > Key: KYLIN-1830 > URL: https://issues.apache.org/jira/browse/KYLIN-1830 > Project: Kylin > Issue Type: Improvement >Reporter: Richard Calaba >Priority: Minor > Labels: newbie > Attachments: kylin.properties, setenv.sh > > > Currently is the KYLIN_JVM_SETTINGS variable stored in the ,/bin/setenv.sh > ... which is not wrong, but as we have also some other memory specific > setting in ./conf/kylin.properties file (like i.e > kylin.job.mapreduce.default.reduce.input.mb or kylin.table.snapshot.max_mb) > it might be good idea to have those performance and sizing related parameters > in one location. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1770) Can't use PreparedStatement with "between and" expression
[
https://issues.apache.org/jira/browse/KYLIN-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367139#comment-15367139
]
Yiming Liu commented on KYLIN-1770:
---
I did some investment. It seems the SqlBetweenOperator does not deriveType like
what SqlInOperator does. So the operandTypeInference is always NULL which will
throw exception when validation.
> Can't use PreparedStatement with "between and" expression
> -
>
> Key: KYLIN-1770
> URL: https://issues.apache.org/jira/browse/KYLIN-1770
> Project: Kylin
> Issue Type: Bug
> Components: Driver - JDBC
>Affects Versions: v1.5.1, v1.5.2
>Reporter: Dayue Gao
>
> Sample code to reproduce:
> {code:java}
> final String sql = "select count(*) from kylin_sales where
> LSTG_SITE_ID between ? and ?";
> try (PreparedStatement stmt = conn.prepareStatement(sql)) {
> stmt.setInt(1, 0);
> stmt.setInt(2, 5);
> try (ResultSet rs = stmt.executeQuery()) {
> printResultSet(rs);
> }
> }
> {code}
> Exception stack trace from server log:
> {noformat}
> java.sql.SQLException: Error while preparing statement [select count(*) from
> kylin_sales where LSTG_SITE_ID between ? and ?]
> at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
> at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:203)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:184)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:85)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:153)
> at
> org.apache.kylin.rest.service.QueryService.execute(QueryService.java:353)
> at
> org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:274)
> at
> org.apache.kylin.rest.service.QueryService.query(QueryService.java:120)
> at
> org.apache.kylin.rest.service.QueryService$$FastClassByCGLIB$$4957273f.invoke()
> at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at
> org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:618)
> at
> org.apache.kylin.rest.service.QueryService$$EnhancerByCGLIB$$8610374f.query()
> at
> org.apache.kylin.rest.controller.QueryController.doQueryWithCache(QueryController.java:192)
> at
> org.apache.kylin.rest.controller.QueryController.prepareQuery(QueryController.java:101)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:213)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:126)
> at
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:578)
> at
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
> at
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
> at
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
> at
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
> at
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at
>
[jira] [Comment Edited] (KYLIN-1830) Put KYLIN_JVM_SETTINGS to kylin.properties
[
https://issues.apache.org/jira/browse/KYLIN-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367134#comment-15367134
]
liyang edited comment on KYLIN-1830 at 7/8/16 2:56 AM:
---
Thanks Richard!
Note there is {{bin/get-properties.sh}} that is also capable of reading
kylin.properties. We shall keep one implementation and stick to it.
was (Author: liyang.g...@gmail.com):
Note there is {{bin/get-properties.sh}} that is also capable of reading
kylin.properties.
> Put KYLIN_JVM_SETTINGS to kylin.properties
> --
>
> Key: KYLIN-1830
> URL: https://issues.apache.org/jira/browse/KYLIN-1830
> Project: Kylin
> Issue Type: Improvement
>Reporter: Richard Calaba
>Priority: Minor
> Labels: newbie
> Attachments: kylin.properties, setenv.sh
>
>
> Currently is the KYLIN_JVM_SETTINGS variable stored in the ,/bin/setenv.sh
> ... which is not wrong, but as we have also some other memory specific
> setting in ./conf/kylin.properties file (like i.e
> kylin.job.mapreduce.default.reduce.input.mb or kylin.table.snapshot.max_mb)
> it might be good idea to have those performance and sizing related parameters
> in one location.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KYLIN-1830) Put KYLIN_JVM_SETTINGS to kylin.properties
[
https://issues.apache.org/jira/browse/KYLIN-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367134#comment-15367134
]
liyang commented on KYLIN-1830:
---
Note there is {{bin/get-properties.sh}} that is also capable of reading
kylin.properties.
> Put KYLIN_JVM_SETTINGS to kylin.properties
> --
>
> Key: KYLIN-1830
> URL: https://issues.apache.org/jira/browse/KYLIN-1830
> Project: Kylin
> Issue Type: Improvement
>Reporter: Richard Calaba
>Priority: Minor
> Labels: newbie
> Attachments: kylin.properties, setenv.sh
>
>
> Currently is the KYLIN_JVM_SETTINGS variable stored in the ,/bin/setenv.sh
> ... which is not wrong, but as we have also some other memory specific
> setting in ./conf/kylin.properties file (like i.e
> kylin.job.mapreduce.default.reduce.input.mb or kylin.table.snapshot.max_mb)
> it might be good idea to have those performance and sizing related parameters
> in one location.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KYLIN-1830) Put KYLIN_JVM_SETTINGS to kylin.properties
[ https://issues.apache.org/jira/browse/KYLIN-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyang updated KYLIN-1830: -- Labels: newbie (was: ) > Put KYLIN_JVM_SETTINGS to kylin.properties > -- > > Key: KYLIN-1830 > URL: https://issues.apache.org/jira/browse/KYLIN-1830 > Project: Kylin > Issue Type: Improvement >Reporter: Richard Calaba >Priority: Minor > Labels: newbie > Attachments: kylin.properties, setenv.sh > > > Currently is the KYLIN_JVM_SETTINGS variable stored in the ,/bin/setenv.sh > ... which is not wrong, but as we have also some other memory specific > setting in ./conf/kylin.properties file (like i.e > kylin.job.mapreduce.default.reduce.input.mb or kylin.table.snapshot.max_mb) > it might be good idea to have those performance and sizing related parameters > in one location. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1855) Should exclude those joins in whose related lookup tables no dimensions are used in cube
[ https://issues.apache.org/jira/browse/KYLIN-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367110#comment-15367110 ] Zhong Yanghong commented on KYLIN-1855: --- For checking joins, it's better to be based on model rather than based on the joins to which all the dimensions used in the cube are related. > Should exclude those joins in whose related lookup tables no dimensions are > used in cube > > > Key: KYLIN-1855 > URL: https://issues.apache.org/jira/browse/KYLIN-1855 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > A cube is based on a model in which a star schema is defined. In some cases, > the cube utilizes only a few lookup tables rather than all. In this case, > when creating the sql for the flat table, those lookup tables should not be > included. Otherwise, it will confuse users when query. If users do query > according to the definition of the flat table, error of no realization will > occur due to lack of the related join. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1855) Should exclude those joins in whose related lookup tables no dimensions are used in cube
[ https://issues.apache.org/jira/browse/KYLIN-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367108#comment-15367108 ] Zhong Yanghong commented on KYLIN-1855: --- Yes, if the SQL includes those lookup tables which not used in cube, "no realization found" error will occur. The reason the cube is based on that model may be that users still want to extract data based on the joins. Thus, it's better to relax the condition for checking joins. > Should exclude those joins in whose related lookup tables no dimensions are > used in cube > > > Key: KYLIN-1855 > URL: https://issues.apache.org/jira/browse/KYLIN-1855 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > A cube is based on a model in which a star schema is defined. In some cases, > the cube utilizes only a few lookup tables rather than all. In this case, > when creating the sql for the flat table, those lookup tables should not be > included. Otherwise, it will confuse users when query. If users do query > according to the definition of the flat table, error of no realization will > occur due to lack of the related join. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1855) Should exclude those joins in whose related lookup tables no dimensions are used in cube
[ https://issues.apache.org/jira/browse/KYLIN-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367105#comment-15367105 ] Shaofeng SHI commented on KYLIN-1855: - Hi Yanghong, Just to confirm whether I get it: Assume I have a model which has 1 fact and 4 lookup tables; Now I create a cube by using that model, but in the cube I only use 2 lookup tables; Now when I query it, if the SQL only has 2 lookup tables, will Kylin report the "no realization found" error? > Should exclude those joins in whose related lookup tables no dimensions are > used in cube > > > Key: KYLIN-1855 > URL: https://issues.apache.org/jira/browse/KYLIN-1855 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > A cube is based on a model in which a star schema is defined. In some cases, > the cube utilizes only a few lookup tables rather than all. In this case, > when creating the sql for the flat table, those lookup tables should not be > included. Otherwise, it will confuse users when query. If users do query > according to the definition of the flat table, error of no realization will > occur due to lack of the related join. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-1859) Use segment "uuid" instead of "name" to seek a segment across the system
Shaofeng SHI created KYLIN-1859: --- Summary: Use segment "uuid" instead of "name" to seek a segment across the system Key: KYLIN-1859 URL: https://issues.apache.org/jira/browse/KYLIN-1859 Project: Kylin Issue Type: Bug Reporter: Shaofeng SHI Assignee: Shaofeng SHI Priority: Minor The "uuid" is unique, while "name" isn't (name is composed by startTime, endTime). It's better to use "uuid" in the job chains. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-1857) Show available memory on UI - in System Tab (and other runtime statistics)
Richard Calaba created KYLIN-1857: - Summary: Show available memory on UI - in System Tab (and other runtime statistics) Key: KYLIN-1857 URL: https://issues.apache.org/jira/browse/KYLIN-1857 Project: Kylin Issue Type: Improvement Affects Versions: v1.5.2, v1.5.2.1 Reporter: Richard Calaba Priority: Minor I have run into situation that Kylin dies (exception in log says heap out of memory) if I try to run 3 parallel cubes with high-cardinality dimensions. It is reproduceable scenario. I have set max snapshot size to 2GB and -Xmx to 16GB. If I run the cube build one-by-one -> Kylin doesn't die. As we have have no idea about memory requirements before we start building the cube(s) then for now it would be beneficial at least to monitor basic Kylin VM statistics, i.e.: -- current memory occupied by snaphots -- total memory allocation & total free memory -- how many (and which) temporary (intermediate) objects (in hive/hbase/filesystem) are created ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-1856) Kylin shows old error in job step output after resume - specifically in #4 Step Name: Build Dimension Dictionary
Richard Calaba created KYLIN-1856: - Summary: Kylin shows old error in job step output after resume - specifically in #4 Step Name: Build Dimension Dictionary Key: KYLIN-1856 URL: https://issues.apache.org/jira/browse/KYLIN-1856 Project: Kylin Issue Type: Bug Affects Versions: v1.5.2, v1.5.2.1 Reporter: Richard Calaba Priority: Minor I have realized that if my job stops with error and I try to recover the error and resume the job - then the latest step starts again from scratch. This is fine by in my opinion the log of the Step should clear as well - now it is showing the error from my previous attempt. Specifically observed in #4 Step Name: Build Dimension Dictionary - but is probbaly generic issue. Ask is: clear the log of the Build Step if job Step is resumed. Already when the job step is restarted, not after it is completed. (if Kylin fails i.e. for out of memmory - it silently dies and analyzing the step log shows wrong error (from previous run) - if it would be empty -> I would know that most probable cause was that Kylin died) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1856) Kylin shows old error in job step output after resume - specifically in #4 Step Name: Build Dimension Dictionary
[ https://issues.apache.org/jira/browse/KYLIN-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Calaba updated KYLIN-1856: -- Description: I have realized that if my job stops with error and I try to recover the error and resume the job - then the latest step starts again from scratch. This is fine but in my opinion the log of the Step should clear as well - now it is showing the error from my previous attempt. Specifically observed in #4 Step Name: Build Dimension Dictionary - but is probbaly generic issue. To correct this: clear the log of the Build Step after the job Step is resumed. Already when the job step is restarted, not after it is completed. (if Kylin fails i.e. for out of memory - it silently dies and analyzing the step log shows wrong error (from previous run) - if it would be empty -> I would know that most probable cause was that Kylin died) was: I have realized that if my job stops with error and I try to recover the error and resume the job - then the latest step starts again from scratch. This is fine but in my opinion the log of the Step should clear as well - now it is showing the error from my previous attempt. Specifically observed in #4 Step Name: Build Dimension Dictionary - but is probbaly generic issue. To correct this: clear the log of the Build Step after the job Step is resumed. Already when the job step is restarted, not after it is completed. (if Kylin fails i.e. for out of memmory - it silently dies and analyzing the step log shows wrong error (from previous run) - if it would be empty -> I would know that most probable cause was that Kylin died) > Kylin shows old error in job step output after resume - specifically in #4 > Step Name: Build Dimension Dictionary > > > Key: KYLIN-1856 > URL: https://issues.apache.org/jira/browse/KYLIN-1856 > Project: Kylin > Issue Type: Bug >Affects Versions: v1.5.2, v1.5.2.1 >Reporter: Richard Calaba >Priority: Minor > > I have realized that if my job stops with error and I try to recover the > error and resume the job - then the latest step starts again from scratch. > This is fine but in my opinion the log of the Step should clear as well - now > it is showing the error from my previous attempt. > Specifically observed in #4 Step Name: Build Dimension Dictionary - but is > probbaly generic issue. > To correct this: clear the log of the Build Step after the job Step is > resumed. Already when the job step is restarted, not after it is completed. > (if Kylin fails i.e. for out of memory - it silently dies and analyzing the > step log shows wrong error (from previous run) - if it would be empty -> I > would know that most probable cause was that Kylin died) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1856) Kylin shows old error in job step output after resume - specifically in #4 Step Name: Build Dimension Dictionary
[ https://issues.apache.org/jira/browse/KYLIN-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Calaba updated KYLIN-1856: -- Description: I have realized that if my job stops with error and I try to recover the error and resume the job - then the latest step starts again from scratch. This is fine but in my opinion the log of the Step should clear as well - now it is showing the error from my previous attempt. Specifically observed in #4 Step Name: Build Dimension Dictionary - but is probbaly generic issue. Ask is: clear the log of the Build Step if job Step is resumed. Already when the job step is restarted, not after it is completed. (if Kylin fails i.e. for out of memmory - it silently dies and analyzing the step log shows wrong error (from previous run) - if it would be empty -> I would know that most probable cause was that Kylin died) was: I have realized that if my job stops with error and I try to recover the error and resume the job - then the latest step starts again from scratch. This is fine by in my opinion the log of the Step should clear as well - now it is showing the error from my previous attempt. Specifically observed in #4 Step Name: Build Dimension Dictionary - but is probbaly generic issue. Ask is: clear the log of the Build Step if job Step is resumed. Already when the job step is restarted, not after it is completed. (if Kylin fails i.e. for out of memmory - it silently dies and analyzing the step log shows wrong error (from previous run) - if it would be empty -> I would know that most probable cause was that Kylin died) > Kylin shows old error in job step output after resume - specifically in #4 > Step Name: Build Dimension Dictionary > > > Key: KYLIN-1856 > URL: https://issues.apache.org/jira/browse/KYLIN-1856 > Project: Kylin > Issue Type: Bug >Affects Versions: v1.5.2, v1.5.2.1 >Reporter: Richard Calaba >Priority: Minor > > I have realized that if my job stops with error and I try to recover the > error and resume the job - then the latest step starts again from scratch. > This is fine but in my opinion the log of the Step should clear as well - now > it is showing the error from my previous attempt. > Specifically observed in #4 Step Name: Build Dimension Dictionary - but is > probbaly generic issue. > Ask is: clear the log of the Build Step if job Step is resumed. Already when > the job step is restarted, not after it is completed. > (if Kylin fails i.e. for out of memmory - it silently dies and analyzing the > step log shows wrong error (from previous run) - if it would be empty -> I > would know that most probable cause was that Kylin died) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1698) Allow bigint(long) as a partition date column
[ https://issues.apache.org/jira/browse/KYLIN-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hongbin ma updated KYLIN-1698: -- Summary: Allow bigint(long) as a partition date column (was: Allow bigint(long) as a date partition column) > Allow bigint(long) as a partition date column > - > > Key: KYLIN-1698 > URL: https://issues.apache.org/jira/browse/KYLIN-1698 > Project: Kylin > Issue Type: Improvement >Affects Versions: v1.5.1 >Reporter: guohuili > > In many cases, hive tables may use long / bigint as date column. Currently > Kylin only support '-MM-dd' or 'MMdd' as date format. This jira is to > support long / bigint format. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KYLIN-1841) Refresh or build different segments of one cube simultaneously
[ https://issues.apache.org/jira/browse/KYLIN-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyang resolved KYLIN-1841. --- Resolution: Duplicate Dup with KYLIN-1771 > Refresh or build different segments of one cube simultaneously > -- > > Key: KYLIN-1841 > URL: https://issues.apache.org/jira/browse/KYLIN-1841 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v1.5.2 >Reporter: Yang Liu >Assignee: Dong Li > Labels: patch > Fix For: Future > > Original Estimate: 504h > Remaining Estimate: 504h > > If some old segments need to be refreshed and a new segment is building at > the same time, we have to wait for the new segment finish building, and then > refresh the old one, sometimes if the refresh of old segments lasts too long, > the upcoming build of a new segment may fail. > so, i'm wondering If Kylin can support refresh or build different segments of > one cube simultaneously -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1842) java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat
[
https://issues.apache.org/jira/browse/KYLIN-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365919#comment-15365919
]
liyang commented on KYLIN-1842:
---
Might relate with hbase.sh
http://apache-kylin.74782.x6.nabble.com/NoClassDefFoundError-org-apache-hive-hcatalog-mapreduce-HCatInputFormat-td1156.html
http://apache-kylin.74782.x6.nabble.com/ava-lang-NoClassDefFoundError-org-apache-hive-hcatalog-mapreduce-HCatInputFo-rmat-td3778.html
> java.lang.NoClassDefFoundError:
> org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> --
>
> Key: KYLIN-1842
> URL: https://issues.apache.org/jira/browse/KYLIN-1842
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v1.5.0
> Environment: Zookeeper-3.4
> Hadoop-2.5
> hbase-1.1.3
> hive-1.2.1
> kylin-1.5.0
>Reporter: raphael5200
>Assignee: Dong Li
> Fix For: 1.x-HBase1.1.3
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> i have a error in kylin;
> Error message:
> 2016-07-01 12:30:07,431 ERROR [pool-5-thread-2]
> execution.AbstractExecutable:116 : error running Executable
> org.apache.kylin.job.exception.ExecuteException:
> java.lang.NoClassDefFoundError:
> org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> at
> org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:81)
> at
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:111)
> at
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:91)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:118)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> ... 6 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720)
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
> ... 13 more
> my conf:
> .bash_profile:
> PATH=$PATH:$HOME/bin
> JAVA_HOME=/usr/java/jdk1.7.0_79
> HADOOP_HOME=/usr/local/hadoop-2.5.1
> ZOOKEEPER_HOME=/usr/local/zookeeper-3.4.6
> HBASE_HOME=/usr/local/hbase-1.1.3
> HIVE_HOME=/usr/local/hive-1.2.1
> HCAT_HOME=$HIVE_HOME/hcatalog
> KYLIN_HOME=/usr/local/kylin-1.5.0-HBase1.1.3
> CATALINA_HOME=$KYLIN_HOME/tomcat
> hive_dependency=$HIVE_HOME/conf:$HIVE_HOME/lib/*:$HCAT_HOME/share/hcatalog/hive-hcatalog-core-1.2.1.jar
> PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin:$HBASE_HOME/bin:$HIVE_HOME/bin:$HCAT_HOME/bin:$CATALINA_HOME/bin:$KYLIN_HOME/bin
> export JAVA_HOME
> export HADOOP_HOME
> export ZOOKEEPER_HOME
> export HBASE_HOME
> export HIVE_HOME
> export HCAT_HOME
> export CATALINA_HOME
> export KYLIN_HOME
> export PATH
> kylin.sh:
> export KYLIN_HOME=/usr/local/kylin-1.5.0-HBase1.1.3
> export
> HBASE_CLASSPATH_PREFIX=${tomcat_root}/bin/bootstrap.jar:${tomcat_root}/bin/tomcat-juli.jar:${tomcat_root}/lib/*:$hive_dependency:$HBASE_CLASSPATH_PREFIX
> kylin.properties:
> kylin.rest.servers=master01:7070
> kylin.job.jar=/tmp/kylin/kylin-job-latest.jar
> kylin.coprocessor.local.jar=/tmp/kylin/kylin-coprocessor-latest.jar
> please help me
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KYLIN-1827) Send mail notification when runtime exception throws during build/merge cube
[ https://issues.apache.org/jira/browse/KYLIN-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365903#comment-15365903 ] Yiming Liu commented on KYLIN-1827: --- All failed builds have email notification. In this case, should Kylin catch the exception and turn the job status to FAILED, or ignore the status update but email notification only? > Send mail notification when runtime exception throws during build/merge cube > > > Key: KYLIN-1827 > URL: https://issues.apache.org/jira/browse/KYLIN-1827 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v1.5.1, v1.5.2 >Reporter: Ma Gang >Assignee: Ma Gang > > Currently mail notification is only sent in the onExecuteFinished() method, > but no notification will be sent when RuntimeException throws, that may cause > user miss some important job build failure, especially for some automation > merge jobs. Sometimes job state update fails(the hbase metastore is > unavailable in a short time), it will make the job always look like in a > running state, but actually it is failed, should send mail to notify user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1855) Should exclude those joins in whose related lookup tables no dimensions are used in cube
[ https://issues.apache.org/jira/browse/KYLIN-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365762#comment-15365762 ] Zhong Yanghong commented on KYLIN-1855: --- Then it's better to refine the condition for checking joins in the method "isMatchedWithJoins" of "CubeCapabilityChecker". [~yangli136] and [~Shaofengshi], do you agree with this? > Should exclude those joins in whose related lookup tables no dimensions are > used in cube > > > Key: KYLIN-1855 > URL: https://issues.apache.org/jira/browse/KYLIN-1855 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > A cube is based on a model in which a star schema is defined. In some cases, > the cube utilizes only a few lookup tables rather than all. In this case, > when creating the sql for the flat table, those lookup tables should not be > included. Otherwise, it will confuse users when query. If users do query > according to the definition of the flat table, error of no realization will > occur due to lack of the related join. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1855) Should exclude those joins in whose related lookup tables no dimensions are used in cube
[ https://issues.apache.org/jira/browse/KYLIN-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365757#comment-15365757 ] Zhong Yanghong commented on KYLIN-1855: --- Just exclude may be not a good solution, because cube is based on model and users still want to filter data by utilizing the joins defined by the model. > Should exclude those joins in whose related lookup tables no dimensions are > used in cube > > > Key: KYLIN-1855 > URL: https://issues.apache.org/jira/browse/KYLIN-1855 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > A cube is based on a model in which a star schema is defined. In some cases, > the cube utilizes only a few lookup tables rather than all. In this case, > when creating the sql for the flat table, those lookup tables should not be > included. Otherwise, it will confuse users when query. If users do query > according to the definition of the flat table, error of no realization will > occur due to lack of the related join. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-1855) Should exclude those joins in whose related lookup tables no dimensions are used in cube
Zhong Yanghong created KYLIN-1855: - Summary: Should exclude those joins in whose related lookup tables no dimensions are used in cube Key: KYLIN-1855 URL: https://issues.apache.org/jira/browse/KYLIN-1855 Project: Kylin Issue Type: Improvement Reporter: Zhong Yanghong Assignee: Zhong Yanghong A cube is based on a model in which a star schema is defined. In some cases, the cube utilizes only a few lookup tables rather than all. In this case, when creating the sql for the flat table, those lookup tables should not be included. Otherwise, it will confuse users when query. If users do query according to the definition of the flat table, error of no realization will occur due to lack of the related join. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1826) kylin support more than one hive based on different hadoop claster
[ https://issues.apache.org/jira/browse/KYLIN-1826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fengYu updated KYLIN-1826: -- Attachment: 0001-KYLIN-1826-support-more-hive-based-on-different-hado.patch I’m sorry for keeping you waiting. I finish my patch and add an source named ID_EXTERNAL_HIVE, one project can only use one hive source, for the sake of modify less code, I add Thread local virable tells which project is using in current thread. However, the patch is a litter big, It can apply or merge to kylin-1.5.2.1, I have test in our env. many times, And It works for more hive client based on two hadoop cluster, what is more, our hadoop engine(kylin calculate engin) is another cluster, and It works fine for me. I wish you can do more test in other env, Hope for your feedback. > kylin support more than one hive based on different hadoop claster > -- > > Key: KYLIN-1826 > URL: https://issues.apache.org/jira/browse/KYLIN-1826 > Project: Kylin > Issue Type: Improvement > Components: Environment >Affects Versions: v1.5.2 >Reporter: fengYu >Assignee: fengYu > Attachments: > 0001-KYLIN-1826-support-more-hive-based-on-different-hado.patch > > > Currently, kylin only support one hive which should run by 'hive' command, > However, when source data located in more than one hive we should deploy more > kylin instance and more than one metastore. which is difficult to manager and > may cause some conflict. > I has been working on it Recently, In our cluster, there are some hive > client(different metastore) which based on different hadoop cluster, I add a > new hive source type which called 'external hive' in kylin 1.5.x > Thanks to kylin Plug-in architecture in 2.x, which make this work easiler. > the main modification are: > 1. add hive root directory in hive config file, external hive client exist in > this directory. hive named by directory name. > 2. add hive-site.xml file while loading hive tables. > 3. store hive name into project, one project can only take one hive as source. > 4. change and add some job to support job building. > I will upload my patch if I finish all my tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
