[jira] [Updated] (KYLIN-4328) Kylin should skip succeed jobs in scheduler
[ https://issues.apache.org/jira/browse/KYLIN-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] GuKe updated KYLIN-4328: Summary: Kylin should skip succeed jobs in scheduler (was: When hbase and kylin that are not in the same IDC and found that the build task became very slow during scheduling. ) > Kylin should skip succeed jobs in scheduler > --- > > Key: KYLIN-4328 > URL: https://issues.apache.org/jira/browse/KYLIN-4328 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v2.2.0, v3.0.0, v2.6.3 > Environment: Centos 7.4 > hbase 1.2.4 > hive 1.1.1 > hadoop 2.7.2 >Reporter: GuKe >Assignee: GuKe >Priority: Major > Fix For: v3.1.0 > > > When hbase and kylin that are not in the same IDC and found that the build > task became very slow during scheduling. > We found that it was caused by the following part of the code. > The method getExecutableManager().GetAllJobIdsInCache() will read all of > jobid,There are currently more than 35,000 jobs in our server,and each jobid > accesses hbase at least twice to read the job state. > While that the most of jobs are succeed status.Those status won't change. > When kylin and hbase services are in the same IDC each visit to hbase Network > Latency is less than 1 ms. > However it takes more than 5 ms to access hbase each time across the IDC so > the delay caused by accessing hbase is considerable. > It takes a long time for scheduling task to run. > So we can add a cache to hold the id of the successful job at the first time > of the service start. > After we modified the code the run time reduced from 10 minutes to 20 seconds. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4305) Streaming Receiver cannot limit income query request or cancel long-running query
[
https://issues.apache.org/jira/browse/KYLIN-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4305:
Sprint: Sprint 52
> Streaming Receiver cannot limit income query request or cancel long-running
> query
> -
>
> Key: KYLIN-4305
> URL: https://issues.apache.org/jira/browse/KYLIN-4305
> Project: Kylin
> Issue Type: Improvement
> Components: Real-time Streaming
>Reporter: Xiaoxiang Yu
>Assignee: Xiaoxiang Yu
>Priority: Major
> Fix For: v3.1.0
>
> Attachments: Jietu20191217-221025.png, after_repair_receiver.jstack,
> image-2019-12-17-22-12-01-098.png, streaming_receiver_jstack.log
>
>
> When under heavy load(high rate of query request), receiver can not stand it,
> and most quey may timeout, but the query processing thread cannot be
> cancelled in receiver side, which will cause receiver's crash. You have to
> restart it.
> kylin.log
> {code:java}
> Caused by: java.lang.RuntimeException: timeout when call stream rpc
> at
> org.apache.kylin.storage.stream.rpc.HttpStreamDataSearchClient$QueuedStreamingTupleIterator.hasNext(HttpStreamDataSearchClient.java:298)
> at com.google.common.collect.Iterators$5.hasNext(Iterators.java:596)
> at
> org.apache.kylin.metadata.tuple.CompoundTupleIterator.hasNext(CompoundTupleIterator.java:52)
> at
> org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:63)
> at Baz$1$1.moveNext(Unknown Source)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy_(EnumerableDefaults.java:825)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy(EnumerableDefaults.java:761)
> at
> org.apache.calcite.linq4j.DefaultEnumerable.groupBy(DefaultEnumerable.java:302)
> at Baz.bind(Unknown Source)
> at
> org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:365)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:301)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:559)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:550)
> at
> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:182)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:667)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:619)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
> ... 83 more
> {code}
> jstack
> {code:java}
> "qtp1901663135-48" #48 prio=5 os_prio=0 tid=0x7f839995f800 nid=0x3cdd
> runnable [0x7f83674fe000]
>java.lang.Thread.State: RUNNABLE
> at java.lang.Thread.yield(Native Method)
> at
> org.apache.kylin.stream.core.query.MultiThreadsResultCollector$1.hasNext(MultiThreadsResultCollector.java:75)
> at
> org.apache.kylin.stream.core.query.RecordsAggregator.aggregate(RecordsAggregator.java:100)
> at
> org.apache.kylin.stream.core.query.StreamingCubeDataSearcher$StreamAggregateSearchResult.iterator(StreamingCubeDataSearcher.java:191)
> at
> org.apache.kylin.stream.server.rest.controller.DataController.query(DataController.java:119)
> at sun.reflect.GeneratedMethodAccessor87.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at
>
[jira] [Updated] (KYLIN-4497) PreparedStatement query do not support "limit ? offset ?"
[
https://issues.apache.org/jira/browse/KYLIN-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4497:
Sprint: Sprint 52
> PreparedStatement query do not support "limit ? offset ?"
> -
>
> Key: KYLIN-4497
> URL: https://issues.apache.org/jira/browse/KYLIN-4497
> Project: Kylin
> Issue Type: Bug
> Components: Driver - JDBC
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v3.1.0
>
>
> Query
> {code:java}
> String sql = "select part_dt, sum(price) from KYLIN_SALES group by part_dt
> order by part_dt limit ? offset ?";
> conn = getConnection();
> statement = conn.prepareStatement(sql);
> statement.setInt(1, 10);
> statement.setInt(2, 0);
> {code}
> Error log
> {code:java}
> 2020-05-15 09:32:36,062 ERROR [Query b87327f4-7299-32f2-4a2e-c19795a42288-44]
> service.QueryService:542 : Exception while executing query
> java.sql.SQLException: Error while preparing statement [select part_dt,
> sum(price) from KYLIN_SALES group by part_dt order by part_dt limit ? offset
> ?]
> at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
> at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:210)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:192)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:89)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:175)
> at
> org.apache.kylin.rest.service.QueryService.createPreparedContext(QueryService.java:1262)
> at
> org.apache.kylin.rest.service.QueryService.access$600(QueryService.java:141)
> at
> org.apache.kylin.rest.service.QueryService$PreparedContextFactory.create(QueryService.java:1301)
> at
> org.apache.kylin.rest.service.QueryService$PreparedContextFactory.create(QueryService.java:1296)
> at
> org.apache.commons.pool2.BaseKeyedPooledObjectFactory.makeObject(BaseKeyedPooledObjectFactory.java:60)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1064)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:358)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:281)
> at
> org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:680)
> at
> org.apache.kylin.rest.service.QueryService.query(QueryService.java:219)
> at
> org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:480)
> at
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:440)
> at
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:378)
> at
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:93)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:209)
> at
> com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:244)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:165)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025)
> at
[jira] [Updated] (KYLIN-4477) Usage of "TLS" is insecure
[
https://issues.apache.org/jira/browse/KYLIN-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4477:
Sprint: Sprint 51
> Usage of "TLS" is insecure
> --
>
> Key: KYLIN-4477
> URL: https://issues.apache.org/jira/browse/KYLIN-4477
> Project: Kylin
> Issue Type: Improvement
>Reporter: Md Mahir Asef Kabir
>Assignee: Md Mahir Asef Kabir
>Priority: Major
> Fix For: v3.1.0
>
>
> *Vulnerability Description:* In
> “engine-mr/src/main/java/org/apache/kylin/engine/mr/common/DefaultSslProtocolSocketFactory.java”
> file the following code was written in
> {code:java}
> private static SSLContext createEasySSLContext()
> {code}
> method -
> {code:java}
> SSLContext context = SSLContext.getInstance("TLS");
> {code}
> The vulnerability is, using "TLS” as the argument to SSLContext.getInstance
> method.
> *Reason it’s vulnerable:* TLS 1.0 is vulnerable to man-in-the-middle attacks.
> For further reference, follow
> [this|https://www.comodo.com/e-commerce/ssl-certificates/tls-1-deprecation.php].
> *Suggested Fix:* Using
> {code:java}
> SSLContext.getInstance("TLSv1.3").
> {code}
> *Feedback:* Please select any of the options down below to help us get an
> idea about how you felt about the suggestion -
> # Liked it and will make the suggested changes
> # Liked it but happy with the existing version
> # Didn’t find the suggestion helpful
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4432) duplicated queries with sytax error take unexpect long time when lazy query enabled
[
https://issues.apache.org/jira/browse/KYLIN-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4432:
Sprint: Sprint 51
> duplicated queries with sytax error take unexpect long time when lazy query
> enabled
> ---
>
> Key: KYLIN-4432
> URL: https://issues.apache.org/jira/browse/KYLIN-4432
> Project: Kylin
> Issue Type: Bug
>Reporter: Congling Xia
>Assignee: Congling Xia
>Priority: Major
> Fix For: v3.1.0
>
>
> Hi! Our Kylin server becomes unresponsive recently. All query threads were
> being TIMED_WAITING so that no more thread in the thread pool of Tomcat could
> response new requests. The server logs are full of "Duplicate SQL request"s,
> and many requests retrying for so long time:
> {code:java}
> $ cat kylin.log.25 | grep 14db5fa0-78e0-4ca6-5d04-92995b2b8367-3133991 | head
> -3
> 2020-03-23 12:15:32,279 INFO [Query
> 14db5fa0-78e0-4ca6-5d04-92995b2b8367-3133991] service.QueryService:571 :
> Duplicated SQL request is running, waiting...
> 2020-03-23 12:15:32,379 INFO [Query
> 14db5fa0-78e0-4ca6-5d04-92995b2b8367-3133991] service.QueryService:571 :
> Duplicated SQL request is running, waiting...
> 2020-03-23 12:15:32,479 INFO [Query
> 14db5fa0-78e0-4ca6-5d04-92995b2b8367-3133991] service.QueryService:571 :
> Duplicated SQL request is running, waiting...
> $ cat kylin.log.25 | grep 14db5fa0-78e0-4ca6-5d04-92995b2b8367-3133991 | tail
> -3
> 2020-03-23 12:26:56,920 INFO [Query
> 14db5fa0-78e0-4ca6-5d04-92995b2b8367-3133991] service.QueryService:571 :
> Duplicated SQL request is running, waiting...
> 2020-03-23 12:26:57,020 INFO [Query
> 14db5fa0-78e0-4ca6-5d04-92995b2b8367-3133991] service.QueryService:571 :
> Duplicated SQL request is running, waiting...
> 2020-03-23 12:26:57,121 INFO [Query
> 14db5fa0-78e0-4ca6-5d04-92995b2b8367-3133991] service.QueryService:571 :
> Duplicated SQL request is running, waiting...
> {code}
> This can be re-produced easily:
> * enable lazy query (KYLIN-2897) by setting environment variable
> kylin.query.cache-enabled=true and kylin.query.lazy-query-enabled=true
> * send the same query with syntax error many times in several minutes
> The first query will quickly response with error message, and the following
> queries will executing for long time and will repeatly output "Duplicated SQL
> request is running, waiting...".
> The code in org.apache.kylin.rest.service.QueryService#searchQueryInCache
> indicates duplicated query will wait at most one minute with default setting
> of LazyQueryWaitingTimeoutMilliSeconds but in fact it waits quite longer than
> that.
> {code:java}
> while (response.isRunning()) {
> // Wait at most one minute
> if (System.currentTimeMillis() - response.getLazyQueryStartTime() >=
> getConfig()
> .getLazyQueryWaitingTimeoutMilliSeconds()) {
> cache.evict(sqlRequest.getCacheKey());
> return null;
> }
> logger.info("Duplicated SQL request is running, waiting...");
> try {
> Thread.sleep(100L);
> } catch (InterruptedException e) {
> }
> wrapper = cache.get(sqlRequest.getCacheKey());
> if (wrapper == null) {
> return null;
> }
> response = (SQLResponse) wrapper.get();
> if (response == null) {
> return null;
> }
> }
> {code}
> After some work-around, I find that the dummy response status remains
> unchanged after execution with exception. So the 2nd query will wait
> LazyQueryWaitingTimeoutMilliSeconds, evict the dummy response created by the
> 1st query from cache, and put a new dummy response into the cache. And the
> 3rd query may use the new dummy response's start time to compute timeout, and
> will wait for another LazyQueryWaitingTimeoutMilliSeconds.
> Suppose we send _n_ identical bad queries to the server quickly. The last
> query may wait for nearly _(n-1)_ times of
> LazyQueryWaitingTimeoutMilliSeconds. That's why the Web container's thread
> pool was exhausted – user program uses bad queries to check availability of
> Kylin service and retries many times in a short time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4355) Add validation for cube re-assignmnet(Realtime OLAP)
[ https://issues.apache.org/jira/browse/KYLIN-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4355: Sprint: Sprint 52 > Add validation for cube re-assignmnet(Realtime OLAP) > > > Key: KYLIN-4355 > URL: https://issues.apache.org/jira/browse/KYLIN-4355 > Project: Kylin > Issue Type: Bug > Components: Real-time Streaming >Affects Versions: v3.0.0 >Reporter: Xiaoxiang Yu >Assignee: Xiaoxiang Yu >Priority: Minor > Fix For: v3.1.0 > > > Case 1. In assignment, specific partition can be assign to more than one > replica set, thus cause receiver consumed duplicate kafka message. > Case 2. In assignment, you can remove all partition for one repilca set, > which is make no sense at all. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4464) Query ... row_number over(order by c1) ... order by c2 ... get wrong order result
[
https://issues.apache.org/jira/browse/KYLIN-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4464:
Sprint: Sprint 51
> Query ... row_number over(order by c1) ... order by c2 ... get wrong order
> result
> -
>
> Key: KYLIN-4464
> URL: https://issues.apache.org/jira/browse/KYLIN-4464
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v3.1.0
>
> Attachments: image-2020-04-20-18-19-40-925.png,
> image-2020-04-20-18-22-34-765.png, image-2020-05-07-10-32-39-983.png
>
>
> *SQL*
> {code:java}
> select t.*, row_number() over (order by t.lstg_format_name) as row_num from (
> select lstg_format_name, sum(price) as GMV from
> KYLIN_SALES
> where lstg_format_name is not null
> group by lstg_format_name
> ) as t
> order by t.GMV desc limit 5{quote}
> {code}
>
> *Kylin result (order by lstg_format_name)*
> !image-2020-04-20-18-19-40-925.png!
>
> *Calcite execution plan*
> {code:java}
> EXECUTION PLAN BEFORE REWRITE
> OLAPToEnumerableConverter
> OLAPWindowRel(window#0=[window(partition {} order by [0 DESC] rows between
> UNBOUNDED PRECEDING and CURRENT ROW aggs [ROW_NUMBER()])], ctx=[],
> groups=[[window(partition {} order by [0 DESC] rows between UNBOUNDED
> PRECEDING and CURRENT ROW aggs [ROW_NUMBER()])]])
> OLAPLimitRel(ctx=[], fetch=[5])
> OLAPSortRel(sort0=[$1], dir0=[DESC], ctx=[])
> OLAPAggregateRel(group=[{0}], GMV=[SUM($1)], ctx=[])
> OLAPProjectRel(LSTG_FORMAT_NAME=[$2], PRICE=[$5], ctx=[])
> OLAPFilterRel(condition=[IS NOT NULL($2)], ctx=[])
> OLAPTableScan(table=[[DEFAULT, KYLIN_SALES]], ctx=[],
> fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]])
> {code}
>
> *Hive result(order by GMV)*
> !image-2020-04-20-18-22-34-765.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4419) Make it possible to change metadata in some cases without rebuilding data
[ https://issues.apache.org/jira/browse/KYLIN-4419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4419: Sprint: Sprint 51 > Make it possible to change metadata in some cases without rebuilding data > - > > Key: KYLIN-4419 > URL: https://issues.apache.org/jira/browse/KYLIN-4419 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4394) Upgrade dependency version for several CVEs
[ https://issues.apache.org/jira/browse/KYLIN-4394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4394: Sprint: Sprint 51 > Upgrade dependency version for several CVEs > --- > > Key: KYLIN-4394 > URL: https://issues.apache.org/jira/browse/KYLIN-4394 > Project: Kylin > Issue Type: Improvement > Components: Security >Reporter: XuCongying >Assignee: Yaqian Zhang >Priority: Major > Fix For: v3.1.0 > > Attachments: apache-kylin_CVE-report.md > > > I noticed some of your libraries contained CVEs. I suggest updating their > versions to increase the security of your project. The following is a > detailed content. > * *Vulnerable Library Version:* org.scala-lang : scala-compiler : 2.11.0 > *CVE ID:* > [CVE-2017-15288|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-15288] > *Import Path:* engine-flink/pom.xml, kylin-it/pom.xml, engine-spark/pom.xml > *Suggested Safe Versions:* 2.11.12, 2.12.10, 2.12.4, 2.12.5, 2.12.6, 2.12.7, > 2.12.8, 2.12.9, 2.13.0, 2.13.0-M1, 2.13.0-M2, 2.13.0-M3, 2.13.0-M3-f73b161, > 2.13.0-M4, 2.13.0-M4-pre-20d3c21, 2.13.0-M5, 2.13.0-M5-1775dba, > 2.13.0-M5-5eef812, 2.13.0-M5-6e0cba7, 2.13.0-RC1, 2.13.0-RC2, 2.13.0-RC3, > 2.13.1 > * *Vulnerable Library Version:* org.apache.tomcat : tomcat-catalina : 7.0.91 > *CVE ID:* > [CVE-2016-8735|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-8735], > [CVE-2019-0232|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-0232], > [CVE-2016-6794|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-6794], > [CVE-2016-6816|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-6816], > [CVE-2016-8745|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-8745], > [CVE-2019-17563|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17563] > *Import Path:* tomcat-ext/pom.xml, server/pom.xml, server-base/pom.xml > *Suggested Safe Versions:* 10.0.0-M1, 7.0.100, 9.0.30, 9.0.31 > * *Vulnerable Library Version:* com.h2database : h2 : 1.4.196 *CVE ID:* > [CVE-2018-10054|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-10054], > > [CVE-2018-14335|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-14335] > *Import Path:* server/pom.xml, kylin-it/pom.xml, source-jdbc/pom.xml, > source-hive/pom.xml, datasource-sdk/pom.xml *Suggested Safe Versions:* > 1.4.198, 1.4.199, 1.4.200 > * *Vulnerable Library Version:* com.google.guava : guava : 14.0 *CVE ID:* > [CVE-2018-10237|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-10237] > *Import Path:* core-storage/pom.xml, stream-receiver/pom.xml, > server/pom.xml, core-cube/pom.xml, core-metadata/pom.xml, jdbc/pom.xml, > tool-assembly/pom.xml, core-metrics/pom.xml *Suggested Safe Versions:* > 24.1.1-android, 24.1.1-jre, 25.0-android, 25.0-jre, 25.1-android, 25.1-jre, > 26.0-android, 26.0-jre, 27.0-android, 27.0-jre, 27.0.1-android, 27.0.1-jre, > 27.1-android, 27.1-jre, 28.0-android, 28.0-jre, 28.1-android, 28.1-jre, > 28.2-android, 28.2-jre > * *Vulnerable Library Version:* org.apache.hive.hcatalog : > hive-hcatalog-core : 1.2.1 *CVE ID:* > [CVE-2015-7521|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-7521] > *Import Path:* metrics-reporter-hive/pom.xml, assembly/pom.xml, > server/pom.xml, kylin-it/pom.xml, source-jdbc/pom.xml, source-hive/pom.xml, > server-base/pom.xml *Suggested Safe Versions:* 1.2.2, 2.0.0, 2.0.1, 2.1.0, > 2.1.1, 2.2.0, 2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.3.4, 2.3.5, 2.3.6, 3.0.0, 3.1.0, > 3.1.1, 3.1.2 > * *Vulnerable Library Version:* org.apache.spark : spark-core_2.11 : 2.3.2 > *CVE ID:* > [CVE-2017-7678|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-7678], > [CVE-2018-3826|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-3826], > [CVE-2018-11770|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-11770], > > [CVE-2019-10099|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-10099] > *Import Path:* server/pom.xml, kylin-it/pom.xml, engine-spark/pom.xml, > storage-hbase/pom.xml *Suggested Safe Versions:* 2.4.5 > * *Vulnerable Library Version:* org.apache.kafka : kafka_2.11 : 1.0.0 *CVE > ID:* > [CVE-2018-1288|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1288], > [CVE-2019-17196|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17196] > *Import Path:* assembly/pom.xml, source-kafka/pom.xml, kylin-it/pom.xml, > stream-source-kafka/pom.xml, metrics-reporter-kafka/pom.xml *Suggested Safe > Versions:* 2.1.1, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0 > * *Vulnerable Library Version:* org.apache.hive : hive-jdbc : 1.2.1 *CVE > ID:* > [CVE-2016-3083|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-3083], > [CVE-2015-7521|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-7521], >
[jira] [Updated] (KYLIN-4281) Precisely set the data type of tuple expression
[ https://issues.apache.org/jira/browse/KYLIN-4281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4281: Sprint: Sprint 51 > Precisely set the data type of tuple expression > --- > > Key: KYLIN-4281 > URL: https://issues.apache.org/jira/browse/KYLIN-4281 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > > Previously to simplify the calculating of sum(case when), all of the binary > calculation is based on BigDecimal, which is not good for all, especially > when dealing with count(distinct case when), whose inner data type may be hll > or bitmap. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-3947) TS conflict when kylin update metadata in JDBC (MySQL)
[
https://issues.apache.org/jira/browse/KYLIN-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110305#comment-17110305
]
nichunen commented on KYLIN-3947:
-
Hi [~inigoml]
Any update?
> TS conflict when kylin update metadata in JDBC (MySQL)
> --
>
> Key: KYLIN-3947
> URL: https://issues.apache.org/jira/browse/KYLIN-3947
> Project: Kylin
> Issue Type: Bug
> Components: Security
>Affects Versions: v2.6.1
>Reporter: Iñigo Martinez
>Assignee: Yaqian Zhang
>Priority: Major
> Fix For: v3.1.0
>
>
> Unable to login in Kylin with admin user. Kylin cluster is composed by two
> nodes, one QUERY node and one ALL node. Our backend for metadata is MySQL
> (not HBASE). Kylin version: 2.6.1
> Solved with same workaround applied in related bug.
> {code:java}
> $KYLIN_HOME/bin/metastore.sh remove /user/admin
> {code}
>
> Tomcat exception follows:
> {code:java}
> Apr 10, 2019 7:30:29 AM org.apache.catalina.core.StandardWrapperValve invoke
> SEVERE: Servlet.service() for servlet [kylin] in context with path [/kylin]
> threw exception
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /user/ADMIN, expect old TS 0, but it is 1554879164739
> at
> org.apache.kylin.common.persistence.JDBCResourceStore$5.execute(JDBCResourceStore.java:482)
> at
> org.apache.kylin.common.persistence.JDBCResourceStore.executeSql(JDBCResourceStore.java:100)
> at
> org.apache.kylin.common.persistence.JDBCResourceStore.checkAndPutResourceInternal(JDBCResourceStore.java:439)
> at
> org.apache.kylin.common.persistence.JDBCResourceStore.checkAndPutResourceImpl(JDBCResourceStore.java:427)
> at
> org.apache.kylin.common.persistence.ResourceStore$6.call(ResourceStore.java:442)
> at
> org.apache.kylin.common.persistence.ResourceStore$6.call(ResourceStore.java:439)
> at
> org.apache.kylin.common.persistence.ExponentialBackoffRetry.doWithRetry(ExponentialBackoffRetry.java:52)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceWithRetry(ResourceStore.java:439)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:427)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResource(ResourceStore.java:421)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResource(ResourceStore.java:401)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResource(ResourceStore.java:380)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192)
> at
> org.apache.kylin.rest.security.KylinUserManager.update(KylinUserManager.java:124)
> at
> org.apache.kylin.rest.service.KylinUserService.updateUser(KylinUserService.java:86)
> at
> org.apache.kylin.rest.service.KylinUserService.createUser(KylinUserService.java:78)
> at
> org.apache.kylin.rest.security.KylinAuthenticationProvider.authenticate(KylinAuthenticationProvider.java:116)
> at
> org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:174)
> at
> org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:199)
> at
> org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:180)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
>
[jira] [Resolved] (KYLIN-4168) Fix sonar reported static code issues phase 2
[ https://issues.apache.org/jira/browse/KYLIN-4168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4168. - Resolution: Fixed > Fix sonar reported static code issues phase 2 > - > > Key: KYLIN-4168 > URL: https://issues.apache.org/jira/browse/KYLIN-4168 > Project: Kylin > Issue Type: Improvement >Reporter: Yaqian Zhang >Priority: Major > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4294) Add http api for metrics
[ https://issues.apache.org/jira/browse/KYLIN-4294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4294: Sprint: Sprint 51 > Add http api for metrics > - > > Key: KYLIN-4294 > URL: https://issues.apache.org/jira/browse/KYLIN-4294 > Project: Kylin > Issue Type: Improvement > Components: REST Service >Affects Versions: v2.6.0 >Reporter: xiang zhang >Assignee: xiang zhang >Priority: Minor > Fix For: v3.1.0 > > Attachments: kylin-4294-instruction.pdf > > > # Expose metrics through http api to facilitate the integration of some > external monitoring components, such as tsdb > # add a python script for tcollector -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4192) Build UHC dictionary with spark
[ https://issues.apache.org/jira/browse/KYLIN-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4192. - Resolution: Fixed > Build UHC dictionary with spark > --- > > Key: KYLIN-4192 > URL: https://issues.apache.org/jira/browse/KYLIN-4192 > Project: Kylin > Issue Type: Improvement >Reporter: wangrupeng >Assignee: wangrupeng >Priority: Major > Fix For: v3.1.0 > > > To enable this, add following property to kylin.properties > kylin.engine.spark-udc-dictionary=true -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4317) Update doc for KYLIN-4104
[ https://issues.apache.org/jira/browse/KYLIN-4317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4317. - Resolution: Fixed > Update doc for KYLIN-4104 > - > > Key: KYLIN-4317 > URL: https://issues.apache.org/jira/browse/KYLIN-4317 > Project: Kylin > Issue Type: Improvement >Reporter: weibin0516 >Assignee: weibin0516 >Priority: Major > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4315) Use metadata numRows in beeline client for quick row counting
[ https://issues.apache.org/jira/browse/KYLIN-4315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4315: Sprint: Sprint 51 > Use metadata numRows in beeline client for quick row counting > - > > Key: KYLIN-4315 > URL: https://issues.apache.org/jira/browse/KYLIN-4315 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Congling Xia >Assignee: Congling Xia >Priority: Major > Fix For: v3.1.0 > > > Hi, I find that in `BeelineHiveClient`, method `getHiveTableRows` uses > "select count(*) from " for table row counting. The method is > invoked in flat intermediate table redistribution step in cube building. > This stats can be loaded in metastore. It costs much less time than scanning > all rows in Hive table. Since intermediate tables are created and inserted by > Kylin, statistics will be automatically calculated and stored in metastore > when > `[hive.stats.autogather|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.stats.autogather]` > is enabled (which is the default setting for Hive). > ref Hive wiki for more detail about `numRows` stats: > [https://cwiki.apache.org/confluence/display/Hive/StatsDev#StatsDev-ExistingTables%E2%80%93ANALYZE] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4330) use nrt streaming build for kafka data, can we use filter function when i set model desinger
[ https://issues.apache.org/jira/browse/KYLIN-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4330: Sprint: Sprint 51 > use nrt streaming build for kafka data, can we use filter function when i set > model desinger > - > > Key: KYLIN-4330 > URL: https://issues.apache.org/jira/browse/KYLIN-4330 > Project: Kylin > Issue Type: New Feature > Components: NRT Streaming >Affects Versions: v2.6.4 > Environment: 阿里云 centos 7 hadoop 2.8.5 >Reporter: kangkang >Priority: Major > Labels: FIlter, model > Fix For: v3.1.0 > > Attachments: 4561578452903_.pic_hd.jpg > > Original Estimate: 336h > Remaining Estimate: 336h > > when i use nrt streaming build for kafka data, can we use filter function > when i set model desinger -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4352) A empty segment will cause incomplete query result in Realtime OLAP
[ https://issues.apache.org/jira/browse/KYLIN-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4352: Sprint: Sprint 52 > A empty segment will cause incomplete query result in Realtime OLAP > --- > > Key: KYLIN-4352 > URL: https://issues.apache.org/jira/browse/KYLIN-4352 > Project: Kylin > Issue Type: Bug > Components: Real-time Streaming >Affects Versions: v3.0.0 >Reporter: Xiaoxiang Yu >Assignee: Xiaoxiang Yu >Priority: Major > Fix For: v3.1.0 > > > When you query hit a streaming cube, you query will divided into two part, > one historical part, one realtime part. > The boundary of two part is decided by the latest READY segment's *End Time* > . But when you have a segment which segment range is from future(created by > mistake), it will cause all following query get incomplete result. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4351) Provided a metadata repair tool for Realtime OLAP
[ https://issues.apache.org/jira/browse/KYLIN-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4351: Sprint: Sprint 52 > Provided a metadata repair tool for Realtime OLAP > - > > Key: KYLIN-4351 > URL: https://issues.apache.org/jira/browse/KYLIN-4351 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: Xiaoxiang Yu >Assignee: Xiaoxiang Yu >Priority: Major > Fix For: v3.1.0 > > > The metatdata of Realtime Receiver cluster is located in Zookeeper, something > it maybe removed by accident, thus cause the fail of query, and build. > In that case, user have to disable and re-enable the streaming cube. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-3121) NPE while executing a query with two left outer joins and floating point expressions on nullable fields
[ https://issues.apache.org/jira/browse/KYLIN-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-3121: Sprint: Sprint 51 > NPE while executing a query with two left outer joins and floating point > expressions on nullable fields > --- > > Key: KYLIN-3121 > URL: https://issues.apache.org/jira/browse/KYLIN-3121 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.2.0 > Environment: HDP 2.5.6, Kylin 2.2.0 >Reporter: Vsevolod Ostapenko >Assignee: nichunen >Priority: Major > Fix For: v3.1.0 > > > Queries that include two (or more) left outer joins and contain floating > point expressions that operate on the fields that contain integer NULL values > (due to left outer join) fail in-flight with NullPointerExceptions. > As an example, the following query generates NPE on either of the two > expressions: > * 100.0 * t2.media_gap_call_count > * 1.0 * NULLIF(t1.active_call_count, 0) > with > t1 > as > ( > select > d1.cell_name, > count(distinct a1.call_id) as active_call_count > from > zetticsdw.a_vl_hourly_v a1 > inner join > zetticsdw.d_cell_v d1 > on > a1.cell_key = d1.cell_key > where > d1.region_3 = 'Mumbai' > and > a1.thedate = '20171011' > and > a1.thehour = '00' > and > a1.active_call_flg = 1 > group by > d1.cell_name > ), > t2 > as > ( > select > d1.cell_name, > count(distinct a1.call_id) as media_gap_call_count > from > zetticsdw.a_vl_hourly_v a1 > inner join > zetticsdw.d_cell_v d1 > on > a1.cell_key = d1.cell_key > where > d1.region_3 = 'Mumbai' > and > a1.thedate='20171011' > and > a1.thehour = '00' > and > a1.media_gap_call_flg = 1 > group by > d1.cell_name > ) > , > t3 > as > ( > select > d1.cell_name, > sum(a1.ow_call_flg) one_way_call_count, > sum(a1.succ_call_flg) successfull_call_count > from > zetticsdw.a_vl_hourly_v a1 > inner join > zetticsdw.d_cell_v d1 > on > a1.cell_key = d1.cell_key > where > d1.region_3 = 'Mumbai' > and > a1.thedate='20171011' > and > a1.thehour = '00' > group by > d1.cell_name > ) > select >t3.cell_name, >t1.active_call_count, >t2.media_gap_call_count, >t3.one_way_call_count, >t3.successfull_call_count, >-- 100 * t2.media_gap_call_count nom, -- > works >-- 1 * NULLIF(t1.active_call_count, 0) denom-- > works >100.0 * t2.media_gap_call_count nom, -- fails, > NPE of one kind >1.0 * NULLIF(t1.active_call_count, 0) denom -- fails, > NPE of different kind >-- 100.0 * COALESCE(t2.media_gap_call_count, 0) nom,-- > works >-- 1.0 * CAST(NULLIF(t1.active_call_count, 0) as DOUBLE) denom -- > works > from >t3 > left outer join >t1 > on >t3.cell_name = t1.cell_name > left outer join >t2 > on >t3.cell_name = t2.cell_name > In the first case (multiplication of an integer field with a NULL value and a > double) kylin log contains a stack trace similar to the following: > null > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) > at > org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:218) > at > org.apache.kylin.rest.service.QueryService.execute(QueryService.java:834) > at > org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:561) > at > org.apache.kylin.rest.service.QueryService.query(QueryService.java:181) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:415) > at > org.apache.kylin.rest.controller.QueryController.query(QueryController.java:78) > at sun.reflect.GeneratedMethodAccessor545.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) > at >
[jira] [Updated] (KYLIN-4120) Failed to query "select * from {lookup}" if a lookup table joined in two different models
[
https://issues.apache.org/jira/browse/KYLIN-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4120:
Sprint: Sprint 52
> Failed to query "select * from {lookup}" if a lookup table joined in two
> different models
> -
>
> Key: KYLIN-4120
> URL: https://issues.apache.org/jira/browse/KYLIN-4120
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v3.0.0-alpha2
>Reporter: nichunen
>Assignee: Xiaoxiang Yu
>Priority: Major
> Fix For: v3.1.0
>
>
> Steps to reproduce:
> 1. Create two models with same lookup table (create snapshot)
> 2. Define different columns as dimensions in the two models
> (Etc: in model ma, with column lookup.c1, lookup.c2; in model mb, with
> column lookup.c2, lookup.c3)
> 3. Create two cubes with the two models, and build the 2 cubes
> 4. Run query with "select * from {lookup}" with the 2 cubes in ready status,
> it should be failed with exception message like
> "
> No model found for OLAPContext,
> CUBE_NOT_CONTAIN_ALL_COLUMN[1_39bd1072:SSB.DATES.D_DAYNUMINWEEK,
> 1_39bd1072:SSB.DATES.D_DAYOFWEEK, 1_39bd1072:SSB.DATES.D_DATE,
> 1_39bd1072:SSB.DATES.D_DAYNUMINMONTH, 1_39bd1072:SSB.DATES.D_DAYNUMINYEAR,
> 1_39bd1072:SSB.DATES.D_MONTH, 1_39bd1072:SSB.DATES.D_MONTHNUMINYEAR],
> CUBE_NOT_READY, CUBE_NOT_READY,
> CUBE_NOT_CONTAIN_ALL_COLUMN[1_39bd1072:SSB.DATES.D_YEARMONTHNUM,
> 1_39bd1072:SSB.DATES.D_YEAR], rel#218:OLAPTableScan.OLAP.[](table=[SSB,
> DATES],ctx=,fields=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]) while executing
> SQL: "select * from DATES LIMIT 5"
> "
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KYLIN-4418) Bug fix for ShardingHash.getShard in HBaseLookupRowEncoder
[ https://issues.apache.org/jira/browse/KYLIN-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4418. - Resolution: Fixed > Bug fix for ShardingHash.getShard in HBaseLookupRowEncoder > -- > > Key: KYLIN-4418 > URL: https://issues.apache.org/jira/browse/KYLIN-4418 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4417) Use hash rather than random for ConvergeCuboidDataPartitioner to avoid potential issue
[ https://issues.apache.org/jira/browse/KYLIN-4417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4417. - Resolution: Fixed > Use hash rather than random for ConvergeCuboidDataPartitioner to avoid > potential issue > -- > > Key: KYLIN-4417 > URL: https://issues.apache.org/jira/browse/KYLIN-4417 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Minor > Fix For: v3.1.0 > > > If use random, similar issue will occur KYLIN-3388 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4414) Make bulkload locality possible
[ https://issues.apache.org/jira/browse/KYLIN-4414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4414: Fix Version/s: v3.1.0 > Make bulkload locality possible > --- > > Key: KYLIN-4414 > URL: https://issues.apache.org/jira/browse/KYLIN-4414 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Minor > Fix For: v3.1.0 > > > # Firstly, we need to do write data directly to the hbase cluster > # Then we can apply patch http://issues.apache.org/jira/browse/HBASE-12596 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4414) Make bulkload locality possible
[ https://issues.apache.org/jira/browse/KYLIN-4414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4414. - Resolution: Fixed > Make bulkload locality possible > --- > > Key: KYLIN-4414 > URL: https://issues.apache.org/jira/browse/KYLIN-4414 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Minor > Fix For: v3.1.0 > > > # Firstly, we need to do write data directly to the hbase cluster > # Then we can apply patch http://issues.apache.org/jira/browse/HBASE-12596 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4415) HTable Creation with Retry
[ https://issues.apache.org/jira/browse/KYLIN-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4415. - Resolution: Fixed > HTable Creation with Retry > -- > > Key: KYLIN-4415 > URL: https://issues.apache.org/jira/browse/KYLIN-4415 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4415) HTable Creation with Retry
[ https://issues.apache.org/jira/browse/KYLIN-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4415: Fix Version/s: v3.1.0 > HTable Creation with Retry > -- > > Key: KYLIN-4415 > URL: https://issues.apache.org/jira/browse/KYLIN-4415 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4416) Disable htable compaction
[ https://issues.apache.org/jira/browse/KYLIN-4416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4416: Fix Version/s: v3.1.0 > Disable htable compaction > - > > Key: KYLIN-4416 > URL: https://issues.apache.org/jira/browse/KYLIN-4416 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Minor > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4416) Disable htable compaction
[ https://issues.apache.org/jira/browse/KYLIN-4416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4416. - Resolution: Fixed > Disable htable compaction > - > > Key: KYLIN-4416 > URL: https://issues.apache.org/jira/browse/KYLIN-4416 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Minor > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4325) ”Value not exists“ when invoke getIdFromValue() for global dictionary field during cube building
[ https://issues.apache.org/jira/browse/KYLIN-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4325. - Resolution: Fixed > ”Value not exists“ when invoke getIdFromValue() for global dictionary field > during cube building > > > Key: KYLIN-4325 > URL: https://issues.apache.org/jira/browse/KYLIN-4325 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > > When building global dictionary, even the field is numeric type, we should > regard it as string rather than number in SelfDefineSortableKey -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4411) Job Engine Improvement
[ https://issues.apache.org/jira/browse/KYLIN-4411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4411. - Resolution: Fixed > Job Engine Improvement > -- > > Key: KYLIN-4411 > URL: https://issues.apache.org/jira/browse/KYLIN-4411 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Minor > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4418) Bug fix for ShardingHash.getShard in HBaseLookupRowEncoder
[ https://issues.apache.org/jira/browse/KYLIN-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110190#comment-17110190 ] ASF subversion and git services commented on KYLIN-4418: Commit 7886a2498178ded085497d6204b4b367f56ce0c8 in kylin's branch refs/heads/master from Zhong, Yanghong [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=7886a24 ] KYLIN-4418 Bug fix for ShardingHash.getShard in HBaseLookupRowEncoder > Bug fix for ShardingHash.getShard in HBaseLookupRowEncoder > -- > > Key: KYLIN-4418 > URL: https://issues.apache.org/jira/browse/KYLIN-4418 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4416) Disable htable compaction
[ https://issues.apache.org/jira/browse/KYLIN-4416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110186#comment-17110186 ] ASF subversion and git services commented on KYLIN-4416: Commit 03274297b1dcdd26a93ee6903e764d92e8847a85 in kylin's branch refs/heads/master from Zhong, Yanghong [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=0327429 ] KYLIN-4416 Disable htable compaction > Disable htable compaction > - > > Key: KYLIN-4416 > URL: https://issues.apache.org/jira/browse/KYLIN-4416 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4325) ”Value not exists“ when invoke getIdFromValue() for global dictionary field during cube building
[ https://issues.apache.org/jira/browse/KYLIN-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4325: Fix Version/s: v3.1.0 > ”Value not exists“ when invoke getIdFromValue() for global dictionary field > during cube building > > > Key: KYLIN-4325 > URL: https://issues.apache.org/jira/browse/KYLIN-4325 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > > When building global dictionary, even the field is numeric type, we should > regard it as string rather than number in SelfDefineSortableKey -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4411) Job Engine Improvement
[ https://issues.apache.org/jira/browse/KYLIN-4411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110183#comment-17110183 ] ASF GitHub Bot commented on KYLIN-4411: --- nichunen merged pull request #1156: URL: https://github.com/apache/kylin/pull/1156 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Job Engine Improvement > -- > > Key: KYLIN-4411 > URL: https://issues.apache.org/jira/browse/KYLIN-4411 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Minor > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4414) Make bulkload locality possible
[ https://issues.apache.org/jira/browse/KYLIN-4414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110187#comment-17110187 ] ASF subversion and git services commented on KYLIN-4414: Commit 981cc273be31bb3099c3de34c78f88f986b0db8d in kylin's branch refs/heads/master from Zhong, Yanghong [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=981cc27 ] KYLIN-4414 add a property jobTempDir for AbstractHadoopJob to indicate the directory for all of the temporary files > Make bulkload locality possible > --- > > Key: KYLIN-4414 > URL: https://issues.apache.org/jira/browse/KYLIN-4414 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Minor > > # Firstly, we need to do write data directly to the hbase cluster > # Then we can apply patch http://issues.apache.org/jira/browse/HBASE-12596 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4325) ”Value not exists“ when invoke getIdFromValue() for global dictionary field during cube building
[ https://issues.apache.org/jira/browse/KYLIN-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110184#comment-17110184 ] ASF subversion and git services commented on KYLIN-4325: Commit 372f4ddf324dde8b1bc430710f7c3a09e7f30151 in kylin's branch refs/heads/master from Zhong, Yanghong [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=372f4dd ] KYLIN-4325 fix TypeFlag for global dictionary field in SelfDefineSortableKey > ”Value not exists“ when invoke getIdFromValue() for global dictionary field > during cube building > > > Key: KYLIN-4325 > URL: https://issues.apache.org/jira/browse/KYLIN-4325 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > > When building global dictionary, even the field is numeric type, we should > regard it as string rather than number in SelfDefineSortableKey -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4415) HTable Creation with Retry
[ https://issues.apache.org/jira/browse/KYLIN-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110185#comment-17110185 ] ASF subversion and git services commented on KYLIN-4415: Commit 20353a339054b71ae50386bf08397c780d2244e7 in kylin's branch refs/heads/master from Zhong, Yanghong [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=20353a3 ] KYLIN-4415 HTable Creation with Retry > HTable Creation with Retry > -- > > Key: KYLIN-4415 > URL: https://issues.apache.org/jira/browse/KYLIN-4415 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4417) Use hash rather than random for ConvergeCuboidDataPartitioner to avoid potential issue
[ https://issues.apache.org/jira/browse/KYLIN-4417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110189#comment-17110189 ] ASF subversion and git services commented on KYLIN-4417: Commit beb976a0ef3a3a6c1d488d4e3beeefaef1f545ce in kylin's branch refs/heads/master from Zhong, Yanghong [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=beb976a ] KYLIN-4417 Use hash rather than random to avoid potential issue in ConvergeCuboidDataPartitioner > Use hash rather than random for ConvergeCuboidDataPartitioner to avoid > potential issue > -- > > Key: KYLIN-4417 > URL: https://issues.apache.org/jira/browse/KYLIN-4417 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Minor > Fix For: v3.1.0 > > > If use random, similar issue will occur KYLIN-3388 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4414) Make bulkload locality possible
[ https://issues.apache.org/jira/browse/KYLIN-4414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110188#comment-17110188 ] ASF subversion and git services commented on KYLIN-4414: Commit 8d2a53fcd7711a843b3f34eae5953a8bfff3d3be in kylin's branch refs/heads/master from Zhong, Yanghong [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=8d2a53f ] KYLIN-4414 bulkload needs to follow locality > Make bulkload locality possible > --- > > Key: KYLIN-4414 > URL: https://issues.apache.org/jira/browse/KYLIN-4414 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Minor > > # Firstly, we need to do write data directly to the hbase cluster > # Then we can apply patch http://issues.apache.org/jira/browse/HBASE-12596 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] nichunen merged pull request #1156: KYLIN-4411 Job Engine Improvement
nichunen merged pull request #1156: URL: https://github.com/apache/kylin/pull/1156 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (KYLIN-4418) Bug fix for ShardingHash.getShard in HBaseLookupRowEncoder
[ https://issues.apache.org/jira/browse/KYLIN-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4418: Fix Version/s: v3.1.0 > Bug fix for ShardingHash.getShard in HBaseLookupRowEncoder > -- > > Key: KYLIN-4418 > URL: https://issues.apache.org/jira/browse/KYLIN-4418 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4417) Use hash rather than random for ConvergeCuboidDataPartitioner to avoid potential issue
[ https://issues.apache.org/jira/browse/KYLIN-4417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4417: Fix Version/s: v3.1.0 > Use hash rather than random for ConvergeCuboidDataPartitioner to avoid > potential issue > -- > > Key: KYLIN-4417 > URL: https://issues.apache.org/jira/browse/KYLIN-4417 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Minor > Fix For: v3.1.0 > > > If use random, similar issue will occur KYLIN-3388 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-3837) hadoop3.0 mapreduce task can not run successfully
[ https://issues.apache.org/jira/browse/KYLIN-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110066#comment-17110066 ] zhaozhangwan commented on KYLIN-3837: - !image-2020-05-18-17-08-51-038.png! > hadoop3.0 mapreduce task can not run successfully > - > > Key: KYLIN-3837 > URL: https://issues.apache.org/jira/browse/KYLIN-3837 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.1 >Reporter: 梁龙 >Priority: Major > Attachments: image-2019-03-04-09-59-28-862.png, > image-2019-03-04-10-00-33-056.png, image-2019-03-04-10-03-04-021.png > > > At the kylin_streaming_cube demo，The first step is to make a mistake. > !image-2019-03-04-09-59-28-862.png! > > Output ： > !image-2019-03-04-10-00-33-056.png! > > and the mapreduce error: > !image-2019-03-04-10-03-04-021.png! > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4500) Timeout waiting for connection from pool
[
https://issues.apache.org/jira/browse/KYLIN-4500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Arki updated KYLIN-4500:
--
Description:
h4. Environment
* Kylin server 3.0.0
* EMR 5.28
h4. Issue
After an extended uptime, both Kylin query server and jobs running on EMR stop
working. The root cause in both cases is:
{noformat}
Caused by: java.io.IOException:
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.SdkClientException: Unable to
execute HTTP request: Timeout waiting for connection from pool
at
com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.getFileStatus(S3NativeFileSystem2.java:257)
~[emrfs-hadoop-assembly-2.37.0.jar:?]{noformat}
Based on
[https://aws.amazon.com/premiumsupport/knowledge-center/emr-timeout-connection-wait/]
increasing the fs.s3.maxConnections setting to 1 is just delaying the
issue thus the underlying issue is likely a connection leak. It also indicates
a leak that restarting the kylin service solves the problem.
A full stack trace from the QueryService is attached.
was:
h4. Environment
* Kylin server 3.0.0
* EMR 5.28
h4. Issue
After an extended uptime, both Kylin query server and jobs running on EMR stop
working. The root cause in both cases is:
{noformat}
Caused by: java.io.IOException:
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.SdkClientException: Unable to
execute HTTP request: Timeout waiting for connection from pool
at
com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.getFileStatus(S3NativeFileSystem2.java:257)
~[emrfs-hadoop-assembly-2.37.0.jar:?]{noformat}
{{Based on
[https://aws.amazon.com/premiumsupport/knowledge-center/emr-timeout-connection-wait/]
increasing the *fs.s3.maxConnections* setting to 1 is just delaying the
issue thus the underlying issue is likely a connection leak. It also indicates
a leak that restarting the kylin service solves the problem.}}
{{A full stack trace from the QueryService is attached.}}
> Timeout waiting for connection from pool
>
>
> Key: KYLIN-4500
> URL: https://issues.apache.org/jira/browse/KYLIN-4500
> Project: Kylin
> Issue Type: Bug
>Reporter: Gabor Arki
>Priority: Major
> Attachments: kylin-connection-timeout.txt
>
>
> h4. Environment
> * Kylin server 3.0.0
> * EMR 5.28
> h4. Issue
> After an extended uptime, both Kylin query server and jobs running on EMR
> stop working. The root cause in both cases is:
> {noformat}
> Caused by: java.io.IOException:
> com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.SdkClientException: Unable
> to execute HTTP request: Timeout waiting for connection from pool
> at
> com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.getFileStatus(S3NativeFileSystem2.java:257)
> ~[emrfs-hadoop-assembly-2.37.0.jar:?]{noformat}
> Based on
> [https://aws.amazon.com/premiumsupport/knowledge-center/emr-timeout-connection-wait/]
> increasing the fs.s3.maxConnections setting to 1 is just delaying the
> issue thus the underlying issue is likely a connection leak. It also
> indicates a leak that restarting the kylin service solves the problem.
> A full stack trace from the QueryService is attached.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-3837) hadoop3.0 mapreduce task can not run successfully
[ https://issues.apache.org/jira/browse/KYLIN-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110061#comment-17110061 ] zhaozhangwan commented on KYLIN-3837: - how to fix this ，i have same problem，thank you !image-2020-05-18-17-05-18-702.png! > hadoop3.0 mapreduce task can not run successfully > - > > Key: KYLIN-3837 > URL: https://issues.apache.org/jira/browse/KYLIN-3837 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.1 >Reporter: 梁龙 >Priority: Major > Attachments: image-2019-03-04-09-59-28-862.png, > image-2019-03-04-10-00-33-056.png, image-2019-03-04-10-03-04-021.png > > > At the kylin_streaming_cube demo，The first step is to make a mistake. > !image-2019-03-04-09-59-28-862.png! > > Output ： > !image-2019-03-04-10-00-33-056.png! > > and the mapreduce error: > !image-2019-03-04-10-03-04-021.png! > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4500) Timeout waiting for connection from pool
[
https://issues.apache.org/jira/browse/KYLIN-4500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Arki updated KYLIN-4500:
--
Description:
h4. Environment
* Kylin server 3.0.0
* EMR 5.28
h4. Issue
After an extended uptime, both Kylin query server and jobs running on EMR stop
working. The root cause in both cases is:
{noformat}
Caused by: java.io.IOException:
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.SdkClientException: Unable to
execute HTTP request: Timeout waiting for connection from pool
at
com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.getFileStatus(S3NativeFileSystem2.java:257)
~[emrfs-hadoop-assembly-2.37.0.jar:?]{noformat}
{{Based on
[https://aws.amazon.com/premiumsupport/knowledge-center/emr-timeout-connection-wait/]
increasing the *fs.s3.maxConnections* setting to 1 is just delaying the
issue thus the underlying issue is likely a connection leak. It also indicates
a leak that restarting the kylin service solves the problem.}}
{{A full stack trace from the QueryService is attached.}}
was:
h4. Environment
* Kylin server 3.0.0
* EMR 5.28
h4. Issue
After an extended uptime, both Kylin query server and jobs running on EMR stop
working. The root cause is both cases is:
{noformat}
Caused by: java.io.IOException:
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.SdkClientException: Unable to
execute HTTP request: Timeout waiting for connection from pool
at
com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.getFileStatus(S3NativeFileSystem2.java:257)
~[emrfs-hadoop-assembly-2.37.0.jar:?]{noformat}
{{Based on
[https://aws.amazon.com/premiumsupport/knowledge-center/emr-timeout-connection-wait/]
increasing the *fs.s3.maxConnections* setting to 1 is just delaying the
issue thus the underlying issue is likely a connection leak. It also indicates
a leak that restarting the kylin service solves the problem.}}
{{A full stack trace from the QueryService is attached.}}
> Timeout waiting for connection from pool
>
>
> Key: KYLIN-4500
> URL: https://issues.apache.org/jira/browse/KYLIN-4500
> Project: Kylin
> Issue Type: Bug
>Reporter: Gabor Arki
>Priority: Major
> Attachments: kylin-connection-timeout.txt
>
>
> h4. Environment
> * Kylin server 3.0.0
> * EMR 5.28
> h4. Issue
> After an extended uptime, both Kylin query server and jobs running on EMR
> stop working. The root cause in both cases is:
> {noformat}
> Caused by: java.io.IOException:
> com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.SdkClientException: Unable
> to execute HTTP request: Timeout waiting for connection from pool
> at
> com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.getFileStatus(S3NativeFileSystem2.java:257)
> ~[emrfs-hadoop-assembly-2.37.0.jar:?]{noformat}
> {{Based on
> [https://aws.amazon.com/premiumsupport/knowledge-center/emr-timeout-connection-wait/]
> increasing the *fs.s3.maxConnections* setting to 1 is just delaying the
> issue thus the underlying issue is likely a connection leak. It also
> indicates a leak that restarting the kylin service solves the problem.}}
> {{A full stack trace from the QueryService is attached.}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4500) Timeout waiting for connection from pool
[
https://issues.apache.org/jira/browse/KYLIN-4500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Arki updated KYLIN-4500:
--
Description:
h4. Environment
* Kylin server 3.0.0
* EMR 5.28
h4. Issue
After an extended uptime, both Kylin query server and jobs running on EMR stop
working. The root cause is both cases is:
{noformat}
Caused by: java.io.IOException:
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.SdkClientException: Unable to
execute HTTP request: Timeout waiting for connection from pool
at
com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.getFileStatus(S3NativeFileSystem2.java:257)
~[emrfs-hadoop-assembly-2.37.0.jar:?]{noformat}
{{Based on
[https://aws.amazon.com/premiumsupport/knowledge-center/emr-timeout-connection-wait/]
increasing the *fs.s3.maxConnections* setting to 1 is just delaying the
issue thus the underlying issue is likely a connection leak. It also indicates
a leak that restarting the kylin service solves the problem.}}
{{A full stack trace from the QueryService is attached.}}
was:
h4. Environment
* Kylin server 3.0.0
* EMR 5.28
h4. Issue
After an extended uptime, both Kylin query server and jobs running on EMR stop
working. The root cause is both cases is:
{{Caused by: java.io.IOException:
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.SdkClientException: Unable to
execute HTTP request: Timeout waiting for connection from pool}}
{{ at
com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.getFileStatus(S3NativeFileSystem2.java:257)
~[emrfs-hadoop-assembly-2.37.0.jar:?]}}
{{Based on
[https://aws.amazon.com/premiumsupport/knowledge-center/emr-timeout-connection-wait/]
increasing the *fs.s3.maxConnections* setting to 1 is just delaying the
issue thus the underlying issue is likely a connection leak. It also indicates
a leak that restarting the kylin service solves the problem.}}
{{A full stack trace from the QueryService is attached.}}
> Timeout waiting for connection from pool
>
>
> Key: KYLIN-4500
> URL: https://issues.apache.org/jira/browse/KYLIN-4500
> Project: Kylin
> Issue Type: Bug
>Reporter: Gabor Arki
>Priority: Major
> Attachments: kylin-connection-timeout.txt
>
>
> h4. Environment
> * Kylin server 3.0.0
> * EMR 5.28
> h4. Issue
> After an extended uptime, both Kylin query server and jobs running on EMR
> stop working. The root cause is both cases is:
> {noformat}
> Caused by: java.io.IOException:
> com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.SdkClientException: Unable
> to execute HTTP request: Timeout waiting for connection from pool
> at
> com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.getFileStatus(S3NativeFileSystem2.java:257)
> ~[emrfs-hadoop-assembly-2.37.0.jar:?]{noformat}
> {{Based on
> [https://aws.amazon.com/premiumsupport/knowledge-center/emr-timeout-connection-wait/]
> increasing the *fs.s3.maxConnections* setting to 1 is just delaying the
> issue thus the underlying issue is likely a connection leak. It also
> indicates a leak that restarting the kylin service solves the problem.}}
> {{A full stack trace from the QueryService is attached.}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4500) Timeout waiting for connection from pool
[
https://issues.apache.org/jira/browse/KYLIN-4500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Arki updated KYLIN-4500:
--
Attachment: kylin-connection-timeout.txt
> Timeout waiting for connection from pool
>
>
> Key: KYLIN-4500
> URL: https://issues.apache.org/jira/browse/KYLIN-4500
> Project: Kylin
> Issue Type: Bug
>Reporter: Gabor Arki
>Priority: Major
> Attachments: kylin-connection-timeout.txt
>
>
> h4. Environment
> * Kylin server 3.0.0
> * EMR 5.28
> h4. Issue
> After an extended uptime, both Kylin query server and jobs running on EMR
> stop working. The root cause is both cases is:
> {noformat}
> Caused by: java.io.IOException:
> com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.SdkClientException: Unable
> to execute HTTP request: Timeout waiting for connection from pool
> at
> com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.getFileStatus(S3NativeFileSystem2.java:257)
> ~[emrfs-hadoop-assembly-2.37.0.jar:?]{noformat}
> {{Based on
> [https://aws.amazon.com/premiumsupport/knowledge-center/emr-timeout-connection-wait/]
> increasing the *fs.s3.maxConnections* setting to 1 is just delaying the
> issue thus the underlying issue is likely a connection leak. It also
> indicates a leak that restarting the kylin service solves the problem.}}
> {{A full stack trace from the QueryService is attached.}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KYLIN-4500) Timeout waiting for connection from pool
Gabor Arki created KYLIN-4500:
-
Summary: Timeout waiting for connection from pool
Key: KYLIN-4500
URL: https://issues.apache.org/jira/browse/KYLIN-4500
Project: Kylin
Issue Type: Bug
Reporter: Gabor Arki
h4. Environment
* Kylin server 3.0.0
* EMR 5.28
h4. Issue
After an extended uptime, both Kylin query server and jobs running on EMR stop
working. The root cause is both cases is:
{{Caused by: java.io.IOException:
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.SdkClientException: Unable to
execute HTTP request: Timeout waiting for connection from pool}}
{{ at
com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.getFileStatus(S3NativeFileSystem2.java:257)
~[emrfs-hadoop-assembly-2.37.0.jar:?]}}
{{Based on
[https://aws.amazon.com/premiumsupport/knowledge-center/emr-timeout-connection-wait/]
increasing the *fs.s3.maxConnections* setting to 1 is just delaying the
issue thus the underlying issue is likely a connection leak. It also indicates
a leak that restarting the kylin service solves the problem.}}
{{A full stack trace from the QueryService is attached.}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4499) Extract kylin server self discovery service from CuratorScheduler
[ https://issues.apache.org/jira/browse/KYLIN-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-4499: -- Description: It's better to extract the kylin server self discovery service to the common module so that other places can benefit from it besides the job module. (was: Currently we should manually set kylin.server.host-address for each individual server rest address, which should be self detected for real kylin server self registry.) > Extract kylin server self discovery service from CuratorScheduler > - > > Key: KYLIN-4499 > URL: https://issues.apache.org/jira/browse/KYLIN-4499 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > > It's better to extract the kylin server self discovery service to the common > module so that other places can benefit from it besides the job module. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4499) Extract kylin server self discovery service from CuratorScheduler
[ https://issues.apache.org/jira/browse/KYLIN-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-4499: -- Description: It's better to extract the kylin server self discovery service to the common module so that other places can benefit from it besides the CuratorScheduler. (was: It's better to extract the kylin server self discovery service to the common module so that other places can benefit from it besides the job module.) > Extract kylin server self discovery service from CuratorScheduler > - > > Key: KYLIN-4499 > URL: https://issues.apache.org/jira/browse/KYLIN-4499 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > > It's better to extract the kylin server self discovery service to the common > module so that other places can benefit from it besides the CuratorScheduler. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4499) Extract kylin server self discovery service from CuratorScheduler
[ https://issues.apache.org/jira/browse/KYLIN-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-4499: -- Summary: Extract kylin server self discovery service from CuratorScheduler (was: Self detect server rest address) > Extract kylin server self discovery service from CuratorScheduler > - > > Key: KYLIN-4499 > URL: https://issues.apache.org/jira/browse/KYLIN-4499 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > > Currently we should manually set kylin.server.host-address for each > individual server rest address, which should be self detected for real kylin > server self registry. -- This message was sent by Atlassian Jira (v8.3.4#803005)
