[jira] [Updated] (KYLIN-3310) Use lint for maven-compiler-plugin
[
https://issues.apache.org/jira/browse/KYLIN-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated KYLIN-3310:
--
Description:
lint helps identify structural problems.
We should enable lint for maven-compiler-plugin
{code}
maven-compiler-plugin
${maven-compiler-plugin.version}
1.8
1.8
-Xlint:all
${compiler.error.flag}
-Xlint:-options
-Xlint:-cast
-Xlint:-deprecation
-Xlint:-processing
-Xlint:-rawtypes
-Xlint:-serial
-Xlint:-try
-Xlint:-unchecked
-Xlint:-varargs
true
false
{code}
was:
lint helps identify structural problems.
We should enable lint for maven-compiler-plugin
{code}
maven-compiler-plugin
${maven-compiler-plugin.version}
1.8
1.8
-Xlint:all
${compiler.error.flag}
-Xlint:-options
-Xlint:-cast
-Xlint:-deprecation
-Xlint:-processing
-Xlint:-rawtypes
-Xlint:-serial
-Xlint:-try
-Xlint:-unchecked
-Xlint:-varargs
true
false
{code}
> Use lint for maven-compiler-plugin
> --
>
> Key: KYLIN-3310
> URL: https://issues.apache.org/jira/browse/KYLIN-3310
> Project: Kylin
> Issue Type: Improvement
> Components: Tools, Build and Test
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
> Fix For: v2.6.0
>
>
> lint helps identify structural problems.
> We should enable lint for maven-compiler-plugin
> {code}
> maven-compiler-plugin
> ${maven-compiler-plugin.version}
>
> 1.8
> 1.8
>
> -Xlint:all
> ${compiler.error.flag}
>
> -Xlint:-options
>
> -Xlint:-cast
> -Xlint:-deprecation
> -Xlint:-processing
> -Xlint:-rawtypes
> -Xlint:-serial
> -Xlint:-try
> -Xlint:-unchecked
> -Xlint:-varargs
>
>
>
>
> true
>
> false
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3631) Utilize Arrays#parallelSort for better performance
Ted Yu created KYLIN-3631: - Summary: Utilize Arrays#parallelSort for better performance Key: KYLIN-3631 URL: https://issues.apache.org/jira/browse/KYLIN-3631 Project: Kylin Issue Type: Task Reporter: Ted Yu Arrays#parallelSort was introduced since Java 1.8 We can utilize Arrays#parallelSort to achieve better performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3610) Upgrade Curator version
Ted Yu created KYLIN-3610: - Summary: Upgrade Curator version Key: KYLIN-3610 URL: https://issues.apache.org/jira/browse/KYLIN-3610 Project: Kylin Issue Type: Task Reporter: Ted Yu Currently we use Curator 2.12.0 Latest Curator release is 4.x We should upgrade Curator dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3585) Ineffective declaration of volatile field in SparkFactDistinct
[
https://issues.apache.org/jira/browse/KYLIN-3585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637139#comment-16637139
]
Ted Yu commented on KYLIN-3585:
---
lgtm
> Ineffective declaration of volatile field in SparkFactDistinct
> --
>
> Key: KYLIN-3585
> URL: https://issues.apache.org/jira/browse/KYLIN-3585
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> In CuboidStatCalculator :
> {code}
> private volatile HLLCounter[] cuboidsHLL;
> {code}
> Though the array is declared volatile, the array elements are not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3445) Upgrade checkstyle version to 8.6
[ https://issues.apache.org/jira/browse/KYLIN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637033#comment-16637033 ] Ted Yu commented on KYLIN-3445: --- Can this be resolved ? > Upgrade checkstyle version to 8.6 > - > > Key: KYLIN-3445 > URL: https://issues.apache.org/jira/browse/KYLIN-3445 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Ted Yu >Priority: Minor > Fix For: v2.6.0 > > > We should upgrade checkstyle version to 8.6+ so that we can use the "match > violation message to this regex" feature for suppression. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3447) Upgrade zookeeper to 3.4.13
[ https://issues.apache.org/jira/browse/KYLIN-3447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3447: -- Description: zookeeper 3.4.13 is being released with the following fixes: ZOOKEEPER-2959 fixes data loss when observer is used ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container / cloud) environment was: zookeeper 3.4.13 is being released with the following fixes: ZOOKEEPER-2959 fixes data loss when observer is used ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container / cloud) environment > Upgrade zookeeper to 3.4.13 > --- > > Key: KYLIN-3447 > URL: https://issues.apache.org/jira/browse/KYLIN-3447 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > > zookeeper 3.4.13 is being released with the following fixes: > ZOOKEEPER-2959 fixes data loss when observer is used > ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container > / cloud) > environment -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3609) NPE from QueryMetricsFacade#updateMetricsToReservoir
Ted Yu created KYLIN-3609:
-
Summary: NPE from QueryMetricsFacade#updateMetricsToReservoir
Key: KYLIN-3609
URL: https://issues.apache.org/jira/browse/KYLIN-3609
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
When running test suite, I saw the following in test output:
{code}
2018-10-02 20:59:20,415 WARN [Query c8f42f2e-8c77-2bfc-97ab-053fbeb7c86e-1]
service.QueryService:423 : Write metric error.
java.lang.NullPointerException
at
org.apache.kylin.rest.metrics.QueryMetricsFacade.updateMetricsToReservoir(QueryMetricsFacade.java:148)
at
org.apache.kylin.rest.metrics.QueryMetricsFacade.updateMetrics(QueryMetricsFacade.java:74)
at
org.apache.kylin.rest.service.QueryService.recordMetric(QueryService.java:505)
at
org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:421)
at
org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:353)
at
org.apache.kylin.rest.controller.QueryController.query(QueryController.java:87)
at
org.apache.kylin.rest.controller.QueryControllerTest.testQueryException(QueryControllerTest.java:63)
{code}
It seems sqlResponse.getResults() returned null.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3608) Move dependency versions to top level pom properties
Ted Yu created KYLIN-3608: - Summary: Move dependency versions to top level pom properties Key: KYLIN-3608 URL: https://issues.apache.org/jira/browse/KYLIN-3608 Project: Kylin Issue Type: Task Reporter: Ted Yu There are some non-top level pom.xml files where dependency version is referenced directly. core-common/pom.xml is an example. We should move all dependency versions to top level pom properties -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3586) Boxing/unboxing to parse a primitive is suboptimal
[
https://issues.apache.org/jira/browse/KYLIN-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635659#comment-16635659
]
Ted Yu commented on KYLIN-3586:
---
There are still a few instances of valueOf calls.
e.g. in Tuple.java :
{code}
return
epicDaysToMillis(Integer.valueOf(row.getValue(partitionCol).toString()));
{code}
> Boxing/unboxing to parse a primitive is suboptimal
> --
>
> Key: KYLIN-3586
> URL: https://issues.apache.org/jira/browse/KYLIN-3586
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Lijun Cao
>Priority: Major
> Fix For: v2.6.0
>
>
> An example is from HBaseLookupRowEncoder:
> {code}
> int valIdx = Integer.valueOf(Bytes.toString(qualifier));
> {code}
> valueOf returns an Integer object which would be unboxed and assigned to
> valIdx.
> Integer.parseInt() should be used instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-2517) Upgrade hbase dependency to 1.4.7
[ https://issues.apache.org/jira/browse/KYLIN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-2517: -- Description: There have been major enhancements / bug fixes since the hbase 1.1.1 release. This issue is to upgrade to 1.4.7 release. was: There have been major enhancements / bug fixes since the hbase 1.1.1 release. This issue is to upgrade to 1.4.6 release. > Upgrade hbase dependency to 1.4.7 > - > > Key: KYLIN-2517 > URL: https://issues.apache.org/jira/browse/KYLIN-2517 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > > There have been major enhancements / bug fixes since the hbase 1.1.1 release. > This issue is to upgrade to 1.4.7 release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3046) Consider introducing log4j-extras
[ https://issues.apache.org/jira/browse/KYLIN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3046: -- Description: log4j-extras allows log rotation as well as compression. https://logging.apache.org/log4j/extras/download.html We should consider using log4j-extras. was: log4j-extras allows log rotation as well as compression. https://logging.apache.org/log4j/extras/download.html We should consider using log4j-extras. > Consider introducing log4j-extras > -- > > Key: KYLIN-3046 > URL: https://issues.apache.org/jira/browse/KYLIN-3046 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > Labels: log > Fix For: Backlog > > > log4j-extras allows log rotation as well as compression. > https://logging.apache.org/log4j/extras/download.html > We should consider using log4j-extras. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2517) Upgrade hbase dependency to 1.4.7
[ https://issues.apache.org/jira/browse/KYLIN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-2517: -- Summary: Upgrade hbase dependency to 1.4.7 (was: Upgrade hbase dependency to 1.4.6) > Upgrade hbase dependency to 1.4.7 > - > > Key: KYLIN-2517 > URL: https://issues.apache.org/jira/browse/KYLIN-2517 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > > There have been major enhancements / bug fixes since the hbase 1.1.1 release. > This issue is to upgrade to 1.4.6 release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3556) Interned string should not be used as lock object
[
https://issues.apache.org/jira/browse/KYLIN-3556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated KYLIN-3556:
--
Description:
In JDBCResourceDAO :
{code}
public void execute(Connection connection) throws SQLException {
synchronized (resPath.intern()) {
{code}
Locking on an interned string can cause unexpected locking collisions with
other part of code.
was:
In JDBCResourceDAO :
{code}
public void execute(Connection connection) throws SQLException {
synchronized (resPath.intern()) {
{code}
Locking on an interned string can cause unexpected locking collisions with
other part of code.
> Interned string should not be used as lock object
> -
>
> Key: KYLIN-3556
> URL: https://issues.apache.org/jira/browse/KYLIN-3556
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.5.0
>Reporter: Ted Yu
>Assignee: Kaige Liu
>Priority: Minor
> Fix For: v2.5.1
>
>
> In JDBCResourceDAO :
> {code}
> public void execute(Connection connection) throws SQLException {
> synchronized (resPath.intern()) {
> {code}
> Locking on an interned string can cause unexpected locking collisions with
> other part of code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3290) Avoid calling Class#newInstance
[ https://issues.apache.org/jira/browse/KYLIN-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3290: -- Description: Class#newInstance is deprecated starting in Java 9 - https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw undeclared checked exceptions. The suggested replacement is getDeclaredConstructor().newInstance(), which wraps the checked exceptions in InvocationException. was: Class#newInstance is deprecated starting in Java 9 - https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw undeclared checked exceptions. The suggested replacement is getDeclaredConstructor().newInstance(), which wraps the checked exceptions in InvocationException. > Avoid calling Class#newInstance > --- > > Key: KYLIN-3290 > URL: https://issues.apache.org/jira/browse/KYLIN-3290 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Assignee: jiatao.tao >Priority: Minor > Labels: jdk > Fix For: v2.6.0 > > > Class#newInstance is deprecated starting in Java 9 - > https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw > undeclared checked exceptions. > The suggested replacement is getDeclaredConstructor().newInstance(), which > wraps the checked exceptions in InvocationException. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3417) Consider replacing ReentrantReadWriteLock with StampedLock
[
https://issues.apache.org/jira/browse/KYLIN-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624443#comment-16624443
]
Ted Yu edited comment on KYLIN-3417 at 10/1/18 4:24 PM:
For phase I, we don't need to use Optimistic read lock.
was (Author: yuzhih...@gmail.com):
For phase I, we don't need to use Optimistic read lock
> Consider replacing ReentrantReadWriteLock with StampedLock
> --
>
> Key: KYLIN-3417
> URL: https://issues.apache.org/jira/browse/KYLIN-3417
> Project: Kylin
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
> Fix For: Backlog
>
>
> ReentrantReadWriteLock's are only the right solution when there is long hold
> time due to expensive I/O.
> It is expensive for readers.
> We should see if the lighter {{StampedLock}} can be used instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-2650) Update to Apache Calcite Avatica 1.17.0
[ https://issues.apache.org/jira/browse/KYLIN-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-2650: -- Summary: Update to Apache Calcite Avatica 1.17.0 (was: Update to Apache Calcite Avatica 1.12.0) > Update to Apache Calcite Avatica 1.17.0 > --- > > Key: KYLIN-2650 > URL: https://issues.apache.org/jira/browse/KYLIN-2650 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Minor > > Apache Calcite Avatica 1.17.0 was released mid-July > https://sematext.com/opensee/m/Calcite/FR3K9IxYty1a4ECo1?subj=+ANNOUNCE+Apache+Calcite+Avatica+1+12+0+released > This issue upgrades Avatica dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2650) Update to Apache Calcite Avatica 1.12.0
[ https://issues.apache.org/jira/browse/KYLIN-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-2650: -- Description: Apache Calcite Avatica 1.17.0 was released mid-July https://sematext.com/opensee/m/Calcite/FR3K9IxYty1a4ECo1?subj=+ANNOUNCE+Apache+Calcite+Avatica+1+12+0+released This issue upgrades Avatica dependency. was: Apache Calcite Avatica 1.12.0 has just been released. https://sematext.com/opensee/m/Calcite/FR3K9IxYty1a4ECo1?subj=+ANNOUNCE+Apache+Calcite+Avatica+1+12+0+released This issue upgrades Avatica dependency. > Update to Apache Calcite Avatica 1.12.0 > --- > > Key: KYLIN-2650 > URL: https://issues.apache.org/jira/browse/KYLIN-2650 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Minor > > Apache Calcite Avatica 1.17.0 was released mid-July > https://sematext.com/opensee/m/Calcite/FR3K9IxYty1a4ECo1?subj=+ANNOUNCE+Apache+Calcite+Avatica+1+12+0+released > This issue upgrades Avatica dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3310) Use lint for maven-compiler-plugin
[
https://issues.apache.org/jira/browse/KYLIN-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560940#comment-16560940
]
Ted Yu edited comment on KYLIN-3310 at 10/1/18 1:40 AM:
Thanks, Jiatao.
was (Author: yuzhih...@gmail.com):
Thanks, Jiatao .
> Use lint for maven-compiler-plugin
> --
>
> Key: KYLIN-3310
> URL: https://issues.apache.org/jira/browse/KYLIN-3310
> Project: Kylin
> Issue Type: Improvement
> Components: Tools, Build and Test
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
> Fix For: v2.6.0
>
>
> lint helps identify structural problems.
> We should enable lint for maven-compiler-plugin
> {code}
> maven-compiler-plugin
> ${maven-compiler-plugin.version}
>
> 1.8
> 1.8
>
> -Xlint:all
> ${compiler.error.flag}
>
> -Xlint:-options
>
> -Xlint:-cast
> -Xlint:-deprecation
> -Xlint:-processing
> -Xlint:-rawtypes
> -Xlint:-serial
> -Xlint:-try
> -Xlint:-unchecked
> -Xlint:-varargs
>
>
>
>
> true
>
> false
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3394) Prepare for Kafka 2.0
[
https://issues.apache.org/jira/browse/KYLIN-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated KYLIN-3394:
--
Description:
Kafka 2.0 has been released.
I got the following when compiling against Kafka 2.0.0:
{code}
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:266:
error: type mismatch;
[ERROR] found : Option[org.apache.kafka.common.Node]
[ERROR] required: org.apache.kafka.common.Node
[ERROR] getBrokerInfoFromCache(zkUtils, cachedBrokerInfo,
List(l)).head.getNode(listenerName)
[ERROR]
^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:273:
error: type mismatch;
[ERROR] found : Seq[Option[org.apache.kafka.common.Node]]
[ERROR] required: Seq[org.apache.kafka.common.Node]
[ERROR] replicaInfo = getBrokerInfoFromCache(zkUtils,
cachedBrokerInfo, replicas).map(_.getNode(listenerName))
[ERROR]
^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:274:
error: type mismatch;
[ERROR] found : Seq[Option[org.apache.kafka.common.Node]]
[ERROR] required: Seq[org.apache.kafka.common.Node]
[ERROR] isrInfo = getBrokerInfoFromCache(zkUtils, cachedBrokerInfo,
inSyncReplicas).map(_.getNode(listenerName))
[ERROR]
^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:476:
error: value getConsumersInGroup is not a member of kafka.utils.ZkUtils
[ERROR] zkUtils.getConsumersInGroup(group).nonEmpty
[ERROR] ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:489:
error: not found: type ZKGroupDirs
[ERROR] val dir = new ZKGroupDirs(group)
[ERROR] ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:507:
error: value getTopicsByConsumerGroup is not a member of kafka.utils.ZkUtils
[ERROR] val topics = zkUtils.getTopicsByConsumerGroup(group)
[ERROR] ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:512:
error: not found: type ZKGroupTopicDirs
[ERROR] val dir = new ZKGroupTopicDirs(group, topic)
[ERROR] ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:528:
error: value getAllConsumerGroupsForTopic is not a member of kafka.utils.ZkUtils
[ERROR] val groups = zkUtils.getAllConsumerGroupsForTopic(topic)
[ERROR] ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:720:
error: value encode is not a member of object kafka.utils.Json
[ERROR] val content =
Json.encode(getConfigChangeZnodeData(sanitizedEntityPath))
[ERROR]^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:733:
error: value encode is not a member of object kafka.utils.Json
[ERROR] zkUtils.updatePersistentPath(entityPath, Json.encode(map))
{code}
was:
Kafka 2.0 is around the corner.
I got the following when compiling against Kafka 2.0.0:
{code}
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:266:
error: type mismatch;
[ERROR] found : Option[org.apache.kafka.common.Node]
[ERROR] required: org.apache.kafka.common.Node
[ERROR] getBrokerInfoFromCache(zkUtils, cachedBrokerInfo,
List(l)).head.getNode(listenerName)
[ERROR]
^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:273:
error: type mismatch;
[ERROR] found : Seq[Option[org.apache.kafka.common.Node]]
[ERROR] required: Seq[org.apache.kafka.common.Node]
[ERROR] replicaInfo = getBrokerInfoFromCache(zkUtils,
cachedBrokerInfo, replicas).map(_.getNode(listenerName))
[ERROR]
^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:274:
error: type mismatch;
[ERROR] found : Seq[Option[org.apache.kafka.common.Node]]
[ERROR] required: Seq[org.apache.kafka.common.Node]
[ERROR] isrInfo = getBrokerInfoFromCache(zkUtils, cachedBrokerInfo,
inSyncReplicas).map(_.getNode(listenerName))
[ERROR]
^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:476:
error: value getConsumersInGroup is not a member of kafka.utils.ZkUtils
[ERROR] zkUtils.getConsumersInGroup(group).nonEmpty
[ERROR] ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:489:
error: not found: type ZKGroupDirs
[ERROR] val dir = new ZKGroupDirs(group)
[ERROR] ^
[jira] [Updated] (KYLIN-3484) Update Hadoop version to 2.7.7
[ https://issues.apache.org/jira/browse/KYLIN-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3484: -- Description: We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and security fixes. (was: We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and security fixes .) > Update Hadoop version to 2.7.7 > -- > > Key: KYLIN-3484 > URL: https://issues.apache.org/jira/browse/KYLIN-3484 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Priority: Minor > > We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and > security fixes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3450) Consider using google re2j
[ https://issues.apache.org/jira/browse/KYLIN-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556761#comment-16556761 ] Ted Yu edited comment on KYLIN-3450 at 10/1/18 1:39 AM: It would be nice to see improvement from using re2j. was (Author: yuzhih...@gmail.com): It would be nice to see improvement from using re2j . > Consider using google re2j > -- > > Key: KYLIN-3450 > URL: https://issues.apache.org/jira/browse/KYLIN-3450 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Minor > > RE2J : https://github.com/google/re2j > For regular expression patterns with a high degree of alternation, using RE2J > would exhibit higher performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3561) Upgrade to JUnit 5
[ https://issues.apache.org/jira/browse/KYLIN-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3561: -- Description: JUnit 5 brings multiple useful features so tests are easier to read and write. We can bump up the dependency version and create new tests with JUnit 5 features. Relevant features of JUnit 5: dynamic test, nested tests, parameterized tests https://twitter.com/nipafx/status/1027095088059559936 was: JUnit 5 brings multiple useful features so tests are easier to read and write. We can bump up the dependency version and create new tests with JUnit 5 features. Relevant features of JUnit 5: dynamic test, nested tests, parameterized tests https://twitter.com/nipafx/status/1027095088059559936 > Upgrade to JUnit 5 > -- > > Key: KYLIN-3561 > URL: https://issues.apache.org/jira/browse/KYLIN-3561 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Priority: Major > > JUnit 5 brings multiple useful features so tests are easier to read and write. > We can bump up the dependency version and create new tests with JUnit 5 > features. > Relevant features of JUnit 5: dynamic test, nested tests, parameterized tests > https://twitter.com/nipafx/status/1027095088059559936 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2690) Upgrade calcite to release 1.18
[ https://issues.apache.org/jira/browse/KYLIN-2690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-2690: -- Summary: Upgrade calcite to release 1.18 (was: Upgrade calcite to release 1.17) > Upgrade calcite to release 1.18 > --- > > Key: KYLIN-2690 > URL: https://issues.apache.org/jira/browse/KYLIN-2690 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > > Calcite 1.16 has been released. > We should consider upgrading to this release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2690) Upgrade calcite to release 1.18
[ https://issues.apache.org/jira/browse/KYLIN-2690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-2690: -- Description: Calcite 1.18 has been released. We should consider upgrading to this release. was: Calcite 1.16 has been released. We should consider upgrading to this release. > Upgrade calcite to release 1.18 > --- > > Key: KYLIN-2690 > URL: https://issues.apache.org/jira/browse/KYLIN-2690 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > > Calcite 1.18 has been released. > We should consider upgrading to this release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3272) Upgrade Spark dependency to 2.3.2
[ https://issues.apache.org/jira/browse/KYLIN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3272: -- Description: Currently Spark 2.1.2 is used. Spark 2.3.2 was released. We should upgrade the dependency to 2.3.2 https://sematext.com/opensee/m/Spark/q3RTtb4Q9LI4F6F?subj=+ANNOUNCE+Announcing+Apache+Spark+2+3+2 was: Currently Spark 2.1.2 is used. Spark 2.3.1 was released. We should upgrade the dependency to 2.3.1 > Upgrade Spark dependency to 2.3.2 > - > > Key: KYLIN-3272 > URL: https://issues.apache.org/jira/browse/KYLIN-3272 > Project: Kylin > Issue Type: Improvement > Components: Spark Engine >Reporter: Ted Yu >Priority: Minor > > Currently Spark 2.1.2 is used. > Spark 2.3.2 was released. > We should upgrade the dependency to 2.3.2 > https://sematext.com/opensee/m/Spark/q3RTtb4Q9LI4F6F?subj=+ANNOUNCE+Announcing+Apache+Spark+2+3+2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3519) Upgrade Jacoco version to 0.8.2
[ https://issues.apache.org/jira/browse/KYLIN-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3519: -- Description: Jacoco 0.8.2 adds Java 11 support: https://github.com/jacoco/jacoco/releases/tag/v0.8.2 Java 11 RC1 is out. We should consider upgrading Jacoco. was: Jacoco 0.8.2 adds Java 11 support: https://github.com/jacoco/jacoco/releases/tag/v0.8.2 Java 11 RC1 is out. We should consider upgrading Jacoco. > Upgrade Jacoco version to 0.8.2 > --- > > Key: KYLIN-3519 > URL: https://issues.apache.org/jira/browse/KYLIN-3519 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Minor > > Jacoco 0.8.2 adds Java 11 support: >https://github.com/jacoco/jacoco/releases/tag/v0.8.2 > Java 11 RC1 is out. > We should consider upgrading Jacoco. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3572) Upgrade commons-dbcp to latest commons-dbcp2
[ https://issues.apache.org/jira/browse/KYLIN-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3572: -- Description: There have been many improvements and a major release in commons-dbcp. The latest version is 2.5.0. This Jira proposes to upgrade the version of commons-dbcp to commons-dbcp2 version 2.5.0 in order to benefit from the improvements/additions. This will involve some code changes in addition to the updated dependencies, as the API has changed slightly to use different terminology was: There have been many improvements and a major release in commons-dbcp. The latest version is 2.5.0. This Jira proposes to upgrade the version of commons-dbcp to commons-dbcp2 version 2.5.0 in order to benefit from the improvements/additions. This will involve some code changes in addition to the updated dependencies, as the API has changed slightly to use different terminology > Upgrade commons-dbcp to latest commons-dbcp2 > > > Key: KYLIN-3572 > URL: https://issues.apache.org/jira/browse/KYLIN-3572 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Priority: Major > > There have been many improvements and a major release in commons-dbcp. The > latest version is 2.5.0. > This Jira proposes to upgrade the version of commons-dbcp to commons-dbcp2 > version 2.5.0 in order to benefit from the improvements/additions. This will > involve some code changes in addition to the updated dependencies, as the API > has changed slightly to use different terminology -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3573) Upgrade Apache parent POM to version 21
[ https://issues.apache.org/jira/browse/KYLIN-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3573: -- Description: Apache parent POM 21 has just been released. It generates .sha512 checksums during release with Maven Release Plugin. We should consider upgrading parent POM. was: Apache parent POM 21 has just been released. It generates .sha512 checksums during release with Maven Release Plugin. We should consider upgrading parent POM. > Upgrade Apache parent POM to version 21 > --- > > Key: KYLIN-3573 > URL: https://issues.apache.org/jira/browse/KYLIN-3573 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Priority: Major > > Apache parent POM 21 has just been released. It generates .sha512 checksums > during release with Maven Release Plugin. > We should consider upgrading parent POM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3523) Upgrade hadoop and hbase dependencies in master-hadoop3.1 branch
[ https://issues.apache.org/jira/browse/KYLIN-3523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3523: -- Description: Hadoop 3.1.1 has been released as stable. hbase 2.0.2 vote has passed. We should upgrade to these two versions in master-hadoop3.1 branch. was: Hadoop 3.1.1 has been released as stable. hbase 2.0.2 vote seems to pass. We should upgrade to these two versions in master-hadoop3.1 branch. > Upgrade hadoop and hbase dependencies in master-hadoop3.1 branch > > > Key: KYLIN-3523 > URL: https://issues.apache.org/jira/browse/KYLIN-3523 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > > Hadoop 3.1.1 has been released as stable. > hbase 2.0.2 vote has passed. > We should upgrade to these two versions in master-hadoop3.1 branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3584) Incorrect lazy initialization of DefaultScheduler.INSTANCE
[
https://issues.apache.org/jira/browse/KYLIN-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630845#comment-16630845
]
Ted Yu commented on KYLIN-3584:
---
DefaultScheduler#init is not involved in {{createInstance}}.
What happens when an un-initialized INSTANCE is handed to the caller of
getInstance ?
> Incorrect lazy initialization of DefaultScheduler.INSTANCE
> --
>
> Key: KYLIN-3584
> URL: https://issues.apache.org/jira/browse/KYLIN-3584
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> Here is related code:
> {code}
> public static DefaultScheduler getInstance() {
> if (INSTANCE == null) {
> INSTANCE = createInstance();
> {code}
> The check against may not be effective.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3586) Boxing/unboxing to parse a primitive is suboptimal
[
https://issues.apache.org/jira/browse/KYLIN-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630670#comment-16630670
]
Ted Yu commented on KYLIN-3586:
---
There are still several {{Integer.valueOf}} calls in the code base.
Can you address them as well ?
Also, the commit log didn't contain JIRA number.
> Boxing/unboxing to parse a primitive is suboptimal
> --
>
> Key: KYLIN-3586
> URL: https://issues.apache.org/jira/browse/KYLIN-3586
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Lijun Cao
>Priority: Major
> Fix For: v2.6.0
>
>
> An example is from HBaseLookupRowEncoder:
> {code}
> int valIdx = Integer.valueOf(Bytes.toString(qualifier));
> {code}
> valueOf returns an Integer object which would be unboxed and assigned to
> valIdx.
> Integer.parseInt() should be used instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3598) Use the "guava beta checker" plugin to stay away from @Beta API
[ https://issues.apache.org/jira/browse/KYLIN-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3598: -- Summary: Use the "guava beta checker" plugin to stay away from @Beta API (was: Use the "guava beta checker" plugin to keep off of @Beta stuff) > Use the "guava beta checker" plugin to stay away from @Beta API > --- > > Key: KYLIN-3598 > URL: https://issues.apache.org/jira/browse/KYLIN-3598 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Priority: Major > > The Guava people publish an Error Prone plugin to detect when stuff that's > annotated with @Beta gets used. Those things shouldn't be used because the > project gives no promises about deprecating before removal. > plugin: > https://github.com/google/guava-beta-checker -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3598) Use the "guava beta checker" plugin to keep off of @Beta stuff
Ted Yu created KYLIN-3598: - Summary: Use the "guava beta checker" plugin to keep off of @Beta stuff Key: KYLIN-3598 URL: https://issues.apache.org/jira/browse/KYLIN-3598 Project: Kylin Issue Type: Task Reporter: Ted Yu The Guava people publish an Error Prone plugin to detect when stuff that's annotated with @Beta gets used. Those things shouldn't be used because the project gives no promises about deprecating before removal. plugin: https://github.com/google/guava-beta-checker -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3590) Missing argument to exception message String in ModelController
[
https://issues.apache.org/jira/browse/KYLIN-3590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16626212#comment-16626212
]
Ted Yu commented on KYLIN-3590:
---
There is similar issue in ProjectController#saveProject
> Missing argument to exception message String in ModelController
> ---
>
> Key: KYLIN-3590
> URL: https://issues.apache.org/jira/browse/KYLIN-3590
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Major
>
> Here is related code:
> {code}
> String.format(Locale.ROOT,
> "Invalid model name %s, only letters, numbers and
> underscore " + "supported."),
> {code}
> There is no argument for '%s'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3590) Missing argument to exception message String in ModelController
Ted Yu created KYLIN-3590:
-
Summary: Missing argument to exception message String in
ModelController
Key: KYLIN-3590
URL: https://issues.apache.org/jira/browse/KYLIN-3590
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
String.format(Locale.ROOT,
"Invalid model name %s, only letters, numbers and
underscore " + "supported."),
{code}
There is no argument for '%s'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3589) Different type is used to check presence in Collection in AclPermissionUtil
Ted Yu created KYLIN-3589:
-
Summary: Different type is used to check presence in Collection in
AclPermissionUtil
Key: KYLIN-3589
URL: https://issues.apache.org/jira/browse/KYLIN-3589
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
public static List transformAuthorities(Collection authorities) {
List ret = new ArrayList();
for (GrantedAuthority auth : authorities) {
if (!authorities.contains(auth.getAuthority())) {
{code}
auth.getAuthority() shouldn't be used to check presence in the Collection since
it is not of GrantedAuthority type.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3588) Potentially duplicate put to RemoveBlackoutRealizationsRule#filters map
Ted Yu created KYLIN-3588:
-
Summary: Potentially duplicate put to
RemoveBlackoutRealizationsRule#filters map
Key: KYLIN-3588
URL: https://issues.apache.org/jira/browse/KYLIN-3588
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
IRealizationFilter filter = filters.get(conf);
if (filter == null) {
try {
Class clz =
ClassUtil.forName(conf.getQueryRealizationFilter(), IRealizationFilter.class);
filter =
clz.getConstructor(KylinConfig.class).newInstance(conf);
} catch (Exception e) {
throw new RuntimeException(e);
}
filters.put(conf, filter);
{code}
Two threads may find out there is no filter for conf, they would proceed with
creation of IRealizationFilter instances and try to put into the map.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3587) Null value is used in IllegalStateException message
Ted Yu created KYLIN-3587:
-
Summary: Null value is used in IllegalStateException message
Key: KYLIN-3587
URL: https://issues.apache.org/jira/browse/KYLIN-3587
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
In ProjectManager :
{code}
TableDesc table = metaMgr.getTableDesc(tableId, projectName);
if (table == null) {
throw new IllegalStateException("Cannot find table '" +
table + "' in metadata manager");
{code}
Since table is null, the exception message is not helpful.
It seems including tableId in the message is better.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3561) Upgrade to JUnit 5
[ https://issues.apache.org/jira/browse/KYLIN-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3561: -- Description: JUnit 5 brings multiple useful features so tests are easier to read and write. We can bump up the dependency version and create new tests with JUnit 5 features. Relevant features of JUnit 5: dynamic test, nested tests, parameterized tests https://twitter.com/nipafx/status/1027095088059559936 was: JUnit 5 brings multiple useful features so tests are easier to read and write. We can bump up the dependency version and create new tests with JUnit 5 features. Relevant features of JUnit 5: dynamic test, nested tests, parameterized tests https://twitter.com/nipafx/status/1027095088059559936 > Upgrade to JUnit 5 > -- > > Key: KYLIN-3561 > URL: https://issues.apache.org/jira/browse/KYLIN-3561 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Priority: Major > > JUnit 5 brings multiple useful features so tests are easier to read and write. > We can bump up the dependency version and create new tests with JUnit 5 > features. > Relevant features of JUnit 5: dynamic test, nested tests, parameterized tests > https://twitter.com/nipafx/status/1027095088059559936 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3586) Boxing/unboxing to parse a primitive is suboptimal
Ted Yu created KYLIN-3586:
-
Summary: Boxing/unboxing to parse a primitive is suboptimal
Key: KYLIN-3586
URL: https://issues.apache.org/jira/browse/KYLIN-3586
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
An example is from HBaseLookupRowEncoder:
{code}
int valIdx = Integer.valueOf(Bytes.toString(qualifier));
{code}
valueOf returns an Integer object which would be unboxed and assigned to valIdx.
Integer.parseInt() should be used instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3585) Ineffective declaration of volatile field in SparkFactDistinct
Ted Yu created KYLIN-3585:
-
Summary: Ineffective declaration of volatile field in
SparkFactDistinct
Key: KYLIN-3585
URL: https://issues.apache.org/jira/browse/KYLIN-3585
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
In CuboidStatCalculator :
{code}
private volatile HLLCounter[] cuboidsHLL;
{code}
Though the array is declared volatile, the array elements are not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3584) Incorrect lazy initialization of DefaultScheduler.INSTANCE
Ted Yu created KYLIN-3584:
-
Summary: Incorrect lazy initialization of DefaultScheduler.INSTANCE
Key: KYLIN-3584
URL: https://issues.apache.org/jira/browse/KYLIN-3584
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
public static DefaultScheduler getInstance() {
if (INSTANCE == null) {
INSTANCE = createInstance();
{code}
The check against may not be effective.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3583) Integer multiplication is performed where Long result is expected
Ted Yu created KYLIN-3583:
-
Summary: Integer multiplication is performed where Long result is
expected
Key: KYLIN-3583
URL: https://issues.apache.org/jira/browse/KYLIN-3583
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
In RecordEventTimeDetail :
{code}
long timeStampForWeekBegin = timeStamp;
timeStampForWeekBegin -= 360 * 24 *
(calendar.get(Calendar.DAY_OF_WEEK) - 1);
{code}
Casting to long should be done first before multiplication.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3582) String comparison should not use == in RecordEvent
Ted Yu created KYLIN-3582:
-
Summary: String comparison should not use == in RecordEvent
Key: KYLIN-3582
URL: https://issues.apache.org/jira/browse/KYLIN-3582
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
for (RecordReserveKeyEnum reserveKey :
RecordReserveKeyEnum.values()) {
if (reserveKey.reserveKey == key) {
{code}
equals() should be used to compare Strings.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3581) compareTo method should be paired with equals method
Ted Yu created KYLIN-3581: - Summary: compareTo method should be paired with equals method Key: KYLIN-3581 URL: https://issues.apache.org/jira/browse/KYLIN-3581 Project: Kylin Issue Type: Bug Reporter: Ted Yu DoggedCubeBuilder$MergeSlot and InMemCubeBuilder$CuboidTask define compareTo method. They should define equals method as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3578) Do not synchronize on the intrinsic locks of high-level concurrency objects
Ted Yu created KYLIN-3578:
-
Summary: Do not synchronize on the intrinsic locks of high-level
concurrency objects
Key: KYLIN-3578
URL: https://issues.apache.org/jira/browse/KYLIN-3578
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
>From
>core-common/src/main/java/org/apache/kylin/common/util/MemoryBudgetController.java
> :
{code}
private final ReentrantLock lock = new ReentrantLock();
...
synchronized (lock) {
{code}
See the following for why such practice is to be avoided:
https://wiki.sei.cmu.edu/confluence/display/java/LCK03-J.+Do+not+synchronize+on+the+intrinsic+locks+of+high-level+concurrency+objects
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3431) Avoid FileInputStream/FileOutputStream
[
https://issues.apache.org/jira/browse/KYLIN-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624744#comment-16624744
]
Ted Yu commented on KYLIN-3431:
---
Thanks, Long.
> Avoid FileInputStream/FileOutputStream
> --
>
> Key: KYLIN-3431
> URL: https://issues.apache.org/jira/browse/KYLIN-3431
> Project: Kylin
> Issue Type: Improvement
>Reporter: Ted Yu
>Priority: Major
> Fix For: Backlog
>
>
> They rely on finalizers (before Java 11), which create unnecessary GC load.
> The alternatives, {{Files.newInputStream}}, are as easy to use and don't have
> this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3417) Consider replacing ReentrantReadWriteLock with StampedLock
[
https://issues.apache.org/jira/browse/KYLIN-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624443#comment-16624443
]
Ted Yu commented on KYLIN-3417:
---
For phase I, we don't need to use Optimistic read lock
> Consider replacing ReentrantReadWriteLock with StampedLock
> --
>
> Key: KYLIN-3417
> URL: https://issues.apache.org/jira/browse/KYLIN-3417
> Project: Kylin
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
> Fix For: Backlog
>
>
> ReentrantReadWriteLock's are only the right solution when there is long hold
> time due to expensive I/O.
> It is expensive for readers.
> We should see if the lighter {{StampedLock}} can be used instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3447) Upgrade zookeeper to 3.4.13
[ https://issues.apache.org/jira/browse/KYLIN-3447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3447: -- Description: zookeeper 3.4.13 is being released with the following fixes: ZOOKEEPER-2959 fixes data loss when observer is used ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container / cloud) environment was: zookeeper 3.4.13 is being released with the following fixes: ZOOKEEPER-2959 fixes data loss when observer is used ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container / cloud) environment > Upgrade zookeeper to 3.4.13 > --- > > Key: KYLIN-3447 > URL: https://issues.apache.org/jira/browse/KYLIN-3447 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > > zookeeper 3.4.13 is being released with the following fixes: > ZOOKEEPER-2959 fixes data loss when observer is used > ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container > / cloud) > environment -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3484) Update Hadoop version to 2.7.7
[ https://issues.apache.org/jira/browse/KYLIN-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3484: -- Description: We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and security fixes . (was: We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and security fixes.) > Update Hadoop version to 2.7.7 > -- > > Key: KYLIN-3484 > URL: https://issues.apache.org/jira/browse/KYLIN-3484 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Priority: Minor > > We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and > security fixes . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3519) Upgrade Jacoco version to 0.8.2
[ https://issues.apache.org/jira/browse/KYLIN-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3519: -- Description: Jacoco 0.8.2 adds Java 11 support: https://github.com/jacoco/jacoco/releases/tag/v0.8.2 Java 11 RC1 is out. We should consider upgrading Jacoco. was: Jacoco 0.8.2 adds Java 11 support: https://github.com/jacoco/jacoco/releases/tag/v0.8.2 Java 11 RC1 is out. We should consider upgrading Jacoco. > Upgrade Jacoco version to 0.8.2 > --- > > Key: KYLIN-3519 > URL: https://issues.apache.org/jira/browse/KYLIN-3519 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Minor > > Jacoco 0.8.2 adds Java 11 support: >https://github.com/jacoco/jacoco/releases/tag/v0.8.2 > Java 11 RC1 is out. > We should consider upgrading Jacoco. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3559) Use Splitter for splitting String
[ https://issues.apache.org/jira/browse/KYLIN-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3559: -- Description: See http://errorprone.info/bugpattern/StringSplitter for why Splitter is preferred . (was: See http://errorprone.info/bugpattern/StringSplitter for why Splitter is preferred.) > Use Splitter for splitting String > - > > Key: KYLIN-3559 > URL: https://issues.apache.org/jira/browse/KYLIN-3559 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Priority: Major > Fix For: v2.6.0 > > > See http://errorprone.info/bugpattern/StringSplitter for why Splitter is > preferred . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3575) Unclosed Connection in DriverTest
Ted Yu created KYLIN-3575:
-
Summary: Unclosed Connection in DriverTest
Key: KYLIN-3575
URL: https://issues.apache.org/jira/browse/KYLIN-3575
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
In testCalciteProps :
{code}
KylinConnection conn = (KylinConnection)
driver.connect("jdbc:kylin:test_url/test_db", props);
{code}
The connection should be closed upon return.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3574) Unclosed lookup table in DictionaryGeneratorCLI#processSegment
Ted Yu created KYLIN-3574:
-
Summary: Unclosed lookup table in
DictionaryGeneratorCLI#processSegment
Key: KYLIN-3574
URL: https://issues.apache.org/jira/browse/KYLIN-3574
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
JoinDesc join =
cubeSeg.getModel().getJoinsTree().getJoinByPKSide(lookup);
cubeMgr.getLookupTable(cubeSeg, join);
{code}
The returned lookup table should be closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3573) Upgrade Apache parent POM to version 21
Ted Yu created KYLIN-3573: - Summary: Upgrade Apache parent POM to version 21 Key: KYLIN-3573 URL: https://issues.apache.org/jira/browse/KYLIN-3573 Project: Kylin Issue Type: Task Reporter: Ted Yu Apache parent POM 21 has just been released. It generates .sha512 checksums during release with Maven Release Plugin. We should consider upgrading parent POM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3572) Upgrade commons-dbcp to latest commons-dbcp2
Ted Yu created KYLIN-3572: - Summary: Upgrade commons-dbcp to latest commons-dbcp2 Key: KYLIN-3572 URL: https://issues.apache.org/jira/browse/KYLIN-3572 Project: Kylin Issue Type: Task Reporter: Ted Yu There have been many improvements and a major release in commons-dbcp. The latest version is 2.5.0. This Jira proposes to upgrade the version of commons-dbcp to commons-dbcp2 version 2.5.0 in order to benefit from the improvements/additions. This will involve some code changes in addition to the updated dependencies, as the API has changed slightly to use different terminology -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2650) Update to Apache Calcite Avatica 1.12.0
[ https://issues.apache.org/jira/browse/KYLIN-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-2650: -- Summary: Update to Apache Calcite Avatica 1.12.0 (was: Update to Apache Calcite Avatica 1.10.0) > Update to Apache Calcite Avatica 1.12.0 > --- > > Key: KYLIN-2650 > URL: https://issues.apache.org/jira/browse/KYLIN-2650 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Minor > > Apache Calcite Avatica 1.10.0 has just been released. > This issue upgrades Avatica dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3095) Use ArrayDeque instead of LinkedList for queue implementation
[
https://issues.apache.org/jira/browse/KYLIN-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated KYLIN-3095:
--
Description:
Use ArrayDeque instead of LinkedList for queue implementation where thread
safety is not needed.
>From https://docs.oracle.com/javase/7/docs/api/java/util/ArrayDeque.html
{quote}
Resizable-array implementation of the Deque interface. Array deques have no
capacity restrictions; they grow as necessary to support usage. They are not
thread-safe; in the absence of external synchronization, they do not support
concurrent access by multiple threads. Null elements are prohibited. This class
is likely to be faster than Stack when used as a stack, and *faster than
LinkedList when used as a queue.*
{quote}
was:
Use ArrayDeque instead of LinkedList for queue implementation where thread
safety is not needed.
>From https://docs.oracle.com/javase/7/docs/api/java/util/ArrayDeque.html
{quote}
Resizable-array implementation of the Deque interface. Array deques have no
capacity restrictions; they grow as necessary to support usage. They are not
thread-safe; in the absence of external synchronization, they do not support
concurrent access by multiple threads. Null elements are prohibited. This class
is likely to be faster than Stack when used as a stack, and *faster than
LinkedList when used as a queue.*
{quote}
> Use ArrayDeque instead of LinkedList for queue implementation
> -
>
> Key: KYLIN-3095
> URL: https://issues.apache.org/jira/browse/KYLIN-3095
> Project: Kylin
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Kaige Liu
>Priority: Minor
> Labels: parallel
> Fix For: Backlog
>
>
> Use ArrayDeque instead of LinkedList for queue implementation where thread
> safety is not needed.
> From https://docs.oracle.com/javase/7/docs/api/java/util/ArrayDeque.html
> {quote}
> Resizable-array implementation of the Deque interface. Array deques have no
> capacity restrictions; they grow as necessary to support usage. They are not
> thread-safe; in the absence of external synchronization, they do not support
> concurrent access by multiple threads. Null elements are prohibited. This
> class is likely to be faster than Stack when used as a stack, and *faster
> than LinkedList when used as a queue.*
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3272) Upgrade Spark dependency to 2.3.1
[ https://issues.apache.org/jira/browse/KYLIN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3272: -- Description: Currently Spark 2.1.2 is used. Spark 2.3.1 was released. We should upgrade the dependency to 2.3.1 was: Currently Spark 2.1.2 is used. Spark 2.3.0 was just released. We should upgrade the dependency to 2.3.0 > Upgrade Spark dependency to 2.3.1 > - > > Key: KYLIN-3272 > URL: https://issues.apache.org/jira/browse/KYLIN-3272 > Project: Kylin > Issue Type: Improvement > Components: Spark Engine >Reporter: Ted Yu >Priority: Minor > > Currently Spark 2.1.2 is used. > Spark 2.3.1 was released. > We should upgrade the dependency to 2.3.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3334) Prepare for Java 10
[
https://issues.apache.org/jira/browse/KYLIN-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473200#comment-16473200
]
Ted Yu edited comment on KYLIN-3334 at 9/18/18 4:17 PM:
Compiling against jdk 11 gives similar error.
was (Author: yuzhih...@gmail.com):
Compiling against jdk 11 gives similar error .
> Prepare for Java 10
> ---
>
> Key: KYLIN-3334
> URL: https://issues.apache.org/jira/browse/KYLIN-3334
> Project: Kylin
> Issue Type: Task
>Reporter: Ted Yu
>Priority: Major
>
> When compiling with Java 10 , MapReduce Engine module fails with
> {code}
> [ERROR] Failed to execute goal on project kylin-engine-mr: Could not resolve
> dependencies for project org.apache.kylin:kylin-engine-mr:jar:2.4.0-SNAPSHOT:
> Could not find artifact jdk.tools:jdk.tools:jar:1.7 at specified path
> /a/jdk-10/../lib/tools.jar -> [Help 1]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3484) Update Hadoop version to 2.7.7
[ https://issues.apache.org/jira/browse/KYLIN-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3484: -- Description: We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and security fixes. (was: We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and security fixes .) > Update Hadoop version to 2.7.7 > -- > > Key: KYLIN-3484 > URL: https://issues.apache.org/jira/browse/KYLIN-3484 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Priority: Minor > > We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and > security fixes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3543) Unclosed Job instance in CreateHTableJob#exportHBaseConfiguration
[
https://issues.apache.org/jira/browse/KYLIN-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16616840#comment-16616840
]
Ted Yu commented on KYLIN-3543:
---
Is it possible to do something for the building cube with MR case ?
> Unclosed Job instance in CreateHTableJob#exportHBaseConfiguration
> -
>
> Key: KYLIN-3543
> URL: https://issues.apache.org/jira/browse/KYLIN-3543
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> out = fs.create(new Path(hbaseConfPath));
> job.getConfiguration().writeXml(out);
> {code}
> The job instance should be closed upon return from the method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3561) Upgrade to JUnit 5
Ted Yu created KYLIN-3561: - Summary: Upgrade to JUnit 5 Key: KYLIN-3561 URL: https://issues.apache.org/jira/browse/KYLIN-3561 Project: Kylin Issue Type: Task Reporter: Ted Yu JUnit 5 brings multiple useful features so tests are easier to read and write. We can bump up the dependency version and create new tests with JUnit 5 features. Relevant features of JUnit 5: dynamic test, nested tests, parameterized tests https://twitter.com/nipafx/status/1027095088059559936 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3171) Support hadoop 3 release
[
https://issues.apache.org/jira/browse/KYLIN-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615047#comment-16615047
]
Ted Yu commented on KYLIN-3171:
---
How would master-hadoop3.1 branch be sync'ed with master branch ?
Currently it lags by few days' commits.
> Support hadoop 3 release
>
>
> Key: KYLIN-3171
> URL: https://issues.apache.org/jira/browse/KYLIN-3171
> Project: Kylin
> Issue Type: Improvement
>Reporter: Ted Yu
>Priority: Major
>
> When compiling against hadoop 3, I got:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.5.1:compile
> (default-compile) on project kylin-engine-mr: Compilation failure:
> Compilation failure:
> [ERROR]
> /a/kylin/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/DefaultSslProtocolSocketFactory.java:[29,36]
> error: package org.apache.commons.httpclient does not exist
> [ERROR]
> /a/kylin/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/DefaultSslProtocolSocketFactory.java:[30,36]
> error: package org.apache.commons.httpclient does not exist
> [ERROR]
> /a/kylin/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/DefaultSslProtocolSocketFactory.java:[31,43]
> error: package org.apache.commons.httpclient.params does not exist
> [ERROR]
> /a/kylin/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/DefaultSslProtocolSocketFactory.java:[32,45]
> error: package org.apache.commons.httpclient.protocol does not exist
> [ERROR]
> /a/kylin/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/DefaultSslProtocolSocketFactory.java:[33,45]
> error: package org.apache.commons.httpclient.protocol does not exist
> [ERROR]
> /a/kylin/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/DefaultSslProtocolSocketFactory.java:[41,56]
> error: cannot find symbol
> [ERROR] symbol: class SecureProtocolSocketFactory
> [ERROR]
> /a/kylin/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/DefaultSslProtocolSocketFactory.java:[94,125]
> error: cannot find symbol
> [ERROR] symbol: class HttpConnectionParams
> [ERROR] location: class DefaultSslProtocolSocketFactory
> [ERROR]
> /a/kylin/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/DefaultSslProtocolSocketFactory.java:[94,196]
> error: cannot find symbol
> [ERROR] symbol: class ConnectTimeoutException
> [ERROR] location: class DefaultSslProtocolSocketFactory
> [ERROR]
> /a/kylin/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/DefaultSslProtocolSocketFactory.java:[105,19]
> error: cannot find symbol
> [ERROR] symbol: variable ControllerThreadSocketFactory
> [ERROR] location: class DefaultSslProtocolSocketFactory
> {code}
> We should allow building against hadoop 3.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3560) Should not depend on personal repository
Ted Yu created KYLIN-3560:
-
Summary: Should not depend on personal repository
Key: KYLIN-3560
URL: https://issues.apache.org/jira/browse/KYLIN-3560
Project: Kylin
Issue Type: Task
Reporter: Ted Yu
In core-common/pom.xml:
{code}
com.github.joshelser
{code}
We shouldn't depend on personal repository.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3559) Use Splitter for splitting String
Ted Yu created KYLIN-3559: - Summary: Use Splitter for splitting String Key: KYLIN-3559 URL: https://issues.apache.org/jira/browse/KYLIN-3559 Project: Kylin Issue Type: Task Reporter: Ted Yu See http://errorprone.info/bugpattern/StringSplitter for why Splitter is preferred. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3310) Use lint for maven-compiler-plugin
[
https://issues.apache.org/jira/browse/KYLIN-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560940#comment-16560940
]
Ted Yu edited comment on KYLIN-3310 at 9/13/18 3:16 AM:
Thanks, Jiatao .
was (Author: yuzhih...@gmail.com):
Thanks, Jiatao.
> Use lint for maven-compiler-plugin
> --
>
> Key: KYLIN-3310
> URL: https://issues.apache.org/jira/browse/KYLIN-3310
> Project: Kylin
> Issue Type: Improvement
> Components: Tools, Build and Test
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
>
> lint helps identify structural problems.
> We should enable lint for maven-compiler-plugin
> {code}
> maven-compiler-plugin
> ${maven-compiler-plugin.version}
>
> 1.8
> 1.8
>
> -Xlint:all
> ${compiler.error.flag}
>
> -Xlint:-options
>
> -Xlint:-cast
> -Xlint:-deprecation
> -Xlint:-processing
> -Xlint:-rawtypes
> -Xlint:-serial
> -Xlint:-try
> -Xlint:-unchecked
> -Xlint:-varargs
>
>
>
>
> true
>
> false
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3557) PreparedStatement should be closed in JDBCResourceDAO#checkTableExists
Ted Yu created KYLIN-3557:
-
Summary: PreparedStatement should be closed in
JDBCResourceDAO#checkTableExists
Key: KYLIN-3557
URL: https://issues.apache.org/jira/browse/KYLIN-3557
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
{code}
final PreparedStatement ps =
connection.prepareStatement(getCheckTableExistsSql(tableName));
final ResultSet rs = ps.executeQuery();
{code}
{{ps}} should be closed upon return.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3556) Interned string should not be used as lock object
Ted Yu created KYLIN-3556:
-
Summary: Interned string should not be used as lock object
Key: KYLIN-3556
URL: https://issues.apache.org/jira/browse/KYLIN-3556
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
In JDBCResourceDAO :
{code}
public void execute(Connection connection) throws SQLException {
synchronized (resPath.intern()) {
{code}
Locking on an interned string can cause unexpected locking collisions with
other part of code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3447) Upgrade zookeeper to 3.4.13
[ https://issues.apache.org/jira/browse/KYLIN-3447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3447: -- Description: zookeeper 3.4.13 is being released with the following fixes: ZOOKEEPER-2959 fixes data loss when observer is used ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container / cloud) environment was: zookeeper 3.4.13 is being released with the following fixes: ZOOKEEPER-2959 fixes data loss when observer is used ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container / cloud) environment > Upgrade zookeeper to 3.4.13 > --- > > Key: KYLIN-3447 > URL: https://issues.apache.org/jira/browse/KYLIN-3447 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > > zookeeper 3.4.13 is being released with the following fixes: > ZOOKEEPER-2959 fixes data loss when observer is used > ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container > / cloud) > environment -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3394) Prepare for Kafka 2.0
[
https://issues.apache.org/jira/browse/KYLIN-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated KYLIN-3394:
--
Description:
Kafka 2.0 is around the corner.
I got the following when compiling against Kafka 2.0.0:
{code}
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:266:
error: type mismatch;
[ERROR] found : Option[org.apache.kafka.common.Node]
[ERROR] required: org.apache.kafka.common.Node
[ERROR] getBrokerInfoFromCache(zkUtils, cachedBrokerInfo,
List(l)).head.getNode(listenerName)
[ERROR]
^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:273:
error: type mismatch;
[ERROR] found : Seq[Option[org.apache.kafka.common.Node]]
[ERROR] required: Seq[org.apache.kafka.common.Node]
[ERROR] replicaInfo = getBrokerInfoFromCache(zkUtils,
cachedBrokerInfo, replicas).map(_.getNode(listenerName))
[ERROR]
^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:274:
error: type mismatch;
[ERROR] found : Seq[Option[org.apache.kafka.common.Node]]
[ERROR] required: Seq[org.apache.kafka.common.Node]
[ERROR] isrInfo = getBrokerInfoFromCache(zkUtils, cachedBrokerInfo,
inSyncReplicas).map(_.getNode(listenerName))
[ERROR]
^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:476:
error: value getConsumersInGroup is not a member of kafka.utils.ZkUtils
[ERROR] zkUtils.getConsumersInGroup(group).nonEmpty
[ERROR] ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:489:
error: not found: type ZKGroupDirs
[ERROR] val dir = new ZKGroupDirs(group)
[ERROR] ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:507:
error: value getTopicsByConsumerGroup is not a member of kafka.utils.ZkUtils
[ERROR] val topics = zkUtils.getTopicsByConsumerGroup(group)
[ERROR] ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:512:
error: not found: type ZKGroupTopicDirs
[ERROR] val dir = new ZKGroupTopicDirs(group, topic)
[ERROR] ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:528:
error: value getAllConsumerGroupsForTopic is not a member of kafka.utils.ZkUtils
[ERROR] val groups = zkUtils.getAllConsumerGroupsForTopic(topic)
[ERROR] ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:720:
error: value encode is not a member of object kafka.utils.Json
[ERROR] val content =
Json.encode(getConfigChangeZnodeData(sanitizedEntityPath))
[ERROR]^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:733:
error: value encode is not a member of object kafka.utils.Json
[ERROR] zkUtils.updatePersistentPath(entityPath, Json.encode(map))
{code}
was:
Kafka 2.0 is around the corner.
I got the following when compiling against Kafka 2.0.0-SNAPSHOT :
{code}
[ERROR]
/a/kylin/kylin-it/src/test/java/org/apache/kylin/provision/MockKafka.java:[79,74]
error: cannot find symbol
[ERROR] symbol: method serverConfig()
[ERROR] location: variable kafkaServer of type KafkaServerStartable
[ERROR]
/a/kylin/kylin-it/src/test/java/org/apache/kylin/provision/MockKafka.java:[79,113]
error: cannot find symbol
[ERROR] symbol: method serverConfig()
[ERROR] location: variable kafkaServer of type KafkaServerStartable
[ERROR]
/a/kylin/kylin-it/src/test/java/org/apache/kylin/provision/MockKafka.java:[79,148]
error: cannot find symbol
[ERROR] symbol: method serverConfig()
[ERROR] location: variable kafkaServer of type KafkaServerStartable
[ERROR]
/a/kylin/kylin-it/src/test/java/org/apache/kylin/provision/MockKafka.java:[98,65]
error: cannot find symbol
{code}
> Prepare for Kafka 2.0
> -
>
> Key: KYLIN-3394
> URL: https://issues.apache.org/jira/browse/KYLIN-3394
> Project: Kylin
> Issue Type: Task
>Reporter: Ted Yu
>Priority: Major
>
> Kafka 2.0 is around the corner.
> I got the following when compiling against Kafka 2.0.0:
> {code}
> [ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:266:
> error: type mismatch;
> [ERROR] found : Option[org.apache.kafka.common.Node]
> [ERROR] required: org.apache.kafka.common.Node
> [ERROR] getBrokerInfoFromCache(zkUtils, cachedBrokerInfo,
> List(l)).head.getNode(listenerName)
> [ERROR]
> ^
> [ERROR]
[jira] [Created] (KYLIN-3543) Unclosed Job instance in CreateHTableJob#exportHBaseConfiguration
Ted Yu created KYLIN-3543:
-
Summary: Unclosed Job instance in
CreateHTableJob#exportHBaseConfiguration
Key: KYLIN-3543
URL: https://issues.apache.org/jira/browse/KYLIN-3543
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
{code}
out = fs.create(new Path(hbaseConfPath));
job.getConfiguration().writeXml(out);
{code}
The job instance should be closed upon return from the method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3445) Upgrade checkstyle version to 8.6
[ https://issues.apache.org/jira/browse/KYLIN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3445: -- Description: We should upgrade checkstyle version to 8.6+ so that we can use the "match violation message to this regex" feature for suppression. (was: We should upgrade checkstyle version to 8.6+ so that we can use the "match violation message to this regex" feature for suppression . ) > Upgrade checkstyle version to 8.6 > - > > Key: KYLIN-3445 > URL: https://issues.apache.org/jira/browse/KYLIN-3445 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Ted Yu >Priority: Minor > > We should upgrade checkstyle version to 8.6+ so that we can use the "match > violation message to this regex" feature for suppression. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3484) Update Hadoop version to 2.7.7
[ https://issues.apache.org/jira/browse/KYLIN-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3484: -- Description: We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and security fixes . (was: We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and security fixes.) > Update Hadoop version to 2.7.7 > -- > > Key: KYLIN-3484 > URL: https://issues.apache.org/jira/browse/KYLIN-3484 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Priority: Minor > > We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and > security fixes . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3447) Upgrade zookeeper to 3.4.13
[ https://issues.apache.org/jira/browse/KYLIN-3447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3447: -- Description: zookeeper 3.4.13 is being released with the following fixes: ZOOKEEPER-2959 fixes data loss when observer is used ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container / cloud) environment was: zookeeper 3.4.13 is being released. ZOOKEEPER-2959 fixes data loss when observer is used ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container / cloud) environment > Upgrade zookeeper to 3.4.13 > --- > > Key: KYLIN-3447 > URL: https://issues.apache.org/jira/browse/KYLIN-3447 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > > zookeeper 3.4.13 is being released with the following fixes: > ZOOKEEPER-2959 fixes data loss when observer is used > ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container > / cloud) > environment -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3484) Update Hadoop version to 2.7.7
[ https://issues.apache.org/jira/browse/KYLIN-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3484: -- Description: We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and security fixes. (was: We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and security fixes .) > Update Hadoop version to 2.7.7 > -- > > Key: KYLIN-3484 > URL: https://issues.apache.org/jira/browse/KYLIN-3484 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Priority: Minor > > We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and > security fixes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3519) Upgrade Jacoco version to 0.8.2
Ted Yu created KYLIN-3519: - Summary: Upgrade Jacoco version to 0.8.2 Key: KYLIN-3519 URL: https://issues.apache.org/jira/browse/KYLIN-3519 Project: Kylin Issue Type: Improvement Reporter: Ted Yu Jacoco 0.8.2 adds Java 11 support: https://github.com/jacoco/jacoco/releases/tag/v0.8.2 Java 11 RC1 is out. We should consider upgrading Jacoco. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3482) Unclosed SetAndUnsetThreadLocalConfig in SparkCubingByLayer
[
https://issues.apache.org/jira/browse/KYLIN-3482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595265#comment-16595265
]
Ted Yu commented on KYLIN-3482:
---
Some more occurrences of calling KylinConfig.setAndUnsetThreadLocalConfig
without closing:
engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkFactDistinct.java
> Unclosed SetAndUnsetThreadLocalConfig in SparkCubingByLayer
> ---
>
> Key: KYLIN-3482
> URL: https://issues.apache.org/jira/browse/KYLIN-3482
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Minor
> Fix For: v2.5.0
>
>
> Here is related code:
> {code}
> KylinConfig kylinConfig =
> AbstractHadoopJob.loadKylinConfigFromHdfs(sConf, metaUrl);
>
> KylinConfig.setAndUnsetThreadLocalConfig(kylinConfig);
> {code}
> The return value from setAndUnsetThreadLocalConfig should be closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3445) Upgrade checkstyle version to 8.6
[ https://issues.apache.org/jira/browse/KYLIN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3445: -- Description: We should upgrade checkstyle version to 8.6+ so that we can use the "match violation message to this regex" feature for suppression . (was: We should upgrade checkstyle version to 8.6+ so that we can use the "match violation message to this regex" feature for suppression. ) > Upgrade checkstyle version to 8.6 > - > > Key: KYLIN-3445 > URL: https://issues.apache.org/jira/browse/KYLIN-3445 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Ted Yu >Priority: Minor > > We should upgrade checkstyle version to 8.6+ so that we can use the "match > violation message to this regex" feature for suppression . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3484) Update Hadoop version to 2.7.7
[ https://issues.apache.org/jira/browse/KYLIN-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3484: -- Description: We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and security fixes . (was: We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and security fixes.) > Update Hadoop version to 2.7.7 > -- > > Key: KYLIN-3484 > URL: https://issues.apache.org/jira/browse/KYLIN-3484 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Priority: Minor > > We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and > security fixes . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3095) Use ArrayDeque instead of LinkedList for queue implementation
[
https://issues.apache.org/jira/browse/KYLIN-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated KYLIN-3095:
--
Description:
Use ArrayDeque instead of LinkedList for queue implementation where thread
safety is not needed.
>From https://docs.oracle.com/javase/7/docs/api/java/util/ArrayDeque.html
{quote}
Resizable-array implementation of the Deque interface. Array deques have no
capacity restrictions; they grow as necessary to support usage. They are not
thread-safe; in the absence of external synchronization, they do not support
concurrent access by multiple threads. Null elements are prohibited. This class
is likely to be faster than Stack when used as a stack, and *faster than
LinkedList when used as a queue.*
{quote}
was:
Use ArrayDeque instead of LinkedList for queue implementation where thread
safety is not needed.
>From https://docs.oracle.com/javase/7/docs/api/java/util/ArrayDeque.html
{quote}
Resizable-array implementation of the Deque interface. Array deques have no
capacity restrictions; they grow as necessary to support usage. They are not
thread-safe; in the absence of external synchronization, they do not support
concurrent access by multiple threads. Null elements are prohibited. This class
is likely to be faster than Stack when used as a stack, and *faster than
LinkedList when used as a queue.*
{quote}
> Use ArrayDeque instead of LinkedList for queue implementation
> -
>
> Key: KYLIN-3095
> URL: https://issues.apache.org/jira/browse/KYLIN-3095
> Project: Kylin
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Kaige Liu
>Priority: Minor
> Labels: parallel
> Fix For: v2.5.0
>
>
> Use ArrayDeque instead of LinkedList for queue implementation where thread
> safety is not needed.
> From https://docs.oracle.com/javase/7/docs/api/java/util/ArrayDeque.html
> {quote}
> Resizable-array implementation of the Deque interface. Array deques have no
> capacity restrictions; they grow as necessary to support usage. They are not
> thread-safe; in the absence of external synchronization, they do not support
> concurrent access by multiple threads. Null elements are prohibited. This
> class is likely to be faster than Stack when used as a stack, and *faster
> than LinkedList when used as a queue.*
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3046) Consider introducing log4j-extras
[ https://issues.apache.org/jira/browse/KYLIN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3046: -- Description: log4j-extras allows log rotation as well as compression. https://logging.apache.org/log4j/extras/download.html We should consider using log4j-extras. was: log4j-extras allows log rotation as well as compression. https://logging.apache.org/log4j/extras/download.html We should consider using log4j-extras. > Consider introducing log4j-extras > -- > > Key: KYLIN-3046 > URL: https://issues.apache.org/jira/browse/KYLIN-3046 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > Labels: log > Fix For: v2.5.0 > > > log4j-extras allows log rotation as well as compression. > https://logging.apache.org/log4j/extras/download.html > We should consider using log4j-extras. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3317) Replace UUID.randomUUID with deterministic PRNG
[
https://issues.apache.org/jira/browse/KYLIN-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated KYLIN-3317:
--
Description:
Currently UUID.randomUUID is called in various places in the code base.
* It is non-deterministic.
* It uses a single secure random for UUID generation. This uses a single JVM
wide lock, and this can lead to lock contention and other performance problems.
We should move to something that is deterministic by using seeded PRNGs
{code}
new UUID(ThreadLocalRandom.current().nextLong(),
ThreadLocalRandom.current().nextLong())
{code}
was:
Currently UUID.randomUUID is called in various places in the code base.
* It is non-deterministic.
* It uses a single secure random for UUID generation. This uses a single JVM
wide lock, and this can lead to lock contention and other performance problems.
We should move to something that is deterministic by using seeded PRNGs
{code}
new UUID(ThreadLocalRandom.current().nextLong(),
ThreadLocalRandom.current().nextLong())
{code}
> Replace UUID.randomUUID with deterministic PRNG
> ---
>
> Key: KYLIN-3317
> URL: https://issues.apache.org/jira/browse/KYLIN-3317
> Project: Kylin
> Issue Type: Task
> Components: Security
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
> Fix For: v2.5.0
>
>
> Currently UUID.randomUUID is called in various places in the code base.
> * It is non-deterministic.
> * It uses a single secure random for UUID generation. This uses a single JVM
> wide lock, and this can lead to lock contention and other performance
> problems.
> We should move to something that is deterministic by using seeded PRNGs
> {code}
> new UUID(ThreadLocalRandom.current().nextLong(),
> ThreadLocalRandom.current().nextLong())
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3450) Consider using google re2j
[ https://issues.apache.org/jira/browse/KYLIN-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556761#comment-16556761 ] Ted Yu edited comment on KYLIN-3450 at 8/9/18 1:55 AM: --- It would be nice to see improvement from using re2j . was (Author: yuzhih...@gmail.com): It would be nice to see improvement from using re2j > Consider using google re2j > -- > > Key: KYLIN-3450 > URL: https://issues.apache.org/jira/browse/KYLIN-3450 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Minor > > RE2J : https://github.com/google/re2j > For regular expression patterns with a high degree of alternation, using RE2J > would exhibit higher performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3484) Update Hadoop version to 2.7.7
Ted Yu created KYLIN-3484: - Summary: Update Hadoop version to 2.7.7 Key: KYLIN-3484 URL: https://issues.apache.org/jira/browse/KYLIN-3484 Project: Kylin Issue Type: Task Reporter: Ted Yu We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and security fixes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3483) Imprecise comparison between double and integer division
Ted Yu created KYLIN-3483:
-
Summary: Imprecise comparison between double and integer division
Key: KYLIN-3483
URL: https://issues.apache.org/jira/browse/KYLIN-3483
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
In HLLCSerializer :
{code}
|| averageNumOfElementsInCounter < (m - 5) / (1 +
registerIndexSize)) {
{code}
The integer division on the right side of < would lose remainder, or
fractional part of the quotient.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3482) Unclosed SetAndUnsetThreadLocalConfig in SparkCubingByLayer
Ted Yu created KYLIN-3482:
-
Summary: Unclosed SetAndUnsetThreadLocalConfig in
SparkCubingByLayer
Key: KYLIN-3482
URL: https://issues.apache.org/jira/browse/KYLIN-3482
Project: Kylin
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
KylinConfig kylinConfig =
AbstractHadoopJob.loadKylinConfigFromHdfs(sConf, metaUrl);
KylinConfig.setAndUnsetThreadLocalConfig(kylinConfig);
{code}
The return value from setAndUnsetThreadLocalConfig should be closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-2517) Upgrade hbase dependency to 1.4.6
[ https://issues.apache.org/jira/browse/KYLIN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-2517: -- Summary: Upgrade hbase dependency to 1.4.6 (was: Upgrade hbase dependency to 1.4.5) > Upgrade hbase dependency to 1.4.6 > - > > Key: KYLIN-2517 > URL: https://issues.apache.org/jira/browse/KYLIN-2517 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > > There have been major enhancements / bug fixes since the hbase 1.1.1 release. > This issue is to upgrade to 1.4.5 release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2517) Upgrade hbase dependency to 1.4.6
[ https://issues.apache.org/jira/browse/KYLIN-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-2517: -- Description: There have been major enhancements / bug fixes since the hbase 1.1.1 release. This issue is to upgrade to 1.4.6 release. was: There have been major enhancements / bug fixes since the hbase 1.1.1 release. This issue is to upgrade to 1.4.5 release. > Upgrade hbase dependency to 1.4.6 > - > > Key: KYLIN-2517 > URL: https://issues.apache.org/jira/browse/KYLIN-2517 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > > There have been major enhancements / bug fixes since the hbase 1.1.1 release. > This issue is to upgrade to 1.4.6 release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3187) JDK APIs using the default locale, time zone or character set should be avoided
[
https://issues.apache.org/jira/browse/KYLIN-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated KYLIN-3187:
--
Description:
Here are a few examples:
{code}
server-base/src/main/java/org/apache/kylin/rest/service/JobService.java:
Calendar calendar = Calendar.getInstance();
storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/HbaseStreamingInput.java:
Calendar cal = Calendar.getInstance();
{code}
Locale should be specified.
See CALCITE-1667 for related information.
was:
Here are a few examples:
{code}
server-base/src/main/java/org/apache/kylin/rest/service/JobService.java:
Calendar calendar = Calendar.getInstance();
storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/HbaseStreamingInput.java:
Calendar cal = Calendar.getInstance();
{code}
Locale should be specified.
See CALCITE-1667 for related information.
> JDK APIs using the default locale, time zone or character set should be
> avoided
> ---
>
> Key: KYLIN-3187
> URL: https://issues.apache.org/jira/browse/KYLIN-3187
> Project: Kylin
> Issue Type: Bug
> Components: REST Service
>Reporter: Ted Yu
>Assignee: Kaige Liu
>Priority: Major
> Labels: usability
> Fix For: v2.5.0
>
>
> Here are a few examples:
> {code}
> server-base/src/main/java/org/apache/kylin/rest/service/JobService.java:
> Calendar calendar = Calendar.getInstance();
> storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/HbaseStreamingInput.java:
> Calendar cal = Calendar.getInstance();
> {code}
> Locale should be specified.
> See CALCITE-1667 for related information.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3046) Consider introducing log4j-extras
[ https://issues.apache.org/jira/browse/KYLIN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3046: -- Description: log4j-extras allows log rotation as well as compression. https://logging.apache.org/log4j/extras/download.html We should consider using log4j-extras. was: log4j-extras allows log rotation as well as compression. https://logging.apache.org/log4j/extras/download.html We should consider using log4j-extras. > Consider introducing log4j-extras > -- > > Key: KYLIN-3046 > URL: https://issues.apache.org/jira/browse/KYLIN-3046 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > Labels: log > Fix For: v2.5.0 > > > log4j-extras allows log rotation as well as compression. > https://logging.apache.org/log4j/extras/download.html > We should consider using log4j-extras. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3203) Upgrade Jacoco release
[ https://issues.apache.org/jira/browse/KYLIN-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3203: -- Description: Jacoco is actively maintained by the community. Here is the latest release: https://github.com/jacoco/jacoco/releases/tag/v0.8.1 We should upgrade to 0.8.1 release was: Jacoco is actively maintained by the community. Here is the latest release: https://github.com/jacoco/jacoco/releases/tag/v0.8.1 We should upgrade to 0.8.1 release > Upgrade Jacoco release > -- > > Key: KYLIN-3203 > URL: https://issues.apache.org/jira/browse/KYLIN-3203 > Project: Kylin > Issue Type: Test > Components: Tools, Build and Test >Reporter: Ted Yu >Assignee: nichunen >Priority: Minor > Fix For: v2.5.0 > > > Jacoco is actively maintained by the community. > Here is the latest release: > https://github.com/jacoco/jacoco/releases/tag/v0.8.1 > We should upgrade to 0.8.1 release -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3290) Avoid calling Class#newInstance
[ https://issues.apache.org/jira/browse/KYLIN-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3290: -- Labels: jdk (was: ) > Avoid calling Class#newInstance > --- > > Key: KYLIN-3290 > URL: https://issues.apache.org/jira/browse/KYLIN-3290 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Assignee: jiatao.tao >Priority: Minor > Labels: jdk > Fix For: v2.5.0 > > > Class#newInstance is deprecated starting in Java 9 - > https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw > undeclared checked exceptions. > The suggested replacement is getDeclaredConstructor().newInstance(), which > wraps the checked exceptions in InvocationException. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3272) Upgrade Spark dependency to 2.3.1
[ https://issues.apache.org/jira/browse/KYLIN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472296#comment-16472296 ] Ted Yu edited comment on KYLIN-3272 at 7/29/18 1:37 AM: Same issue reported in SPARK-21569. was (Author: yuzhih...@gmail.com): Same issue reported in SPARK-21569 . > Upgrade Spark dependency to 2.3.1 > - > > Key: KYLIN-3272 > URL: https://issues.apache.org/jira/browse/KYLIN-3272 > Project: Kylin > Issue Type: Improvement > Components: Spark Engine >Reporter: Ted Yu >Priority: Minor > > Currently Spark 2.1.2 is used. > Spark 2.3.0 was just released. > We should upgrade the dependency to 2.3.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3310) Use lint for maven-compiler-plugin
[
https://issues.apache.org/jira/browse/KYLIN-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560940#comment-16560940
]
Ted Yu commented on KYLIN-3310:
---
Thanks, Jiatao.
> Use lint for maven-compiler-plugin
> --
>
> Key: KYLIN-3310
> URL: https://issues.apache.org/jira/browse/KYLIN-3310
> Project: Kylin
> Issue Type: Improvement
> Components: Tools, Build and Test
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
>
> lint helps identify structural problems.
> We should enable lint for maven-compiler-plugin
> {code}
> maven-compiler-plugin
> ${maven-compiler-plugin.version}
>
> 1.8
> 1.8
>
> -Xlint:all
> ${compiler.error.flag}
>
> -Xlint:-options
>
> -Xlint:-cast
> -Xlint:-deprecation
> -Xlint:-processing
> -Xlint:-rawtypes
> -Xlint:-serial
> -Xlint:-try
> -Xlint:-unchecked
> -Xlint:-varargs
>
>
>
>
> true
>
> false
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3394) Prepare for Kafka 2.0
[
https://issues.apache.org/jira/browse/KYLIN-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560938#comment-16560938
]
Ted Yu commented on KYLIN-3394:
---
Kafka 2.0 RC has passed.
> Prepare for Kafka 2.0
> -
>
> Key: KYLIN-3394
> URL: https://issues.apache.org/jira/browse/KYLIN-3394
> Project: Kylin
> Issue Type: Task
>Reporter: Ted Yu
>Priority: Major
>
> Kafka 2.0 is around the corner.
> I got the following when compiling against Kafka 2.0.0-SNAPSHOT :
> {code}
> [ERROR]
> /a/kylin/kylin-it/src/test/java/org/apache/kylin/provision/MockKafka.java:[79,74]
> error: cannot find symbol
> [ERROR] symbol: method serverConfig()
> [ERROR] location: variable kafkaServer of type KafkaServerStartable
> [ERROR]
> /a/kylin/kylin-it/src/test/java/org/apache/kylin/provision/MockKafka.java:[79,113]
> error: cannot find symbol
> [ERROR] symbol: method serverConfig()
> [ERROR] location: variable kafkaServer of type KafkaServerStartable
> [ERROR]
> /a/kylin/kylin-it/src/test/java/org/apache/kylin/provision/MockKafka.java:[79,148]
> error: cannot find symbol
> [ERROR] symbol: method serverConfig()
> [ERROR] location: variable kafkaServer of type KafkaServerStartable
> [ERROR]
> /a/kylin/kylin-it/src/test/java/org/apache/kylin/provision/MockKafka.java:[98,65]
> error: cannot find symbol
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3431) Avoid FileInputStream/FileOutputStream
[
https://issues.apache.org/jira/browse/KYLIN-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated KYLIN-3431:
--
Description:
They rely on finalizers (before Java 11), which create unnecessary GC load.
The alternatives, {{Files.newInputStream}}, are as easy to use and don't have
this issue.
was:They rely on finalizers (before Java 11), which create unnecessary GC
load. The alternatives, {{Files.newInputStream}}, are as easy to use and don't
have this issue.
> Avoid FileInputStream/FileOutputStream
> --
>
> Key: KYLIN-3431
> URL: https://issues.apache.org/jira/browse/KYLIN-3431
> Project: Kylin
> Issue Type: Improvement
>Reporter: Ted Yu
>Priority: Major
> Fix For: v2.5.0
>
>
> They rely on finalizers (before Java 11), which create unnecessary GC load.
> The alternatives, {{Files.newInputStream}}, are as easy to use and don't have
> this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3445) Upgrade checkstyle version to 8.6
[ https://issues.apache.org/jira/browse/KYLIN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3445: -- Component/s: Tools, Build and Test > Upgrade checkstyle version to 8.6 > - > > Key: KYLIN-3445 > URL: https://issues.apache.org/jira/browse/KYLIN-3445 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Ted Yu >Priority: Minor > > We should upgrade checkstyle version to 8.6+ so that we can use the "match > violation message to this regex" feature for suppression. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3317) Replace UUID.randomUUID with deterministic PRNG
[
https://issues.apache.org/jira/browse/KYLIN-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated KYLIN-3317:
--
Description:
Currently UUID.randomUUID is called in various places in the code base.
* It is non-deterministic.
* It uses a single secure random for UUID generation. This uses a single JVM
wide lock, and this can lead to lock contention and other performance problems.
We should move to something that is deterministic by using seeded PRNGs
{code}
new UUID(ThreadLocalRandom.current().nextLong(),
ThreadLocalRandom.current().nextLong())
{code}
was:
Currently UUID.randomUUID is called in various places in the code base.
* It is non-deterministic.
* It uses a single secure random for UUID generation. This uses a single JVM
wide lock, and this can lead to lock contention and other performance problems.
We should move to something that is deterministic by using seeded PRNGs
{code}
new UUID(ThreadLocalRandom.current().nextLong(),
ThreadLocalRandom.current().nextLong())
{code}
> Replace UUID.randomUUID with deterministic PRNG
> ---
>
> Key: KYLIN-3317
> URL: https://issues.apache.org/jira/browse/KYLIN-3317
> Project: Kylin
> Issue Type: Task
> Components: Security
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
> Fix For: v2.5.0
>
>
> Currently UUID.randomUUID is called in various places in the code base.
> * It is non-deterministic.
> * It uses a single secure random for UUID generation. This uses a single JVM
> wide lock, and this can lead to lock contention and other performance
> problems.
> We should move to something that is deterministic by using seeded PRNGs
> {code}
> new UUID(ThreadLocalRandom.current().nextLong(),
> ThreadLocalRandom.current().nextLong())
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3417) Consider replacing ReentrantReadWriteLock with StampedLock
[
https://issues.apache.org/jira/browse/KYLIN-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556844#comment-16556844
]
Ted Yu commented on KYLIN-3417:
---
Thanks for taking this.
> Consider replacing ReentrantReadWriteLock with StampedLock
> --
>
> Key: KYLIN-3417
> URL: https://issues.apache.org/jira/browse/KYLIN-3417
> Project: Kylin
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
> Fix For: v2.5.0
>
>
> ReentrantReadWriteLock's are only the right solution when there is long hold
> time due to expensive I/O.
> It is expensive for readers.
> We should see if the lighter {{StampedLock}} can be used instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
