Re: How to reflect last hour data into Hive and Kylin Insights query window
I am not really sure. But I think it is the Query cache make your query result unchanged. The config entry is kylin.query.cache-enabled , is turn on by default. This doc links is https://kylin.apache.org/5.0/docs/configuration/query_cache -- Best wishes to you ! From ：Xiaoxiang Yu At 2023-11-17 09:48:55, "Nam Đỗ Duy" wrote: >Hello Team, hello Xiaoxiang, can you please help me with this urgent >issue... > >(this is public email group so in general I neglect your specific name from >greeting of first email in the threads, but in fact most of time Xiaoxiang >actively answers my issues, thank you very much) > >On Thu, Nov 16, 2023 at 2:59 PM Nam Đỗ Duy wrote: > >> Dear Dev Team, please kindly advise this scenario >> >> 1. I have a fact table and I use Kylin insights window to query it and get >> 5 million rows. >> >> 2. Then I use following command to load X rows (last hour data) from >> parquet into Hive table >> >> LOAD DATA LOCAL INPATH >> '/opt/LastHour/factUserEventDF_2023_11_16.parquet/14' INTO TABLE >> factUserEvent; >> >> 3. Then I open Kylin insights window to query it but it still returned >> previous number (5 million rows) not adding the last hour data of X rows >> which I previously loaded from parquet into hive in step 2) >> >> Can you advise the way to make table refresh and updated? >> >> Thank you very much >>
Re: How to reflect last hour data into Hive and Kylin Insights query window
Hello Team, hello Xiaoxiang, can you please help me with this urgent issue... (this is public email group so in general I neglect your specific name from greeting of first email in the threads, but in fact most of time Xiaoxiang actively answers my issues, thank you very much) On Thu, Nov 16, 2023 at 2:59 PM Nam Đỗ Duy wrote: > Dear Dev Team, please kindly advise this scenario > > 1. I have a fact table and I use Kylin insights window to query it and get > 5 million rows. > > 2. Then I use following command to load X rows (last hour data) from > parquet into Hive table > > LOAD DATA LOCAL INPATH > '/opt/LastHour/factUserEventDF_2023_11_16.parquet/14' INTO TABLE > factUserEvent; > > 3. Then I open Kylin insights window to query it but it still returned > previous number (5 million rows) not adding the last hour data of X rows > which I previously loaded from parquet into hive in step 2) > > Can you advise the way to make table refresh and updated? > > Thank you very much >
[jira] [Created] (KYLIN-5727) update spark version to 3.2.0-kylin-4.6.11.0
Hongrong Cao created KYLIN-5727: --- Summary: update spark version to 3.2.0-kylin-4.6.11.0 Key: KYLIN-5727 URL: https://issues.apache.org/jira/browse/KYLIN-5727 Project: Kylin Issue Type: Improvement Affects Versions: 5.0-beta Reporter: Hongrong Cao Assignee: Hongrong Cao Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5726) Remove unnecessary execution goal for deploy plugin
Hongrong Cao created KYLIN-5726: --- Summary: Remove unnecessary execution goal for deploy plugin Key: KYLIN-5726 URL: https://issues.apache.org/jira/browse/KYLIN-5726 Project: Kylin Issue Type: Improvement Affects Versions: 5.0-beta Reporter: Hongrong Cao Assignee: Hongrong Cao Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5725) After turning on the email alert configuration, the operation task prompts an unknown error, and the query underpressing also reports an error.
Hongrong Cao created KYLIN-5725: --- Summary: After turning on the email alert configuration, the operation task prompts an unknown error, and the query underpressing also reports an error. Key: KYLIN-5725 URL: https://issues.apache.org/jira/browse/KYLIN-5725 Project: Kylin Issue Type: Bug Affects Versions: 5.0-beta Reporter: Hongrong Cao Assignee: sibing.zhang Fix For: 5.0.0 Steps to Reproduce 1:Configure to enable email alerts 2:Create a model and initiate a build task. 3:In the task list operation task restart, pause, terminate 4:Query sql, expect query downpressure Actual result 3:Operation error report 4:Query error -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5724) Support local debug & glue hive client cache
Hongrong Cao created KYLIN-5724: --- Summary: Support local debug & glue hive client cache Key: KYLIN-5724 URL: https://issues.apache.org/jira/browse/KYLIN-5724 Project: Kylin Issue Type: Improvement Affects Versions: 5.0-beta Reporter: Hongrong Cao Assignee: Hongrong Cao Fix For: 5.0.0 update hadoop-aws-3.3.1.jar and aws-glue-datacatalog-spark-client-3.4.0-SNAPSHOT.jar to spark jars. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5723) Optimize Calcite plan to convert spark logical plan
Hongrong Cao created KYLIN-5723:
---
Summary: Optimize Calcite plan to convert spark logical plan
Key: KYLIN-5723
URL: https://issues.apache.org/jira/browse/KYLIN-5723
Project: Kylin
Issue Type: Improvement
Affects Versions: 5.0-beta
Reporter: Hongrong Cao
Assignee: Hongrong Cao
Fix For: 5.0.0
The problem with calcite plan to spark LogicalPlan is that it uses the
DataFrame interface, which means that it parses and builds the LogicalPlan at
the same time.
During the process of calcite plan to spark plan, the select agg and other
operators execute the following methods, where the execution of
qe.assertAnalyzed() is of interest
{quote}def ofRows(sparkSession: SparkSession, logicalPlan: LogicalPlan):
DataFrame =
sparkSession.withActive {
val qe = sparkSession.sessionState.executePlan(logicalPlan)
qe.assertAnalyzed()
new Dataset[Row](qe, RowEncoder(qe.analyzed.schema))
}
{quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KYLIN-5722) Optimize KylinConfExt
Hongrong Cao created KYLIN-5722: --- Summary: Optimize KylinConfExt Key: KYLIN-5722 URL: https://issues.apache.org/jira/browse/KYLIN-5722 Project: Kylin Issue Type: Improvement Affects Versions: 5.0-beta Reporter: Hongrong Cao Assignee: Hongrong Cao Fix For: 5.0.0 KylinConfigExt every time create a new instance to get the configuration, you will get it through the org.apache.kylin.common.KylinConfigExt#getSubstitutor method, which will new a HashMap every time and then do the element copy action, the method is time consuming and needs to be optimized. This method is time-consuming and needs to be optimized. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5721) The input parameters of some date functions support string format
Hongrong Cao created KYLIN-5721: --- Summary: The input parameters of some date functions support string format Key: KYLIN-5721 URL: https://issues.apache.org/jira/browse/KYLIN-5721 Project: Kylin Issue Type: Improvement Affects Versions: 5.0-beta Reporter: Hongrong Cao Assignee: Hongrong Cao Fix For: 5.0.0 date_trunc date_trunc minute date_trunc extract These functions hopefully also implement date strings as arguments. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5720) the query node returns to the login page after clicking on the model page
Hongrong Cao created KYLIN-5720: --- Summary: the query node returns to the login page after clicking on the model page Key: KYLIN-5720 URL: https://issues.apache.org/jira/browse/KYLIN-5720 Project: Kylin Issue Type: Bug Affects Versions: 5.0-beta Reporter: Hongrong Cao Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5719) Occasional initialization failure of the V3 global dictionary file, resulting in failure of build jobs
Hongrong Cao created KYLIN-5719: --- Summary: Occasional initialization failure of the V3 global dictionary file, resulting in failure of build jobs Key: KYLIN-5719 URL: https://issues.apache.org/jira/browse/KYLIN-5719 Project: Kylin Issue Type: Improvement Affects Versions: 5.0-beta Reporter: Hongrong Cao Assignee: Yaguang Jia Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5718) Support automatic merging of V3 dictionary files
Hongrong Cao created KYLIN-5718: --- Summary: Support automatic merging of V3 dictionary files Key: KYLIN-5718 URL: https://issues.apache.org/jira/browse/KYLIN-5718 Project: Kylin Issue Type: Improvement Affects Versions: 5.0-beta Reporter: Hongrong Cao Assignee: Yaguang Jia Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5717) The to_date function is not aligned with SparkSQL. When converting to a time format such as yyyyMMdd, the result of the query hitting the snapshot is incorrect
Hongrong Cao created KYLIN-5717: --- Summary: The to_date function is not aligned with SparkSQL. When converting to a time format such as MMdd, the result of the query hitting the snapshot is incorrect Key: KYLIN-5717 URL: https://issues.apache.org/jira/browse/KYLIN-5717 Project: Kylin Issue Type: Bug Affects Versions: 5.0-beta Reporter: Hongrong Cao Assignee: pengfei.zhan Fix For: 5.0.0 The to_date function is not aligned with SparkSQL. When converting to a time format such as MMdd, the result of the query hitting the snapshot is incorrect； -- This message was sent by Atlassian Jira (v8.20.10#820010)
How to reflect last hour data into Hive and Kylin Insights query window
Dear Dev Team, please kindly advise this scenario 1. I have a fact table and I use Kylin insights window to query it and get 5 million rows. 2. Then I use following command to load X rows (last hour data) from parquet into Hive table LOAD DATA LOCAL INPATH '/opt/LastHour/factUserEventDF_2023_11_16.parquet/14' INTO TABLE factUserEvent; 3. Then I open Kylin insights window to query it but it still returned previous number (5 million rows) not adding the last hour data of X rows which I previously loaded from parquet into hive in step 2) Can you advise the way to make table refresh and updated? Thank you very much
