[jira] [Commented] (IMPALA-6818) Rethink data-stream sender/receiver startup sequencing
[
https://issues.apache.org/jira/browse/IMPALA-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520935#comment-16520935
]
Michael Ho commented on IMPALA-6818:
[~dhecht] and I discussed about an alternative which may allow us to get rid of
this timeout option.
When a sender issues a {{TransmitData()}} RPC before the receiver is created,
the data stream manager should just reply to the sender with an error message
that it's not found. The sender will then keep pinging the destination fragment
periodically until the receiver shows up or until the sender's fragment is
cancelled. There are some races which may still occur:
- The receiver is closed before the sender manages to send the first row batch
to the receiver. In which case, the sender will keep pinging the destination
fragment until the sender fragment is cancelled. The assumption is that the
receiver will only be closed prematurely before receiving all the data from the
sender when (1) the query is cancelled or (2) the ancestor executors of the
exchange node has a limit and closes the exchange node once the limit is
reached. In case (2), it's expected that the report status thread of the
sender's fragment will notice that all results have been returned and cancel
the sender fragment (I believe this may not yet be the case until IMPALA-5119
is fixed. [~dhecht], please correct me if I misunderstood anything).
> Rethink data-stream sender/receiver startup sequencing
> --
>
> Key: IMPALA-6818
> URL: https://issues.apache.org/jira/browse/IMPALA-6818
> Project: IMPALA
> Issue Type: Sub-task
> Components: Distributed Exec
>Reporter: Dan Hecht
>Assignee: Michael Ho
>Priority: Major
>
> IMPALA-1599 introduced parallel fragment startup, which is good for startup
> latency. However, it meant that data-stream senders can start before
> receivers, and there is a timeout to handle the case when the receiver never
> shows up:
> {code:java}
> Sender timed out waiting for receiver fragment instance{code}
> We see this timeout fairly regularly (e.g. when a host has a spike in load
> and does not process the exec rpc for a while). Let's rethink how this works
> to see if we can make it robust but being careful to not sacrifice startup
> time too much.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7203) Support functions in LocalCatalog
Todd Lipcon created IMPALA-7203: --- Summary: Support functions in LocalCatalog Key: IMPALA-7203 URL: https://issues.apache.org/jira/browse/IMPALA-7203 Project: IMPALA Issue Type: Sub-task Reporter: Todd Lipcon Assignee: Todd Lipcon -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7202) Add WIDTH_BUCKET() function to the decimal fuzz test
[ https://issues.apache.org/jira/browse/IMPALA-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Taras Bobrovytsky updated IMPALA-7202: -- Component/s: Infrastructure > Add WIDTH_BUCKET() function to the decimal fuzz test > > > Key: IMPALA-7202 > URL: https://issues.apache.org/jira/browse/IMPALA-7202 > Project: IMPALA > Issue Type: Task > Components: Infrastructure >Reporter: Taras Bobrovytsky >Priority: Major > > We need to add the new WIDTH_BUCKET() function to the decimal fuzz test for > better coverage. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7202) Add WIDTH_BUCKET() function to the decimal fuzz test
Taras Bobrovytsky created IMPALA-7202: - Summary: Add WIDTH_BUCKET() function to the decimal fuzz test Key: IMPALA-7202 URL: https://issues.apache.org/jira/browse/IMPALA-7202 Project: IMPALA Issue Type: Task Reporter: Taras Bobrovytsky We need to add the new WIDTH_BUCKET() function to the decimal fuzz test for better coverage. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7201) Support DDL in LocalCatalog using existing catalogd
Todd Lipcon created IMPALA-7201: --- Summary: Support DDL in LocalCatalog using existing catalogd Key: IMPALA-7201 URL: https://issues.apache.org/jira/browse/IMPALA-7201 Project: IMPALA Issue Type: Sub-task Reporter: Todd Lipcon Need some changes to ensure that create table, create view, drop view, etc, can work. The initial implementation will still RPC out to catalogd, which will perform the mutations. At some point we may want to move this work to the impalad itself, but for now keeping the code with as little change as possible is preferred. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7141) Extract interfaces for partition pruning prior to fetching partitions
[ https://issues.apache.org/jira/browse/IMPALA-7141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved IMPALA-7141. - Resolution: Fixed Fix Version/s: Impala 3.1.0 Resolved in 747e0a949316b8e14fa6ad123ffc34d9c450608b and 0063e0c5d7747aca39a8d9c53cc1c52eed5c884b > Extract interfaces for partition pruning prior to fetching partitions > - > > Key: IMPALA-7141 > URL: https://issues.apache.org/jira/browse/IMPALA-7141 > Project: IMPALA > Issue Type: Sub-task >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Fix For: Impala 3.1.0 > > > In the LocalCatalog, we want to only fetch the partitions that are referenced > by a query -- i.e. we must prune partitions based only on the partition names > and not the entire partition objects. However, the PartitionPruner > implementation currently expect to be able to fetch the full map of > HdfsPartition objects from the table and work on them as is. > This JIRA is to do some refactorings such that the PartitionPruner interacts > with a slightly more restricted interface that only exposes the minimal > interaction points with the table and the partition map. Once it has computed > a list of remaining partitions, it can then instruct the table to fully load > them to yield the resulting full Partition objects. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7135) Skeleton implementation of LocalCatalog
[ https://issues.apache.org/jira/browse/IMPALA-7135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved IMPALA-7135. - Resolution: Fixed Assignee: Todd Lipcon Fix Version/s: Impala 3.1.0 853b6a9c83b7f136e5df200a579ff39b3db42b8c > Skeleton implementation of LocalCatalog > --- > > Key: IMPALA-7135 > URL: https://issues.apache.org/jira/browse/IMPALA-7135 > Project: IMPALA > Issue Type: Sub-task >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Major > Fix For: Impala 3.1.0 > > > This subtask encompasses adding the skeleton for the non-catalogd-based > FeCatalog implementation. The goal for this JIRA is to just add the main > classes and a simple unit test to verify that it can load the list of > databases and tables. Additional subtasks will be filed for follow-up work > such as: > - hooking it up to the Frontend and adding the appropriate configuration > flag, etc. > - loading partition info, files, functions, cache pools, etc > - implementing caching -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7199) Need to have scripts to generate coverage
[ https://issues.apache.org/jira/browse/IMPALA-7199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe McDonnell updated IMPALA-7199: -- Attachment: coverage_be_tests.tar.gz > Need to have scripts to generate coverage > - > > Key: IMPALA-7199 > URL: https://issues.apache.org/jira/browse/IMPALA-7199 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Affects Versions: Impala 3.1.0 >Reporter: Joe McDonnell >Assignee: Joe McDonnell >Priority: Major > Attachments: coverage_be_tests.tar.gz > > > Code coverage can be a useful means to verify that tests are exercising the > code as expected. It would be useful to have a helper script to make this > process as simple as possible. Now that gcovr is pip installable, that is one > option for generating coverage reports. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7175) In a local FS build, test_native_functions_race thinks there are 2 impalads where there should be 1
[
https://issues.apache.org/jira/browse/IMPALA-7175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vuk Ercegovac resolved IMPALA-7175.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
IMPALA-7175: deflake check for failed impalad
test_native_functions_race checks for an impalad crash by testing whether the
number of impalads at the start and end of the test is the same. A recent run
was flaky since the number of impalads at the start was incorrectly found to be
2. This fix tries to make the test most robust by determining the number of
impalads based on how many can evaluate a trivial test query. For these tests,
its assumed that the number of coordinators is the same as the number of
impalads in the cluster.
Change-Id: I97c6b398e43c6abb1df2b1783c26159137f14fa4
Reviewed-on: [http://gerrit.cloudera.org:8080/10745]
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> In a local FS build, test_native_functions_race thinks there are 2 impalads
> where there should be 1
> ---
>
> Key: IMPALA-7175
> URL: https://issues.apache.org/jira/browse/IMPALA-7175
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Tianyi Wang
>Assignee: Vuk Ercegovac
>Priority: Critical
> Labels: broken-build
> Fix For: Impala 3.1.0
>
>
> In TestUdfExecution.test_native_functions_race, the test checks the number of
> impalads at the beginning and end of the test. In a local build there should
> be only 1 impalad but somehow the test found 2 at the beginning of the test
> and failed.
> {noformat}
> Stacktrace
> query_test/test_udfs.py:379: in test_native_functions_race
> assert len(cluster.impalads) == exp_num_impalads
> E assert 1 == 2
> E+ where 1 = len([ 0xc9ffa90>])
> E+where [ 0xc9ffa90>] = 0x6a5d510>.impalads
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-7198) Impala hints in docs are wrong
[
https://issues.apache.org/jira/browse/IMPALA-7198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Rodoni closed IMPALA-7198.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> Impala hints in docs are wrong
> --
>
> Key: IMPALA-7198
> URL: https://issues.apache.org/jira/browse/IMPALA-7198
> Project: IMPALA
> Issue Type: Bug
> Components: Docs
>Affects Versions: Impala 3.0
>Reporter: Lars Volker
>Assignee: Alex Rodoni
>Priority: Major
> Fix For: Impala 3.1.0
>
>
> The "CLUSTER" and "NOCLUSTER" hints in this page are spelled wrong. They
> should actually be "CLUSTERED" and "NOCLUSTERED".
> https://impala.apache.org/docs/build/html/topics/impala_hints.html
> {noformat}
> CREATE /* +CLUSTER|NOCLUSTER */
> table_clauses
> AS SELECT remainder_of_query;
> CREATE -- +CLUSTER|NOCLUSTER
> table_clauses
> AS SELECT remainder_of_query;
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7200) Local filesystem dataload fails due to missing FILESYSTEM_PREFIX
[
https://issues.apache.org/jira/browse/IMPALA-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell reassigned IMPALA-7200:
-
Assignee: Joe McDonnell
> Local filesystem dataload fails due to missing FILESYSTEM_PREFIX
>
>
> Key: IMPALA-7200
> URL: https://issues.apache.org/jira/browse/IMPALA-7200
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Joe McDonnell
>Assignee: Joe McDonnell
>Priority: Blocker
> Labels: broken-build
>
> A recent local filesystem run failed during dataload with the following
> output:
> {noformat}
> 14:27:43 Running custom post-load steps (logging to
> /data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/logs/data_loading/custom-post-load-steps.log)...
>
> 14:27:49 FAILED (Took: 0 min 6 sec)
> 14:27:49 'custom-post-load-steps' failed. Tail of log:
> 14:27:49 Log for command 'custom-post-load-steps'
> 14:27:49 put: `/test-warehouse/': No such file or directory:
> `file:///test-warehouse'{noformat}
> Local filesystem tests rely on the FILESYSTEM_PREFIX being added to add HDFS
> calls. The fix for IMPALA-3307 included this line in
> testdata/bin/create-load-data.sh:
> {noformat}
> # IMPALA-3307: Upload test time-zone database
> hadoop fs -Ddfs.block.size=1048576 -put -f ${IMPALA_HOME}/testdata/tzdb
> /test-warehouse/{noformat}
> This needs to use ${FILESYSTEM_PREFIX}/test-warehouse rather than
> /test-warehouse. Local runs on /tmp. This is likely to impact s3 as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7200) Local filesystem dataload fails due to missing FILESYSTEM_PREFIX
Joe McDonnell created IMPALA-7200:
-
Summary: Local filesystem dataload fails due to missing
FILESYSTEM_PREFIX
Key: IMPALA-7200
URL: https://issues.apache.org/jira/browse/IMPALA-7200
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Affects Versions: Impala 3.1.0
Reporter: Joe McDonnell
A recent local filesystem run failed during dataload with the following output:
{noformat}
14:27:43 Running custom post-load steps (logging to
/data/jenkins/workspace/impala-asf-master-core-local/repos/Impala/logs/data_loading/custom-post-load-steps.log)...
14:27:49 FAILED (Took: 0 min 6 sec)
14:27:49 'custom-post-load-steps' failed. Tail of log:
14:27:49 Log for command 'custom-post-load-steps'
14:27:49 put: `/test-warehouse/': No such file or directory:
`file:///test-warehouse'{noformat}
Local filesystem tests rely on the FILESYSTEM_PREFIX being added to add HDFS
calls. The fix for IMPALA-3307 included this line in
testdata/bin/create-load-data.sh:
{noformat}
# IMPALA-3307: Upload test time-zone database
hadoop fs -Ddfs.block.size=1048576 -put -f ${IMPALA_HOME}/testdata/tzdb
/test-warehouse/{noformat}
This needs to use ${FILESYSTEM_PREFIX}/test-warehouse rather than
/test-warehouse. Local runs on /tmp. This is likely to impact s3 as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-6013) Remove InProcessImpalaServer
[ https://issues.apache.org/jira/browse/IMPALA-6013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520762#comment-16520762 ] Sailesh Mukil commented on IMPALA-6013: --- [~tarmstrong] Yes, that sounds about right. > Remove InProcessImpalaServer > > > Key: IMPALA-6013 > URL: https://issues.apache.org/jira/browse/IMPALA-6013 > Project: IMPALA > Issue Type: Task > Components: Backend >Reporter: Sailesh Mukil >Assignee: Sailesh Mukil >Priority: Major > Labels: refactor > > Now that we've refactored CreateImpalaServer() such that it can be used in > tests (IMPALA-4786), we should get rid of InProcessImpalaServer, which was > only historically there because the ImpalaServer couldn't be used in tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-5900) document -fe_service_threads
[
https://issues.apache.org/jira/browse/IMPALA-5900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Rodoni closed IMPALA-5900.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> document -fe_service_threads
>
>
> Key: IMPALA-5900
> URL: https://issues.apache.org/jira/browse/IMPALA-5900
> Project: IMPALA
> Issue Type: Task
> Components: Docs
>Affects Versions: Impala 2.7.0
>Reporter: Michael Brown
>Assignee: Alex Rodoni
>Priority: Major
> Fix For: Impala 3.1.0
>
>
> Fairly often users are confused about the need to bump up
> {{-fe_service_threads}}. The latest example is IMPALA-5897. It could be
> helpful to add documentation about this option. I noticed that there is
> nothing in documentation about it currently:
> {noformat}
> mikeb@mikeb-desktop:0:~/Impala/docs (asf-master) $ git grep fe_service
> mikeb@mikeb-desktop:1:~/Impala/docs (asf-master) $
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-5900) document -fe_service_threads
[
https://issues.apache.org/jira/browse/IMPALA-5900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520750#comment-16520750
]
Alex Rodoni commented on IMPALA-5900:
-
https://gerrit.cloudera.org/#/c/10795/
> document -fe_service_threads
>
>
> Key: IMPALA-5900
> URL: https://issues.apache.org/jira/browse/IMPALA-5900
> Project: IMPALA
> Issue Type: Task
> Components: Docs
>Affects Versions: Impala 2.7.0
>Reporter: Michael Brown
>Assignee: Alex Rodoni
>Priority: Major
> Fix For: Impala 3.1.0
>
>
> Fairly often users are confused about the need to bump up
> {{-fe_service_threads}}. The latest example is IMPALA-5897. It could be
> helpful to add documentation about this option. I noticed that there is
> nothing in documentation about it currently:
> {noformat}
> mikeb@mikeb-desktop:0:~/Impala/docs (asf-master) $ git grep fe_service
> mikeb@mikeb-desktop:1:~/Impala/docs (asf-master) $
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7199) Need to have scripts to generate coverage
Joe McDonnell created IMPALA-7199: - Summary: Need to have scripts to generate coverage Key: IMPALA-7199 URL: https://issues.apache.org/jira/browse/IMPALA-7199 Project: IMPALA Issue Type: Bug Components: Infrastructure Affects Versions: Impala 3.1.0 Reporter: Joe McDonnell Assignee: Joe McDonnell Code coverage can be a useful means to verify that tests are exercising the code as expected. It would be useful to have a helper script to make this process as simple as possible. Now that gcovr is pip installable, that is one option for generating coverage reports. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7161) Bootstrap's handling of JAVA_HOME needs improvement
[
https://issues.apache.org/jira/browse/IMPALA-7161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved IMPALA-7161.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
commit 28b4ad14f61bf6f0195b731195767d3a59e92ed8
Author: Joe McDonnell
Date: Tue Jun 12 14:46:58 2018 -0700
IMPALA-7161: Fix impala-config.sh's handling of JAVA_HOME
It is common for developers to specify JAVA_HOME in
bin/impala-config-local.sh, so wait until after it is
sourced to validate JAVA_HOME.
Also, try harder to auto-detect the system's JAVA_HOME
in case it has not been specified in the environment.
Here is a run through of different scenarios:
1. Not set in environment, not set in impala-config-local.sh:
Didn't work before, now tries to autodetect by looking
for javac on the PATH
2. Set in environment, not set in impala-config-local.sh:
No change
3. Not set in environment, set in impala-config-local.sh:
Didn't work before, now works
4. Set in environment and set in impala-config-local.sh:
This used to be potentially inconsistent (i.e. JAVA comes
from the environment's JAVA_HOME, but JAVA_HOME is
overwritten by impala-config-local.sh), now it always
uses the value from impala-config-local.sh.
Change-Id: Idf3521b4f44fdbdc841a90fd00c477c9423a75bb
Reviewed-on: http://gerrit.cloudera.org:8080/10702
Reviewed-by: Philip Zeyliger
Tested-by: Impala Public Jenkins
> Bootstrap's handling of JAVA_HOME needs improvement
> ---
>
> Key: IMPALA-7161
> URL: https://issues.apache.org/jira/browse/IMPALA-7161
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.13.0, Impala 3.1.0
>Reporter: Joe McDonnell
>Assignee: Joe McDonnell
>Priority: Major
> Fix For: Impala 3.1.0
>
>
> bin/bootstrap_system.sh installs the Java SDK and sets JAVA_HOME in the
> current shell. It also adds a command to the bin/impala-config-local.sh to
> export JAVA_HOME there. This doesn't do the job.
> bin/impala-config.sh tests for JAVA_HOME at the very start of the script,
> before it has sourced bin/impala-config-local.sh. So, the user doesn't have a
> way of developing over the long term without manually setting up JAVA_HOME.
> bin/impala-config.sh also doesn't detect the system JAVA_HOME. For Ubuntu
> 16.04, this is fairly simple and if a developer has their system JDK set up
> appropriately, it would make sense to use it. For example:
>
> {noformat}
> # If javac exists, then the system has a Java SDK (JRE does not have javac).
> # Follow the symbolic links and use this to determine the system's JAVA_HOME.
> if [ -L /usr/bin/javac ]; then
> SYSTEM_JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:bin/javac::")
> fi
> export JAVA_HOME="${JAVA_HOME:-${SYSTEM_JAVA_HOME}}"{noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7193) Local filesystem failes with fs.defaultFS (file:/tmp) is not supported
[
https://issues.apache.org/jira/browse/IMPALA-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved IMPALA-7193.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
commit a541ac603912e789cbe0e34f5cb581e2116a94ee
Author: Joe McDonnell
Date: Thu Jun 21 10:56:02 2018 -0700
IMPALA-7193: Fix cluster startup args in create-load-data.sh
The minicluster args for dataload changed to
a bash array in IMPALA-7119, and this requires
a special syntax to derefence and get the whole
array.
This fixes the invocation to use the right
syntax ($BASH_VAR[@] rather than $BASH_VAR).
Change-Id: Ie9a24c0e9fa34e43697b16b48cf219f47f30c0cc
Reviewed-on: http://gerrit.cloudera.org:8080/10782
Reviewed-by: Philip Zeyliger
Tested-by: Impala Public Jenkins
> Local filesystem failes with fs.defaultFS (file:/tmp) is not supported
> --
>
> Key: IMPALA-7193
> URL: https://issues.apache.org/jira/browse/IMPALA-7193
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 3.1.0
>Reporter: Tianyi Wang
>Assignee: Joe McDonnell
>Priority: Critical
> Labels: broken-build
> Fix For: Impala 3.1.0
>
>
> Impala failed to start on a local file system
> {noformat}
> I0619 07:25:34.034348 13157 status.cc:125] Currently configured default
> filesystem: ProxyLocalFileSystem. fs.defaultFS (file:/tmp) is not supported.
> @ 0x18a01f9 impala::Status::Status()
> @ 0x1dd30b6 impala::Frontend::ValidateSettings()
> @ 0x1dee2aa impala::ImpalaServer::ImpalaServer()
> @ 0x1dea8db ImpaladMain()
> @ 0x185c380 main
> @ 0x7f373f829c05 __libc_start_main
> @ 0x185c1f1 (unknown)
> E0619 07:25:34.034366 13157 impala-server.cc:286] Currently configured
> default filesystem: ProxyLocalFileSystem. fs.defaultFS (file:/tmp) is not
> supported.
> E0619 07:25:34.034384 13157 impala-server.cc:289] Aborting Impala Server
> startup due to improper configuration. Impalad exiting.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7119) HBase tests failing with RetriesExhausted and "RuntimeException: couldn't retrieve HBase table"
[
https://issues.apache.org/jira/browse/IMPALA-7119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell resolved IMPALA-7119.
---
Resolution: Fixed
Fix Version/s: Impala 3.1.0
commit 147e962f2dc7507d36cde696640bd76e8821b37c
Author: Joe McDonnell
Date: Fri Jun 8 11:20:42 2018 -0700
IMPALA-7119: Restart whole minicluster when HDFS replication stalls
After loading data, we wait for HDFS to replicate
all of the blocks appropriately. If this takes too long,
we restart HDFS. However, HBase can fail if HDFS is
restarted and HBase is unable to write its logs.
In general, there is no real reason to keep HBase
and the other minicluster components running while
restarting HDFS.
This changes the HDFS health check to restart the
whole minicluster and Impala rather than just HDFS.
Testing:
- Tested with a modified version that always does
the restart in the HDFS health check and verified
that the tests pass
Change-Id: I58ffe301708c78c26ee61aa754a06f46c224c6e2
Reviewed-on: http://gerrit.cloudera.org:8080/10665
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> HBase tests failing with RetriesExhausted and "RuntimeException: couldn't
> retrieve HBase table"
> ---
>
> Key: IMPALA-7119
> URL: https://issues.apache.org/jira/browse/IMPALA-7119
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 2.13.0
>Reporter: Tim Armstrong
>Assignee: Joe McDonnell
>Priority: Major
> Labels: broken-build, flaky
> Fix For: Impala 3.1.0
>
>
> 64820211a2d30238093f1c4cd03bc268e3a01638
> {noformat}
>
> metadata.test_compute_stats.TestHbaseComputeStats.test_hbase_compute_stats_incremental[exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000,
> 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: hbase/none]
>
> metadata.test_compute_stats.TestHbaseComputeStats.test_hbase_compute_stats[exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000,
> 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: hbase/none]
> query_test.test_mt_dop.TestMtDop.test_mt_dop[mt_dop: 1 | exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
> 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: hbase/none]
> query_test.test_mt_dop.TestMtDop.test_compute_stats[mt_dop: 1 |
> exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
> 0} | table_format: hbase/none]
>
> query_test.test_hbase_queries.TestHBaseQueries.test_hbase_scan_node[exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
> 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: hbase/none]
> query_test.test_queries.TestHdfsQueries.test_file_partitions[exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
> 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: hbase/none]
> query_test.test_mt_dop.TestMtDop.test_mt_dop[mt_dop: 0 | exec_option:
> {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
> 'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
> 'exec_single_node_rows_threshold': 0} | table_format: hbase/none]
> query_test.test_observability.TestObservability.test_scan_summary
> query_test.test_mt_dop.TestMtDop.test_compute_stats[mt_dop: 0 |
> exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
> 0} | table_format: hbase/none]
> failure.test_failpoints.TestFailpoints.test_failpoints[table_format:
> hbase/none | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
> 0} | mt_dop: 4 | location: GETNEXT_SCANNER | action: FAIL | query: select 1
> from alltypessmall order by id limit 100]
> failure.test_failpoints.TestFailpoints.test_failpoints[table_format:
> hbase/none | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> 'abort_on_error': 1, 'debug_action': None,
[jira] [Work started] (IMPALA-7198) Impala hints in docs are wrong
[
https://issues.apache.org/jira/browse/IMPALA-7198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-7198 started by Alex Rodoni.
---
> Impala hints in docs are wrong
> --
>
> Key: IMPALA-7198
> URL: https://issues.apache.org/jira/browse/IMPALA-7198
> Project: IMPALA
> Issue Type: Bug
> Components: Docs
>Affects Versions: Impala 3.0
>Reporter: Lars Volker
>Assignee: Alex Rodoni
>Priority: Major
>
> The "CLUSTER" and "NOCLUSTER" hints in this page are spelled wrong. They
> should actually be "CLUSTERED" and "NOCLUSTERED".
> https://impala.apache.org/docs/build/html/topics/impala_hints.html
> {noformat}
> CREATE /* +CLUSTER|NOCLUSTER */
> table_clauses
> AS SELECT remainder_of_query;
> CREATE -- +CLUSTER|NOCLUSTER
> table_clauses
> AS SELECT remainder_of_query;
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7198) Impala hints in docs are wrong
Lars Volker created IMPALA-7198:
---
Summary: Impala hints in docs are wrong
Key: IMPALA-7198
URL: https://issues.apache.org/jira/browse/IMPALA-7198
Project: IMPALA
Issue Type: Bug
Components: Docs
Affects Versions: Impala 3.0
Reporter: Lars Volker
Assignee: Alex Rodoni
The "CLUSTER" and "NOCLUSTER" hints in this page are spelled wrong. They should
actually be "CLUSTERED" and "NOCLUSTERED".
https://impala.apache.org/docs/build/html/topics/impala_hints.html
{noformat}
CREATE /* +CLUSTER|NOCLUSTER */
table_clauses
AS SELECT remainder_of_query;
CREATE -- +CLUSTER|NOCLUSTER
table_clauses
AS SELECT remainder_of_query;
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7187) Fix test_group_impersonation when running inside Docker
[ https://issues.apache.org/jira/browse/IMPALA-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fredy Wijaya resolved IMPALA-7187. -- Resolution: Fixed Fix Version/s: Impala 3.1.0 Impala 2.13.0 > Fix test_group_impersonation when running inside Docker > --- > > Key: IMPALA-7187 > URL: https://issues.apache.org/jira/browse/IMPALA-7187 > Project: IMPALA > Issue Type: Bug > Components: Infrastructure >Affects Versions: Impala 3.0, Impala 2.12.0 >Reporter: Fredy Wijaya >Assignee: Fredy Wijaya >Priority: Major > Labels: broken-build > Fix For: Impala 2.13.0, Impala 3.1.0 > > > test_group_impersonation runs fine inside a standard Docker container, but it > fails when running with test-with-docker.py. There maybe something a bit > special with the way test-with-docker.py runs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7196) Impala is not avoiding footers even after mentioning 'skip.footer.line.count' in ddl
[ https://issues.apache.org/jira/browse/IMPALA-7196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Volker updated IMPALA-7196: Issue Type: New Feature (was: Bug) > Impala is not avoiding footers even after mentioning 'skip.footer.line.count' > in ddl > > > Key: IMPALA-7196 > URL: https://issues.apache.org/jira/browse/IMPALA-7196 > Project: IMPALA > Issue Type: New Feature > Components: Backend >Affects Versions: Impala 2.11.0 >Reporter: Naga Venkata Giridhar >Priority: Major > > we have noticed that skip.footer.line.count is not working in impala and we > are getting different counts in impala and hive -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7196) Impala is not avoiding footers even after mentioning 'skip.footer.line.count' in ddl
[ https://issues.apache.org/jira/browse/IMPALA-7196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Volker updated IMPALA-7196: Issue Type: Bug (was: New Feature) > Impala is not avoiding footers even after mentioning 'skip.footer.line.count' > in ddl > > > Key: IMPALA-7196 > URL: https://issues.apache.org/jira/browse/IMPALA-7196 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.11.0 >Reporter: Naga Venkata Giridhar >Priority: Major > > we have noticed that skip.footer.line.count is not working in impala and we > are getting different counts in impala and hive -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7197) Make process handling during tests more robust
[ https://issues.apache.org/jira/browse/IMPALA-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Csaba Ringhofer updated IMPALA-7197: Description: tests/common/impala_cluster.py contains some helper classes for starting/stopping/listing impala daemons. These work well in the general case, but do not check for things that could go wrong. Some examples: - start() just starts the process and does not wait for it to be actually responsive - this means that a crash/freeze during startup will not be noticed during the startup, but in a later (possible unrelated) test/operation that tries to reach the daemon - kill() does not wait for the process to terminate, which may mislead tests that list/count Impala daemons (this is a possible cause of IMPALA-7175) - kill() does not check if the PID it kills is actually an Impala daemon - the process may have crashed and the OS may have given the same PID to a new process (this is very unlikely, but the results are potentially really weird) It is probably not possible to make these functions completely synchronous, but reducing the chance for issues can save us from some long investigations in the future. Note that the impala processes are detached from the test process, so no notification is sent if they are terminated. was: tests/common/impala_cluster.py contains some helper classes for starting/stopping/listing impala daemons. These work well in the general case, but do not check for things that could go wrong. Some examples: - start() just starts the process and does not wait for it to be actually responsive - this means that a crash/freeze during startup will not be noticed during the startup, but in a later (possible unrelated) test/operation that tries to reach the daemon - kill() does not wait for the process to terminate, which may mislead tests that list/count Impala daemons (this is a possible cause of IMPALA-7175) - kill() does not check if the PID it kills is actually an Impala daemon - the process may have crashed and the OS may have given the same PID to a new process (this is very unlikely, but the results are potentially really weird) It is probably not possible to make these functions completely synchronous, but reducing the chance for issues can save us from some long investigations in the future. Note that the impala processes are detached from the test the process, so no notification is sent if they are terminated. > Make process handling during tests more robust > -- > > Key: IMPALA-7197 > URL: https://issues.apache.org/jira/browse/IMPALA-7197 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Reporter: Csaba Ringhofer >Priority: Major > Labels: ramp-up > > tests/common/impala_cluster.py contains some helper classes for > starting/stopping/listing impala daemons. These work well in the general > case, but do not check for things that could go wrong. Some examples: > - start() just starts the process and does not wait for it to be actually > responsive - this means that a crash/freeze during startup will not be > noticed during the startup, but in a later (possible unrelated) > test/operation that tries to reach the daemon > - kill() does not wait for the process to terminate, which may mislead tests > that list/count Impala daemons (this is a possible cause of IMPALA-7175) > - kill() does not check if the PID it kills is actually an Impala daemon - > the process may have crashed and the OS may have given the same PID to a new > process (this is very unlikely, but the results are potentially really weird) > It is probably not possible to make these functions completely synchronous, > but reducing the chance for issues can save us from some long investigations > in the future. > Note that the impala processes are detached from the test process, so no > notification is sent if they are terminated. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7197) Make process handling during tests more robust
[ https://issues.apache.org/jira/browse/IMPALA-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Csaba Ringhofer updated IMPALA-7197: Labels: ramp-up (was: ) > Make process handling during tests more robust > -- > > Key: IMPALA-7197 > URL: https://issues.apache.org/jira/browse/IMPALA-7197 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Reporter: Csaba Ringhofer >Priority: Major > Labels: ramp-up > > tests/common/impala_cluster.py contains some helper classes for > starting/stopping/listing impala daemons. These work well in the general > case, but do not check for things that could go wrong. Some examples: > - start() just starts the process and does not wait for it to be actually > responsive - this means that a crash/freeze during startup will not be > noticed during the startup, but in a later (possible unrelated) > test/operation that tries to reach the daemon > - kill() does not wait for the process to terminate, which may mislead tests > that list/count Impala daemons (this is a possible cause of IMPALA-7175) > - kill() does not check if the PID it kills is actually an Impala daemon - > the process may have crashed and the OS may have given the same PID to a new > process (this is very unlikely, but the results are potentially really weird) > It is probably not possible to make these functions completely synchronous, > but reducing the chance for issues can save us from some long investigations > in the future. > Note that the impala processes are detached from the test the process, so no > notification is sent if they are terminated. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7197) Make process handling during tests more robust
[ https://issues.apache.org/jira/browse/IMPALA-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Csaba Ringhofer updated IMPALA-7197: Description: tests/common/impala_cluster.py contains some helper classes for starting/stopping/listing impala daemons. These work well in the general case, but do not check for things that could go wrong. Some examples: - start() just starts the process and does not wait for it to be actually responsive - this means that a crash/freeze during startup will not be noticed during the startup, but in a later (possible unrelated) test/operation that tries to reach the daemon - kill() does not wait for the process to terminate, which may mislead tests that list/count Impala daemons (this is a possible cause of IMPALA-7175) - kill() does not check if the PID it kills is actually an Impala daemon - the process may have crashed and the OS may have given the same PID to a new process (this is very unlikely, but the results are potentially really weird) It is probably not possible to make these functions completely synchronous, but reducing the chance for issues can save us from some long investigations in the future. Note that the impala processes are detached from the test the process, so no notification is sent if they are terminated. was: tests/common/impala_cluster.py contains some helper classes for starting/stopping/listing impala daemons. These work well in the general case, but do not check for things that could go wrong. Some examples: -start() just starts the process and does not wait for it to be actually responsive - this means that a crash/freeze during startup will not be noticed during the startup, but in a later (possible unrelated) test/operation that tries to reach the daemon - kill() does not wait for the process to terminate, which may mislead tests that list/count Impala daemons (this is a possible cause of IMPALA-7175) - kill() does not check if the PID it kills is actually an Impala daemon - the process may have crashed and the OS may have given the same PID to a new process (this is very unlikely, but the results are potentially really weird) It is probably not possible to make these functions completely synchronous, but reducing the chance for issues can save us from some long investigations in the future. Note that the impala processes are detached from the test the process, so no notification is sent if they are terminated. > Make process handling during tests more robust > -- > > Key: IMPALA-7197 > URL: https://issues.apache.org/jira/browse/IMPALA-7197 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Reporter: Csaba Ringhofer >Priority: Major > > tests/common/impala_cluster.py contains some helper classes for > starting/stopping/listing impala daemons. These work well in the general > case, but do not check for things that could go wrong. Some examples: > - start() just starts the process and does not wait for it to be actually > responsive - this means that a crash/freeze during startup will not be > noticed during the startup, but in a later (possible unrelated) > test/operation that tries to reach the daemon > - kill() does not wait for the process to terminate, which may mislead tests > that list/count Impala daemons (this is a possible cause of IMPALA-7175) > - kill() does not check if the PID it kills is actually an Impala daemon - > the process may have crashed and the OS may have given the same PID to a new > process (this is very unlikely, but the results are potentially really weird) > It is probably not possible to make these functions completely synchronous, > but reducing the chance for issues can save us from some long investigations > in the future. > Note that the impala processes are detached from the test the process, so no > notification is sent if they are terminated. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7197) Make process handling during tests more robust
Csaba Ringhofer created IMPALA-7197: --- Summary: Make process handling during tests more robust Key: IMPALA-7197 URL: https://issues.apache.org/jira/browse/IMPALA-7197 Project: IMPALA Issue Type: Improvement Components: Infrastructure Reporter: Csaba Ringhofer tests/common/impala_cluster.py contains some helper classes for starting/stopping/listing impala daemons. These work well in the general case, but do not check for things that could go wrong. Some examples: -start() just starts the process and does not wait for it to be actually responsive - this means that a crash/freeze during startup will not be noticed during the startup, but in a later (possible unrelated) test/operation that tries to reach the daemon - kill() does not wait for the process to terminate, which may mislead tests that list/count Impala daemons (this is a possible cause of IMPALA-7175) - kill() does not check if the PID it kills is actually an Impala daemon - the process may have crashed and the OS may have given the same PID to a new process (this is very unlikely, but the results are potentially really weird) It is probably not possible to make these functions completely synchronous, but reducing the chance for issues can save us from some long investigations in the future. Note that the impala processes are detached from the test the process, so no notification is sent if they are terminated. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-7189) Impala save as Kudu, Column Type is TimeStamp ,Time is always eight hours short
[
https://issues.apache.org/jira/browse/IMPALA-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520036#comment-16520036
]
Xu Zhang edited comment on IMPALA-7189 at 6/22/18 8:43 AM:
---
[~twmarshall]
Thanks！I’ll try as you say 。
I have called _operation and maintenance department ,the machine‘s timezone is
set to CST ._
_Product env and Development env are same ._
_Finally , I try to save this data as int type in impala .I will convert int
to timestamp before i querying them ._
_In this case , i need test it's performance ._
__
was (Author: xu.zhang5):
[~twmarshall]
Thanks！I’ll try as you say 。
I have called _operation and maintenance department ,the product environment
machine‘s timezone is set to CST ._
_Product env and Development env are same ._
_Finally , I try to save this data as int type in impala .I will convert int
to timestamp before i querying them ._
_In this case , i need test it's performance ._
__
> Impala save as Kudu, Column Type is TimeStamp ,Time is always eight hours
> short
> ---
>
> Key: IMPALA-7189
> URL: https://issues.apache.org/jira/browse/IMPALA-7189
> Project: IMPALA
> Issue Type: Bug
> Components: Clients
>Affects Versions: Impala 2.11.0
> Environment: CDH 5.14.2
> HUE 4.0
>Reporter: Xu Zhang
>Priority: Major
> Attachments: image-2018-06-21-09-48-33-924.png
>
>
> Data in Kudu like this:
> login_time: 146567226800
> user_id : 171734
>
> login_time eq 2016-06-12 03:11:08
>
> Select this data in impala :
> login_time: 2016-06-11 19:11:08
> user_id : 171734
>
>
>
> How did i save the datas :
> code
> start-
> Dataset
> consignorRows=sparkSession.createDataFrame(consignorUserActionJavaRDD,
> ConsignorUserAction.class);
>
> kuduContext.upsertRows(consignorRows, consignorKuduTableName);
> code
> end-
>
>
> ConsignorUserAction has four properties ,include login_time as Timestamp
> I use debug console and trace to the final step , KuduContext 《
> case DataTypes.TimestampType => operation.getRow.addLong(kuduIdx,
> KuduRelation.timestampToMicros(row.getTimestamp(sparkIdx)))
> 》 ,Before commit this data, The "login_time" is alway right !
>
> Then,Use Kudu Scan :
> --Code
> start---
> KuduClient client = new
> KuduClient.KuduClientBuilder("avatartest:7051").build();
> try {
> KuduTable table = client.openTable("impala::default.consignor_action");
> List projectColumns = new ArrayList<>(1);
> projectColumns.add("login_time");
> KuduScanner scanner = client.newScannerBuilder(table)
> .setProjectedColumnNames(projectColumns)
> .build();
> while (scanner.hasMoreRows()) {
> RowResultIterator results = scanner.nextRows();
> while (results.hasNext())
> { RowResult result = results.next(); System.out.println(result.getLong(0)); }
> }
>
> --Code
> end---
> I got this: 146567226800,That's right .
>
>
> HUE QUERY PAGE:
>
>
> !image-2018-06-21-09-48-33-924.png!
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
