[jira] [Created] (IMPALA-7968) Convert Analyzer AST to JSON for test verification
Paul Rogers created IMPALA-7968:
---
Summary: Convert Analyzer AST to JSON for test verification
Key: IMPALA-7968
URL: https://issues.apache.org/jira/browse/IMPALA-7968
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Affects Versions: Impala 3.1.0
Reporter: Paul Rogers
Assignee: Paul Rogers
The FE code provides the {{PlannerTest}} which parses a SQL statement, analyzes
it, plans the query, and writes selected portions of the plan to a file to be
compared with a "golden" copy. Very helpful.
At present, the Analyzer itself has no such test framework. This ticket asks to
provide one. Key pieces:
* JSON serializer to write the AST to JSON in a compact format for testing.
* Potentially a YAML form of the above if the JSON grows too bulky
* Serialization code in the AST objects to produce the JSON
* Additions to the PlannerTest framework to add an AST section and an AST test
framework
* The actual AST tests
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7967) Incorrect decimal size in V2 for a numeric const cast to BIGINT
Paul Rogers created IMPALA-7967:
---
Summary: Incorrect decimal size in V2 for a numeric const cast to
BIGINT
Key: IMPALA-7967
URL: https://issues.apache.org/jira/browse/IMPALA-7967
Project: IMPALA
Issue Type: Bug
Components: Frontend
Affects Versions: Impala 3.1.0
Reporter: Paul Rogers
Assignee: Paul Rogers
Decimal version 2 introduces revised rules for computing decimal width. For
example:
{noformat}
CAST(1 AS DECIMAL(10, 0)) + CAST(2 AS DECIMAL(19,0)) --> DECIMAL(20,0)
{noformat}
The FE uses rules to convert from one type to another. The rule to convert from
{{BIGINT}} to {{DECIMAL}} is:
{noformat}
BIGINT --> DECIMAL(19,0)
{noformat}
Put these two together:
{noformat}
CAST(1 AS DECIMAL(10, 0)) + CAST(2 AS BIGINT)
{noformat}
The result should be {{DECIMAL(20,0)}}. But, because of a bug in the way
constant folding works, the result is actually {{DECIMAL(11,0)}} as seen in
{{AnalyzeExprsTest.TestDecimalArithmetic()}}:
{code:java}
testDecimalExpr(decimal_10_0 + " + cast(1 as bigint)",
ScalarType.createDecimalType(11, 0));
{code}
It seems one reason the bug was not caught is that the unit tests only check
for constants, not for columns. Modify the tests to work against
{{functional.alltypes}} (they currently work without a table), and substitute
{{bigint_col}} for {{CAST(2 AS BIGINT)}}.
Expected that the code should have followed the rules described above whether
the values are a column, or a constant explicitly cast to the same type as the
column. (Constants without out a cast should follow rules for their "natural
type" which is what appears to be incorrectly happening in the test case above.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7966) Analyzer does not follow Decimal V1 type propagation rules
Paul Rogers created IMPALA-7966:
---
Summary: Analyzer does not follow Decimal V1 type propagation rules
Key: IMPALA-7966
URL: https://issues.apache.org/jira/browse/IMPALA-7966
Project: IMPALA
Issue Type: Bug
Components: Frontend
Affects Versions: Impala 3.1.0
Reporter: Paul Rogers
Assignee: Paul Rogers
Consider {{AnalyzeExprsTest.TestDecimalArithmetic()}}:
{code:java}
// Operators between decimal and numeric types should be supported. The int
// should be cast to the appropriate decimal (e.g. tinyint -> decimal(3,0)).
testDecimalExpr(decimal_10_0 + " + cast(1 as tinyint)",
ScalarType.createDecimalType(11, 0));
testDecimalExpr(decimal_10_0 + " + cast(1 as smallint)",
ScalarType.createDecimalType(11, 0));
testDecimalExpr(decimal_10_0 + " + cast(1 as int)",
ScalarType.createDecimalType(11, 0));
testDecimalExpr(decimal_10_0 + " + cast(1 as bigint)",
ScalarType.createDecimalType(20, 0));
testDecimalExpr(decimal_10_0 + " + cast(1 as float)",
ScalarType.createDecimalType(38, 9), ScalarType.createDecimalType(38,
8));
testDecimalExpr(decimal_10_0 + " + cast(1 as double)",
ScalarType.createDecimalType(38, 17), ScalarType.createDecimalType(38,
16));
{code}
This is rather cryptic: it means that the given expression should evaluate, in
both Decimal V1 and V2, to the given type. The last two tests have different
types for V1 and V2.
However, the above should follow the V1 rules as explained in
{{Expr.convertNumericLiteralsFromDecimal()}}:
{noformat}
* This function applies a heuristic that casts literal child exprs of this
expr from
* decimal to floating point in certain circumstances to reduce processing
cost. In
* earlier versions of Impala's decimal support, it was much slower than
floating point
* arithmetic.
{noformat}
There appears to be some bug that prevents the above from being applied. Once
the rules are applied (though a change elsewhere that somehow enabled this
path), the tests above all produce {{DOUBLE}} types. That is, the test should
be:
{code:java}
testDecimalExpr(decimal_10_0 + " + cast(1 as tinyint)",
Type.DOUBLE, ScalarType.createDecimalType(11, 0));
testDecimalExpr(decimal_10_0 + " + cast(1 as smallint)",
Type.DOUBLE, ScalarType.createDecimalType(11, 0));
testDecimalExpr(decimal_10_0 + " + cast(1 as int)",
Type.DOUBLE, ScalarType.createDecimalType(11, 0));
testDecimalExpr(decimal_10_0 + " + cast(1 as bigint)",
Type.DOUBLE, ScalarType.createDecimalType(11, 0));
testDecimalExpr(decimal_10_0 + " + cast(1 as float)",
Type.DOUBLE, ScalarType.createDecimalType(38, 8));
testDecimalExpr(decimal_10_0 + " + cast(1 as double)",
Type.DOUBLE, ScalarType.createDecimalType(38, 16));
{code}
The issue is marked as major because it may affect customer queries that
expected the incorrect V1 behavior.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-7534) Handle invalidation races in CatalogdMetaProvider cache
[ https://issues.apache.org/jira/browse/IMPALA-7534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrian Ng reassigned IMPALA-7534: - Assignee: Paul Rogers > Handle invalidation races in CatalogdMetaProvider cache > --- > > Key: IMPALA-7534 > URL: https://issues.apache.org/jira/browse/IMPALA-7534 > Project: IMPALA > Issue Type: Sub-task >Reporter: Todd Lipcon >Assignee: Paul Rogers >Priority: Major > > There is a well-known race in Guava's LoadingCache that we are using for > CatalogdMetaProvider which we are not currently handling: > - thread 1 gets a cache miss and makes a request to fetch some data from the > catalogd. It fetches the catalog object with version 1 and then gets context > switched out or otherwise slow > - thread 2 receives an invalidation for the same object, because it has > changed to v2. It calls 'invalidate' on the cache, but nothing is yet cached. > - thread 1 puts back v1 of the object into the cache > In essence we've "missed" an invalidation. This is also described in this > nice post: https://softwaremill.com/race-condition-cache-guava-caffeine/ > The race is quite unlikely but could cause some unexpected results that are > hard to reason about, so we should look into a fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7965) FE should use "permissive" constant folding evaluation
Paul Rogers created IMPALA-7965:
---
Summary: FE should use "permissive" constant folding evaluation
Key: IMPALA-7965
URL: https://issues.apache.org/jira/browse/IMPALA-7965
Project: IMPALA
Issue Type: Bug
Components: Frontend
Affects Versions: Impala 3.1.0
Reporter: Paul Rogers
Assignee: Paul Rogers
The FE implements "constant folding" the conversion of constant expressions
into constant literals. For example, {{1 + 1}} is constant-folded to {{2}}.
Subtle cases occur when the BE cannot evaluate the expression. For example:
* DECIMAL-value overflow or underflow e.g. {{CAST(10 AS DECIMAL(1,0))}}
* Illegal casts e.g. {{CAST('foo' AS TIMESTAMP)}}
* Result strings that contain non-ASCII characters e.g. {{hex('D3')}}
In this case, the FE does not fail the query, it just leaves the original SQL
and let's the BE fail the query at runtime. (Doing so seems more of a bug than
a feature, but let's set that issue aside at present.)
There are cases in which the error is benign. Consider this case from
{{ExprRewriteRulesTest.TestFoldConstantsRule()}}:
Note that a side rule is that the result of constant folding should be the
result of the BE executing the entire expression. From
{{ExprRewriteRulesTest.TestFoldConstantsRule()}}:
{code:java}
// Tests correct handling of strings with chars > 127. Should not be folded.
verifyRewrite("hex(unhex(hex(unhex('D3'", null);
{code}
This test says that the following SQL expression cannot be evaluated on the BE:
{code:sql}
hex(unhex(hex(unhex('D3'
{code}
But, the BE can happily evaluate the whole thing, and produce a result of
{{'D3'}}. What fails is:
{code:sql}
unhex('D3')
{code}
But not:
{code:sql}
hex(unhex('D3'))
{code}
Similar reasoning applies to:
{code:sql}
FALSE AND CAST(20 AS DECIMAL(1,0))
{code}
In the present code, the entire test expression fails. But, under the, "let the
BE decide" rule, the expression:
{code:sql}
hex(unhex(hex(unhex('D3'
{code}
Can be rewritten to:
{code:sql}
hex('D3')
{code}
The inner three functions are evaluated properly on the BE. Only the outer
function fails and is preserved in the AST.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-3600) Add missing admission control test coverage
[ https://issues.apache.org/jira/browse/IMPALA-3600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikramjeet Vig resolved IMPALA-3600. Resolution: Fixed > Add missing admission control test coverage > --- > > Key: IMPALA-3600 > URL: https://issues.apache.org/jira/browse/IMPALA-3600 > Project: IMPALA > Issue Type: Task > Components: Backend >Affects Versions: Impala 2.5.0 >Reporter: Matthew Jacobs >Assignee: Bikramjeet Vig >Priority: Major > Labels: admission-control > > Some test debt from 2.5: > * -Test case setting mem limit high enough for accounting to overflow- > * -Add test for require_username- > * -Submit queries with every kind of rejection- > * Changing pool cfg while running > * Verify profile string for every kind of queued reason -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7696) llvm crash in custom cluster test
[
https://issues.apache.org/jira/browse/IMPALA-7696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bikramjeet Vig resolved IMPALA-7696.
Resolution: Cannot Reproduce
I am closing this based on the lack of relevant logging, Tim's observation and
the rarity of this symptom.
> llvm crash in custom cluster test
> -
>
> Key: IMPALA-7696
> URL: https://issues.apache.org/jira/browse/IMPALA-7696
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.1.0
>Reporter: Michael Brown
>Assignee: Bikramjeet Vig
>Priority: Critical
> Labels: crash
> Attachments: 07015206-bbc7-414c-b9e5f0a6-3f8194e3.dmp_dumped
>
>
> [~bikramjeet.vig] please take a look at this crash that happened on GVO.
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/3346/
> {noformat}
> Crash reason: SIGSEGV
> Crash address: 0x0
> Process uptime: not available
> Thread 202 (crashed)
> 0 libc-2.23.so + 0x16efe5
> rax = 0x rdx = 0x0010
> rcx = 0x7fab7869cfe5 rbx = 0x
> rsi = 0x046edc91 rdi = 0x0010
> rbp = 0x0665d290 rsp = 0x7faae72d5468
> r8 = 0x0081r9 = 0x0a92f5c0
> r10 = 0x0001 r11 = 0x7fab786c2f90
> r12 = 0x0010 r13 = 0x0086
> r14 = 0x046edc81 r15 = 0x07731b00
> rip = 0x7fab7869cfe5
> Found by: given as instruction pointer in context
> 1 impalad!llvm::cl::generic_parser_base::findOption(llvm::StringRef) + 0x54
> rsp = 0x7faae72d5470 rip = 0x04392784
> Found by: stack scanning
> 2 impalad!llvm::PassNameParser::passRegistered(llvm::PassInfo const*) + 0x67
> rbx = 0x0665d288 rbp = 0x0af15250
> rsp = 0x7faae72d54a0 r12 = 0x046edc81
> r13 = 0x0432d430 r14 = 0x0665d290
> rip = 0x0432e817
> Found by: call frame info
> 3 impalad!llvm::PassRegistry::registerPass(llvm::PassInfo const&, bool) +
> 0x342
> rbx = 0x0af15250 rbp = 0x0777d720
> rsp = 0x7faae72d54f0 r12 = 0x0777d720
> r13 = 0x0432d430 r14 = 0x0010
> r15 = 0x07731b00 rip = 0x043347f2
> Found by: call frame info
> 4 impalad!initializeMachineCombinerPassOnce(llvm::PassRegistry&) + 0x92
> rbx = 0x0af15250 rbp = 0x07731b00
> rsp = 0x7faae72d5560 r12 = 0x7faae72d5590
> r13 = 0x r14 = 0x
> r15 = 0x rip = 0x039f9a62
> Found by: call frame info
> 5 libpthread-2.23.so + 0xea99
> rbx = 0x0664773c rbp = 0x7fab790ebac0
> rsp = 0x7faae72d5580 r12 = 0x7faae72d5590
> r13 = 0x r14 = 0x
> r15 = 0x rip = 0x7fab78906a99
> Found by: call frame info
> 6 impalad!llvm::APInt::operator[](unsigned int) const [APInt.h : 1098 +
> 0x1e]
> rbp = 0x7fab790ebac0 rsp = 0x7faae72d5590
> rip = 0x02157216
> Found by: stack scanning
> 7 libpthread-2.23.so + 0xeae0
> rbp = 0x7fab790ebac0 rsp = 0x7faae72d5598
> rip = 0x7fab78906ae0
> Found by: stack scanning
> 8 impalad!llvm::initializeMachineCombinerPass(llvm::PassRegistry&) + 0x5e
> rbp = 0x7fab790ebac0 rsp = 0x7faae72d55d0
> rip = 0x039fa06e
> Found by: stack scanning
> 9 impalad!llvm::initializeCodeGen(llvm::PassRegistry&) + 0x109
> rbp = 0x7fab790ebac0 rsp = 0x7faae72d55f0
> rip = 0x03ad33f9
> Found by: call frame info
> 10
> impalad!llvm::TargetPassConfig::TargetPassConfig(llvm::LLVMTargetMachine&,
> llvm::legacy::PassManagerBase&) + 0xc2
> rbx = 0x0af21720 rbp = 0x7fab790ebac0
> rsp = 0x7faae72d5600 rip = 0x03aa3432
> Found by: call frame info
> 11
> impalad!llvm::X86TargetMachine::createPassConfig(llvm::legacy::PassManagerBase&)
> + 0x25
> rbx = 0x0af21720 rbp = 0x0aef5380
> rsp = 0x7faae72d5620 rip = 0x03546395
> Found by: call frame info
> 12 impalad!addPassesToGenerateCode(llvm::LLVMTargetMachine*,
> llvm::legacy::PassManagerBase&, bool, void const*, void const*, void const*,
> void const*) + 0x28
> rbx = 0x0bc8c268 rbp = 0x7faae72d57a0
> rsp = 0x7faae72d5640 r12 = 0x0aef5380
> rip = 0x039e0438
> Found by: call frame info
> 13
> impalad!llvm::LLVMTargetMachine::addPassesToEmitMC(llvm::legacy::PassManagerBase&,
> llvm::MCContext*&, llvm::raw_pwrite_stream&, bool) + 0x37
> rbx = 0x0bc8c268 rbp = 0x7faae72d57a0
[jira] [Created] (IMPALA-7964) Inconsistent typing of pure-constant expressions
Paul Rogers created IMPALA-7964:
---
Summary: Inconsistent typing of pure-constant expressions
Key: IMPALA-7964
URL: https://issues.apache.org/jira/browse/IMPALA-7964
Project: IMPALA
Issue Type: Bug
Components: Frontend
Affects Versions: Impala 3.1.0
Reporter: Paul Rogers
Assignee: Paul Rogers
Impala implements a very subtle rule for computing the type of constant
expressions. See IMPALA-4213. The desired rules when performing constant
folding are:
* Pure-literal numeric expressions take the natural (smallest) type of the
result.
* All other expressions preserve the type of the expression.
Examples:
{noformat}
-- simple literal
100 -> 100:TINYINT
-- explicit cast
CAST(100 AS SMALLINT) --> 100:SMALLINT
-- pure literal expression
2 + 3 --> 5:TINYINT
-- addition is done at next wider type up from that of tinyint_col
tinyint_col + 3 --> SMALLINT
-- explicit type given, math done with that type, mimics above
CAST(2 AS TINYINT) + 3 --> 5:SMALLINT
-- Impala uses C++ semantics, with overflow
CAST(257 AS TINYINT) --> 2:TINYINT
{noformat}
At present, some parts of the code follow these rules, but the constant folding
rule does not. As a result, {{PlannerTest}} fails for subtle changes in
unrelated areas. Some {{PlannerTest}} results, such as the Kudu test mentioned
in IMPALA-4213 do observe the rules. But, some constant folding test *do not*
observe the rules.
Expected that the rules would be followed on all code paths to ensure
consistent results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7963) test_empty_build_joins failed with hdfs timeout
Thomas Tauber-Marshall created IMPALA-7963:
--
Summary: test_empty_build_joins failed with hdfs timeout
Key: IMPALA-7963
URL: https://issues.apache.org/jira/browse/IMPALA-7963
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 3.2.0
Reporter: Thomas Tauber-Marshall
Assignee: Joe McDonnell
Seen in an exhaustive build on centos6:
{noformat}
05:39:09 TestJoinQueries.test_empty_build_joins[batch_size: 1 | protocol:
beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold':
0} | table_format: parquet/none]
05:39:09 [gw3] linux2 -- Python 2.6.6
/data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/bin/../infra/python/env/bin/python
05:39:09 query_test/test_join_queries.py:97: in test_empty_build_joins
05:39:09 self.run_test_case('QueryTest/empty-build-joins', new_vector)
05:39:09 common/impala_test_suite.py:467: in run_test_case
05:39:09 result = self.__execute_query(target_impalad_client, query,
user=user)
05:39:09 common/impala_test_suite.py:688: in __execute_query
05:39:09 return impalad_client.execute(query, user=user)
05:39:09 common/impala_connection.py:170: in execute
05:39:09 return self.__beeswax_client.execute(sql_stmt, user=user)
05:39:09 beeswax/impala_beeswax.py:182: in execute
05:39:09 handle = self.__execute_query(query_string.strip(), user=user)
05:39:09 beeswax/impala_beeswax.py:359: in __execute_query
05:39:09 self.wait_for_finished(handle)
05:39:09 beeswax/impala_beeswax.py:380: in wait_for_finished
05:39:09 raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
05:39:09 E ImpalaBeeswaxException: ImpalaBeeswaxException:
05:39:09 EQuery aborted:hdfsOpenFile() for
hdfs://localhost:20500/test-warehouse/alltypestiny/year=2009/month=2/090201.txt
failed to finish before the 300 second timeout
05:39:09 - Captured stderr call
-
05:39:09 -- executing against localhost:21000
05:39:09 use functional_parquet;
05:39:09
05:39:09 -- 2018-12-11 03:11:34,797 INFO MainThread: Started query
d747763f9d663cd7:9abd4b99
05:39:09 SET batch_size=1;
05:39:09 SET num_nodes=0;
05:39:09 SET disable_codegen_rows_threshold=0;
05:39:09 SET disable_codegen=False;
05:39:09 SET abort_on_error=1;
05:39:09 SET exec_single_node_rows_threshold=0;
05:39:09 -- executing against localhost:21000
05:39:09 select straight_join atp.id
05:39:09 from alltypes atp
05:39:09 inner join functional.alltypestiny att on atp.id = att.id
05:39:09 where att.int_col = 999;
05:39:09
05:39:09 -- 2018-12-11 03:11:34,816 INFO MainThread: Started query
5045de8553c5843c:bdc6aa1c
05:39:09 -- executing against localhost:21000
05:39:09 select straight_join atp.id
05:39:09 from alltypes atp
05:39:09 right join functional.alltypestiny att on atp.id = att.id
05:39:09 where att.int_col = 999;
05:39:09
05:39:09 -- 2018-12-11 03:11:35,519 INFO MainThread: Started query
124ef451a3f65d09:f2ae4a5d
{noformat}
Presumably caused by IMPALA-7738
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7962) Figure out how to handle localtime in docker minicluster containers
[
https://issues.apache.org/jira/browse/IMPALA-7962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719294#comment-16719294
]
Philip Zeyliger commented on IMPALA-7962:
-
See the {{test-with-docker.py}} source code (grep for {{localtime}}) for one
approach. There are some "interesting" behaviors here.
> Figure out how to handle localtime in docker minicluster containers
> ---
>
> Key: IMPALA-7962
> URL: https://issues.apache.org/jira/browse/IMPALA-7962
> Project: IMPALA
> Issue Type: Sub-task
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
>
> The timezone from the host is not inherited by the container - it is instead
> mapped to /usr/share/zoneinfo/Etc/UTC
> We should figure out if we need to fix this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Reopened] (IMPALA-5929) Remove useless explicit casts to string
[
https://issues.apache.org/jira/browse/IMPALA-5929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bikramjeet Vig reopened IMPALA-5929:
> Remove useless explicit casts to string
> ---
>
> Key: IMPALA-5929
> URL: https://issues.apache.org/jira/browse/IMPALA-5929
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 2.5.0, Impala 2.6.0, Impala 2.7.0, Impala 2.8.0,
> Impala 2.9.0, Impala 2.10.0
>Reporter: Alexander Behm
>Assignee: Bikramjeet Vig
>Priority: Major
> Labels: newbie
> Fix For: Impala 2.12.0
>
>
> Some BI tools like to generate expressions that look like this:
> {code}
> cast(numeric_col as string) = '123456'
> {code}
> Casting and comparing as string is expensive and we should convert such
> expressions to:
> {code}
> numeric_col = 123456
> {code}
> Such transformations may be applicable in other situations as well. We should
> be careful about transforming inequality predicates because the string and
> numeric comparison might not always be equivalent. For example, the following
> transformation would be wrong:
> {code}
> cast(numeric_col as string) = '0123456'
> {code}
> is *not* equivalent to
> {code}
> numeric_col = 123456
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-7960) wrong results when comparing timestamp casted to varchar of smaller length to a string literal in a binary predicate
[
https://issues.apache.org/jira/browse/IMPALA-7960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bikramjeet Vig resolved IMPALA-7960.
Resolution: Fixed
Fix Version/s: Impala 3.2.0
> wrong results when comparing timestamp casted to varchar of smaller length to
> a string literal in a binary predicate
>
>
> Key: IMPALA-7960
> URL: https://issues.apache.org/jira/browse/IMPALA-7960
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 2.12.0
>Reporter: Bikramjeet Vig
>Assignee: Bikramjeet Vig
>Priority: Blocker
> Labels: correctness
> Fix For: Impala 3.2.0
>
>
> Expression rewriting seems to identify this as a redundant cast and removes
> it.
> Steps to re-create:
> {noformat}
> select * from (select cast('2018-12-11 09:59:37' as timestamp) as ts) tbl
> where cast(ts as varchar(10)) = '2018-12-11';
> {noformat}
> output:
> {noformat}
> Fetched 0 row(s)
> {noformat}
> Now disable expression re-writes.
> {noformat}
> set ENABLE_EXPR_REWRITES=false;
> select * from (select cast('2018-12-11 09:59:37' as timestamp) as ts) tbl
> where cast(ts as varchar(10)) = '2018-12-11';
> {noformat}
> output:
> {noformat}
> +-+
> | ts |
> +-+
> | 2018-12-11 09:59:37 |
> +-+
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7960) wrong results when comparing timestamp casted to varchar of smaller length to a string literal in a binary predicate
[
https://issues.apache.org/jira/browse/IMPALA-7960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719240#comment-16719240
]
ASF subversion and git services commented on IMPALA-7960:
-
Commit 899bfd10a2a42d40634cdb8c9c670494b3c8945e in impala's branch
refs/heads/master from [~bikram.sngh91]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=899bfd1 ]
IMPALA-7960: Revert "IMPALA-5929: Remove redundant explicit casts to string"
The fix for IMPALA-5929 introduced a bug that produced wrong results.
This bug is detailed in IMPALA-7960. Reverting for now.
This reverts commit 545163bb0a5c86aa02652d0557871f5b694a6c82.
Change-Id: I6f0da62a7ff86f05859a2acbec13a726a9bd6f4c
Reviewed-on: http://gerrit.cloudera.org:8080/12073
Reviewed-by: Zoram Thanga
Tested-by: Impala Public Jenkins
> wrong results when comparing timestamp casted to varchar of smaller length to
> a string literal in a binary predicate
>
>
> Key: IMPALA-7960
> URL: https://issues.apache.org/jira/browse/IMPALA-7960
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 2.12.0
>Reporter: Bikramjeet Vig
>Assignee: Bikramjeet Vig
>Priority: Blocker
> Labels: correctness
>
> Expression rewriting seems to identify this as a redundant cast and removes
> it.
> Steps to re-create:
> {noformat}
> select * from (select cast('2018-12-11 09:59:37' as timestamp) as ts) tbl
> where cast(ts as varchar(10)) = '2018-12-11';
> {noformat}
> output:
> {noformat}
> Fetched 0 row(s)
> {noformat}
> Now disable expression re-writes.
> {noformat}
> set ENABLE_EXPR_REWRITES=false;
> select * from (select cast('2018-12-11 09:59:37' as timestamp) as ts) tbl
> where cast(ts as varchar(10)) = '2018-12-11';
> {noformat}
> output:
> {noformat}
> +-+
> | ts |
> +-+
> | 2018-12-11 09:59:37 |
> +-+
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7907) Multiple toSql() bugs in ScalarFunction
[
https://issues.apache.org/jira/browse/IMPALA-7907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719232#comment-16719232
]
ASF subversion and git services commented on IMPALA-7907:
-
Commit d94a474658cb5f6114bc3cfb9a5426bdf1610e11 in impala's branch
refs/heads/master from [~paul-rogers]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=d94a474 ]
IMPALA-7907: Fix ScalarFunction.toSql()
Each AST statement node has a toSql() implementation. The code for
ScalarFunction and ToSqlUtils has a number of issues:
* If Location or Symbol are not set, they are shown as 'null'. Better to
omit these clauses if the items are not available. This is mostly an
issue during testing.
* The generated SQL does not follow the CREATE TABLE syntax. For
example, the signature and return value are provided for Java
functions, but should not be.
* Unlike other statements, this one is generated with a trailing newline.
* ToSql.getCreateFunctionSql() fails to separate functions with the
semi-colon statement separator.
These are all minor issues, but we might as well fix the code to work as
intended.
Testing:
* Added a new unit tests to verify the behavior (no tested existed
previously.)
* Re-ran all FE tests.
Change-Id: Id34d6df97760a11c299092dff8edbdb7033bce1c
Reviewed-on: http://gerrit.cloudera.org:8080/12014
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Multiple toSql() bugs in ScalarFunction
> ---
>
> Key: IMPALA-7907
> URL: https://issues.apache.org/jira/browse/IMPALA-7907
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 3.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> Each AST statement node has a {{toSql()}} implementation. The code for
> {{ScalarFunction}} and ToSqlUtils has a number of issues:
> * If Location or Symbol are not set, they are shown as {{'null'}}. Better to
> omit these clauses if the items are not available.
> * The generated SQL does not follow the [Impala CREATE TABLE|] syntax. For
> example, the signature and return value are provided for JAVA functions, but
> should not be.
> * Unlike other statements, this one is generated with a trailing newline.
> * {{ToSql.getCreateFunctionSql()}} fails to separate functions with the
> semi-colon statement terminator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7212) Deprecate --use_krpc flag
[
https://issues.apache.org/jira/browse/IMPALA-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719236#comment-16719236
]
ASF subversion and git services commented on IMPALA-7212:
-
Commit 0eae772f92ca6041fbf99fb1db9c48d7af6401e1 in impala's branch
refs/heads/master from Michael Ho
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=0eae772 ]
IMPALA-7212: Remove dead code data-stream-mgr.cc
Dead code which was accidentally left out in the last
patch of IMPALA-7212.
Testing done: Built Impala debug and release builds
Change-Id: I047e2a01b835936f1066d4d7f87194dcc6857542
Reviewed-on: http://gerrit.cloudera.org:8080/12064
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Deprecate --use_krpc flag
> -
>
> Key: IMPALA-7212
> URL: https://issues.apache.org/jira/browse/IMPALA-7212
> Project: IMPALA
> Issue Type: Sub-task
> Components: Distributed Exec
>Affects Versions: Impala 3.0
>Reporter: Michael Ho
>Assignee: Michael Ho
>Priority: Major
> Fix For: Impala 3.1.0
>
>
> {{--use_krpc}} was introduced in 2.9.0 during development of adopting KRPC
> for DataStream services (IMPALA-2567). It was initially hidden and set to
> false as it wasn't fully implemented yet. Since 2.12.0 and 3.0, this flag is
> set to true by default. Going forward, we will port more services to KRPC and
> it's become very burdensome to keep both Thrift RPC and KRPC implementation
> for converted RPCs. The best route forward is to drop this flag for 3.0+
> releases.
> cc'ing [~dhecht], [~sailesh], [~lv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-5929) Remove useless explicit casts to string
[
https://issues.apache.org/jira/browse/IMPALA-5929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719238#comment-16719238
]
ASF subversion and git services commented on IMPALA-5929:
-
Commit 899bfd10a2a42d40634cdb8c9c670494b3c8945e in impala's branch
refs/heads/master from [~bikram.sngh91]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=899bfd1 ]
IMPALA-7960: Revert "IMPALA-5929: Remove redundant explicit casts to string"
The fix for IMPALA-5929 introduced a bug that produced wrong results.
This bug is detailed in IMPALA-7960. Reverting for now.
This reverts commit 545163bb0a5c86aa02652d0557871f5b694a6c82.
Change-Id: I6f0da62a7ff86f05859a2acbec13a726a9bd6f4c
Reviewed-on: http://gerrit.cloudera.org:8080/12073
Reviewed-by: Zoram Thanga
Tested-by: Impala Public Jenkins
> Remove useless explicit casts to string
> ---
>
> Key: IMPALA-5929
> URL: https://issues.apache.org/jira/browse/IMPALA-5929
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 2.5.0, Impala 2.6.0, Impala 2.7.0, Impala 2.8.0,
> Impala 2.9.0, Impala 2.10.0
>Reporter: Alexander Behm
>Assignee: Bikramjeet Vig
>Priority: Major
> Labels: newbie
> Fix For: Impala 2.12.0
>
>
> Some BI tools like to generate expressions that look like this:
> {code}
> cast(numeric_col as string) = '123456'
> {code}
> Casting and comparing as string is expensive and we should convert such
> expressions to:
> {code}
> numeric_col = 123456
> {code}
> Such transformations may be applicable in other situations as well. We should
> be careful about transforming inequality predicates because the string and
> numeric comparison might not always be equivalent. For example, the following
> transformation would be wrong:
> {code}
> cast(numeric_col as string) = '0123456'
> {code}
> is *not* equivalent to
> {code}
> numeric_col = 123456
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7939) Impala shell not displaying results for a CTE query.
[
https://issues.apache.org/jira/browse/IMPALA-7939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719234#comment-16719234
]
ASF subversion and git services commented on IMPALA-7939:
-
Commit 00ddac32dc8ae49e49e4ba0b417779117522d870 in impala's branch
refs/heads/master from [~fredyw]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=00ddac3 ]
IMPALA-7939: Fix issue where CTE is categorized as DML statement
The logic that checks whether a CTE is DML or SELECT uses shlex that
splits the statement into tokens and check if any of the tokens matches
the DML regular expression. Before this patch, the shlex was set to
posix=True, which means the quotes are stripped from the token, e.g.
select a from foo where a = 'update' becomes
['select', 'a', 'from', 'foo', 'where', 'a', '=', 'update'].
As a result, any token that contains "insert", "delete", "upsert", and
"update" in it will be categorized as DML even though the token is part
of string literal value.
This patch fixes the issue by setting posix=False in shlex that
preserves the quotes. For example:
['select', 'a', 'from', 'foo', 'where', 'a', '=', '"update"']
Testing:
- Added a new shell test
- Ran all shell tests
Change-Id: I011b8e73a0477ac6b2357725452458f972785ae7
Reviewed-on: http://gerrit.cloudera.org:8080/12052
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Impala shell not displaying results for a CTE query.
>
>
> Key: IMPALA-7939
> URL: https://issues.apache.org/jira/browse/IMPALA-7939
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.1.0
>Reporter: Anuj Phadke
>Assignee: Fredy Wijaya
>Priority: Critical
> Fix For: Impala 3.2.0
>
>
> {noformat}
> 1.
> [localhost:21000] > CREATE TABLE test (cp_master_id STRING, data_status
> STRING) stored as parquet;
> Query: CREATE TABLE test (cp_master_id STRING, data_status STRING) stored as
> parquet
> Fetched 0 row(s) in 0.03s
> 2.
> [localhost:21000] > insert into test (cp_master_id, data_status) values
> ('111','NEWREC');
> Query: insert into test (cp_master_id, data_status) values
> ('111','NEWREC')
> Query submitted at: 2018-12-06 14:02:21 (Coordinator:
> [http://anuj-OptiPlex-9020:25000|http://anuj-optiplex-9020:25000/])
> Query progress can be monitored at:
> [http://anuj-OptiPlex-9020:25000/query_plan?query_id=14182abf0a7e4bb:8788cdab|http://anuj-optiplex-9020:25000/query_plan?query_id=14182abf0a7e4bb:8788cdab]
> Modified 1 row(s) in 4.14s
> ***
> 3.
> [localhost:21000] > WITH tbl_incoming
> > AS
> > (
> > SELECT cp_master_id, data_status FROM test WHERE (data_status = 'NEWREC'
> OR data_status='UPDATE')
> > )
> > select * from test;
> Query: WITH tbl_incoming
> AS
> (
> SELECT cp_master_id, data_status FROM test WHERE (data_status = 'NEWREC' OR
> data_status='UPDATE')
> )
> select * from test
> Modified 0 row(s) in 0.12s
> [localhost:21000] >
>
> 4.
> [localhost:21000] > SELECT cp_master_id, data_status FROM test WHERE
> (data_status = 'NEWREC' OR data_status='UPDATE');
> Query: SELECT cp_master_id, data_status FROM test WHERE (data_status =
> 'NEWREC' OR data_status='UPDATE')
> Query submitted at: 2018-12-06 14:05:48 (Coordinator:
> [http://anuj-OptiPlex-9020:25000|http://anuj-optiplex-9020:25000/])
> Query progress can be monitored at:
> [http://anuj-OptiPlex-9020:25000/query_plan?query_id=4b49d50ec0b973c1:ce474a15|http://anuj-optiplex-9020:25000/query_plan?query_id=4b49d50ec0b973c1:ce474a15]
> ++---+
> |cp_master_id|data_status|
> ++---+
> |111|NEWREC|
> ++---+
> Fetched 1 row(s) in 0.12s
> {noformat}
>
> I think the bug is in the regex here -
> https://github.com/apache/impala/blob/master/shell/impala_shell.py#L1157
> It matches for "UPDATE" -
> https://github.com/apache/impala/blob/master/shell/impala_shell.py#L139
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-5929) Remove useless explicit casts to string
[
https://issues.apache.org/jira/browse/IMPALA-5929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719239#comment-16719239
]
ASF subversion and git services commented on IMPALA-5929:
-
Commit 899bfd10a2a42d40634cdb8c9c670494b3c8945e in impala's branch
refs/heads/master from [~bikram.sngh91]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=899bfd1 ]
IMPALA-7960: Revert "IMPALA-5929: Remove redundant explicit casts to string"
The fix for IMPALA-5929 introduced a bug that produced wrong results.
This bug is detailed in IMPALA-7960. Reverting for now.
This reverts commit 545163bb0a5c86aa02652d0557871f5b694a6c82.
Change-Id: I6f0da62a7ff86f05859a2acbec13a726a9bd6f4c
Reviewed-on: http://gerrit.cloudera.org:8080/12073
Reviewed-by: Zoram Thanga
Tested-by: Impala Public Jenkins
> Remove useless explicit casts to string
> ---
>
> Key: IMPALA-5929
> URL: https://issues.apache.org/jira/browse/IMPALA-5929
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 2.5.0, Impala 2.6.0, Impala 2.7.0, Impala 2.8.0,
> Impala 2.9.0, Impala 2.10.0
>Reporter: Alexander Behm
>Assignee: Bikramjeet Vig
>Priority: Major
> Labels: newbie
> Fix For: Impala 2.12.0
>
>
> Some BI tools like to generate expressions that look like this:
> {code}
> cast(numeric_col as string) = '123456'
> {code}
> Casting and comparing as string is expensive and we should convert such
> expressions to:
> {code}
> numeric_col = 123456
> {code}
> Such transformations may be applicable in other situations as well. We should
> be careful about transforming inequality predicates because the string and
> numeric comparison might not always be equivalent. For example, the following
> transformation would be wrong:
> {code}
> cast(numeric_col as string) = '0123456'
> {code}
> is *not* equivalent to
> {code}
> numeric_col = 123456
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7914) Introduce AST base class/interface for statement-like nodes
[
https://issues.apache.org/jira/browse/IMPALA-7914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719233#comment-16719233
]
ASF subversion and git services commented on IMPALA-7914:
-
Commit 80a4e6d3187d4dde6381e229cd409aa8ab13d9b7 in impala's branch
refs/heads/master from [~paul-rogers]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=80a4e6d ]
IMPALA-7914: Base class for statement-like AST nodes
In order to integrate expression rewrites into the analysis phase, the
expression analyze() operation must be able to replace one expression
node with another. Statements, however, are analyzed in place. The two
types of parse nodes thus need different analyze() semantics.
To prepare for that goal, this patch introduces a new StmtNode class
as the base for all statement-like AST nodes. The existing analyze()
method moves to StmtNode. While Expr still defines this method for now,
the future goal is to change the Expr analyze() semantics.
Tests: This is purely a code restructuring, no functional changes. Reran
all FE tests.
Change-Id: Ie565ff02ad74f805a667017ba9bc8c0a2697a97b
Reviewed-on: http://gerrit.cloudera.org:8080/12018
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Introduce AST base class/interface for statement-like nodes
> ---
>
> Key: IMPALA-7914
> URL: https://issues.apache.org/jira/browse/IMPALA-7914
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
>Affects Versions: Impala 3.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
>
> The front-end is based on an abstract syntax tree (AST). The parser creates
> the AST, the analyzer decorates it with semantic information, and the planner
> creates a plan for it.
> At present, the class hierarchy looks like this:
> {noformat}
> ParseNode
> |-- Expr
> | |--
> |-- FromClause
> |--
> |-- StatementBase
> | |-- SelectStmt
> | |--
> {noformat}
> This is a nuisance because the only common base class for all statement-like
> nodes is the {{ParseNode}}, which is also the common base class or
> expressions. However, expressions and statement-like nodes behave differently
> during analysis, SQL generation, and so on.
> We propose to refactor the tree to introduce a new {{StmtNode}} interface or
> class that defines the statement-like semantics, leaving {{Expr}} to define
> the expression-like semantics. The methods then move out of {{ParseNode}}.
> This change all allow revising the analysis step as follows:
> * Analysis of statement-like nodes is done "in place"
> * Analysis of expression nodes may result in replacing one node with a
> rewritten version
> Similarly, when generating SQL:
> * Statements provide the option of generating before- or after-rewrite SQL.
> * Expressions can only generate SQL for what they are; they have no concept
> of before- or after- rewrites.
> Specifically:
> {noformat}
> ParseNode
> |-- Expr
> | |--
> |-- StmtNode
> | |-- FromClause
> | |--
> | |-- StatementBase
> | | |-- SelectStmt
> | | |--
> {noformat}
> It may be useful to introduce a {{ClauseNode}}, but we'll see if that is
> actually helpful as we do the refactoring:
> {noformat}
> |-- StmtNode
> | |-- ClauseNode
> | | |-- FromClause
> | | |--
> | |-- StatementBase
> | | |-- SelectStmt
> | | |--
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7960) wrong results when comparing timestamp casted to varchar of smaller length to a string literal in a binary predicate
[
https://issues.apache.org/jira/browse/IMPALA-7960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719237#comment-16719237
]
ASF subversion and git services commented on IMPALA-7960:
-
Commit 899bfd10a2a42d40634cdb8c9c670494b3c8945e in impala's branch
refs/heads/master from [~bikram.sngh91]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=899bfd1 ]
IMPALA-7960: Revert "IMPALA-5929: Remove redundant explicit casts to string"
The fix for IMPALA-5929 introduced a bug that produced wrong results.
This bug is detailed in IMPALA-7960. Reverting for now.
This reverts commit 545163bb0a5c86aa02652d0557871f5b694a6c82.
Change-Id: I6f0da62a7ff86f05859a2acbec13a726a9bd6f4c
Reviewed-on: http://gerrit.cloudera.org:8080/12073
Reviewed-by: Zoram Thanga
Tested-by: Impala Public Jenkins
> wrong results when comparing timestamp casted to varchar of smaller length to
> a string literal in a binary predicate
>
>
> Key: IMPALA-7960
> URL: https://issues.apache.org/jira/browse/IMPALA-7960
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 2.12.0
>Reporter: Bikramjeet Vig
>Assignee: Bikramjeet Vig
>Priority: Blocker
> Labels: correctness
>
> Expression rewriting seems to identify this as a redundant cast and removes
> it.
> Steps to re-create:
> {noformat}
> select * from (select cast('2018-12-11 09:59:37' as timestamp) as ts) tbl
> where cast(ts as varchar(10)) = '2018-12-11';
> {noformat}
> output:
> {noformat}
> Fetched 0 row(s)
> {noformat}
> Now disable expression re-writes.
> {noformat}
> set ENABLE_EXPR_REWRITES=false;
> select * from (select cast('2018-12-11 09:59:37' as timestamp) as ts) tbl
> where cast(ts as varchar(10)) = '2018-12-11';
> {noformat}
> output:
> {noformat}
> +-+
> | ts |
> +-+
> | 2018-12-11 09:59:37 |
> +-+
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7212) Deprecate --use_krpc flag
[
https://issues.apache.org/jira/browse/IMPALA-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719235#comment-16719235
]
ASF subversion and git services commented on IMPALA-7212:
-
Commit 0eae772f92ca6041fbf99fb1db9c48d7af6401e1 in impala's branch
refs/heads/master from Michael Ho
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=0eae772 ]
IMPALA-7212: Remove dead code data-stream-mgr.cc
Dead code which was accidentally left out in the last
patch of IMPALA-7212.
Testing done: Built Impala debug and release builds
Change-Id: I047e2a01b835936f1066d4d7f87194dcc6857542
Reviewed-on: http://gerrit.cloudera.org:8080/12064
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Deprecate --use_krpc flag
> -
>
> Key: IMPALA-7212
> URL: https://issues.apache.org/jira/browse/IMPALA-7212
> Project: IMPALA
> Issue Type: Sub-task
> Components: Distributed Exec
>Affects Versions: Impala 3.0
>Reporter: Michael Ho
>Assignee: Michael Ho
>Priority: Major
> Fix For: Impala 3.1.0
>
>
> {{--use_krpc}} was introduced in 2.9.0 during development of adopting KRPC
> for DataStream services (IMPALA-2567). It was initially hidden and set to
> false as it wasn't fully implemented yet. Since 2.12.0 and 3.0, this flag is
> set to true by default. Going forward, we will port more services to KRPC and
> it's become very burdensome to keep both Thrift RPC and KRPC implementation
> for converted RPCs. The best route forward is to drop this flag for 3.0+
> releases.
> cc'ing [~dhecht], [~sailesh], [~lv]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7961) Concurrent catalog heavy workloads can cause queries with SYNC_DDL to fail fast
[
https://issues.apache.org/jira/browse/IMPALA-7961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bharath v updated IMPALA-7961:
--
Description:
When catalog server is under heavy load with concurrent updates to objects,
queries with SYNC_DDL can fail with the following message.
*User facing error message:*
{noformat}
ERROR: CatalogException: Couldn't retrieve the catalog topic version for the
SYNC_DDL operation after 3 attempts.The operation has been successfully
executed but its effects may have not been broadcast to all the coordinators.
{noformat}
*Exception from the catalog server log:*
{noformat}
I1031 00:00:49.168761 1127039 CatalogServiceCatalog.java:1903] Operation using
SYNC_DDL is waiting for catalog topic version: 236535. Time to identify topic
version (msec): 1088
I1031 00:00:49.168824 1125528 CatalogServiceCatalog.java:1903] Operation using
SYNC_DDL is waiting for catalog topic version: 236535. Time to identify topic
version (msec): 12625
I1031 00:00:49.168851 1131986 jni-util.cc:230]
org.apache.impala.catalog.CatalogException: Couldn't retrieve the catalog topic
version for the SYNC_DDL operation after 3 attempts.The operation has been
successfully executed but its effects may have not been broadcast to all the
coordinators.
at
org.apache.impala.catalog.CatalogServiceCatalog.waitForSyncDdlVersion(CatalogServiceCatalog.java:1891)
at
org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:336)
at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:146)
{noformat}
*What this means*
The Catalog operation is actually successful (the change has been committed to
HMS and Catalog server cache) but the Catalog server noticed that it is taking
longer than expected time for it to broadcast the changes (for whatever reason)
and instead of hanging in there, it fails fast. The coordinators are expected
to eventually sync up in the background.
*Problem*
- This violates the contract of the SYNC_DDL query option since the query
returns early.
- This is a behavioral regression from pre IMPALA-5058 state where the queries
would wait forever for SYNC_DDL based changes to propagate.
*Notes*
- Usual suspect here is heavily concurrent catalog operations with long
running DDLs.
- Introduced by IMPALA-5058
- My understanding is that this also applies to the Catalog V2 (or
LocalCatalog mode) since we still rely on the CatalogServer for DDL
orchestration and hence it takes this codepath.
Please refer to the jira comment for technical explanation as to why this is
happening (to be updated).
was:
When catalog server is under heavy load with concurrent updates to objects,
queries with SYNC_DDL can fail with the following message.
*User facing error message:*
{noformat}
ERROR: CatalogException: Couldn't retrieve the catalog topic version for the
SYNC_DDL operation after 3 attempts.The operation has been successfully
executed but its effects may have not been broadcast to all the coordinators.
{noformat}
*Exception from the catalog server log:*
{noformat}
I1031 00:00:49.168761 1127039 CatalogServiceCatalog.java:1903] Operation using
SYNC_DDL is waiting for catalog topic version: 236535. Time to identify topic
version (msec): 1088
I1031 00:00:49.168824 1125528 CatalogServiceCatalog.java:1903] Operation using
SYNC_DDL is waiting for catalog topic version: 236535. Time to identify topic
version (msec): 12625
I1031 00:00:49.168851 1131986 jni-util.cc:230]
org.apache.impala.catalog.CatalogException: Couldn't retrieve the catalog topic
version for the SYNC_DDL operation after 3 attempts.The operation has been
successfully executed but its effects may have not been broadcast to all the
coordinators.
at
org.apache.impala.catalog.CatalogServiceCatalog.waitForSyncDdlVersion(CatalogServiceCatalog.java:1891)
at
org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:336)
at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:146)
{noformat}
*What this means*
The Catalog operation is actually successful (the change has been committed to
HMS and Catalog server cache) but the Catalog server noticed that it is taking
longer than expected time for it to broadcast the changes (for whatever reason)
and instead of hanging in there, it fails fast. The coordinators are expected
to eventually sync up in the background.
*Problem*
- This violates the contract of the SYNC_DDL query option since the query
returns early.
- This is a behavioral regression from pre IMPALA-5058 state where the queries
would wait forever for SYNC_DDL based changes to propagate.
*Notes*
- Usual suspect here is heavily concurrent catalog operations with long
running DDLs.
- Introduced by IMPALA-5058
- My understanding is that this also applies to the Catalog V2 (or
LocalCatalog mode) since we still rely on the
