[jira] [Resolved] (IMPALA-6381) test_exchange_delays.py failed on isilon build due to sender timeout

[Tim Armstrong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IMPALA-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Armstrong resolved IMPALA-6381.
---
   Resolution: Fixed
Fix Version/s: Impala 2.12.0

Increased the timeout, hopefully that should prevent it reoccurring

IMPALA-6381: increase test_exchange_delays timeout for isilon

Change-Id: Ie82030403fa238b673b0a3ccdc7731b0d78b63af
Reviewed-on: http://gerrit.cloudera.org:8080/8993
Reviewed-by: Alex Behm 
Tested-by: Impala Public Jenkins
---

> test_exchange_delays.py failed on isilon build due to sender timeout
> 
>
> Key: IMPALA-6381
> URL: https://issues.apache.org/jira/browse/IMPALA-6381
> Project: IMPALA
>  Issue Type: Bug
>  Components: Infrastructure
>Affects Versions: Impala 2.11.0
>Reporter: Zoram Thanga
>Assignee: Tim Armstrong
> Fix For: Impala 2.12.0
>
>
> Please feel free to re-assign, etc.
> Isilon builds have been failing because of this:
> Error Message
> AssertionError: Expected exception: Sender timed out waiting for receiver 
> fragment instance
> Stacktrace
> self = 
> vector = 
> @pytest.mark.execute_serially
> @CustomClusterTestSuite.with_args(
> "--stress_datastream_recvr_delay_ms={0}".format(DELAY_MS)
> + " --datastream_sender_timeout_ms=1")
> def test_exchange_large_delay(self, vector):
>   """Test delays in registering data stream receivers where all of the 
> batches sent
> will time out before the receiver registers. Before IMPALA-2987, this 
> scenario
> resulted in the query hanging.
> """
>   self.run_test_case('QueryTest/exchange-delays', vector)
> 
>   # Test the special case when no batches are sent and the EOS message 
> times out.
> > self.run_test_case('QueryTest/exchange-delays-zero-rows', vector)
> custom_cluster/test_exchange_delays.py:58: 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IMPALA-6384) RequestPoolService doesn't honor custom user -> group mapping overrides in HDFS config

[bharath v (JIRA)]

bharath v created IMPALA-6384:
-

 Summary: RequestPoolService doesn't honor custom user -> group 
mapping overrides in HDFS config
 Key: IMPALA-6384
 URL: https://issues.apache.org/jira/browse/IMPALA-6384
 Project: IMPALA
  Issue Type: Bug
  Components: Frontend
Affects Versions: Impala 2.10.0, Impala 2.9.0, Impala 2.8.0, Impala 2.7.0, 
Impala 2.6.0
Reporter: bharath v
Assignee: bharath v


{noformat}
 public RequestPoolService(final String fsAllocationPath, final String 
llamaSitePath) {
Preconditions.checkNotNull(fsAllocationPath);
running_ = new AtomicBoolean(false);
allocationConf_ = new AtomicReference();
URL fsAllocationURL = getURL(fsAllocationPath);
if (fsAllocationURL == null) {
  throw new IllegalArgumentException(
  "Unable to find allocation configuration file: " + fsAllocationPath);
}
Configuration allocConf = new Configuration(false);   < Doesn't load 
the default Hadoop configs
allocConf.set(FairSchedulerConfiguration.ALLOCATION_FILE, 
fsAllocationURL.getPath());
allocLoader_ = new AllocationFileLoaderService();
allocLoader_.init(allocConf);
{noformat}

The affect of this bug is that {{LDAPGroupsMapping}} cannot be used with fair 
scheduler group mapping. It only loads the default 
{{ShellBasedUnixGroupsMapping}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IMPALA-6383) Memory from previous row groups can accumulate in Parquet scanner

[Tim Armstrong (JIRA)]

Tim Armstrong created IMPALA-6383:
-

 Summary: Memory from previous row groups can accumulate in Parquet 
scanner
 Key: IMPALA-6383
 URL: https://issues.apache.org/jira/browse/IMPALA-6383
 Project: IMPALA
  Issue Type: Bug
  Components: Backend
Affects Versions: Impala 2.10.0, Impala 2.11.0, Impala 2.12.0
Reporter: Tim Armstrong
Assignee: Tim Armstrong


I ran across this bug when working on porting scanners to the new buffer pool. 
Before that the only symptom of the failures was excessive memory consumption, 
but with the reservations they become easy-to-detect hard failures.

The problem is in HdfsParquetScanner::NextRowGroup(), which calls InitColumns() 
on column readers, which starts scans, which allocate memory. The problem is 
that, if the row group is skipped because of dictionary predicates or some 
other error, the scans aren't cancelled and the I/O buffers aren't releated.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (IMPALA-5014) DECIMAL V2 round when casting to/from DECIMAL, part 2

[Taras Bobrovytsky (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IMPALA-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Taras Bobrovytsky resolved IMPALA-5014.
---
   Resolution: Fixed
Fix Version/s: Impala 2.12.0

{code}
commit c86b0a9736ee1e19b95a2d06771ca2ab8577950f
Author: Taras Bobrovytsky 
Date:   Thu Dec 21 15:47:06 2017 -0800

IMPALA-5014: Part 2: Round when casting decimal to timestamp

When there are too many digits to the right of the dot in a decimal, we
would always truncate when casting to timestamp. In this patch we change
the behavior to round instead of truncating when decimal_v2 is enabled.

Testing:
- Added some EE tests, ran BE tests on my machine.

Change-Id: I8fb3a7d976ab980b8572d7e9524850572bad57da
Reviewed-on: http://gerrit.cloudera.org:8080/8969
Reviewed-by: Tim Armstrong 
Tested-by: Impala Public Jenkins
{code}

> DECIMAL V2 round when casting to/from DECIMAL, part 2
> -
>
> Key: IMPALA-5014
> URL: https://issues.apache.org/jira/browse/IMPALA-5014
> Project: IMPALA
>  Issue Type: Bug
>  Components: Backend
>Affects Versions: Impala 2.0
>Reporter: Dan Hecht
>Assignee: Taras Bobrovytsky
>  Labels: correctness, downgraded, ramp-up
> Fix For: Impala 2.12.0
>
>
> This is the continuation of IMPALA-4810 to finish the remaining cases:
> ||From Type||To Type||
> |STRING|DECIMAL|
> |DECIMAL|TIMESTAMP|
> See IMPALA-4810 for more background.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IMPALA-6382) Impalad crashes on SELECT query when spill buffer is set on certain values

[Xinran Tinney (JIRA)]

Xinran Tinney created IMPALA-6382:
-

 Summary: Impalad crashes on SELECT query when spill buffer is set 
on certain values
 Key: IMPALA-6382
 URL: https://issues.apache.org/jira/browse/IMPALA-6382
 Project: IMPALA
  Issue Type: Bug
 Environment: Impala mini cluster
Reporter: Xinran Tinney


After starting the minicluster and run bin/impala-shell.sh, do 
set MIN_SPILLABLE_BUFFER_SIZE= 4611686018427387904;
set DEFAULT_SPILLABLE_BUFFER_SIZE=4611686018427387904;
use tpch;
select distinct l_comment from lineitem;
then impalad is crashed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (IMPALA-4168) Adopt Oracle-style hint placement for INSERT statements

[Dimitris Tsirogiannis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/IMPALA-4168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dimitris Tsirogiannis resolved IMPALA-4168.
---
   Resolution: Fixed
Fix Version/s: Impala 2.12.0

Change-Id: Ied7629d70197a0270cdc0853e00cc021fdb4dc20
Reviewed-on: http://gerrit.cloudera.org:8080/8676
Reviewed-by: Dimitris Tsirogiannis 
Tested-by: Impala Public Jenkins
---
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
M testdata/workloads/functional-planner/queries/PlannerTest/insert.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu-upsert.test
8 files changed, 265 insertions(+), 84 deletions(-)

> Adopt Oracle-style hint placement for INSERT statements
> ---
>
> Key: IMPALA-4168
> URL: https://issues.apache.org/jira/browse/IMPALA-4168
> Project: IMPALA
>  Issue Type: Improvement
>  Components: Frontend
>Affects Versions: Impala 2.2, Impala 2.3.0, Impala 2.5.0, Impala 2.4.0, 
> Impala 2.6.0, Impala 2.7.0
>Reporter: Alexander Behm
>Assignee: Jinchul Kim
>  Labels: incompatibility, ramp-up
> Fix For: Impala 2.12.0
>
>
> For consistency with Oracle we should consider accepting hints in the same 
> places in SQL statements. For example, our current INSERT statements accepts 
> hints right before the SELECT portion:
> {code}
> INSERT INTO t PARTITIONED(year,month) /*+ hints */ SELECT * FROM src;
> {code}
> The proposal is to accept hints immediately after INSERT like Oracle does:
> {code}
> INSERT /*+ hints */ INTO t PARTITIONED(year,month) SELECT * FROM src;
> {code}
> Ideally, we would not accept hints in multiple places to avoid confusion and 
> to reduce the code and testing burden. Ceasing to recognize the old hint 
> placement is a backwards incompatible change.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IMPALA-6381) test_exchange_delays.py failed on isilon build due to sender timeout

[Zoram Thanga (JIRA)]

Zoram Thanga created IMPALA-6381:


 Summary: test_exchange_delays.py failed on isilon build due to 
sender timeout
 Key: IMPALA-6381
 URL: https://issues.apache.org/jira/browse/IMPALA-6381
 Project: IMPALA
  Issue Type: Bug
  Components: Infrastructure
Affects Versions: Impala 2.11.0
Reporter: Zoram Thanga
Assignee: Tim Armstrong


Please feel free to re-assign, etc.

Isilon builds have been failing because of this:

Error Message
AssertionError: Expected exception: Sender timed out waiting for receiver 
fragment instance
Stacktrace
self = 
vector = 

@pytest.mark.execute_serially
@CustomClusterTestSuite.with_args(
"--stress_datastream_recvr_delay_ms={0}".format(DELAY_MS)
+ " --datastream_sender_timeout_ms=1")
def test_exchange_large_delay(self, vector):
  """Test delays in registering data stream receivers where all of the 
batches sent
will time out before the receiver registers. Before IMPALA-2987, this 
scenario
resulted in the query hanging.
"""
  self.run_test_case('QueryTest/exchange-delays', vector)

  # Test the special case when no batches are sent and the EOS message 
times out.
> self.run_test_case('QueryTest/exchange-delays-zero-rows', vector)

custom_cluster/test_exchange_delays.py:58: 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)