[
https://issues.apache.org/jira/browse/IMPALA-8849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903351#comment-16903351
]
Tim Armstrong commented on IMPALA-8849:
---------------------------------------
So it looks like it only happens when stats are computed and the column is a
VARCHAR.
{noformat}
[localhost:21000] tpcds> use tpcds_10000_text;
Query: use tpcds_10000_text
[localhost:21000] tpcds_10000_text> explain select c_customer_id from
tpcds_10000_text.customer;
Query: explain select c_customer_id from tpcds_10000_text.customer
+------------------------------------------------------------+
| Explain String |
+------------------------------------------------------------+
| Max Per-Host Resource Reservation: Memory=8.00MB Threads=3 |
| Per-Host Resource Estimates: Memory=48MB |
| |
| PLAN-ROOT SINK |
| | |
| 01:EXCHANGE [UNPARTITIONED] |
| | |
| 00:SCAN HDFS [tpcds_10000_text.customer] |
| HDFS partitions=1/1 files=1 size=12.50MB |
| row-size=-1B cardinality=100.00K |
+------------------------------------------------------------+
{noformat}
Here's a repro in my dev env:
{noformat}
[localhost:21000] tpcds> create external table repro_cust(c_customer_sk INT,
c_customer_id VARCHAR(16)) row format delimited fields terminated by '|' WITH
SERDEPROPERTIES ('field.delim'='|', 'serialization.format'='|') STORED AS
TEXTFILE LOCATION 'hdfs://localhost:20500/test-warehouse/tpcds.customer';
Query: create external table repro_cust(c_customer_sk INT, c_customer_id
VARCHAR(16)) row format delimited fields terminated by '|' WITH SERDEPROPERTIES
('field.delim'='|', 'serialization.format'='|') STORED AS TEXTFILE LOCATION
'hdfs://localhost:20500/test-warehouse/tpcds.customer'
+-------------------------+
| summary |
+-------------------------+
| Table has been created. |
+-------------------------+
Fetched 1 row(s) in 0.39s
[localhost:21000] tpcds> compute stats repro_cust;
Query: compute stats repro_cust
+-----------------------------------------+
| summary |
+-----------------------------------------+
| Updated 1 partition(s) and 2 column(s). |
+-----------------------------------------+
Fetched 1 row(s) in 3.62s
[localhost:21000] tpcds> explain select c_customer_id from repro_cust;
Query: explain select c_customer_id from repro_cust
+------------------------------------------------------------+
| Explain String |
+------------------------------------------------------------+
| Max Per-Host Resource Reservation: Memory=8.00MB Threads=3 |
| Per-Host Resource Estimates: Memory=48MB |
| |
| PLAN-ROOT SINK |
| | |
| 01:EXCHANGE [UNPARTITIONED] |
| | |
| 00:SCAN HDFS [tpcds.repro_cust] |
| HDFS partitions=1/1 files=1 size=12.60MB |
| row-size=-1B cardinality=100.00K |
+------------------------------------------------------------+
Fetched 10 row(s) in 0.01s
{noformat}
> IllegalStateException in HashJoinNode because of missing memory estimate
> ------------------------------------------------------------------------
>
> Key: IMPALA-8849
> URL: https://issues.apache.org/jira/browse/IMPALA-8849
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Affects Versions: Impala 3.3.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Blocker
> Attachments: query4.sql
>
>
> [~grahn] reports the below error when running the attached TPC-DS query on
> uncompressed text with stats.
> {noformat}
> I0808 03:27:16.621085 150090 jni-util.cc:288]
> 8945e9f76f33dca0:8bf31c4700000000] java.lang.IllegalStateException: Mem
> estimate must be set
> at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
> at
> org.apache.impala.planner.ResourceProfileBuilder.build(ResourceProfileBuilder.java:73)
> at
> org.apache.impala.planner.HashJoinNode.computeNodeResourceProfile(HashJoinNode.java:252)
> at
> org.apache.impala.planner.PlanFragment.computeResourceProfile(PlanFragment.java:238)
> at org.apache.impala.planner.Planner.computeResourceReqs(Planner.java:416)
> at
> org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1162)
> at
> org.apache.impala.service.Frontend.getPlannedExecRequest(Frontend.java:1477)
> at
> org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1341)
> at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1236)
> at
> org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1206)
> at
> org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:154)
> I0808 03:27:16.621861 150090 status.cc:124]
> 8945e9f76f33dca0:8bf31c4700000000] IllegalStateException: Mem estimate must
> be set
> @ 0xb9af99
> @ 0x11f785e
> @ 0x10ab9a3
> @ 0x10d0f54
> @ 0x10e2a2c
> @ 0x1125f8d
> @ 0x1458c54
> @ 0x145810c
> @ 0xb68769
> @ 0xf91ca0
> @ 0xf8791e
> @ 0xf887b1
> @ 0x127cd2f
> @ 0x127d8d9
> @ 0x1ac0709
> @ 0x7f11ea4d7dd4
> @ 0x7f11e6f1e02c
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
