[jira] [Commented] (IMPALA-8849) IllegalStateException in HashJoinNode because of missing memory estimate
[ https://issues.apache.org/jira/browse/IMPALA-8849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16908144#comment-16908144 ] ASF subversion and git services commented on IMPALA-8849: - Commit c3a67b67faaebf18735fa36d35c80d8c11043f7f in impala's branch refs/heads/master from Tim Armstrong [ https://gitbox.apache.org/repos/asf?p=impala.git;h=c3a67b6 ] IMPALA-8849: fix IllegalStateException with VARCHAR The bug is that the serialized size wasn't populated for VARCHAR in a case when it should have been. It appears a condition was simply not updated when VARCHAR was added. Other code assumed that the serialized size was populated when the other size field was populated, which is a reasonable invariant. I documented the invariant in the class and added validation that the invariant held. Defining and checking invariants led to discovering various other minor issues where the sizes were set incorrect for fixed-length types or not set for variable-length types: * CHAR was not consistently treated as a fixed-length type. * avgSerializedSize_ was not always updated with avgSize_ Testing: Added a regression test for this specific case. Adding the assertions resulted in other cases showing up related bugs. Change-Id: Ie45e386cb09e31f4b7cdc82b7734dbecb4464534 Reviewed-on: http://gerrit.cloudera.org:8080/14062 Tested-by: Impala Public Jenkins Reviewed-by: Csaba Ringhofer > IllegalStateException in HashJoinNode because of missing memory estimate > > > Key: IMPALA-8849 > URL: https://issues.apache.org/jira/browse/IMPALA-8849 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Affects Versions: Impala 3.3.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Blocker > Fix For: Impala 3.3.0 > > Attachments: query4.sql > > > [~grahn] reports the below error when running the attached TPC-DS query on > uncompressed text with stats. > {noformat} > I0808 03:27:16.621085 150090 jni-util.cc:288] > 8945e9f76f33dca0:8bf31c47] java.lang.IllegalStateException: Mem > estimate must be set > at com.google.common.base.Preconditions.checkState(Preconditions.java:149) > at > org.apache.impala.planner.ResourceProfileBuilder.build(ResourceProfileBuilder.java:73) > at > org.apache.impala.planner.HashJoinNode.computeNodeResourceProfile(HashJoinNode.java:252) > at > org.apache.impala.planner.PlanFragment.computeResourceProfile(PlanFragment.java:238) > at org.apache.impala.planner.Planner.computeResourceReqs(Planner.java:416) > at > org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1162) > at > org.apache.impala.service.Frontend.getPlannedExecRequest(Frontend.java:1477) > at > org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1341) > at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1236) > at > org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1206) > at > org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:154) > I0808 03:27:16.621861 150090 status.cc:124] > 8945e9f76f33dca0:8bf31c47] IllegalStateException: Mem estimate must > be set > @ 0xb9af99 > @ 0x11f785e > @ 0x10ab9a3 > @ 0x10d0f54 > @ 0x10e2a2c > @ 0x1125f8d > @ 0x1458c54 > @ 0x145810c > @ 0xb68769 > @ 0xf91ca0 > @ 0xf8791e > @ 0xf887b1 > @ 0x127cd2f > @ 0x127d8d9 > @ 0x1ac0709 > @ 0x7f11ea4d7dd4 > @ 0x7f11e6f1e02c > {noformat} -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8849) IllegalStateException in HashJoinNode because of missing memory estimate
[ https://issues.apache.org/jira/browse/IMPALA-8849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903356#comment-16903356 ] Tim Armstrong commented on IMPALA-8849: --- Reverting this commit fixed the issue for me commit c2516d220da8e532b6ebdb6f3a12e7ad97c4f597 Author: Csaba Ringhofer Date: Wed Apr 17 15:01:07 2019 +0200 IMPALA-8409: Fix row-size for STRING columns with unknown stats Explain returned row-size=11B for STRING columns without statistics. The issue was caused by adding -1 (meaning unknown) to the 12 byte slot size (sizeof(StringValue)). The code in TupleDescriptor.java tried to handle this by checking if the size is -1, but it was already 11 at this point. There is more potential for cleanup, but I wanted to keep this change minimal. Testing: - revived some tests in CatalogTest.java that were removed in 2013 due to flakiness - added an EE test that checks row size with and without stats - fixed a similar test, test_explain_validate_cardinality_estimates (the format of the line it looks for has changed, which lead to skipping the actual verification and accepting everything) - ran core FE and EE tests Change-Id: I866acf10b2c011a735dee019f4bc29358f2ec4e5 Reviewed-on: http://gerrit.cloudera.org:8080/13190 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > IllegalStateException in HashJoinNode because of missing memory estimate > > > Key: IMPALA-8849 > URL: https://issues.apache.org/jira/browse/IMPALA-8849 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Affects Versions: Impala 3.3.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Blocker > Attachments: query4.sql > > > [~grahn] reports the below error when running the attached TPC-DS query on > uncompressed text with stats. > {noformat} > I0808 03:27:16.621085 150090 jni-util.cc:288] > 8945e9f76f33dca0:8bf31c47] java.lang.IllegalStateException: Mem > estimate must be set > at com.google.common.base.Preconditions.checkState(Preconditions.java:149) > at > org.apache.impala.planner.ResourceProfileBuilder.build(ResourceProfileBuilder.java:73) > at > org.apache.impala.planner.HashJoinNode.computeNodeResourceProfile(HashJoinNode.java:252) > at > org.apache.impala.planner.PlanFragment.computeResourceProfile(PlanFragment.java:238) > at org.apache.impala.planner.Planner.computeResourceReqs(Planner.java:416) > at > org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1162) > at > org.apache.impala.service.Frontend.getPlannedExecRequest(Frontend.java:1477) > at > org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1341) > at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1236) > at > org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1206) > at > org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:154) > I0808 03:27:16.621861 150090 status.cc:124] > 8945e9f76f33dca0:8bf31c47] IllegalStateException: Mem estimate must > be set > @ 0xb9af99 > @ 0x11f785e > @ 0x10ab9a3 > @ 0x10d0f54 > @ 0x10e2a2c > @ 0x1125f8d > @ 0x1458c54 > @ 0x145810c > @ 0xb68769 > @ 0xf91ca0 > @ 0xf8791e > @ 0xf887b1 > @ 0x127cd2f > @ 0x127d8d9 > @ 0x1ac0709 > @ 0x7f11ea4d7dd4 > @ 0x7f11e6f1e02c > {noformat} -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8849) IllegalStateException in HashJoinNode because of missing memory estimate
[ https://issues.apache.org/jira/browse/IMPALA-8849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903351#comment-16903351 ] Tim Armstrong commented on IMPALA-8849: --- So it looks like it only happens when stats are computed and the column is a VARCHAR. {noformat} [localhost:21000] tpcds> use tpcds_1_text; Query: use tpcds_1_text [localhost:21000] tpcds_1_text> explain select c_customer_id from tpcds_1_text.customer; Query: explain select c_customer_id from tpcds_1_text.customer ++ | Explain String | ++ | Max Per-Host Resource Reservation: Memory=8.00MB Threads=3 | | Per-Host Resource Estimates: Memory=48MB | || | PLAN-ROOT SINK | | | | | 01:EXCHANGE [UNPARTITIONED]| | | | | 00:SCAN HDFS [tpcds_1_text.customer] | |HDFS partitions=1/1 files=1 size=12.50MB| |row-size=-1B cardinality=100.00K| ++ {noformat} Here's a repro in my dev env: {noformat} [localhost:21000] tpcds> create external table repro_cust(c_customer_sk INT, c_customer_id VARCHAR(16)) row format delimited fields terminated by '|' WITH SERDEPROPERTIES ('field.delim'='|', 'serialization.format'='|') STORED AS TEXTFILE LOCATION 'hdfs://localhost:20500/test-warehouse/tpcds.customer'; Query: create external table repro_cust(c_customer_sk INT, c_customer_id VARCHAR(16)) row format delimited fields terminated by '|' WITH SERDEPROPERTIES ('field.delim'='|', 'serialization.format'='|') STORED AS TEXTFILE LOCATION 'hdfs://localhost:20500/test-warehouse/tpcds.customer' +-+ | summary | +-+ | Table has been created. | +-+ Fetched 1 row(s) in 0.39s [localhost:21000] tpcds> compute stats repro_cust; Query: compute stats repro_cust +-+ | summary | +-+ | Updated 1 partition(s) and 2 column(s). | +-+ Fetched 1 row(s) in 3.62s [localhost:21000] tpcds> explain select c_customer_id from repro_cust; Query: explain select c_customer_id from repro_cust ++ | Explain String | ++ | Max Per-Host Resource Reservation: Memory=8.00MB Threads=3 | | Per-Host Resource Estimates: Memory=48MB | || | PLAN-ROOT SINK | | | | | 01:EXCHANGE [UNPARTITIONED]| | | | | 00:SCAN HDFS [tpcds.repro_cust]| |HDFS partitions=1/1 files=1 size=12.60MB| |row-size=-1B cardinality=100.00K| ++ Fetched 10 row(s) in 0.01s {noformat} > IllegalStateException in HashJoinNode because of missing memory estimate > > > Key: IMPALA-8849 > URL: https://issues.apache.org/jira/browse/IMPALA-8849 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Affects Versions: Impala 3.3.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Blocker > Attachments: query4.sql > > > [~grahn] reports the below error when running the attached TPC-DS query on > uncompressed text with stats. > {noformat} > I0808 03:27:16.621085 150090 jni-util.cc:288] > 8945e9f76f33dca0:8bf31c47] java.lang.IllegalStateException: Mem > estimate must be set > at com.google.common.base.Preconditions.checkState(Preconditions.java:149) > at > org.apache.impala.planner.ResourceProfileBuilder.build(ResourceProfileBuilder.java:73) > at > org.apache.impala.planner.HashJoinNode.computeNodeResourceProfile(HashJoinNode.java:252) > at > org.apache.impala.planner.PlanFragment.computeResourceProfile(PlanFragment.java:238) > at org.apache.impala.planner.Planner.computeResourceReqs(Planner.java:416) > at > org.apache.impala.service.Frontend.createExecRequest(Frontend.jav
[jira] [Commented] (IMPALA-8849) IllegalStateException in HashJoinNode because of missing memory estimate
[ https://issues.apache.org/jira/browse/IMPALA-8849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903342#comment-16903342 ] Tim Armstrong commented on IMPALA-8849: --- {noformat} [nightly6x-unsecure-1.nightly6x-unsecure.root.hwx.site:21000] default> describe tpcds.customer; Query: describe tpcds.customer ++-+-+ | name | type| comment | ++-+-+ | c_customer_sk | int | | | c_customer_id | varchar(16) | | | c_current_cdemo_sk | int | | | c_current_hdemo_sk | int | | | c_current_addr_sk | int | | | c_first_shipto_date_sk | int | | | c_first_sales_date_sk | int | | | c_salutation | varchar(10) | | | c_first_name | varchar(20) | | | c_last_name| varchar(30) | | | c_preferred_cust_flag | varchar(1) | | | c_birth_day| int | | | c_birth_month | int | | | c_birth_year | int | | | c_birth_country| varchar(20) | | | c_login| varchar(13) | | | c_email_address| varchar(50) | | | c_last_review_date_sk | int | | ++-+-+ Fetched 18 row(s) in 0.01s [nightly6x-unsecure-1.nightly6x-unsecure.root.hwx.site:21000] default> explain select c_customer_id from tpcds.customer; Query: explain select c_customer_id from tpcds.customer ++ | Explain String | ++ | Max Per-Host Resource Reservation: Memory=8.00MB Threads=3 | | Per-Host Resource Estimates: Memory=48MB | || | PLAN-ROOT SINK | | | | | 01:EXCHANGE [UNPARTITIONED]| | | | | 00:SCAN HDFS [tpcds.customer] | |HDFS partitions=1/1 files=1 size=12.50MB| |row-size=-1B cardinality=100.00K| ++ Fetched 10 row(s) in 0.02s [nightly6x-unsecure-1.nightly6x-unsecure.root.hwx.site:21000] default> show column stats tpcds.customer; Query: show column stats tpcds.customer ++-+--++--+---+ | Column | Type| #Distinct Values | #Nulls | Max Size | Avg Size | ++-+--++--+---+ | c_customer_sk | INT | 10 | 0 | 4| 4 | | c_customer_id | VARCHAR(16) | 10 | 0 | 16 | 16| | c_current_cdemo_sk | INT | 91558| 3438 | 4| 4 | | c_current_hdemo_sk | INT | 7376 | 3431 | 4| 4 | | c_current_addr_sk | INT | 42003| 0 | 4| 4 | | c_first_shipto_date_sk | INT | 3754 | 3443 | 4| 4 | | c_first_sales_date_sk | INT | 3734 | 3518 | 4| 4 | | c_salutation | VARCHAR(10) | 6| 3410 | 4| 3.24143433228 | | c_first_name | VARCHAR(20) | 4013 | 3492 | 11 | 5.839499950408936 | | c_last_name| VARCHAR(30) | 4951 | 3497 | 13 | 6.124800205230713 | | c_preferred_cust_flag | VARCHAR(1) | 2| 3426 | 1| 1 | | c_birth_day| INT | 31 | 3461 | 4| 4 | | c_birth_month | INT | 12 | 3449 | 4| 4 | | c_birth_year | INT | 67 | 3453 | 4| 4 | | c_birth_country| VARCHAR(20) | 207 | 3439 | 20 | 8.708800315856934 | | c_login| VARCHAR(13) | 0| 10 | 0| 0 | | c_email_address| VARCHAR(50) | 10 | 3521 | 46 | 27.45219993591309 | | c_last_review_date_sk | INT | 350 | 3484 | 4| 4 | ++-+--++--+---+ {
[jira] [Commented] (IMPALA-8849) IllegalStateException in HashJoinNode because of missing memory estimate
[ https://issues.apache.org/jira/browse/IMPALA-8849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16903290#comment-16903290 ] Tim Armstrong commented on IMPALA-8849: --- Looks like HdfsScanNode can produce a negative row size estimate, which would explain the negative memory estimate. {noformat} I0808 20:25:37.342036 110162 DistributedPlanner.java:498] 114bef3456667eb2:458377fe] 36:SCAN HDFS [tpcds.customer , RANDOM] HDFS partitions=1/1 files=1 size=12.50MB stored statistics: table: rows=100.00K size=12.50MB columns: all extrapolated-rows=disabled max-scan-range-rows=100.00K mem-estimate=invalid mem-reservation=invalid thread-reservation=invalid tuple-ids=78 row-size=-3B cardinality=100.00K in pipelines: {noformat} > IllegalStateException in HashJoinNode because of missing memory estimate > > > Key: IMPALA-8849 > URL: https://issues.apache.org/jira/browse/IMPALA-8849 > Project: IMPALA > Issue Type: Bug > Components: Frontend >Affects Versions: Impala 3.3.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Blocker > Attachments: query4.sql > > > [~grahn] reports the below error when running the attached TPC-DS query on > uncompressed text with stats. > {noformat} > I0808 03:27:16.621085 150090 jni-util.cc:288] > 8945e9f76f33dca0:8bf31c47] java.lang.IllegalStateException: Mem > estimate must be set > at com.google.common.base.Preconditions.checkState(Preconditions.java:149) > at > org.apache.impala.planner.ResourceProfileBuilder.build(ResourceProfileBuilder.java:73) > at > org.apache.impala.planner.HashJoinNode.computeNodeResourceProfile(HashJoinNode.java:252) > at > org.apache.impala.planner.PlanFragment.computeResourceProfile(PlanFragment.java:238) > at org.apache.impala.planner.Planner.computeResourceReqs(Planner.java:416) > at > org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1162) > at > org.apache.impala.service.Frontend.getPlannedExecRequest(Frontend.java:1477) > at > org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1341) > at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1236) > at > org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1206) > at > org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:154) > I0808 03:27:16.621861 150090 status.cc:124] > 8945e9f76f33dca0:8bf31c47] IllegalStateException: Mem estimate must > be set > @ 0xb9af99 > @ 0x11f785e > @ 0x10ab9a3 > @ 0x10d0f54 > @ 0x10e2a2c > @ 0x1125f8d > @ 0x1458c54 > @ 0x145810c > @ 0xb68769 > @ 0xf91ca0 > @ 0xf8791e > @ 0xf887b1 > @ 0x127cd2f > @ 0x127d8d9 > @ 0x1ac0709 > @ 0x7f11ea4d7dd4 > @ 0x7f11e6f1e02c > {noformat} -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org