[
https://issues.apache.org/jira/browse/TAJO-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jaehwa Jung updated TAJO-2047:
------------------------------
Description:
Query hangs while executing CTAS for partition table on aws as following:
* Root directory exists on s3
* Dataset : tpc-h 1G
* Query
{code:xml}
CREATE TABLE LINEITEM_P (L_ORDERKEY bigint, L_PARTKEY bigint, L_SUPPKEY bigint,
L_LINENUMBER bigint, L_QUANTITY double, L_EXTENDEDPRICE double, L_DISCOUNT
double, L_TAX double, L_RETURNFLAG text, L_LINESTATUS text, L_COMMITDATE text,
L_RECEIPTDATE text, L_SHIPINSTRUCT text, L_SHIPMODE text, L_COMMENT text)
USING TEXT WITH ('text.delimiter'='|')
PARTITION BY COLUMN (L_SHIPDATE text) AS
SELECT L_ORDERKEY, L_PARTKEY, L_SUPPKEY, L_LINENUMBER, L_QUANTITY,
L_EXTENDEDPRICE, L_DISCOUNT, L_TAX, L_RETURNFLAG, L_LINESTATUS, L_COMMITDATE,
L_RECEIPTDATE, L_SHIPINSTRUCT, L_SHIPMODE, L_COMMENT, L_SHIPDATE FROM LINEITEM;
{code}
* TSql logs
{code:xml}
Progress: 99%, response time: 1157.913 sec
Progress: 99%, response time: 1158.914 sec
Progress: 100%, response time: 1159.914 sec
Progress: 100%, response time: 1160.915 sec
Progress: 100%, response time: 1161.916 sec
Progress: 100%, response time: 1162.917 sec
Progress: 100%, response time: 1163.917 sec
Progress: 100%, response time: 1164.918 sec
Progress: 100%, response time: 1165.919 sec
Progress: 100%, response time: 1166.919 sec
Progress: 100%, response time: 1167.92 sec
Progress: 100%, response time: 1168.921 sec
Progress: 100%, response time: 1169.921 sec
Progress: 100%, response time: 1170.922 sec
Progress: 100%, response time: 1171.923 sec
Progress: 100%, response time: 1172.924 sec
Progress: 100%, response time: 1173.924 sec
Progress: 100%, response time: 1174.925 sec
{code}
* TajoWorker logs
{code:xml}
2016-01-08 07:50:56,158 INFO org.apache.tajo.querymaster.Stage: Stage completed
- eb_1452238256580_0001_000002 (total=6, success=6, killed=0)
2016-01-08 07:50:56,159 INFO org.apache.tajo.querymaster.Query: Processing
q_1452238256580_0001 of type STAGE_COMPLETED
2016-01-08 07:50:56,159 INFO
org.apache.tajo.engine.planner.global.ParallelExecutionQueue: Next executable
block eb_1452238256580_0001_000003
2016-01-08 07:50:56,159 INFO org.apache.tajo.querymaster.Query: Complete
Stage[eb_1452238256580_0001_000002], State: SUCCEEDED, 2/2.
2016-01-08 07:50:56,159 INFO org.apache.tajo.querymaster.Query: Processing
q_1452238256580_0001 of type QUERY_COMPLETED
2016-01-08 07:50:56,161 INFO org.apache.tajo.worker.ExecutionBlockContext:
Worker's task counter - total:6, succeeded: 6, killed: 0, failed: 0
2016-01-08 07:50:56,194 INFO org.apache.tajo.worker.TaskManager: Stopped
execution block:eb_1452238256580_0001_000002
{code}
I found that query hangs on FileTableSpace::commitTable. We need to implement
commitTable in S3TableSpace by considering s3 features.
was:
Query hangs while executing CTAS for partition table on aws as following:
* Root directory exists on s3
* Dataset : tpc-h 1G
* Query
{code:xml}
CREATE TABLE LINEITEM_P (L_ORDERKEY bigint, L_PARTKEY bigint, L_SUPPKEY bigint,
L_LINENUMBER bigint, L_QUANTITY double, L_EXTENDEDPRICE double, L_DISCOUNT
double, L_TAX double, L_RETURNFLAG text, L_LINESTATUS text, L_COMMITDATE text,
L_RECEIPTDATE text, L_SHIPINSTRUCT text, L_SHIPMODE text, L_COMMENT text)
USING TEXT WITH ('text.delimiter'='|')
PARTITION BY COLUMN (L_SHIPDATE text) AS
SELECT L_ORDERKEY, L_PARTKEY, L_SUPPKEY, L_LINENUMBER, L_QUANTITY,
L_EXTENDEDPRICE, L_DISCOUNT, L_TAX, L_RETURNFLAG, L_LINESTATUS, L_COMMITDATE,
L_RECEIPTDATE, L_SHIPINSTRUCT, L_SHIPMODE, L_COMMENT, L_SHIPDATE FROM LINEITEM;
{code}
* TSql logs
{code:xml}
Progress: 99%, response time: 1157.913 sec
Progress: 99%, response time: 1158.914 sec
Progress: 100%, response time: 1159.914 sec
Progress: 100%, response time: 1160.915 sec
Progress: 100%, response time: 1161.916 sec
Progress: 100%, response time: 1162.917 sec
Progress: 100%, response time: 1163.917 sec
Progress: 100%, response time: 1164.918 sec
Progress: 100%, response time: 1165.919 sec
Progress: 100%, response time: 1166.919 sec
Progress: 100%, response time: 1167.92 sec
Progress: 100%, response time: 1168.921 sec
Progress: 100%, response time: 1169.921 sec
Progress: 100%, response time: 1170.922 sec
Progress: 100%, response time: 1171.923 sec
Progress: 100%, response time: 1172.924 sec
Progress: 100%, response time: 1173.924 sec
Progress: 100%, response time: 1174.925 sec
{code}
* TajoWorker logs
{code:xml}
2016-01-08 07:50:56,158 INFO org.apache.tajo.querymaster.Stage: Stage completed
- eb_1452238256580_0001_000002 (total=6, success=6, killed=0)
2016-01-08 07:50:56,159 INFO org.apache.tajo.querymaster.Query: Processing
q_1452238256580_0001 of type STAGE_COMPLETED
2016-01-08 07:50:56,159 INFO
org.apache.tajo.engine.planner.global.ParallelExecutionQueue: Next executable
block eb_1452238256580_0001_000003
2016-01-08 07:50:56,159 INFO org.apache.tajo.querymaster.Query: Complete
Stage[eb_1452238256580_0001_000002], State: SUCCEEDED, 2/2.
2016-01-08 07:50:56,159 INFO org.apache.tajo.querymaster.Query: Processing
q_1452238256580_0001 of type QUERY_COMPLETED
2016-01-08 07:50:56,161 INFO org.apache.tajo.worker.ExecutionBlockContext:
Worker's task counter - total:6, succeeded: 6, killed: 0, failed: 0
2016-01-08 07:50:56,194 INFO org.apache.tajo.worker.TaskManager: Stopped
execution block:eb_1452238256580_0001_000002
{code}
I found that query hangs on FileTableSpace::commitTable. We need to implement
commitTable in S3FileTableSpace by considering s3 features.
> Query hangs occasionally while executing CTAS for partition table on aws
> ------------------------------------------------------------------------
>
> Key: TAJO-2047
> URL: https://issues.apache.org/jira/browse/TAJO-2047
> Project: Tajo
> Issue Type: Sub-task
> Components: S3, Storage
> Reporter: Jaehwa Jung
> Assignee: Jaehwa Jung
>
> Query hangs while executing CTAS for partition table on aws as following:
> * Root directory exists on s3
> * Dataset : tpc-h 1G
> * Query
> {code:xml}
> CREATE TABLE LINEITEM_P (L_ORDERKEY bigint, L_PARTKEY bigint, L_SUPPKEY
> bigint, L_LINENUMBER bigint, L_QUANTITY double, L_EXTENDEDPRICE double,
> L_DISCOUNT double, L_TAX double, L_RETURNFLAG text, L_LINESTATUS text,
> L_COMMITDATE text, L_RECEIPTDATE text, L_SHIPINSTRUCT text, L_SHIPMODE text,
> L_COMMENT text)
> USING TEXT WITH ('text.delimiter'='|')
> PARTITION BY COLUMN (L_SHIPDATE text) AS
> SELECT L_ORDERKEY, L_PARTKEY, L_SUPPKEY, L_LINENUMBER, L_QUANTITY,
> L_EXTENDEDPRICE, L_DISCOUNT, L_TAX, L_RETURNFLAG, L_LINESTATUS, L_COMMITDATE,
> L_RECEIPTDATE, L_SHIPINSTRUCT, L_SHIPMODE, L_COMMENT, L_SHIPDATE FROM
> LINEITEM;
> {code}
> * TSql logs
> {code:xml}
> Progress: 99%, response time: 1157.913 sec
> Progress: 99%, response time: 1158.914 sec
> Progress: 100%, response time: 1159.914 sec
> Progress: 100%, response time: 1160.915 sec
> Progress: 100%, response time: 1161.916 sec
> Progress: 100%, response time: 1162.917 sec
> Progress: 100%, response time: 1163.917 sec
> Progress: 100%, response time: 1164.918 sec
> Progress: 100%, response time: 1165.919 sec
> Progress: 100%, response time: 1166.919 sec
> Progress: 100%, response time: 1167.92 sec
> Progress: 100%, response time: 1168.921 sec
> Progress: 100%, response time: 1169.921 sec
> Progress: 100%, response time: 1170.922 sec
> Progress: 100%, response time: 1171.923 sec
> Progress: 100%, response time: 1172.924 sec
> Progress: 100%, response time: 1173.924 sec
> Progress: 100%, response time: 1174.925 sec
> {code}
> * TajoWorker logs
> {code:xml}
> 2016-01-08 07:50:56,158 INFO org.apache.tajo.querymaster.Stage: Stage
> completed - eb_1452238256580_0001_000002 (total=6, success=6, killed=0)
> 2016-01-08 07:50:56,159 INFO org.apache.tajo.querymaster.Query: Processing
> q_1452238256580_0001 of type STAGE_COMPLETED
> 2016-01-08 07:50:56,159 INFO
> org.apache.tajo.engine.planner.global.ParallelExecutionQueue: Next executable
> block eb_1452238256580_0001_000003
> 2016-01-08 07:50:56,159 INFO org.apache.tajo.querymaster.Query: Complete
> Stage[eb_1452238256580_0001_000002], State: SUCCEEDED, 2/2.
> 2016-01-08 07:50:56,159 INFO org.apache.tajo.querymaster.Query: Processing
> q_1452238256580_0001 of type QUERY_COMPLETED
> 2016-01-08 07:50:56,161 INFO org.apache.tajo.worker.ExecutionBlockContext:
> Worker's task counter - total:6, succeeded: 6, killed: 0, failed: 0
> 2016-01-08 07:50:56,194 INFO org.apache.tajo.worker.TaskManager: Stopped
> execution block:eb_1452238256580_0001_000002
> {code}
> I found that query hangs on FileTableSpace::commitTable. We need to implement
> commitTable in S3TableSpace by considering s3 features.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
