[
https://issues.apache.org/jira/browse/IMPALA-10773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17455376#comment-17455376
]
ASF subversion and git services commented on IMPALA-10773:
----------------------------------------------------------
Commit 157086cb80cb54c6565da25cc327621f6b2b8513 in impala's branch
refs/heads/master from Fucun Chu
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=157086c ]
IMPALA-10771: Add Tencent COS support
This patch adds support for COS(Cloud Object Storage). Using the
hadoop-cos, the implementation is similar to other remote FileSystems.
New flags for COS:
- num_cos_io_threads: Number of COS I/O threads. Defaults to be 16.
Follow-up:
- Support for caching COS file handles will be addressed in
IMPALA-10772.
- test_concurrent_inserts and test_failing_inserts in
test_acid_stress.py are skipped due to slow file listing on
COS (IMPALA-10773).
Tests:
- Upload hdfs test data to a COS bucket. Modify all locations in HMS
DB to point to the COS bucket. Remove some hdfs caching params.
Run CORE tests.
Change-Id: Idce135a7591d1b4c74425e365525be3086a39821
Reviewed-on: http://gerrit.cloudera.org:8080/17503
Reviewed-by: Joe McDonnell <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Stress insert tests timeout on COS
> ----------------------------------
>
> Key: IMPALA-10773
> URL: https://issues.apache.org/jira/browse/IMPALA-10773
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Reporter: Fucun Chu
> Priority: Major
>
> Like [IMPALA-10563|https://issues.apache.org/jira/browse/IMPALA-10563], the
> following tests also failed with TimeoutError when running on COS:
> {code}
> stress/test_acid_stress.py::TestConcurrentAcidInserts::test_concurrent_inserts[unique_database0]
> FAILED
> stress/test_acid_stress.py::TestFailingAcidInserts::test_failing_inserts[unique_database0]
> FAILED
> =================================== FAILURES
> ===================================
> _____ TestConcurrentAcidInserts.test_concurrent_inserts[unique_database0]
> ______
> stress/test_acid_stress.py:257: in test_concurrent_inserts
> run_tasks(writers + checkers)
> /home/impdev/Impala/tests/stress/stress_util.py:43: in run_tasks
> pool.map_async(Task.run, tasks).get(timeout_seconds)
> /home/impdev/Impala/toolchain/toolchain-packages-gcc7.5.0/python-2.7.16/lib/python2.7/multiprocessing/pool.py:568:
> in
> get
> raise TimeoutError
> E TimeoutError
> ________ TestFailingAcidInserts.test_failing_inserts[unique_database0]
> _________
> stress/test_acid_stress.py:387: in test_failing_inserts
> self._run_test_failing_inserts(unique_database, is_partitioned)
> stress/test_acid_stress.py:376: in _run_test_failing_inserts
> run_tasks(writers + checkers)
> stress/stress_util.py:45: in run_tasks
> pool.map_async(Task.run, tasks).get(timeout_seconds)
> /home/impdev/Impala/toolchain/toolchain-packages-gcc7.5.0/python-2.7.16/lib/python2.7/multiprocessing/pool.py:568:
> in
> get
> raise TimeoutError
> E TimeoutError
> {code}
> Found some error logs in HMS's log:
> {code}
> 2021-06-26T16:43:47,875 INFO [pool-8-thread-167] txn.TxnHandler: Allocated
> writeId: 1 for txnId: 4011
> 2021-06-26T16:43:47,876 INFO [pool-8-thread-167] txn.TxnHandler: Allocated
> write ids for dbName=test_concurrent_insert
> s_8933345c, tblName=test_concurrent_inserts (txnIds: [4011])
> 2021-06-26T16:43:47,879 INFO [main] thrift.TCustomSocket: Buffer size for
> TSocket is: 8192
> 2021-06-26T16:43:47,884 INFO [pool-8-thread-195] txn.TxnHandler: Added
> entries to MIN_HISTORY_LEVEL for current txns:
> ([4014]) with min_open_txn: 4010
> 2021-06-26T16:43:47,883 ERROR [pool-8-thread-162] txn.TxnHandler: Exception
> during write ids allocation for request=All
> ocateTableWriteIdsRequest(dbName:test_concurrent_inserts_8933345c,
> tableName:test_concurrent_inserts, txnIds:[4010]). W
> ill retry if possible.
> org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique
> constraint "NEXT_WRITE_ID_IDX"
> Detail: Key ("NWI_DATABASE",
> "NWI_TABLE")=(test_concurrent_inserts_8933345c, test_concurrent_inserts)
> already exists.
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2532)
> ~[postgresql-42.2
> .14.jar:42.2.14]
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2267)
> ~[postgresql-42.2.14.ja
> r:42.2.14]
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:312)
> ~[postgresql-42.2.14.jar:42.2.1
> 4]
> at
> org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:448)
> ~[postgresql-42.2.14.jar:42.2.14]
> /ERROR
>
> at
> org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> [hive-exec-3.1.3000
> .7.2.11.0-94.jar:3.1.3000.7.2.11.0-94]
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> [hive-exec-3.1.3000
> .7.2.11.0-94.jar:3.1.3000.7.2.11.0-94]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [?:1.8.0_292]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [?:1.8.0_292]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
> 2021-06-26T16:43:47,888 WARN [pool-8-thread-162] txn.TxnHandler: Retryable
> error detected in allocateTableWriteIds(All
> ocateTableWriteIdsRequest(dbName:test_concurrent_inserts_8933345c,
> tableName:test_concurrent_inserts, txnIds:[4010])).
> Will wait 2000ms and retry up to 10 times. Error: ERROR: duplicate key
> value violates unique constraint "NEXT_WRITE_I
> D_IDX"
> Detail: Key ("NWI_DATABASE",
> "NWI_TABLE")=(test_concurrent_inserts_8933345c, test_concurrent_inserts)
> already exists.
> 2021-06-26T16:43:47,888 INFO [pool-8-thread-179] txn.TxnHandler: Added
> entries to MIN_HISTORY_LEVEL for current txns:
> ([4015]) with min_open_txn: 4010
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
