[
https://issues.apache.org/jira/browse/IMPALA-10557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17369623#comment-17369623
]
ASF subversion and git services commented on IMPALA-10557:
----------------------------------------------------------
Commit fcaea30b151d89f412816a8e49d5feeef6964a0f in impala's branch
refs/heads/master from wzhou-code
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=fcaea30 ]
IMPALA-10557: Support Kudu's multi-row transaction
Kudu added multi-row transaction so Impala could run query that inserts
multiple rows into Kudu's table in the context of a single transaction.
Kudu provides new Java/C++ client APIs to open/commit/rollback
transaction, create session with transaction, serialize/deserialize
metadata of transaction object. Kudu transaction object has built-in
heartbeater.
This patch added Impala support to use Kudu's multiple-row transaction.
- Added a new query option to enable Kudu's transaction.
- When the query option is set, a new Kudu transaction should be
started for "insert", "CTAS" and "UPDATE/UPSERT/DELETE" statements
by Impala's frontend of coordinator.
- The Kudu transaction objects are kept in KuduTransactionManager until
the transactions are going to be aborted or committed.
- Frontend serialize the transaction metadata into a transaction token
and pass to executors.
- Executors deserialize the transaction token and ingest via that
transaction handle. For Kudu session in the context of a transaction,
return the first error if there are any pending errors for the Kudu
session so that the Kudu transaction will be aborted.
Since Kudu does not support transaction for "UPDATE/UPSERT/DELETE"
statements now, Kudu returns error which causes transaction to be
aborted.
- Coordinator commits the transaction if everything goes well.
Otherwise, aborts the transaction.
Also changed code to store KuduClient as shared pointer since KuduClient
has to be passed as a shared pointer when KuduTransaction::Deserialize()
is called.
Testing:
- Added new e-to-e tests for Kudu transaction.
- Passed core test.
Change-Id: I876ada48991afdff5d61b5d6a0417571aba7cb34
Reviewed-on: http://gerrit.cloudera.org:8080/17553
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Allow Impala to use Kudu's multi-row transaction implementation
> ---------------------------------------------------------------
>
> Key: IMPALA-10557
> URL: https://issues.apache.org/jira/browse/IMPALA-10557
> Project: IMPALA
> Issue Type: New Feature
> Components: Clients
> Reporter: Andrew Wong
> Assignee: Wenzhe Zhou
> Priority: Major
>
> Once Kudu's support for multi-row transactions materializes, it'd be great if
> Impala could expose a query option or session-level flag to indicate that
> Impala should run queries that insert into Kudu in the context of a single
> transaction.
> When the query option is set, similar to how transactions are opened for Hive
> ACID tables, a new Kudu transaction should be started by Impala's Frontend,
> and the resulting KuduTransaction object should be serialized and passed as
> context to the Impala executors, which should open their Kudu sessions using
> the {{KuduTransaction::CreateSession()}} API instead of the usual
> {{KuduClient::NewSession()}}.
> Errors returned in the session should be propagated to the Impala Frontend
> before committing – it seems reasonable to expect that any session errors
> should result in Impala aborting the transaction.
> The {{KuduTransaction}} handle in the frontend has a heartbeater built in, so
> unlike Hive ACID transactions, which require manual tracking of heartbeating,
> no additional manual requests are needed to maintain the transaction.
> Kudu's Java client API is defined in [this
> patch|https://gerrit.cloudera.org/c/16894/], and C++ client API is defined in
> [this patch|https://gerrit.cloudera.org/c/16710/]. The implementation isn't
> fully operational quite yet – once [this patch to automatically register
> participants|https://gerrit.cloudera.org/c/17037/] lands, Impala developers
> should at least be able to start adding basic testing non-overlapping cases
> (partition locking is still in progress).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
