[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449094#comment-16449094
]
Jason Dere commented on HIVE-17645:
---
Ported to branch-3
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Jason Dere
>Priority: Major
> Labels: mm-gap-2
> Fix For: 3.0.0
>
> Attachments: HIVE-17645.1.patch, HIVE-17645.2.patch,
> HIVE-17645.3.patch
>
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443205#comment-16443205
]
Eugene Koifman commented on HIVE-17645:
---
HIVE-19240 for backport
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Jason Dere
>Priority: Major
> Labels: mm-gap-2
> Fix For: 3.1.0
>
> Attachments: HIVE-17645.1.patch, HIVE-17645.2.patch,
> HIVE-17645.3.patch
>
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443192#comment-16443192
]
Jason Dere commented on HIVE-17645:
---
Agreed, I didn’t commit this so didn’t occur to me. Can you port?
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Jason Dere
>Priority: Major
> Labels: mm-gap-2
> Fix For: 3.1.0
>
> Attachments: HIVE-17645.1.patch, HIVE-17645.2.patch,
> HIVE-17645.3.patch
>
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443184#comment-16443184
]
Eugene Koifman commented on HIVE-17645:
---
[~jdere], shouldn't this go to 3.0 as well?
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Jason Dere
>Priority: Major
> Labels: mm-gap-2
> Fix For: 3.1.0
>
> Attachments: HIVE-17645.1.patch, HIVE-17645.2.patch,
> HIVE-17645.3.patch
>
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438764#comment-16438764
]
Eugene Koifman commented on HIVE-17645:
---
+1
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Jason Dere
>Priority: Major
> Labels: mm-gap-2
> Attachments: HIVE-17645.1.patch, HIVE-17645.2.patch,
> HIVE-17645.3.patch
>
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438622#comment-16438622
]
Sankar Hariappan commented on HIVE-17645:
-
[~jdere],
You don't have to wait up for HIVE-18988. It will take some more time to go in.
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Jason Dere
>Priority: Major
> Labels: mm-gap-2
> Attachments: HIVE-17645.1.patch, HIVE-17645.2.patch,
> HIVE-17645.3.patch
>
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438418#comment-16438418
]
Jason Dere commented on HIVE-17645:
---
Tests look good. [~ekoifman] can you review when you get a chance? Or should I
rebase once HIVE-18988 is in?
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Jason Dere
>Priority: Major
> Labels: mm-gap-2
> Attachments: HIVE-17645.1.patch, HIVE-17645.2.patch,
> HIVE-17645.3.patch
>
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438313#comment-16438313
]
Hive QA commented on HIVE-17645:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12918620/HIVE-17645.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 61 failed/errored test(s), 13760 tests
executed
*Failed tests:*
{noformat}
TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed
out) (batchId=247)
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=149)
[mapreduce2.q,orc_llap_counters1.q,bucket6.q,insert_into1.q,empty_dir_in_table.q,parquet_struct_type_vectorization.q,orc_merge1.q,parquet_types_vectorization.q,orc_merge_diff_fs.q,llap_stats.q,llapdecider.q,llap_nullscan.q,orc_ppd_basic.q,rcfile_merge4.q,orc_merge3.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438296#comment-16438296
]
Hive QA commented on HIVE-17645:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
54s{color} | {color:red} ql: The patch generated 7 new + 1295 unchanged - 0
fixed = 1302 total (was 1295) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10199/dev-support/hive-personality.sh
|
| git revision | master / 24969a8 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10199/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10199/yetus/patch-asflicense-problems.txt
|
| modules | C: llap-ext-client ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10199/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Jason Dere
>Priority: Major
> Labels: mm-gap-2
> Attachments: HIVE-17645.1.patch, HIVE-17645.2.patch,
> HIVE-17645.3.patch
>
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433414#comment-16433414
]
Sankar Hariappan commented on HIVE-17645:
-
[~ekoifman], [~jdere],
The changes in replication related code looks good to me.
However, Repl bootstrap doesn't need txnMgr anymore. It will be removed as part
of HIVE-18988.
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Jason Dere
>Priority: Major
> Labels: mm-gap-2
> Attachments: HIVE-17645.1.patch, HIVE-17645.2.patch
>
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433374#comment-16433374
]
Jason Dere commented on HIVE-17645:
---
No need for changes in TestAcidOnTez, it's the changes related to TxnMgr usage
and LLAP external client which fix the test.
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Jason Dere
>Priority: Major
> Labels: mm-gap-2
> Attachments: HIVE-17645.1.patch, HIVE-17645.2.patch
>
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433347#comment-16433347
]
Eugene Koifman commented on HIVE-17645:
---
I think SessionState.getTxnMgr() needs a detailed comment explaining that it
should not generally be used and what should be used instead.
I don't see TestAcidOnTez in the patch. Should it be?
CC [~sankarh], [~anishek] - there are some minor Replication changes
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Jason Dere
>Priority: Major
> Labels: mm-gap-2
> Attachments: HIVE-17645.1.patch, HIVE-17645.2.patch
>
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433288#comment-16433288
]
Jason Dere commented on HIVE-17645:
---
[~ekoifman] can you review this patch?
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Jason Dere
>Priority: Major
> Labels: mm-gap-2
> Attachments: HIVE-17645.1.patch, HIVE-17645.2.patch
>
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433286#comment-16433286
]
Jason Dere commented on HIVE-17645:
---
Updating patch - missed a couple of uses of SessionState.getTxnMgr() from
CalcitePlanner/MaterializedViewRebuildSemanticAnalyzer.
Also adding a couple of fixes to fix TestAcidOnTez which also depend on the
rest of this patch.
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Jason Dere
>Priority: Major
> Labels: mm-gap-2
> Attachments: HIVE-17645.1.patch, HIVE-17645.2.patch
>
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431045#comment-16431045
]
Jason Dere commented on HIVE-17645:
---
RB at https://reviews.apache.org/r/66514/
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Jason Dere
>Priority: Major
> Labels: mm-gap-2
> Attachments: HIVE-17645.1.patch
>
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431042#comment-16431042
]
Jason Dere commented on HIVE-17645:
---
Adding a patch to try to replace the usage of SessionState.getTxnMgr() in the
non-test code. There are still a couple of tests that use
SessionState.getTxnMgr(), so I have not removed this yet.
[~ekoifman] [~sershe] does this look ok?
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Jason Dere
>Priority: Major
> Labels: mm-gap-2
> Attachments: HIVE-17645.1.patch
>
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428759#comment-16428759
]
Sergey Shelukhin commented on HIVE-17645:
-
If there's a reasonable way to get the txn manager everywhere, session API
should just be removed.
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Jason Dere
>Priority: Major
> Labels: mm-gap-2
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428748#comment-16428748
]
Jason Dere commented on HIVE-17645:
---
[~sershe] brought this to my attention as well, plus the fact that the
sessionstate txn manager is being called in some of the semantic analyzer code:
- SemanticAnalyzer.genFileSinkPlan()
- ReplicationSemanticAnalyzer.analyzeReplLoad()
- LoadSemanticAnalyzer.analyzeExternal()
- ImportSemanticAnalyzer.prepareImport()
- DDLSemanticAnalyzer.handleTransactionalTable()
As Sergey mentions earlier, most of these (with the exception of
genFileSinkPlan) are probably ok since they are related t DDL/import/repl,
which would not happen for HIVE-17482. But probably good to fix these so that
all examples of getting the txn manager use the same call.
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Labels: mm-gap-2
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424902#comment-16424902
]
Sergey Shelukhin commented on HIVE-17645:
-
I see code everywhere using session txn manager, not just in MM tables.
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Labels: mm-gap-2
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211914#comment-16211914
]
Eugene Koifman commented on HIVE-17645:
---
There is a single LM for the warehouse. Having multiple TMs is like having
multiple Sessions - they can lock the same resources as long as requested locks
are compatible.
The issue is that each TM has it's own transaction context, i.e. in a different
transaction. For Spark reads, each Query Fragment uses a separate TM (and
ValidTxnList) so from the overall query perspective this creates Read Committed
semantics.
(to be continued)
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
> Labels: mm-gap-2
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211886#comment-16211886
]
Sergey Shelukhin commented on HIVE-17645:
-
Spark+ACID only uses the hacky mode for select queries so it should be ok as
long as we don't get it from the session for selects.
However, a larger concern I have is this... how does it work at all if a
different TxnManager has non shared state with the main one? They'd be able to
take locks separately in parallel for the same things.
And if they don't have non-shared state (rely on the same metastore DB, ZK/DB
lock paths, etc) then what's the problem with getting a different txn manager?
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
> Labels: mm-gap-2
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211844#comment-16211844
]
Eugene Koifman commented on HIVE-17645:
---
Ask [~jdere]
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
> Labels: mm-gap-2
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17645) MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
[
https://issues.apache.org/jira/browse/HIVE-17645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211822#comment-16211822
]
Sergey Shelukhin commented on HIVE-17645:
-
Hmm.. how can it be valid to have multiple txn managers in a single HS2?
> MM tables patch conflicts with HIVE-17482 (Spark/Acid integration)
> --
>
> Key: HIVE-17645
> URL: https://issues.apache.org/jira/browse/HIVE-17645
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>
> MM code introduces
> {noformat}
> HiveTxnManager txnManager = SessionState.get().getTxnMgr()
> {noformat}
> in a number of places (e.g _DDLTask.generateAddMmTasks(Table tbl)_).
> HIVE-17482 adds a mode where a TransactionManager not associated with the
> session should be used. This will need to be addressed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
