[jira] [Commented] (HIVE-18285) StatsTask uses a cached ql.metadata.Table object
[ https://issues.apache.org/jira/browse/HIVE-18285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16300971#comment-16300971 ] Eugene Koifman commented on HIVE-18285: --- this patch should not add any time to compilation since it only changes where StatsTask gets the Table object when it runs. I believe this happens on the client side and long after compilation is done. bq. Secondly, logic to convert table type based on config to me seems to belong to compiler. Metastore shall not govern that. Table is a metadata object and whether a table supports Acid ops is a property of this metadata object. Seems like Metastore is exactly the place to govern metadata objects. Normally compiler just reads metadata objects. Finally, perhaps most importantly: A large part of acid is in the metastore. The entire state of compactor, lock manger, transaction manager is currently managed by the metastore. This is very specific to Hive. If you don't think Hive specific logic belongs in the metastore, where do you see all this logic going once metastore becomes standalone? > StatsTask uses a cached ql.metadata.Table object > > > Key: HIVE-18285 > URL: https://issues.apache.org/jira/browse/HIVE-18285 > Project: Hive > Issue Type: Bug > Components: Metastore, Statistics >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18285.01.patch > > > this then causes BasicStatsTask.aggregateStats(Hive) to call > Hive.alterTable() with a stale Table object. (It misses any changes made by > any MetaStorePreEventListener) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18285) StatsTask uses a cached ql.metadata.Table object
[ https://issues.apache.org/jira/browse/HIVE-18285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16300771#comment-16300771 ] Ashutosh Chauhan commented on HIVE-18285: - I agree with Zoltan. We don't want to make extra metastore calls. All modifications on metadata objects shall be made on compiler side. Metastore should just be used to read and persist metadata objects. This is because of 2 reasons : a) Making metastore calls add latencies to compilation. b) Having modifications being done on metadata objects in 2 different processes makes it hard to reason about what processes are making what changes. Secondly, logic to convert table type based on config to me seems to belong to compiler. Metastore shall not govern that. Lastly, with standalone-metastore ACID concept seems like specific to Hive and may not be applicable to other engines. So, Hive compiler seems like a logical place for that logic, instead of metastore which is deemed to be used by multiple engines. > StatsTask uses a cached ql.metadata.Table object > > > Key: HIVE-18285 > URL: https://issues.apache.org/jira/browse/HIVE-18285 > Project: Hive > Issue Type: Bug > Components: Metastore, Statistics >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18285.01.patch > > > this then causes BasicStatsTask.aggregateStats(Hive) to call > Hive.alterTable() with a stale Table object. (It misses any changes made by > any MetaStorePreEventListener) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18285) StatsTask uses a cached ql.metadata.Table object
[ https://issues.apache.org/jira/browse/HIVE-18285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298207#comment-16298207 ] Zoltan Haindrich commented on HIVE-18285: - I just thinked about adding a testcase which triggers the path you have just found that it's broken... set a preevent listener which changes somethingand then show that the listener is respected by the statstask. but anyway I'm not against this change: +1 > StatsTask uses a cached ql.metadata.Table object > > > Key: HIVE-18285 > URL: https://issues.apache.org/jira/browse/HIVE-18285 > Project: Hive > Issue Type: Bug > Components: Metastore, Statistics >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18285.01.patch > > > this then causes BasicStatsTask.aggregateStats(Hive) to call > Hive.alterTable() with a stale Table object. (It misses any changes made by > any MetaStorePreEventListener) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18285) StatsTask uses a cached ql.metadata.Table object
[ https://issues.apache.org/jira/browse/HIVE-18285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297124#comment-16297124 ] Eugene Koifman commented on HIVE-18285: --- [~kgyrtkirk] a test to make sure someone doesn't start caching these object? I'm not sure how to do that. > StatsTask uses a cached ql.metadata.Table object > > > Key: HIVE-18285 > URL: https://issues.apache.org/jira/browse/HIVE-18285 > Project: Hive > Issue Type: Bug > Components: Metastore, Statistics >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18285.01.patch > > > this then causes BasicStatsTask.aggregateStats(Hive) to call > Hive.alterTable() with a stale Table object. (It misses any changes made by > any MetaStorePreEventListener) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18285) StatsTask uses a cached ql.metadata.Table object
[ https://issues.apache.org/jira/browse/HIVE-18285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296522#comment-16296522 ] Zoltan Haindrich commented on HIVE-18285: - There were some efforts to reduce the used metastore calls; but ddltask sends the table object to metastore - and since statstask needs to update the stats (which are currently stored in properties ; sends a second table call I think this could be avoided later somehow ; when statstask makes less metastore calls ; I've opened HIVE-18302. I don't feel right to change the table object fundamentally in metastore listeners...I've found examples where inputformat have been changed to some other... [~ekoifman]: Could you please add a testcase to protect this usecase in the future? > StatsTask uses a cached ql.metadata.Table object > > > Key: HIVE-18285 > URL: https://issues.apache.org/jira/browse/HIVE-18285 > Project: Hive > Issue Type: Bug > Components: Metastore, Statistics >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18285.01.patch > > > this then causes BasicStatsTask.aggregateStats(Hive) to call > Hive.alterTable() with a stale Table object. (It misses any changes made by > any MetaStorePreEventListener) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18285) StatsTask uses a cached ql.metadata.Table object
[ https://issues.apache.org/jira/browse/HIVE-18285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296026#comment-16296026 ] Ashutosh Chauhan commented on HIVE-18285: - [~kgyrtkirk] Can you please review this? > StatsTask uses a cached ql.metadata.Table object > > > Key: HIVE-18285 > URL: https://issues.apache.org/jira/browse/HIVE-18285 > Project: Hive > Issue Type: Bug > Components: Metastore, Statistics >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-18285.01.patch > > > this then causes BasicStatsTask.aggregateStats(Hive) to call > Hive.alterTable() with a stale Table object. (It misses any changes made by > any MetaStorePreEventListener) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18285) StatsTask uses a cached ql.metadata.Table object
[
https://issues.apache.org/jira/browse/HIVE-18285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293399#comment-16293399
]
Hive QA commented on HIVE-18285:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12902421/HIVE-18285.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 11503 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[quotedid_smb]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[archive_partspec2]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_authorization_create_no_grant]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_notin_implicit_gby]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_10]
(batchId=138)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketsortoptimize_insert_7]
(batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=113)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=209)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=226)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8277/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8277/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8277/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 21 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12902421 - PreCommit-HIVE-Build
> StatsTask uses a cached ql.metadata.Table object
>
>
> Key: HIVE-18285
> URL: https://issues.apache.org/jira/browse/HIVE-18285
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Statistics
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18285.01.patch
>
>
> this then causes BasicStatsTask.aggregateStats(Hive) to call
> Hive.alterTable() with a stale Table object. (It misses any changes made by
> any MetaStorePreEventListener)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18285) StatsTask uses a cached ql.metadata.Table object
[
https://issues.apache.org/jira/browse/HIVE-18285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293363#comment-16293363
]
Hive QA commented on HIVE-18285:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / f52e8b4 |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8277/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> StatsTask uses a cached ql.metadata.Table object
>
>
> Key: HIVE-18285
> URL: https://issues.apache.org/jira/browse/HIVE-18285
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Statistics
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18285.01.patch
>
>
> this then causes BasicStatsTask.aggregateStats(Hive) to call
> Hive.alterTable() with a stale Table object. (It misses any changes made by
> any MetaStorePreEventListener)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
