[jira] [Commented] (HIVE-22191) Simplify SemanticAnalyzer by removing unused code
[
https://issues.apache.org/jira/browse/HIVE-22191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939228#comment-16939228
]
Laszlo Bodor commented on HIVE-22191:
-
+1
> Simplify SemanticAnalyzer by removing unused code
> -
>
> Key: HIVE-22191
> URL: https://issues.apache.org/jira/browse/HIVE-22191
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Akos Dombi
>Assignee: Akos Dombi
>Priority: Major
> Attachments: HIVE-22191.2.patch, HIVE-22191.3.patch,
> HIVE-22191.4.patch, HIVE-22191.5.patch, HIVE-22191.5.patch, HIVE-22191.patch
>
>
> Simplify {{SemanticAnalyzer}} by:
> - Remove dead code
> - Simplify returning statements
> - Use interfaces types for parameters/fields/variables where it is
> straightforward to migrate
> - Make visibility stricter where it is possible
> - Check logging to use parametrised logging
> - Removing unnecessary keywords (e.g.: {{static}})
> - Some code part could be simplified by using Java 8 features
> I think this is crucial step as this class already contains 15000+ lines of
> code which is screaming for splitting into more reasonable classes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21164) ACID: explore how we can avoid a move step during inserts/compaction
[ https://issues.apache.org/jira/browse/HIVE-21164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21164: Attachment: HIVE-21164.11.patch > ACID: explore how we can avoid a move step during inserts/compaction > > > Key: HIVE-21164 > URL: https://issues.apache.org/jira/browse/HIVE-21164 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-21164.1.patch, HIVE-21164.10.patch, > HIVE-21164.11.patch, HIVE-21164.11.patch, HIVE-21164.2.patch, > HIVE-21164.3.patch, HIVE-21164.4.patch, HIVE-21164.5.patch, > HIVE-21164.6.patch, HIVE-21164.7.patch, HIVE-21164.8.patch, HIVE-21164.9.patch > > > Currently, we write compacted data to a temporary location and then move the > files to a final location, which is an expensive operation on some cloud file > systems. Since HIVE-20823 is already in, it can control the visibility of > compacted data for the readers. Therefore, we can perhaps avoid writing data > to a temporary location and directly write compacted data to the intended > final path. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22217) Better Logging for Hive JAR Reload
[ https://issues.apache.org/jira/browse/HIVE-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22217: Resolution: Fixed Status: Resolved (was: Patch Available) > Better Logging for Hive JAR Reload > -- > > Key: HIVE-22217 > URL: https://issues.apache.org/jira/browse/HIVE-22217 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.2.0, 2.3.6 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22217.1.patch > > > Troubleshooting Hive Reloadable Auxiliary JARs has always been difficult. > Add logging to at least confirm which JAR files are being loaded. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22217) Better Logging for Hive JAR Reload
[ https://issues.apache.org/jira/browse/HIVE-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22217: Fix Version/s: 4.0.0 > Better Logging for Hive JAR Reload > -- > > Key: HIVE-22217 > URL: https://issues.apache.org/jira/browse/HIVE-22217 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.2.0, 2.3.6 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22217.1.patch > > > Troubleshooting Hive Reloadable Auxiliary JARs has always been difficult. > Add logging to at least confirm which JAR files are being loaded. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22217) Better Logging for Hive JAR Reload
[ https://issues.apache.org/jira/browse/HIVE-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934990#comment-16934990 ] Laszlo Bodor commented on HIVE-22217: - pushed to master, thanks [~belugabehr] for the patch and [~ngangam] for the review! > Better Logging for Hive JAR Reload > -- > > Key: HIVE-22217 > URL: https://issues.apache.org/jira/browse/HIVE-22217 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.2.0, 2.3.6 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22217.1.patch > > > Troubleshooting Hive Reloadable Auxiliary JARs has always been difficult. > Add logging to at least confirm which JAR files are being loaded. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21164) ACID: explore how we can avoid a move step during inserts/compaction
[ https://issues.apache.org/jira/browse/HIVE-21164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21164: Attachment: HIVE-21164.01.patch > ACID: explore how we can avoid a move step during inserts/compaction > > > Key: HIVE-21164 > URL: https://issues.apache.org/jira/browse/HIVE-21164 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-21164.1.patch, HIVE-21164.10.patch, > HIVE-21164.11.patch, HIVE-21164.2.patch, HIVE-21164.3.patch, > HIVE-21164.4.patch, HIVE-21164.5.patch, HIVE-21164.6.patch, > HIVE-21164.7.patch, HIVE-21164.8.patch, HIVE-21164.9.patch > > > Currently, we write compacted data to a temporary location and then move the > files to a final location, which is an expensive operation on some cloud file > systems. Since HIVE-20823 is already in, it can control the visibility of > compacted data for the readers. Therefore, we can perhaps avoid writing data > to a temporary location and directly write compacted data to the intended > final path. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21164) ACID: explore how we can avoid a move step during inserts/compaction
[ https://issues.apache.org/jira/browse/HIVE-21164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21164: Attachment: (was: HIVE-21164.01.patch) > ACID: explore how we can avoid a move step during inserts/compaction > > > Key: HIVE-21164 > URL: https://issues.apache.org/jira/browse/HIVE-21164 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-21164.1.patch, HIVE-21164.10.patch, > HIVE-21164.11.patch, HIVE-21164.2.patch, HIVE-21164.3.patch, > HIVE-21164.4.patch, HIVE-21164.5.patch, HIVE-21164.6.patch, > HIVE-21164.7.patch, HIVE-21164.8.patch, HIVE-21164.9.patch > > > Currently, we write compacted data to a temporary location and then move the > files to a final location, which is an expensive operation on some cloud file > systems. Since HIVE-20823 is already in, it can control the visibility of > compacted data for the readers. Therefore, we can perhaps avoid writing data > to a temporary location and directly write compacted data to the intended > final path. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21164) ACID: explore how we can avoid a move step during inserts/compaction
[ https://issues.apache.org/jira/browse/HIVE-21164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21164: Attachment: HIVE-21164.11.patch > ACID: explore how we can avoid a move step during inserts/compaction > > > Key: HIVE-21164 > URL: https://issues.apache.org/jira/browse/HIVE-21164 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-21164.1.patch, HIVE-21164.10.patch, > HIVE-21164.11.patch, HIVE-21164.2.patch, HIVE-21164.3.patch, > HIVE-21164.4.patch, HIVE-21164.5.patch, HIVE-21164.6.patch, > HIVE-21164.7.patch, HIVE-21164.8.patch, HIVE-21164.9.patch > > > Currently, we write compacted data to a temporary location and then move the > files to a final location, which is an expensive operation on some cloud file > systems. Since HIVE-20823 is already in, it can control the visibility of > compacted data for the readers. Therefore, we can perhaps avoid writing data > to a temporary location and directly write compacted data to the intended > final path. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22202) Remove unnecessary code from ObjectStore.java
[ https://issues.apache.org/jira/browse/HIVE-22202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22202: Component/s: Standalone Metastore Metastore > Remove unnecessary code from ObjectStore.java > - > > Key: HIVE-22202 > URL: https://issues.apache.org/jira/browse/HIVE-22202 > Project: Hive > Issue Type: Task > Components: Hive, Metastore, Standalone Metastore >Reporter: Hankó Gergely >Assignee: Hankó Gergely >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22202.1.patch, HIVE-22202.2.patch, > HIVE-22202.3.patch > > > Remove unnecessary code > * remove unnecessary/unused variables > * remove unused private methods > * remove unnecessary throws declarations > * remove unnecessary conditionals that are guaranteed to always give the > same result > * simplify collection handling > ** use member sort() instead of Collections.sort() > ** use addAll() instead of adding one-by-one in a loop > * remove Javadoc that doesn't give additional information > * use try with resource where possible > * remove unnecessary explicit type arguments > * remove redundant warning suppression -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22202) Remove unnecessary code from ObjectStore.java
[ https://issues.apache.org/jira/browse/HIVE-22202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22202: Fix Version/s: 4.0.0 > Remove unnecessary code from ObjectStore.java > - > > Key: HIVE-22202 > URL: https://issues.apache.org/jira/browse/HIVE-22202 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Hankó Gergely >Assignee: Hankó Gergely >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22202.1.patch, HIVE-22202.2.patch, > HIVE-22202.3.patch > > > Remove unnecessary code > * remove unnecessary/unused variables > * remove unused private methods > * remove unnecessary throws declarations > * remove unnecessary conditionals that are guaranteed to always give the > same result > * simplify collection handling > ** use member sort() instead of Collections.sort() > ** use addAll() instead of adding one-by-one in a loop > * remove Javadoc that doesn't give additional information > * use try with resource where possible > * remove unnecessary explicit type arguments > * remove redundant warning suppression -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22202) Remove unnecessary code from ObjectStore.java
[ https://issues.apache.org/jira/browse/HIVE-22202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22202: Resolution: Fixed Status: Resolved (was: Patch Available) > Remove unnecessary code from ObjectStore.java > - > > Key: HIVE-22202 > URL: https://issues.apache.org/jira/browse/HIVE-22202 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Hankó Gergely >Assignee: Hankó Gergely >Priority: Minor > Attachments: HIVE-22202.1.patch, HIVE-22202.2.patch, > HIVE-22202.3.patch > > > Remove unnecessary code > * remove unnecessary/unused variables > * remove unused private methods > * remove unnecessary throws declarations > * remove unnecessary conditionals that are guaranteed to always give the > same result > * simplify collection handling > ** use member sort() instead of Collections.sort() > ** use addAll() instead of adding one-by-one in a loop > * remove Javadoc that doesn't give additional information > * use try with resource where possible > * remove unnecessary explicit type arguments > * remove redundant warning suppression -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22202) Remove unnecessary code from ObjectStore.java
[ https://issues.apache.org/jira/browse/HIVE-22202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934200#comment-16934200 ] Laszlo Bodor commented on HIVE-22202: - pushed to master, thanks [~ghanko] for the patch! > Remove unnecessary code from ObjectStore.java > - > > Key: HIVE-22202 > URL: https://issues.apache.org/jira/browse/HIVE-22202 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Hankó Gergely >Assignee: Hankó Gergely >Priority: Minor > Attachments: HIVE-22202.1.patch, HIVE-22202.2.patch, > HIVE-22202.3.patch > > > Remove unnecessary code > * remove unnecessary/unused variables > * remove unused private methods > * remove unnecessary throws declarations > * remove unnecessary conditionals that are guaranteed to always give the > same result > * simplify collection handling > ** use member sort() instead of Collections.sort() > ** use addAll() instead of adding one-by-one in a loop > * remove Javadoc that doesn't give additional information > * use try with resource where possible > * remove unnecessary explicit type arguments > * remove redundant warning suppression -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22202) Remove unnecessary code from ObjectStore.java
[ https://issues.apache.org/jira/browse/HIVE-22202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932255#comment-16932255 ] Laszlo Bodor commented on HIVE-22202: - thanks for the clarification +1 > Remove unnecessary code from ObjectStore.java > - > > Key: HIVE-22202 > URL: https://issues.apache.org/jira/browse/HIVE-22202 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Hankó Gergely >Assignee: Hankó Gergely >Priority: Minor > Attachments: HIVE-22202.1.patch, HIVE-22202.2.patch, > HIVE-22202.3.patch > > > Remove unnecessary code > * remove unnecessary/unused variables > * remove unused private methods > * remove unnecessary throws declarations > * remove unnecessary conditionals that are guaranteed to always give the > same result > * simplify collection handling > ** use member sort() instead of Collections.sort() > ** use addAll() instead of adding one-by-one in a loop > * remove Javadoc that doesn't give additional information > * use try with resource where possible > * remove unnecessary explicit type arguments > * remove redundant warning suppression -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22202) Remove unnecessary code from ObjectStore.java
[ https://issues.apache.org/jira/browse/HIVE-22202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932244#comment-16932244 ] Laszlo Bodor edited comment on HIVE-22202 at 9/18/19 9:42 AM: -- [~ghanko]: thanks for this refactor I can't see why this line is removed: https://github.com/abstractdog/hive/commit/3c92843f042ecdff89aa473f0f3b6bf2f7dd61a8#diff-33821c5e69c8a323e566c485d0b3db7bL948 other than that, it looks good to me was (Author: abstractdog): [~ghanko]: thanks for this refactor I can't why this line is removed: https://github.com/abstractdog/hive/commit/3c92843f042ecdff89aa473f0f3b6bf2f7dd61a8#diff-33821c5e69c8a323e566c485d0b3db7bL948 other than that, it looks good to me > Remove unnecessary code from ObjectStore.java > - > > Key: HIVE-22202 > URL: https://issues.apache.org/jira/browse/HIVE-22202 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Hankó Gergely >Assignee: Hankó Gergely >Priority: Minor > Attachments: HIVE-22202.1.patch, HIVE-22202.2.patch, > HIVE-22202.3.patch > > > Remove unnecessary code > * remove unnecessary/unused variables > * remove unused private methods > * remove unnecessary throws declarations > * remove unnecessary conditionals that are guaranteed to always give the > same result > * simplify collection handling > ** use member sort() instead of Collections.sort() > ** use addAll() instead of adding one-by-one in a loop > * remove Javadoc that doesn't give additional information > * use try with resource where possible > * remove unnecessary explicit type arguments > * remove redundant warning suppression -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22202) Remove unnecessary code from ObjectStore.java
[ https://issues.apache.org/jira/browse/HIVE-22202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932244#comment-16932244 ] Laszlo Bodor commented on HIVE-22202: - [~ghanko]: thanks for this refactor I can't why this line is removed: https://github.com/abstractdog/hive/commit/3c92843f042ecdff89aa473f0f3b6bf2f7dd61a8#diff-33821c5e69c8a323e566c485d0b3db7bL948 other than that, it looks good to me > Remove unnecessary code from ObjectStore.java > - > > Key: HIVE-22202 > URL: https://issues.apache.org/jira/browse/HIVE-22202 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Hankó Gergely >Assignee: Hankó Gergely >Priority: Minor > Attachments: HIVE-22202.1.patch, HIVE-22202.2.patch, > HIVE-22202.3.patch > > > Remove unnecessary code > * remove unnecessary/unused variables > * remove unused private methods > * remove unnecessary throws declarations > * remove unnecessary conditionals that are guaranteed to always give the > same result > * simplify collection handling > ** use member sort() instead of Collections.sort() > ** use addAll() instead of adding one-by-one in a loop > * remove Javadoc that doesn't give additional information > * use try with resource where possible > * remove unnecessary explicit type arguments > * remove redundant warning suppression -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22211) Change maven phase to generate test sources
[ https://issues.apache.org/jira/browse/HIVE-22211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931468#comment-16931468 ] Laszlo Bodor commented on HIVE-22211: - +1 > Change maven phase to generate test sources > --- > > Key: HIVE-22211 > URL: https://issues.apache.org/jira/browse/HIVE-22211 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22211.01.patch > > > Some protobuf files are generated in the wrong phase; so I get compile errors > because they are not there for eclipse... -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22210) Vectorization may reuse computation output columns involved in filtering
[
https://issues.apache.org/jira/browse/HIVE-22210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931288#comment-16931288
]
Laszlo Bodor commented on HIVE-22210:
-
+1, pending tests
> Vectorization may reuse computation output columns involved in filtering
>
>
> Key: HIVE-22210
> URL: https://issues.apache.org/jira/browse/HIVE-22210
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22210.01.patch
>
>
> running the following test with TestMiniLlapLocalCliDriver leads to an
> unexpected results; the coalesce calculated inside the subquery has a value
> of 1 instead of the correct(922) value.
> {code}
> drop table if exists u_table_4;
> create table u_table_4(smallint_col_22 smallint, int_col_5 int);
> insert into u_table_4 values(238,922);
> drop table u_table_7;
> create table u_table_7 ( bigint_col_3 bigint, int_col_10 int);
> insert into u_table_7 values (571,198);
> drop table u_table_19;
> create table u_table_19 (bigint_col_18 bigint ,int_col_19 int, STRING_COL_7
> string);
> insert into u_table_19 values (922,5,'500');
> set hive.mapjoin.full.outer=true;
> set hive.auto.convert.join=true;
> set hive.query.results.cache.enabled=false;
> set hive.merge.nway.joins=true;
> set hive.vectorized.execution.enabled=true;
> --explain analyze
> SELECT
> a5.int_col,
> 922 as expected,
> COALESCE(a5.int_col, a5.aa) as expected2,
> a5.int_col_3 as reality
> FROMu_table_19 a1
> FULL OUTER JOIN
> (
>SELECT a2.int_col_5 AS int_col,
> a2.smallint_col_22 as aa,
> COALESCE(a2.int_col_5, a2.smallint_col_22) AS
> int_col_3
>FROM u_table_4 a2
> ) a5
> ON (
> a1.bigint_col_18) = (a5.int_col_3)
> INNER JOIN
> (
> SELECT a3.bigint_col_3
> AS int_col,
> Cast (COALESCE(a3.bigint_col_3,
> a3.bigint_col_3, a3.int_col_10) AS BIGINT) * Cast (a3.bigint_col_3 AS BIGINT)
> AS int_col_3
> FROM u_table_7 a3
> WHEREbigint_col_3=571
> ) a4
> ON (a1.int_col_19=5)
> OR ((a5.int_col_3) IN (a4.int_col, 10))
> where
> a1.STRING_COL_7='500'
> ORDER BYint_col DESC nulls last limit 100
> ;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-22059) hive-exec jar doesn't contain (fasterxml) jackson library
[
https://issues.apache.org/jira/browse/HIVE-22059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931146#comment-16931146
]
Laszlo Bodor commented on HIVE-22059:
-
+1
> hive-exec jar doesn't contain (fasterxml) jackson library
> -
>
> Key: HIVE-22059
> URL: https://issues.apache.org/jira/browse/HIVE-22059
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-22059.01.patch, HIVE-22059.02.patch,
> HIVE-22059.03.patch, HIVE-22059.04.patch, HIVE-22059.04.patch,
> HIVE-22059.05.patch, HIVE-22059.06.patch
>
>
> While deploying master branch into a container I've noticed that the jackson
> libraries are not 100% sure that are available at runtime - this is probably
> due to the fact that we are still using the "old" codehaus jackson and also
> the "new" fasterxml one.
> {code:java}
> ]Vertex killed, vertexName=Reducer 2,
> vertexId=vertex_1564408646590_0005_1_01, diagnostics=[Vertex received Kill in
> INITED state., Vertex vertex_1564408646590_0005_1_01 [Reducer 2]
> killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to
> VERTEX_FAILURE. failedVertices:1 killedVertices:1
> INFO : Completed executing
> command(queryId=vagrant_20190729141949_8d8c7f0d-0ac4-4d76-ba12-6ec01561b040);
> Time taken: 5.127 seconds
> INFO : Concurrency mode is disabled, not creating a lock manager
> Error: Error while processing statement: FAILED: Execution Error, return code
> 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed,
> vertexName=Map 1, vertexId=vertex_1564408646590_0005_1_00,
> diagnostics=[Vertex vertex_1564408646590_0005_1_00 [Map 1] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: _dummy_table initializer failed,
> vertex=vertex_1564408646590_0005_1_00 [Map 1],
> java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper
> at org.apache.hadoop.hive.ql.exec.Utilities.(Utilities.java:226)
> at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:428)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:508)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:488)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:337)
> at
> org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:122)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ClassNotFoundException:
> com.fasterxml.jackson.databind.ObjectMapper
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 19 more
> ]Vertex killed, vertexName=Reducer 2,
> vertexId=vertex_1564408646590_0005_1_01, diagnostics=[Vertex received Kill in
> INITED state., Vertex vertex_1564408646590_0005_1_01 [Reducer 2]
> killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to
> VERTEX_FAILURE. failedVertices:1 killedVertices:1 (state=08S01,code=2)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-22182) SemanticAnalyzer populates map which is not used at all
[
https://issues.apache.org/jira/browse/HIVE-22182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931129#comment-16931129
]
Laszlo Bodor commented on HIVE-22182:
-
pushed to master, thanks [~adombi] for the patch!
> SemanticAnalyzer populates map which is not used at all
> ---
>
> Key: HIVE-22182
> URL: https://issues.apache.org/jira/browse/HIVE-22182
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Akos Dombi
>Assignee: Akos Dombi
>Priority: Major
> Attachments: HIVE-22182.patch
>
>
> The {{SemanticAnalyzer}} class has two map fields which are populated but
> never queried:
> * {{topToTable}}
> * {{topToTableProps}}
> Last usage was removed in HIVE-11752.
> h3. Purpose of this task
> - Remove unused {{topToTable}} and {{topToTableProps}} fields
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22182) SemanticAnalyzer populates map which is not used at all
[
https://issues.apache.org/jira/browse/HIVE-22182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-22182:
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> SemanticAnalyzer populates map which is not used at all
> ---
>
> Key: HIVE-22182
> URL: https://issues.apache.org/jira/browse/HIVE-22182
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Akos Dombi
>Assignee: Akos Dombi
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22182.patch
>
>
> The {{SemanticAnalyzer}} class has two map fields which are populated but
> never queried:
> * {{topToTable}}
> * {{topToTableProps}}
> Last usage was removed in HIVE-11752.
> h3. Purpose of this task
> - Remove unused {{topToTable}} and {{topToTableProps}} fields
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-21604) preCommit job should not be triggered on non-patch attachments
[ https://issues.apache.org/jira/browse/HIVE-21604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21604: Resolution: Fixed Status: Resolved (was: Patch Available) > preCommit job should not be triggered on non-patch attachments > -- > > Key: HIVE-21604 > URL: https://issues.apache.org/jira/browse/HIVE-21604 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21604.01.patch, HIVE-21604.01.patch, files.txt > > > latest example: HIVE-21604 > https://issues.apache.org/jira/browse/HIVE-14669?focusedCommentId=16815520=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16815520 > I think one should be able to upload any kind of attachments (e.g. > screenshot) without triggering the precommit job > 2 possible ways: > 1. strict: enable only .patch (should work) > (2. lenient: introduce blacklist, .png ...) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-21604) preCommit job should not be triggered on non-patch attachments
[ https://issues.apache.org/jira/browse/HIVE-21604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927460#comment-16927460 ] Laszlo Bodor commented on HIVE-21604: - pushed to master, thanks [~kgyrtkirk] for the review! > preCommit job should not be triggered on non-patch attachments > -- > > Key: HIVE-21604 > URL: https://issues.apache.org/jira/browse/HIVE-21604 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21604.01.patch, HIVE-21604.01.patch, files.txt > > > latest example: HIVE-21604 > https://issues.apache.org/jira/browse/HIVE-14669?focusedCommentId=16815520=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16815520 > I think one should be able to upload any kind of attachments (e.g. > screenshot) without triggering the precommit job > 2 possible ways: > 1. strict: enable only .patch (should work) > (2. lenient: introduce blacklist, .png ...) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-21510) Vectorization: add support for and/or for (constant,column) cases
[
https://issues.apache.org/jira/browse/HIVE-21510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927459#comment-16927459
]
Laszlo Bodor commented on HIVE-21510:
-
pushed to master, thanks [~kgyrtkirk] for the review!
> Vectorization: add support for and/or for (constant,column) cases
> -
>
> Key: HIVE-21510
> URL: https://issues.apache.org/jira/browse/HIVE-21510
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21510.01.patch, HIVE-21510.02.patch,
> HIVE-21510.03.patch, HIVE-21510.04.patch
>
>
> After HIVE-21001 some selectExpressions will start using VectorUDFAdaptor for
> "null and x" expressions. Because right now there are 2-3 places from which
> rewrite will be done to the form of "null and/or x" form; it would be better
> to support it.
> {code}
> [...]
> selectExpressions: VectorUDFAdaptor((null and dt1 is null))
> [...]
> usesVectorUDFAdaptor: true
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-21510) Vectorization: add support for and/or for (constant,column) cases
[
https://issues.apache.org/jira/browse/HIVE-21510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-21510:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Vectorization: add support for and/or for (constant,column) cases
> -
>
> Key: HIVE-21510
> URL: https://issues.apache.org/jira/browse/HIVE-21510
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21510.01.patch, HIVE-21510.02.patch,
> HIVE-21510.03.patch, HIVE-21510.04.patch
>
>
> After HIVE-21001 some selectExpressions will start using VectorUDFAdaptor for
> "null and x" expressions. Because right now there are 2-3 places from which
> rewrite will be done to the form of "null and/or x" form; it would be better
> to support it.
> {code}
> [...]
> selectExpressions: VectorUDFAdaptor((null and dt1 is null))
> [...]
> usesVectorUDFAdaptor: true
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-22182) SemanticAnalyzer populates map which is not used at all
[
https://issues.apache.org/jira/browse/HIVE-22182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927339#comment-16927339
]
Laszlo Bodor commented on HIVE-22182:
-
+1
> SemanticAnalyzer populates map which is not used at all
> ---
>
> Key: HIVE-22182
> URL: https://issues.apache.org/jira/browse/HIVE-22182
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Akos Dombi
>Assignee: Akos Dombi
>Priority: Major
> Attachments: HIVE-22182.patch
>
>
> The {{SemanticAnalyzer}} class has two map fields which are populated but
> never queried:
> * {{topToTable}}
> * {{topToTableProps}}
> Last usage was removed in HIVE-11752.
> h3. Purpose of this task
> - Remove unused {{topToTable}} and {{topToTableProps}} fields
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-21604) preCommit job should not be triggered on non-patch attachments
[ https://issues.apache.org/jira/browse/HIVE-21604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21604: Attachment: HIVE-21604.01.patch > preCommit job should not be triggered on non-patch attachments > -- > > Key: HIVE-21604 > URL: https://issues.apache.org/jira/browse/HIVE-21604 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21604.01.patch, HIVE-21604.01.patch, files.txt > > > latest example: HIVE-21604 > https://issues.apache.org/jira/browse/HIVE-14669?focusedCommentId=16815520=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16815520 > I think one should be able to upload any kind of attachments (e.g. > screenshot) without triggering the precommit job > 2 possible ways: > 1. strict: enable only .patch (should work) > (2. lenient: introduce blacklist, .png ...) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-21942) Remove useless MetastoreType enum from AbstractCliConfig
[ https://issues.apache.org/jira/browse/HIVE-21942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925679#comment-16925679 ] Laszlo Bodor commented on HIVE-21942: - pushed to master, thanks [~imolnar] for the patch! > Remove useless MetastoreType enum from AbstractCliConfig > > > Key: HIVE-21942 > URL: https://issues.apache.org/jira/browse/HIVE-21942 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Laszlo Bodor >Assignee: Imre Molnar >Priority: Major > Labels: newbie > Fix For: 4.0.0 > > Attachments: HIVE-21942.1.patch > > > I'm not 100% aware of historical reasons, but this single-value enum seems > totally useless at the moment. > https://github.com/apache/hive/blob/86a15600ae610e1c6017019883ef9344585572ef/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/AbstractCliConfig.java#L50-L52 > According to HIVE-14854, it may have made sense when there was an initiative > of hbase metastore, but it's not in the scope anymore AFAIK. > https://github.com/apache/hive/commit/474425aa62e3f25b119419439373aa684c6c2121 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-21942) Remove useless MetastoreType enum from AbstractCliConfig
[ https://issues.apache.org/jira/browse/HIVE-21942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21942: Resolution: Fixed Status: Resolved (was: Patch Available) > Remove useless MetastoreType enum from AbstractCliConfig > > > Key: HIVE-21942 > URL: https://issues.apache.org/jira/browse/HIVE-21942 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Laszlo Bodor >Assignee: Imre Molnar >Priority: Major > Labels: newbie > Fix For: 4.0.0 > > Attachments: HIVE-21942.1.patch > > > I'm not 100% aware of historical reasons, but this single-value enum seems > totally useless at the moment. > https://github.com/apache/hive/blob/86a15600ae610e1c6017019883ef9344585572ef/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/AbstractCliConfig.java#L50-L52 > According to HIVE-14854, it may have made sense when there was an initiative > of hbase metastore, but it's not in the scope anymore AFAIK. > https://github.com/apache/hive/commit/474425aa62e3f25b119419439373aa684c6c2121 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (HIVE-21604) preCommit job should not be triggered on non-patch attachments
[ https://issues.apache.org/jira/browse/HIVE-21604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor reassigned HIVE-21604: --- Assignee: Laszlo Bodor > preCommit job should not be triggered on non-patch attachments > -- > > Key: HIVE-21604 > URL: https://issues.apache.org/jira/browse/HIVE-21604 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-21604.01.patch, files.txt > > > latest example: HIVE-21604 > https://issues.apache.org/jira/browse/HIVE-14669?focusedCommentId=16815520=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16815520 > I think one should be able to upload any kind of attachments (e.g. > screenshot) without triggering the precommit job > 2 possible ways: > 1. strict: enable only .patch (should work) > (2. lenient: introduce blacklist, .png ...) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (HIVE-21604) preCommit job should not be triggered on non-patch attachments
[
https://issues.apache.org/jira/browse/HIVE-21604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925542#comment-16925542
]
Laszlo Bodor edited comment on HIVE-21604 at 9/9/19 10:02 AM:
--
tested with attached files.txt
original script: grep -o '"/jira/secure/attachment/[0-9]*/[^"]*' files.txt |
grep -v -e 'htm[l]*$' | sort | grep -o '/jira/secure/attachment/[0-9]*/[^"]*'
{code}
/jira/secure/attachment/12964860/files.txt
/jira/secure/attachment/12964869/01_mvn_out.png
/jira/secure/attachment/12964870/02_hive_log.png
/jira/secure/attachment/12965605/Screen%20Shot%202019-04-11%20at%204.54.41%20PM.png
/jira/secure/attachment/12965681/HIVE-14469.01.patch
/jira/secure/attachment/12965721/HIVE-14469.02.patch
/jira/secure/attachment/12967773/HIVE-14469.03.patch
{code}
modified script: grep -o '"/jira/secure/attachment/[0-9]*/[^"]*\.patch*'
files.txt | sort | grep -o '/jira/secure/attachment/[0-9]*/[^"]*\.patch*'
{code}
/jira/secure/attachment/12965681/HIVE-14469.01.patch
/jira/secure/attachment/12965721/HIVE-14469.02.patch
/jira/secure/attachment/12967773/HIVE-14469.03.patch
{code}
I think the original "grep -v" wasn't satisfying (only filtered html files)
with the current solution, it doesn't seem too strict to force devs to upload
only .patch files for patches
[~vihangk1]: could you please review the patch?
was (Author: abstractdog):
tested with attached files.txt
original script: grep -o '"/jira/secure/attachment/[0-9]*/[^"]*' files.txt |
grep -v -e 'htm[l]*$' | sort | grep -o '/jira/secure/attachment/[0-9]*/[^"]*'
{code}
/jira/secure/attachment/12964860/files.txt
/jira/secure/attachment/12964869/01_mvn_out.png
/jira/secure/attachment/12964870/02_hive_log.png
/jira/secure/attachment/12965605/Screen%20Shot%202019-04-11%20at%204.54.41%20PM.png
/jira/secure/attachment/12965681/HIVE-14469.01.patch
/jira/secure/attachment/12965721/HIVE-14469.02.patch
/jira/secure/attachment/12967773/HIVE-14469.03.patch
{code}
modified script: grep -o '"/jira/secure/attachment/[0-9]*/[^"]*\.patch*'
files.txt | sort | grep -o '/jira/secure/attachment/[0-9]*/[^"]*\.patch*'
{code}
/jira/secure/attachment/12965681/HIVE-14469.01.patch
/jira/secure/attachment/12965721/HIVE-14469.02.patch
/jira/secure/attachment/12967773/HIVE-14469.03.patch
{code}
I think the original "grep -v" wasn't satisfying (only filtered html files), as
it doesn't seem too strict to force devs to upload .patch files
[~vihangk1]: could you please review the patch?
> preCommit job should not be triggered on non-patch attachments
> --
>
> Key: HIVE-21604
> URL: https://issues.apache.org/jira/browse/HIVE-21604
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21604.01.patch, files.txt
>
>
> latest example: HIVE-21604
> https://issues.apache.org/jira/browse/HIVE-14669?focusedCommentId=16815520=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16815520
> I think one should be able to upload any kind of attachments (e.g.
> screenshot) without triggering the precommit job
> 2 possible ways:
> 1. strict: enable only .patch (should work)
> (2. lenient: introduce blacklist, .png ...)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-21604) preCommit job should not be triggered on non-patch attachments
[ https://issues.apache.org/jira/browse/HIVE-21604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21604: Status: Patch Available (was: Open) > preCommit job should not be triggered on non-patch attachments > -- > > Key: HIVE-21604 > URL: https://issues.apache.org/jira/browse/HIVE-21604 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-21604.01.patch, files.txt > > > latest example: HIVE-21604 > https://issues.apache.org/jira/browse/HIVE-14669?focusedCommentId=16815520=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16815520 > I think one should be able to upload any kind of attachments (e.g. > screenshot) without triggering the precommit job > 2 possible ways: > 1. strict: enable only .patch (should work) > (2. lenient: introduce blacklist, .png ...) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-21604) preCommit job should not be triggered on non-patch attachments
[ https://issues.apache.org/jira/browse/HIVE-21604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21604: Fix Version/s: 4.0.0 > preCommit job should not be triggered on non-patch attachments > -- > > Key: HIVE-21604 > URL: https://issues.apache.org/jira/browse/HIVE-21604 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21604.01.patch, files.txt > > > latest example: HIVE-21604 > https://issues.apache.org/jira/browse/HIVE-14669?focusedCommentId=16815520=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16815520 > I think one should be able to upload any kind of attachments (e.g. > screenshot) without triggering the precommit job > 2 possible ways: > 1. strict: enable only .patch (should work) > (2. lenient: introduce blacklist, .png ...) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-21604) preCommit job should not be triggered on non-patch attachments
[
https://issues.apache.org/jira/browse/HIVE-21604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925542#comment-16925542
]
Laszlo Bodor commented on HIVE-21604:
-
tested with attached files.txt
original script: grep -o '"/jira/secure/attachment/[0-9]*/[^"]*' files.txt |
grep -v -e 'htm[l]*$' | sort | grep -o '/jira/secure/attachment/[0-9]*/[^"]*'
{code}
/jira/secure/attachment/12964860/files.txt
/jira/secure/attachment/12964869/01_mvn_out.png
/jira/secure/attachment/12964870/02_hive_log.png
/jira/secure/attachment/12965605/Screen%20Shot%202019-04-11%20at%204.54.41%20PM.png
/jira/secure/attachment/12965681/HIVE-14469.01.patch
/jira/secure/attachment/12965721/HIVE-14469.02.patch
/jira/secure/attachment/12967773/HIVE-14469.03.patch
{code}
modified script: grep -o '"/jira/secure/attachment/[0-9]*/[^"]*\.patch*'
files.txt | sort | grep -o '/jira/secure/attachment/[0-9]*/[^"]*\.patch*'
{code}
/jira/secure/attachment/12965681/HIVE-14469.01.patch
/jira/secure/attachment/12965721/HIVE-14469.02.patch
/jira/secure/attachment/12967773/HIVE-14469.03.patch
{code}
I think the original "grep -v" wasn't satisfying (only filtered html files), as
it doesn't seem too strict to force devs to upload .patch files
[~vihangk1]: could you please review the patch
> preCommit job should not be triggered on non-patch attachments
> --
>
> Key: HIVE-21604
> URL: https://issues.apache.org/jira/browse/HIVE-21604
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21604.01.patch, files.txt
>
>
> latest example: HIVE-21604
> https://issues.apache.org/jira/browse/HIVE-14669?focusedCommentId=16815520=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16815520
> I think one should be able to upload any kind of attachments (e.g.
> screenshot) without triggering the precommit job
> 2 possible ways:
> 1. strict: enable only .patch (should work)
> (2. lenient: introduce blacklist, .png ...)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Comment Edited] (HIVE-21604) preCommit job should not be triggered on non-patch attachments
[
https://issues.apache.org/jira/browse/HIVE-21604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925542#comment-16925542
]
Laszlo Bodor edited comment on HIVE-21604 at 9/9/19 10:00 AM:
--
tested with attached files.txt
original script: grep -o '"/jira/secure/attachment/[0-9]*/[^"]*' files.txt |
grep -v -e 'htm[l]*$' | sort | grep -o '/jira/secure/attachment/[0-9]*/[^"]*'
{code}
/jira/secure/attachment/12964860/files.txt
/jira/secure/attachment/12964869/01_mvn_out.png
/jira/secure/attachment/12964870/02_hive_log.png
/jira/secure/attachment/12965605/Screen%20Shot%202019-04-11%20at%204.54.41%20PM.png
/jira/secure/attachment/12965681/HIVE-14469.01.patch
/jira/secure/attachment/12965721/HIVE-14469.02.patch
/jira/secure/attachment/12967773/HIVE-14469.03.patch
{code}
modified script: grep -o '"/jira/secure/attachment/[0-9]*/[^"]*\.patch*'
files.txt | sort | grep -o '/jira/secure/attachment/[0-9]*/[^"]*\.patch*'
{code}
/jira/secure/attachment/12965681/HIVE-14469.01.patch
/jira/secure/attachment/12965721/HIVE-14469.02.patch
/jira/secure/attachment/12967773/HIVE-14469.03.patch
{code}
I think the original "grep -v" wasn't satisfying (only filtered html files), as
it doesn't seem too strict to force devs to upload .patch files
[~vihangk1]: could you please review the patch?
was (Author: abstractdog):
tested with attached files.txt
original script: grep -o '"/jira/secure/attachment/[0-9]*/[^"]*' files.txt |
grep -v -e 'htm[l]*$' | sort | grep -o '/jira/secure/attachment/[0-9]*/[^"]*'
{code}
/jira/secure/attachment/12964860/files.txt
/jira/secure/attachment/12964869/01_mvn_out.png
/jira/secure/attachment/12964870/02_hive_log.png
/jira/secure/attachment/12965605/Screen%20Shot%202019-04-11%20at%204.54.41%20PM.png
/jira/secure/attachment/12965681/HIVE-14469.01.patch
/jira/secure/attachment/12965721/HIVE-14469.02.patch
/jira/secure/attachment/12967773/HIVE-14469.03.patch
{code}
modified script: grep -o '"/jira/secure/attachment/[0-9]*/[^"]*\.patch*'
files.txt | sort | grep -o '/jira/secure/attachment/[0-9]*/[^"]*\.patch*'
{code}
/jira/secure/attachment/12965681/HIVE-14469.01.patch
/jira/secure/attachment/12965721/HIVE-14469.02.patch
/jira/secure/attachment/12967773/HIVE-14469.03.patch
{code}
I think the original "grep -v" wasn't satisfying (only filtered html files), as
it doesn't seem too strict to force devs to upload .patch files
[~vihangk1]: could you please review the patch
> preCommit job should not be triggered on non-patch attachments
> --
>
> Key: HIVE-21604
> URL: https://issues.apache.org/jira/browse/HIVE-21604
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21604.01.patch, files.txt
>
>
> latest example: HIVE-21604
> https://issues.apache.org/jira/browse/HIVE-14669?focusedCommentId=16815520=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16815520
> I think one should be able to upload any kind of attachments (e.g.
> screenshot) without triggering the precommit job
> 2 possible ways:
> 1. strict: enable only .patch (should work)
> (2. lenient: introduce blacklist, .png ...)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-21604) preCommit job should not be triggered on non-patch attachments
[ https://issues.apache.org/jira/browse/HIVE-21604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21604: Attachment: HIVE-21604.01.patch > preCommit job should not be triggered on non-patch attachments > -- > > Key: HIVE-21604 > URL: https://issues.apache.org/jira/browse/HIVE-21604 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Laszlo Bodor >Priority: Major > Attachments: HIVE-21604.01.patch, files.txt > > > latest example: HIVE-21604 > https://issues.apache.org/jira/browse/HIVE-14669?focusedCommentId=16815520=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16815520 > I think one should be able to upload any kind of attachments (e.g. > screenshot) without triggering the precommit job > 2 possible ways: > 1. strict: enable only .patch (should work) > (2. lenient: introduce blacklist, .png ...) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-21604) preCommit job should not be triggered on non-patch attachments
[ https://issues.apache.org/jira/browse/HIVE-21604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21604: Attachment: files.txt > preCommit job should not be triggered on non-patch attachments > -- > > Key: HIVE-21604 > URL: https://issues.apache.org/jira/browse/HIVE-21604 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Laszlo Bodor >Priority: Major > Attachments: HIVE-21604.01.patch, files.txt > > > latest example: HIVE-21604 > https://issues.apache.org/jira/browse/HIVE-14669?focusedCommentId=16815520=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16815520 > I think one should be able to upload any kind of attachments (e.g. > screenshot) without triggering the precommit job > 2 possible ways: > 1. strict: enable only .patch (should work) > (2. lenient: introduce blacklist, .png ...) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (HIVE-21503) Vectorization: query with regex gives incorrect results with vectorization
[
https://issues.apache.org/jira/browse/HIVE-21503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor reassigned HIVE-21503:
---
Assignee: (was: Laszlo Bodor)
> Vectorization: query with regex gives incorrect results with vectorization
> --
>
> Key: HIVE-21503
> URL: https://issues.apache.org/jira/browse/HIVE-21503
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.3.4
>Reporter: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-21503.01.WIP.patch
>
>
> i see wrong results with vectorization. Without vectorization, it works fine.
> {noformat}
> e.g
> WHEN x like '%radio%' THEN 'radio'
> WHEN x like '%tv%' THEN 'tv'
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22149: Resolution: Fixed Status: Resolved (was: Patch Available) > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22149.01.patch, HIVE-22149.01.patch, > HIVE-22149.01.patch, HIVE-22149.02.patch, HIVE-22149.03.patch, > HIVE-22149.03.patch, HIVE-22149.03.patch, metrics_hiveserver2.log, > metrics_metastore.log > > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925467#comment-16925467 ] Laszlo Bodor commented on HIVE-22149: - pushed to master, thanks for the review [~kgyrtkirk]! > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22149.01.patch, HIVE-22149.01.patch, > HIVE-22149.01.patch, HIVE-22149.02.patch, HIVE-22149.03.patch, > HIVE-22149.03.patch, HIVE-22149.03.patch, metrics_hiveserver2.log, > metrics_metastore.log > > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22149: Component/s: Standalone Metastore > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22149.01.patch, HIVE-22149.01.patch, > HIVE-22149.01.patch, HIVE-22149.02.patch, HIVE-22149.03.patch, > HIVE-22149.03.patch, HIVE-22149.03.patch, metrics_hiveserver2.log, > metrics_metastore.log > > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22149: Fix Version/s: 4.0.0 > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22149.01.patch, HIVE-22149.01.patch, > HIVE-22149.01.patch, HIVE-22149.02.patch, HIVE-22149.03.patch, > HIVE-22149.03.patch, HIVE-22149.03.patch, metrics_hiveserver2.log, > metrics_metastore.log > > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-21510) Vectorization: add support for and/or for (constant,column) cases
[
https://issues.apache.org/jira/browse/HIVE-21510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924839#comment-16924839
]
Laszlo Bodor commented on HIVE-21510:
-
could you please review [~kgyrtkirk]?
> Vectorization: add support for and/or for (constant,column) cases
> -
>
> Key: HIVE-21510
> URL: https://issues.apache.org/jira/browse/HIVE-21510
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21510.01.patch, HIVE-21510.02.patch,
> HIVE-21510.03.patch, HIVE-21510.04.patch
>
>
> After HIVE-21001 some selectExpressions will start using VectorUDFAdaptor for
> "null and x" expressions. Because right now there are 2-3 places from which
> rewrite will be done to the form of "null and/or x" form; it would be better
> to support it.
> {code}
> [...]
> selectExpressions: VectorUDFAdaptor((null and dt1 is null))
> [...]
> usesVectorUDFAdaptor: true
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22149: Attachment: HIVE-22149.03.patch > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-22149.01.patch, HIVE-22149.01.patch, > HIVE-22149.01.patch, HIVE-22149.02.patch, HIVE-22149.03.patch, > HIVE-22149.03.patch, HIVE-22149.03.patch, metrics_hiveserver2.log, > metrics_metastore.log > > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22149: Attachment: HIVE-22149.03.patch > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-22149.01.patch, HIVE-22149.01.patch, > HIVE-22149.01.patch, HIVE-22149.02.patch, HIVE-22149.03.patch, > HIVE-22149.03.patch, metrics_hiveserver2.log, metrics_metastore.log > > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22149: Attachment: HIVE-22149.03.patch > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-22149.01.patch, HIVE-22149.01.patch, > HIVE-22149.01.patch, HIVE-22149.02.patch, HIVE-22149.03.patch, > metrics_hiveserver2.log, metrics_metastore.log > > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-21510) Vectorization: add support for and/or for (constant,column) cases
[
https://issues.apache.org/jira/browse/HIVE-21510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924602#comment-16924602
]
Laszlo Bodor commented on HIVE-21510:
-
need to fix asf license headers
> Vectorization: add support for and/or for (constant,column) cases
> -
>
> Key: HIVE-21510
> URL: https://issues.apache.org/jira/browse/HIVE-21510
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21510.01.patch, HIVE-21510.02.patch,
> HIVE-21510.03.patch, HIVE-21510.04.patch
>
>
> After HIVE-21001 some selectExpressions will start using VectorUDFAdaptor for
> "null and x" expressions. Because right now there are 2-3 places from which
> rewrite will be done to the form of "null and/or x" form; it would be better
> to support it.
> {code}
> [...]
> selectExpressions: VectorUDFAdaptor((null and dt1 is null))
> [...]
> usesVectorUDFAdaptor: true
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-21510) Vectorization: add support for and/or for (constant,column) cases
[
https://issues.apache.org/jira/browse/HIVE-21510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-21510:
Attachment: HIVE-21510.04.patch
> Vectorization: add support for and/or for (constant,column) cases
> -
>
> Key: HIVE-21510
> URL: https://issues.apache.org/jira/browse/HIVE-21510
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21510.01.patch, HIVE-21510.02.patch,
> HIVE-21510.03.patch, HIVE-21510.04.patch
>
>
> After HIVE-21001 some selectExpressions will start using VectorUDFAdaptor for
> "null and x" expressions. Because right now there are 2-3 places from which
> rewrite will be done to the form of "null and/or x" form; it would be better
> to support it.
> {code}
> [...]
> selectExpressions: VectorUDFAdaptor((null and dt1 is null))
> [...]
> usesVectorUDFAdaptor: true
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22149: Attachment: HIVE-22149.02.patch > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-22149.01.patch, HIVE-22149.01.patch, > HIVE-22149.01.patch, HIVE-22149.02.patch, metrics_hiveserver2.log, > metrics_metastore.log > > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-21510) Vectorization: add support for and/or for (constant,column) cases
[
https://issues.apache.org/jira/browse/HIVE-21510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-21510:
Attachment: HIVE-21510.03.patch
> Vectorization: add support for and/or for (constant,column) cases
> -
>
> Key: HIVE-21510
> URL: https://issues.apache.org/jira/browse/HIVE-21510
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21510.01.patch, HIVE-21510.02.patch,
> HIVE-21510.03.patch
>
>
> After HIVE-21001 some selectExpressions will start using VectorUDFAdaptor for
> "null and x" expressions. Because right now there are 2-3 places from which
> rewrite will be done to the form of "null and/or x" form; it would be better
> to support it.
> {code}
> [...]
> selectExpressions: VectorUDFAdaptor((null and dt1 is null))
> [...]
> usesVectorUDFAdaptor: true
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-21510) Vectorization: add support for and/or for (constant,column) cases
[
https://issues.apache.org/jira/browse/HIVE-21510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-21510:
Attachment: HIVE-21510.02.patch
> Vectorization: add support for and/or for (constant,column) cases
> -
>
> Key: HIVE-21510
> URL: https://issues.apache.org/jira/browse/HIVE-21510
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21510.01.patch, HIVE-21510.02.patch
>
>
> After HIVE-21001 some selectExpressions will start using VectorUDFAdaptor for
> "null and x" expressions. Because right now there are 2-3 places from which
> rewrite will be done to the form of "null and/or x" form; it would be better
> to support it.
> {code}
> [...]
> selectExpressions: VectorUDFAdaptor((null and dt1 is null))
> [...]
> usesVectorUDFAdaptor: true
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-21510) Vectorization: add support for and/or for (constant,column) cases
[
https://issues.apache.org/jira/browse/HIVE-21510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-21510:
Attachment: (was: HIVE-21510.02.patch)
> Vectorization: add support for and/or for (constant,column) cases
> -
>
> Key: HIVE-21510
> URL: https://issues.apache.org/jira/browse/HIVE-21510
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21510.01.patch
>
>
> After HIVE-21001 some selectExpressions will start using VectorUDFAdaptor for
> "null and x" expressions. Because right now there are 2-3 places from which
> rewrite will be done to the form of "null and/or x" form; it would be better
> to support it.
> {code}
> [...]
> selectExpressions: VectorUDFAdaptor((null and dt1 is null))
> [...]
> usesVectorUDFAdaptor: true
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-21510) Vectorization: add support for and/or for (constant,column) cases
[
https://issues.apache.org/jira/browse/HIVE-21510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-21510:
Attachment: HIVE-21510.02.patch
> Vectorization: add support for and/or for (constant,column) cases
> -
>
> Key: HIVE-21510
> URL: https://issues.apache.org/jira/browse/HIVE-21510
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21510.01.patch, HIVE-21510.02.patch
>
>
> After HIVE-21001 some selectExpressions will start using VectorUDFAdaptor for
> "null and x" expressions. Because right now there are 2-3 places from which
> rewrite will be done to the form of "null and/or x" form; it would be better
> to support it.
> {code}
> [...]
> selectExpressions: VectorUDFAdaptor((null and dt1 is null))
> [...]
> usesVectorUDFAdaptor: true
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-21942) Remove useless MetastoreType enum from AbstractCliConfig
[ https://issues.apache.org/jira/browse/HIVE-21942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924008#comment-16924008 ] Laszlo Bodor commented on HIVE-21942: - +1 > Remove useless MetastoreType enum from AbstractCliConfig > > > Key: HIVE-21942 > URL: https://issues.apache.org/jira/browse/HIVE-21942 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: Laszlo Bodor >Assignee: Imre Molnar >Priority: Major > Labels: newbie > Fix For: 4.0.0 > > Attachments: HIVE-21942.1.patch > > > I'm not 100% aware of historical reasons, but this single-value enum seems > totally useless at the moment. > https://github.com/apache/hive/blob/86a15600ae610e1c6017019883ef9344585572ef/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/AbstractCliConfig.java#L50-L52 > According to HIVE-14854, it may have made sense when there was an initiative > of hbase metastore, but it's not in the scope anymore AFAIK. > https://github.com/apache/hive/commit/474425aa62e3f25b119419439373aa684c6c2121 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22149: Attachment: HIVE-22149.01.patch > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-22149.01.patch, HIVE-22149.01.patch, > HIVE-22149.01.patch, metrics_hiveserver2.log, metrics_metastore.log > > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923722#comment-16923722 ] Laszlo Bodor commented on HIVE-22149: - tests seem to pass locally, reuploading 01.patch > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-22149.01.patch, HIVE-22149.01.patch, > HIVE-22149.01.patch, metrics_hiveserver2.log, metrics_metastore.log > > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (HIVE-21431) Vectorization: ltrim throws ArrayIndexOutOfBounds in corner cases
[
https://issues.apache.org/jira/browse/HIVE-21431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor reassigned HIVE-21431:
---
Assignee: (was: Laszlo Bodor)
> Vectorization: ltrim throws ArrayIndexOutOfBounds in corner cases
> -
>
> Key: HIVE-21431
> URL: https://issues.apache.org/jira/browse/HIVE-21431
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.3.4
>Reporter: Rajesh Balamohan
>Priority: Major
>
> In corner cases, {{ltrim}} with string columns throws
> arraryindexoutofboundsexception with vectorization enabled. {{HIVE-19565}}
> seem to fix corner cases. But in another corner case, {{length[]}} was all
> {{0}} and this causes {{-1}} to be returned in the length to be set in the
> target vector. I will check if i can get a easier repro for this.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923114#comment-16923114 ] Laszlo Bodor commented on HIVE-22149: - unfortunately, adding this to metastore common would not add any value, as probably this logic is used only at metastore metrics (once in the metastore codebase) > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-22149.01.patch, HIVE-22149.01.patch, > metrics_hiveserver2.log, metrics_metastore.log > > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22149: Attachment: HIVE-22149.01.patch > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-22149.01.patch, HIVE-22149.01.patch, > metrics_hiveserver2.log, metrics_metastore.log > > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922412#comment-16922412 ] Laszlo Bodor commented on HIVE-22149: - e.g. Cloudera Manager relies on codahale metrics, and I faced a situation where I had to decide whether I'll change CM code in order to integrate to metastore service properly or unify these keys with the corresponding keys in hiveserver2 metric file (I did the latter) unfortunately, as standalone-metastore should be able to be built independently from other parts of hive, I cannot refactor this logic to a common place > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-22149.01.patch, metrics_hiveserver2.log, > metrics_metastore.log > > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22149: Attachment: HIVE-22149.01.patch > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-22149.01.patch, metrics_hiveserver2.log, > metrics_metastore.log > > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22149: Attachment: (was: HIVE-22149.01.patch) > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-22149.01.patch, metrics_hiveserver2.log, > metrics_metastore.log > > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22149: Attachment: HIVE-22149.01.patch > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-22149.01.patch, metrics_hiveserver2.log, > metrics_metastore.log > > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22149: Status: Patch Available (was: Open) > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-22149.01.patch, metrics_hiveserver2.log, > metrics_metastore.log > > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22149: Attachment: metrics_metastore.log metrics_hiveserver2.log > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Attachments: metrics_hiveserver2.log, metrics_metastore.log > > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-21887) Multiple implementations of PersistenceManager are on the classpath
[ https://issues.apache.org/jira/browse/HIVE-21887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16917757#comment-16917757 ] Laszlo Bodor commented on HIVE-21887: - +1 > Multiple implementations of PersistenceManager are on the classpath > --- > > Key: HIVE-21887 > URL: https://issues.apache.org/jira/browse/HIVE-21887 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21887.01.patch, HIVE-21887.01.patch, > HIVE-21887.01.patch, HIVE-21887.02.patch, HIVE-21887.03.patch > > > While writing some test I've just bumped into that PersistenceManager is not > always AutoCloseable ; however when I was using it from the metastore it was. > it turned out that: > * 'standalone-metastore' uses: org.datanucleus:javax.jdo:3.2.0-m3 > * 'metastore' uses: javax.jdo:jdo-api:3.0.1 > the problem is that both dependency contains the jdo api; and sometimes > jdo-api is earlier on the classpath -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-21944) Remove unused methods, fields and variables from Vectorizer
[ https://issues.apache.org/jira/browse/HIVE-21944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21944: Fix Version/s: 4.0.0 > Remove unused methods, fields and variables from Vectorizer > --- > > Key: HIVE-21944 > URL: https://issues.apache.org/jira/browse/HIVE-21944 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Trivial > Fix For: 4.0.0 > > Attachments: HIVE-21944.1.patch, HIVE-21944.1.patch, > HIVE-21944.1.patch, HIVE-21944.1.patch > > > It seems there are many unused fields, variables and methods in > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer class. Removing them > would make the code easier to understand. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-21944) Remove unused methods, fields and variables from Vectorizer
[ https://issues.apache.org/jira/browse/HIVE-21944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21944: Resolution: Fixed Status: Resolved (was: Patch Available) > Remove unused methods, fields and variables from Vectorizer > --- > > Key: HIVE-21944 > URL: https://issues.apache.org/jira/browse/HIVE-21944 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Trivial > Fix For: 4.0.0 > > Attachments: HIVE-21944.1.patch, HIVE-21944.1.patch, > HIVE-21944.1.patch, HIVE-21944.1.patch > > > It seems there are many unused fields, variables and methods in > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer class. Removing them > would make the code easier to understand. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-21944) Remove unused methods, fields and variables from Vectorizer
[ https://issues.apache.org/jira/browse/HIVE-21944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21944: Component/s: Vectorization > Remove unused methods, fields and variables from Vectorizer > --- > > Key: HIVE-21944 > URL: https://issues.apache.org/jira/browse/HIVE-21944 > Project: Hive > Issue Type: Improvement > Components: Hive, Vectorization >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Trivial > Fix For: 4.0.0 > > Attachments: HIVE-21944.1.patch, HIVE-21944.1.patch, > HIVE-21944.1.patch, HIVE-21944.1.patch > > > It seems there are many unused fields, variables and methods in > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer class. Removing them > would make the code easier to understand. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-21944) Remove unused methods, fields and variables from Vectorizer
[ https://issues.apache.org/jira/browse/HIVE-21944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916649#comment-16916649 ] Laszlo Bodor commented on HIVE-21944: - pushed to master, thanks [~isuller] for the patch! > Remove unused methods, fields and variables from Vectorizer > --- > > Key: HIVE-21944 > URL: https://issues.apache.org/jira/browse/HIVE-21944 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Trivial > Attachments: HIVE-21944.1.patch, HIVE-21944.1.patch, > HIVE-21944.1.patch, HIVE-21944.1.patch > > > It seems there are many unused fields, variables and methods in > org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer class. Removing them > would make the code easier to understand. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (HIVE-21944) Remove unused methods, fields and variables from Vectorizer
[
https://issues.apache.org/jira/browse/HIVE-21944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896565#comment-16896565
]
Laszlo Bodor edited comment on HIVE-21944 at 8/27/19 11:53 AM:
---
[~isuller]: I've just reviewed your another patch and found 1 line which is
contradictory to that.
In
[HIVE-21962.2.patch|https://issues.apache.org/jira/secure/attachment/12975370/HIVE-21962.2.patch#file-6]
you refactored something in vectorizer which can avoid a needless shallow copy
(as far as I can understand):
from
{code}
mapWork.setVectorizationEnabledConditionsMet(new
ArrayList(enabledConditionsMetSet));
{code}
to
{code}
mapWork.setVectorizationEnabledConditionsMet(enabledConditionsMetSet);
{code}
however here the last patch seems to touch the same code part, but doesn't care
about that
https://issues.apache.org/jira/secure/attachment/12975371/HIVE-21944.1.patch
Apart from that, this looks good to me. +1
was (Author: abstractdog):
[~isuller]: I've just reviewed your another patch and found 1 line which is
contradictory to that.
In
[HIVE-21962.2.patch|https://issues.apache.org/jira/secure/attachment/12975370/HIVE-21962.2.patch#file-6]
you refactored something in vectorizer which can avoid a needless shallow copy
(as far as I can understand):
from
{code}
mapWork.setVectorizationEnabledConditionsMet(new
ArrayList(enabledConditionsMetSet));
{code}
to
{code}
mapWork.setVectorizationEnabledConditionsMet(enabledConditionsMetSet);
{code}
however here the last patch seems to touch the same code part, but doesn't care
about that
https://issues.apache.org/jira/secure/attachment/12975371/HIVE-21944.1.patch
Apart from that, this looks good to me.
> Remove unused methods, fields and variables from Vectorizer
> ---
>
> Key: HIVE-21944
> URL: https://issues.apache.org/jira/browse/HIVE-21944
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Trivial
> Attachments: HIVE-21944.1.patch, HIVE-21944.1.patch,
> HIVE-21944.1.patch, HIVE-21944.1.patch
>
>
> It seems there are many unused fields, variables and methods in
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer class. Removing them
> would make the code easier to understand.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-21962) Replacing ArrayList params with List in and around PlanUtils and MapWork
[ https://issues.apache.org/jira/browse/HIVE-21962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916644#comment-16916644 ] Laszlo Bodor commented on HIVE-21962: - pushed to master, thanks [~isuller] for the patch! > Replacing ArrayList params with List in and around PlanUtils and MapWork > > > Key: HIVE-21962 > URL: https://issues.apache.org/jira/browse/HIVE-21962 > Project: Hive > Issue Type: Improvement >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > Attachments: HIVE-21962.1.patch, HIVE-21962.1.patch, > HIVE-21962.2.patch, HIVE-21962.2.patch > > > Using the implementing class is usually a bad practice. OO suggest to use the > least restrictive interface instead. ArrayList is used in many-many methods > as a parameter - this is just a tiny part of this work. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-21962) Replacing ArrayList params with List in and around PlanUtils and MapWork
[ https://issues.apache.org/jira/browse/HIVE-21962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21962: Resolution: Fixed Status: Resolved (was: Patch Available) > Replacing ArrayList params with List in and around PlanUtils and MapWork > > > Key: HIVE-21962 > URL: https://issues.apache.org/jira/browse/HIVE-21962 > Project: Hive > Issue Type: Improvement >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > Attachments: HIVE-21962.1.patch, HIVE-21962.1.patch, > HIVE-21962.2.patch, HIVE-21962.2.patch > > > Using the implementing class is usually a bad practice. OO suggest to use the > least restrictive interface instead. ArrayList is used in many-many methods > as a parameter - this is just a tiny part of this work. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22149: Description: While fixing HIVE-22140 I found some really annoying differences between the codahale metric file structures between hiveserver2 and metastore, e.g. open_connections: can be found in "counters" for hs2, but in "gauges" for ms threads count: it's a proper "threads.count" for hs2, but a really ambiguous "count" for ms so I realized that "memory." and "threads." prefix is completely absent in ms metrics file, which is misleading > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > > While fixing HIVE-22140 I found some really annoying differences between the > codahale metric file structures between hiveserver2 and metastore, e.g. > open_connections: can be found in "counters" for hs2, but in "gauges" for ms > threads count: it's a proper "threads.count" for hs2, but a really ambiguous > "count" for ms > so I realized that "memory." and "threads." prefix is completely absent in ms > metrics file, which is misleading -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22140) Metrics: unify codahale metric frequency unit between metastore and hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-22140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916637#comment-16916637
]
Laszlo Bodor commented on HIVE-22140:
-
thanks for the review [~kgyrtkirk], I've pushed this to master
> Metrics: unify codahale metric frequency unit between metastore and
> hiveserver2
> ---
>
> Key: HIVE-22140
> URL: https://issues.apache.org/jira/browse/HIVE-22140
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22140.01.patch, HIVE-22140.01.patch
>
>
> Currently, different default value and even different unit are used for json
> metric file generation frequency (for metastore and hiveserver2), which is
> confusing, especially in a situation when a metastore config can fall back to
> a corresponding hive config (metastore.metrics.file.frequency ->
> hive.service.metrics.file.frequency).
> I think ms would be appropriate for both of them, without even changing the
> default 1minute for metastore (so changing it to 6ms is my proposal)
> hive.service.metrics.file.frequency
> Hiveserver2:
> https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> {code:java}
> HIVE_METRICS_JSON_FILE_INTERVAL("hive.service.metrics.file.frequency",
> "5000ms",
> new TimeValidator(TimeUnit.MILLISECONDS),
> "For metric class
> org.apache.hadoop.hive.common.metrics.metrics2.JsonFileMetricsReporter, " +
> "the frequency of updating JSON metrics file."),
> {code}
> Metastore:
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
> {code:java}
> METRICS_JSON_FILE_INTERVAL("metastore.metrics.file.frequency",
> "hive.service.metrics.file.frequency", 1, TimeUnit.MINUTES,
> "For json metric reporter, the frequency of updating JSON metrics
> file."),
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (HIVE-22149) Metastore: Unify codahale metrics.log json structure between hiveserver2 and metastore services
[ https://issues.apache.org/jira/browse/HIVE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor reassigned HIVE-22149: --- Assignee: Laszlo Bodor > Metastore: Unify codahale metrics.log json structure between hiveserver2 and > metastore services > --- > > Key: HIVE-22149 > URL: https://issues.apache.org/jira/browse/HIVE-22149 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22140) Metrics: unify codahale metric frequency unit between metastore and hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-22140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-22140:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Metrics: unify codahale metric frequency unit between metastore and
> hiveserver2
> ---
>
> Key: HIVE-22140
> URL: https://issues.apache.org/jira/browse/HIVE-22140
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22140.01.patch, HIVE-22140.01.patch
>
>
> Currently, different default value and even different unit are used for json
> metric file generation frequency (for metastore and hiveserver2), which is
> confusing, especially in a situation when a metastore config can fall back to
> a corresponding hive config (metastore.metrics.file.frequency ->
> hive.service.metrics.file.frequency).
> I think ms would be appropriate for both of them, without even changing the
> default 1minute for metastore (so changing it to 6ms is my proposal)
> hive.service.metrics.file.frequency
> Hiveserver2:
> https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> {code:java}
> HIVE_METRICS_JSON_FILE_INTERVAL("hive.service.metrics.file.frequency",
> "5000ms",
> new TimeValidator(TimeUnit.MILLISECONDS),
> "For metric class
> org.apache.hadoop.hive.common.metrics.metrics2.JsonFileMetricsReporter, " +
> "the frequency of updating JSON metrics file."),
> {code}
> Metastore:
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
> {code:java}
> METRICS_JSON_FILE_INTERVAL("metastore.metrics.file.frequency",
> "hive.service.metrics.file.frequency", 1, TimeUnit.MINUTES,
> "For json metric reporter, the frequency of updating JSON metrics
> file."),
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22140) Metrics: unify codahale metric frequency unit between metastore and hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-22140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-22140:
Summary: Metrics: unify codahale metric frequency unit between metastore
and hiveserver2 (was: Metrics: unify codehale metric frequency unit between
metastore and hiveserver2)
> Metrics: unify codahale metric frequency unit between metastore and
> hiveserver2
> ---
>
> Key: HIVE-22140
> URL: https://issues.apache.org/jira/browse/HIVE-22140
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22140.01.patch, HIVE-22140.01.patch
>
>
> Currently, different default value and even different unit are used for json
> metric file generation frequency (for metastore and hiveserver2), which is
> confusing, especially in a situation when a metastore config can fall back to
> a corresponding hive config (metastore.metrics.file.frequency ->
> hive.service.metrics.file.frequency).
> I think ms would be appropriate for both of them, without even changing the
> default 1minute for metastore (so changing it to 6ms is my proposal)
> hive.service.metrics.file.frequency
> Hiveserver2:
> https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> {code:java}
> HIVE_METRICS_JSON_FILE_INTERVAL("hive.service.metrics.file.frequency",
> "5000ms",
> new TimeValidator(TimeUnit.MILLISECONDS),
> "For metric class
> org.apache.hadoop.hive.common.metrics.metrics2.JsonFileMetricsReporter, " +
> "the frequency of updating JSON metrics file."),
> {code}
> Metastore:
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
> {code:java}
> METRICS_JSON_FILE_INTERVAL("metastore.metrics.file.frequency",
> "hive.service.metrics.file.frequency", 1, TimeUnit.MINUTES,
> "For json metric reporter, the frequency of updating JSON metrics
> file."),
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22140) Metrics: unify codehale metric frequency unit between metastore and hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-22140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-22140:
Attachment: HIVE-22140.01.patch
> Metrics: unify codehale metric frequency unit between metastore and
> hiveserver2
> ---
>
> Key: HIVE-22140
> URL: https://issues.apache.org/jira/browse/HIVE-22140
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22140.01.patch, HIVE-22140.01.patch
>
>
> Currently, different default value and even different unit are used for json
> metric file generation frequency (for metastore and hiveserver2), which is
> confusing, especially in a situation when a metastore config can fall back to
> a corresponding hive config (metastore.metrics.file.frequency ->
> hive.service.metrics.file.frequency).
> I think ms would be appropriate for both of them, without even changing the
> default 1minute for metastore (so changing it to 6ms is my proposal)
> hive.service.metrics.file.frequency
> Hiveserver2:
> https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> {code:java}
> HIVE_METRICS_JSON_FILE_INTERVAL("hive.service.metrics.file.frequency",
> "5000ms",
> new TimeValidator(TimeUnit.MILLISECONDS),
> "For metric class
> org.apache.hadoop.hive.common.metrics.metrics2.JsonFileMetricsReporter, " +
> "the frequency of updating JSON metrics file."),
> {code}
> Metastore:
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
> {code:java}
> METRICS_JSON_FILE_INTERVAL("metastore.metrics.file.frequency",
> "hive.service.metrics.file.frequency", 1, TimeUnit.MINUTES,
> "For json metric reporter, the frequency of updating JSON metrics
> file."),
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22140) Metrics: unify codehale metric frequency unit between metastore and hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-22140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-22140:
Description:
Currently, different default value and even different unit are used for json
metric file generation frequency (for metastore and hiveserver2), which is
confusing, especially in a situation when a metastore config can fall back to a
corresponding hive config (metastore.metrics.file.frequency ->
hive.service.metrics.file.frequency).
I think ms would be appropriate for both of them, without even changing the
default 1minute for metastore (so changing it to 6ms is my proposal)
hive.service.metrics.file.frequency
Hiveserver2:
https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
{code:java}
HIVE_METRICS_JSON_FILE_INTERVAL("hive.service.metrics.file.frequency",
"5000ms",
new TimeValidator(TimeUnit.MILLISECONDS),
"For metric class
org.apache.hadoop.hive.common.metrics.metrics2.JsonFileMetricsReporter, " +
"the frequency of updating JSON metrics file."),
{code}
Metastore:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
{code:java}
METRICS_JSON_FILE_INTERVAL("metastore.metrics.file.frequency",
"hive.service.metrics.file.frequency", 1, TimeUnit.MINUTES,
"For json metric reporter, the frequency of updating JSON metrics
file."),
{code}
was:
Currently, different default value and even different unit are used for json
metric file generation frequency (for metastore and hiveserver2), which is
confusing, especially in a situation when a metastore config can fall back to a
corresponding hive config (metastore.metrics.file.frequency ->
hive.service.metrics.file.frequency).
I think ms would be appropriate for both of them, without even changing the
default 1m for metastore (so changing it to 6ms)
hive.service.metrics.file.frequency
Hiveserver2:
https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
{code:java}
HIVE_METRICS_JSON_FILE_INTERVAL("hive.service.metrics.file.frequency",
"5000ms",
new TimeValidator(TimeUnit.MILLISECONDS),
"For metric class
org.apache.hadoop.hive.common.metrics.metrics2.JsonFileMetricsReporter, " +
"the frequency of updating JSON metrics file."),
{code}
Metastore:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
{code:java}
METRICS_JSON_FILE_INTERVAL("metastore.metrics.file.frequency",
"hive.service.metrics.file.frequency", 1, TimeUnit.MINUTES,
"For json metric reporter, the frequency of updating JSON metrics
file."),
{code}
> Metrics: unify codehale metric frequency unit between metastore and
> hiveserver2
> ---
>
> Key: HIVE-22140
> URL: https://issues.apache.org/jira/browse/HIVE-22140
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22140.01.patch
>
>
> Currently, different default value and even different unit are used for json
> metric file generation frequency (for metastore and hiveserver2), which is
> confusing, especially in a situation when a metastore config can fall back to
> a corresponding hive config (metastore.metrics.file.frequency ->
> hive.service.metrics.file.frequency).
> I think ms would be appropriate for both of them, without even changing the
> default 1minute for metastore (so changing it to 6ms is my proposal)
> hive.service.metrics.file.frequency
> Hiveserver2:
> https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> {code:java}
> HIVE_METRICS_JSON_FILE_INTERVAL("hive.service.metrics.file.frequency",
> "5000ms",
> new TimeValidator(TimeUnit.MILLISECONDS),
> "For metric class
> org.apache.hadoop.hive.common.metrics.metrics2.JsonFileMetricsReporter, " +
> "the frequency of updating JSON metrics file."),
> {code}
> Metastore:
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
> {code:java}
> METRICS_JSON_FILE_INTERVAL("metastore.metrics.file.frequency",
> "hive.service.metrics.file.frequency", 1, TimeUnit.MINUTES,
> "For json metric reporter, the frequency of updating JSON metrics
> file."),
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22140) Metrics: unify codehale metric frequency unit between metastore and hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-22140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-22140:
Fix Version/s: 4.0.0
> Metrics: unify codehale metric frequency unit between metastore and
> hiveserver2
> ---
>
> Key: HIVE-22140
> URL: https://issues.apache.org/jira/browse/HIVE-22140
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22140.01.patch
>
>
> Currently, different default value and even different unit are used for json
> metric file generation frequency (for metastore and hiveserver2), which is
> confusing, especially in a situation when a metastore config can fall back to
> a corresponding hive config (metastore.metrics.file.frequency ->
> hive.service.metrics.file.frequency).
> I think ms would be appropriate for both of them, without even changing the
> default 1m for metastore (so changing it to 6ms)
> hive.service.metrics.file.frequency
> Hiveserver2:
> https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> {code:java}
> HIVE_METRICS_JSON_FILE_INTERVAL("hive.service.metrics.file.frequency",
> "5000ms",
> new TimeValidator(TimeUnit.MILLISECONDS),
> "For metric class
> org.apache.hadoop.hive.common.metrics.metrics2.JsonFileMetricsReporter, " +
> "the frequency of updating JSON metrics file."),
> {code}
> Metastore:
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
> {code:java}
> METRICS_JSON_FILE_INTERVAL("metastore.metrics.file.frequency",
> "hive.service.metrics.file.frequency", 1, TimeUnit.MINUTES,
> "For json metric reporter, the frequency of updating JSON metrics
> file."),
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22140) Metrics: unify codehale metric frequency unit between metastore and hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-22140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-22140:
Description:
Currently, different default value and even different unit are used for json
metric file generation frequency (for metastore and hiveserver2), which is
confusing, especially in a situation when a metastore config can fall back to a
corresponding hive config (metastore.metrics.file.frequency ->
hive.service.metrics.file.frequency).
I think ms would be appropriate for both of them, without even changing the
default 1m for metastore (so changing it to 6ms)
hive.service.metrics.file.frequency
Hiveserver2:
https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
{code:java}
HIVE_METRICS_JSON_FILE_INTERVAL("hive.service.metrics.file.frequency",
"5000ms",
new TimeValidator(TimeUnit.MILLISECONDS),
"For metric class
org.apache.hadoop.hive.common.metrics.metrics2.JsonFileMetricsReporter, " +
"the frequency of updating JSON metrics file."),
{code}
Metastore:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
{code:java}
METRICS_JSON_FILE_INTERVAL("metastore.metrics.file.frequency",
"hive.service.metrics.file.frequency", 1, TimeUnit.MINUTES,
"For json metric reporter, the frequency of updating JSON metrics
file."),
{code}
was:
Currently, different default value and even different unit are used for json
metric file generation frequency (for metastore and hiveserver2), which is
confusing, especially in a situation when a metastore config can fall back to a
corresponding hive config.
I think ms would be appropriate for both of them, without even changing the
default 1m for metastore (so changing it to 6ms)
hive.service.metrics.file.frequency
Hiveserver2:
https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
{code:java}
HIVE_METRICS_JSON_FILE_INTERVAL("hive.service.metrics.file.frequency",
"5000ms",
new TimeValidator(TimeUnit.MILLISECONDS),
"For metric class
org.apache.hadoop.hive.common.metrics.metrics2.JsonFileMetricsReporter, " +
"the frequency of updating JSON metrics file."),
{code}
Metastore:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
{code:java}
METRICS_JSON_FILE_INTERVAL("metastore.metrics.file.frequency",
"hive.service.metrics.file.frequency", 1, TimeUnit.MINUTES,
"For json metric reporter, the frequency of updating JSON metrics
file."),
{code}
> Metrics: unify codehale metric frequency unit between metastore and
> hiveserver2
> ---
>
> Key: HIVE-22140
> URL: https://issues.apache.org/jira/browse/HIVE-22140
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-22140.01.patch
>
>
> Currently, different default value and even different unit are used for json
> metric file generation frequency (for metastore and hiveserver2), which is
> confusing, especially in a situation when a metastore config can fall back to
> a corresponding hive config (metastore.metrics.file.frequency ->
> hive.service.metrics.file.frequency).
> I think ms would be appropriate for both of them, without even changing the
> default 1m for metastore (so changing it to 6ms)
> hive.service.metrics.file.frequency
> Hiveserver2:
> https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> {code:java}
> HIVE_METRICS_JSON_FILE_INTERVAL("hive.service.metrics.file.frequency",
> "5000ms",
> new TimeValidator(TimeUnit.MILLISECONDS),
> "For metric class
> org.apache.hadoop.hive.common.metrics.metrics2.JsonFileMetricsReporter, " +
> "the frequency of updating JSON metrics file."),
> {code}
> Metastore:
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
> {code:java}
> METRICS_JSON_FILE_INTERVAL("metastore.metrics.file.frequency",
> "hive.service.metrics.file.frequency", 1, TimeUnit.MINUTES,
> "For json metric reporter, the frequency of updating JSON metrics
> file."),
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HIVE-22140) Metrics: unify codehale metric frequency unit between metastore and hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-22140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914185#comment-16914185
]
Laszlo Bodor commented on HIVE-22140:
-
could you please review [~alangates]?
small patch
> Metrics: unify codehale metric frequency unit between metastore and
> hiveserver2
> ---
>
> Key: HIVE-22140
> URL: https://issues.apache.org/jira/browse/HIVE-22140
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-22140.01.patch
>
>
> Currently, different default value and even different unit are used for json
> metric file generation frequency (for metastore and hiveserver2), which is
> confusing, especially in a situation when a metastore config can fall back to
> a corresponding hive config.
> I think ms would be appropriate for both of them, without even changing the
> default 1m for metastore (so changing it to 6ms)
> hive.service.metrics.file.frequency
> Hiveserver2:
> https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> {code:java}
> HIVE_METRICS_JSON_FILE_INTERVAL("hive.service.metrics.file.frequency",
> "5000ms",
> new TimeValidator(TimeUnit.MILLISECONDS),
> "For metric class
> org.apache.hadoop.hive.common.metrics.metrics2.JsonFileMetricsReporter, " +
> "the frequency of updating JSON metrics file."),
> {code}
> Metastore:
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
> {code:java}
> METRICS_JSON_FILE_INTERVAL("metastore.metrics.file.frequency",
> "hive.service.metrics.file.frequency", 1, TimeUnit.MINUTES,
> "For json metric reporter, the frequency of updating JSON metrics
> file."),
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22140) Metrics: unify codehale metric frequency unit between metastore and hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-22140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-22140:
Status: Patch Available (was: Open)
> Metrics: unify codehale metric frequency unit between metastore and
> hiveserver2
> ---
>
> Key: HIVE-22140
> URL: https://issues.apache.org/jira/browse/HIVE-22140
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-22140.01.patch
>
>
> Currently, different default value and even different unit are used for json
> metric file generation frequency (for metastore and hiveserver2), which is
> confusing, especially in a situation when a metastore config can fall back to
> a corresponding hive config.
> I think ms would be appropriate for both of them, without even changing the
> default 1m for metastore (so changing it to 6ms)
> hive.service.metrics.file.frequency
> Hiveserver2:
> https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> {code:java}
> HIVE_METRICS_JSON_FILE_INTERVAL("hive.service.metrics.file.frequency",
> "5000ms",
> new TimeValidator(TimeUnit.MILLISECONDS),
> "For metric class
> org.apache.hadoop.hive.common.metrics.metrics2.JsonFileMetricsReporter, " +
> "the frequency of updating JSON metrics file."),
> {code}
> Metastore:
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
> {code:java}
> METRICS_JSON_FILE_INTERVAL("metastore.metrics.file.frequency",
> "hive.service.metrics.file.frequency", 1, TimeUnit.MINUTES,
> "For json metric reporter, the frequency of updating JSON metrics
> file."),
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22140) Metrics: unify codehale metric frequency unit between metastore and hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-22140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-22140:
Attachment: HIVE-22140.01.patch
> Metrics: unify codehale metric frequency unit between metastore and
> hiveserver2
> ---
>
> Key: HIVE-22140
> URL: https://issues.apache.org/jira/browse/HIVE-22140
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-22140.01.patch
>
>
> Currently, different default value and even different unit are used for json
> metric file generation frequency (for metastore and hiveserver2), which is
> confusing, especially in a situation when a metastore config can fall back to
> a corresponding hive config.
> I think ms would be appropriate for both of them, without even changing the
> default 1m for metastore (so changing it to 6ms)
> hive.service.metrics.file.frequency
> Hiveserver2:
> https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> {code:java}
> HIVE_METRICS_JSON_FILE_INTERVAL("hive.service.metrics.file.frequency",
> "5000ms",
> new TimeValidator(TimeUnit.MILLISECONDS),
> "For metric class
> org.apache.hadoop.hive.common.metrics.metrics2.JsonFileMetricsReporter, " +
> "the frequency of updating JSON metrics file."),
> {code}
> Metastore:
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
> {code:java}
> METRICS_JSON_FILE_INTERVAL("metastore.metrics.file.frequency",
> "hive.service.metrics.file.frequency", 1, TimeUnit.MINUTES,
> "For json metric reporter, the frequency of updating JSON metrics
> file."),
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22140) Metrics: unify codehale metric frequency unit between metastore and hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-22140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-22140:
Summary: Metrics: unify codehale metric frequency unit between metastore
and hiveserver2 (was: Metrics: unify codehale metric frequency unit and
default between metastore and hiveserver2)
> Metrics: unify codehale metric frequency unit between metastore and
> hiveserver2
> ---
>
> Key: HIVE-22140
> URL: https://issues.apache.org/jira/browse/HIVE-22140
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
>
> Currently, different default value and even different unit are used for json
> metric file generation frequency (for metastore and hiveserver2), which is
> confusing, especially in a situation when a metastore config can fall back to
> a corresponding hive config.
> I think ms would be appropriate for both of them, without even changing the
> default 1m for metastore (so changing it to 6ms)
> hive.service.metrics.file.frequency
> Hiveserver2:
> https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> {code:java}
> HIVE_METRICS_JSON_FILE_INTERVAL("hive.service.metrics.file.frequency",
> "5000ms",
> new TimeValidator(TimeUnit.MILLISECONDS),
> "For metric class
> org.apache.hadoop.hive.common.metrics.metrics2.JsonFileMetricsReporter, " +
> "the frequency of updating JSON metrics file."),
> {code}
> Metastore:
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
> {code:java}
> METRICS_JSON_FILE_INTERVAL("metastore.metrics.file.frequency",
> "hive.service.metrics.file.frequency", 1, TimeUnit.MINUTES,
> "For json metric reporter, the frequency of updating JSON metrics
> file."),
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22140) Metrics: unify codehale metric frequency unit and default between metastore and hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-22140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-22140:
Description:
Currently, different default value and even different unit are used for json
metric file generation frequency (for metastore and hiveserver2), which is
confusing, especially in a situation when a metastore config can fall back to a
corresponding hive config.
I think ms would be appropriate for both of them, without even changing the
default 1m for metastore (so changing it to 6ms)
hive.service.metrics.file.frequency
Hiveserver2:
https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
{code:java}
HIVE_METRICS_JSON_FILE_INTERVAL("hive.service.metrics.file.frequency",
"5000ms",
new TimeValidator(TimeUnit.MILLISECONDS),
"For metric class
org.apache.hadoop.hive.common.metrics.metrics2.JsonFileMetricsReporter, " +
"the frequency of updating JSON metrics file."),
{code}
Metastore:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
{code:java}
METRICS_JSON_FILE_INTERVAL("metastore.metrics.file.frequency",
"hive.service.metrics.file.frequency", 1, TimeUnit.MINUTES,
"For json metric reporter, the frequency of updating JSON metrics
file."),
{code}
was:
Currently, different default value and even different unit are used for json
metric file generation frequency, which is confusing, especially in a situation
when a metastore config can fall back to hive a corresponding hive config.
I think ms would be appropriate for both of them, without even changing the
default 1m for metastore (so changing it to 6ms)
Hiveserver2:
https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
{code:java}
HIVE_METRICS_JSON_FILE_INTERVAL("hive.service.metrics.file.frequency",
"5000ms",
new TimeValidator(TimeUnit.MILLISECONDS),
"For metric class
org.apache.hadoop.hive.common.metrics.metrics2.JsonFileMetricsReporter, " +
"the frequency of updating JSON metrics file."),
{code}
Metastore:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
{code:java}
METRICS_JSON_FILE_INTERVAL("metastore.metrics.file.frequency",
"hive.service.metrics.file.frequency", 1, TimeUnit.MINUTES,
"For json metric reporter, the frequency of updating JSON metrics
file."),
{code}
> Metrics: unify codehale metric frequency unit and default between metastore
> and hiveserver2
> ---
>
> Key: HIVE-22140
> URL: https://issues.apache.org/jira/browse/HIVE-22140
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Laszlo Bodor
>Priority: Major
>
> Currently, different default value and even different unit are used for json
> metric file generation frequency (for metastore and hiveserver2), which is
> confusing, especially in a situation when a metastore config can fall back to
> a corresponding hive config.
> I think ms would be appropriate for both of them, without even changing the
> default 1m for metastore (so changing it to 6ms)
> hive.service.metrics.file.frequency
> Hiveserver2:
> https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> {code:java}
> HIVE_METRICS_JSON_FILE_INTERVAL("hive.service.metrics.file.frequency",
> "5000ms",
> new TimeValidator(TimeUnit.MILLISECONDS),
> "For metric class
> org.apache.hadoop.hive.common.metrics.metrics2.JsonFileMetricsReporter, " +
> "the frequency of updating JSON metrics file."),
> {code}
> Metastore:
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
> {code:java}
> METRICS_JSON_FILE_INTERVAL("metastore.metrics.file.frequency",
> "hive.service.metrics.file.frequency", 1, TimeUnit.MINUTES,
> "For json metric reporter, the frequency of updating JSON metrics
> file."),
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (HIVE-22140) Metrics: unify codehale metric frequency unit and default between metastore and hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-22140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor reassigned HIVE-22140:
---
Assignee: Laszlo Bodor
> Metrics: unify codehale metric frequency unit and default between metastore
> and hiveserver2
> ---
>
> Key: HIVE-22140
> URL: https://issues.apache.org/jira/browse/HIVE-22140
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
>
> Currently, different default value and even different unit are used for json
> metric file generation frequency (for metastore and hiveserver2), which is
> confusing, especially in a situation when a metastore config can fall back to
> a corresponding hive config.
> I think ms would be appropriate for both of them, without even changing the
> default 1m for metastore (so changing it to 6ms)
> hive.service.metrics.file.frequency
> Hiveserver2:
> https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> {code:java}
> HIVE_METRICS_JSON_FILE_INTERVAL("hive.service.metrics.file.frequency",
> "5000ms",
> new TimeValidator(TimeUnit.MILLISECONDS),
> "For metric class
> org.apache.hadoop.hive.common.metrics.metrics2.JsonFileMetricsReporter, " +
> "the frequency of updating JSON metrics file."),
> {code}
> Metastore:
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
> {code:java}
> METRICS_JSON_FILE_INTERVAL("metastore.metrics.file.frequency",
> "hive.service.metrics.file.frequency", 1, TimeUnit.MINUTES,
> "For json metric reporter, the frequency of updating JSON metrics
> file."),
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22140) Metrics: unify codehale metric frequency unit and default between metastore and hiveserver2
[
https://issues.apache.org/jira/browse/HIVE-22140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-22140:
Description:
Currently, different default value and even different unit are used for json
metric file generation frequency, which is confusing, especially in a situation
when a metastore config can fall back to hive a corresponding hive config.
I think ms would be appropriate for both of them, without even changing the
default 1m for metastore (so changing it to 6ms)
Hiveserver2:
https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
{code:java}
HIVE_METRICS_JSON_FILE_INTERVAL("hive.service.metrics.file.frequency",
"5000ms",
new TimeValidator(TimeUnit.MILLISECONDS),
"For metric class
org.apache.hadoop.hive.common.metrics.metrics2.JsonFileMetricsReporter, " +
"the frequency of updating JSON metrics file."),
{code}
Metastore:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
{code:java}
METRICS_JSON_FILE_INTERVAL("metastore.metrics.file.frequency",
"hive.service.metrics.file.frequency", 1, TimeUnit.MINUTES,
"For json metric reporter, the frequency of updating JSON metrics
file."),
{code}
was:
Currently different default and even different unit is used for json metric
file generation frequency, which is confusing.
For
> Metrics: unify codehale metric frequency unit and default between metastore
> and hiveserver2
> ---
>
> Key: HIVE-22140
> URL: https://issues.apache.org/jira/browse/HIVE-22140
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Laszlo Bodor
>Priority: Major
>
> Currently, different default value and even different unit are used for json
> metric file generation frequency, which is confusing, especially in a
> situation when a metastore config can fall back to hive a corresponding hive
> config.
> I think ms would be appropriate for both of them, without even changing the
> default 1m for metastore (so changing it to 6ms)
> Hiveserver2:
> https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> {code:java}
> HIVE_METRICS_JSON_FILE_INTERVAL("hive.service.metrics.file.frequency",
> "5000ms",
> new TimeValidator(TimeUnit.MILLISECONDS),
> "For metric class
> org.apache.hadoop.hive.common.metrics.metrics2.JsonFileMetricsReporter, " +
> "the frequency of updating JSON metrics file."),
> {code}
> Metastore:
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
> {code:java}
> METRICS_JSON_FILE_INTERVAL("metastore.metrics.file.frequency",
> "hive.service.metrics.file.frequency", 1, TimeUnit.MINUTES,
> "For json metric reporter, the frequency of updating JSON metrics
> file."),
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HIVE-22140) Metrics: unify codehale metric frequency unit and default between metastore and hiveserver2
[ https://issues.apache.org/jira/browse/HIVE-22140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22140: Summary: Metrics: unify codehale metric frequency unit and default between metastore and hiveserver2 (was: Metrics: unify codehale metric frequency unit and default) > Metrics: unify codehale metric frequency unit and default between metastore > and hiveserver2 > --- > > Key: HIVE-22140 > URL: https://issues.apache.org/jira/browse/HIVE-22140 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Laszlo Bodor >Priority: Major > > Currently different default and even different unit is used for json metric > file generation frequency, which is confusing. > For -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22140) Metrics: unify codehale metric frequency unit and default
[ https://issues.apache.org/jira/browse/HIVE-22140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22140: Description: Currently different default and even different unit is used for json metric file generation frequency, which is confusing. For > Metrics: unify codehale metric frequency unit and default > - > > Key: HIVE-22140 > URL: https://issues.apache.org/jira/browse/HIVE-22140 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Laszlo Bodor >Priority: Major > > Currently different default and even different unit is used for json metric > file generation frequency, which is confusing. > For -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-21828) Tez: Use a pre-parsed TezConfiguration from DagUtils
[ https://issues.apache.org/jira/browse/HIVE-21828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914063#comment-16914063 ] Laszlo Bodor commented on HIVE-21828: - pushed as https://github.com/apache/hive/commit/9482a0c0319427d74af6829d7766c16e44938500 thanks [~amagyar], [~gopalv]! > Tez: Use a pre-parsed TezConfiguration from DagUtils > > > Key: HIVE-21828 > URL: https://issues.apache.org/jira/browse/HIVE-21828 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21828.1.patch, HIVE-21828.2.patch, > HIVE-21828.5.patch, HIVE-21828.6.patch, HIVE-21828.7.patch, > HIVE-21828.8.patch, HIVE-21828.9.patch > > > The HS2 tez-site.xml does not change dynamically - the XML parsed components > of the config can be obtained statically and kept across sessions. > This allows for the replacing of "new TezConfiguration()" with a HS2 local > version instead. > The configuration object however has to reference the right resource file > (i.e location of tez-site.xml) without reparsing it for each query. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-21550) TestObjectStore tests are flaky - A lock could not be obtained within the time requested
[
https://issues.apache.org/jira/browse/HIVE-21550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907246#comment-16907246
]
Laszlo Bodor commented on HIVE-21550:
-
thanks [~vgarg], looking at current state of TestObjectStore:
https://github.com/apache/hive/blob/4510efd15f44cc4c217bbc65ad2147c14261bccc/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java#L147
NPE is thrown while calling a method of a recently created object, not really
sure how this is possible
> TestObjectStore tests are flaky - A lock could not be obtained within the
> time requested
> -
>
> Key: HIVE-21550
> URL: https://issues.apache.org/jira/browse/HIVE-21550
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21550.01.patch, HIVE-21550.02.patch,
> HIVE-21550.repro.patch,
> TEST-230_UTBatch_standalone-metastore__metastore-server_20_tests-TEST-org.apache.hadoop.hive.metastore.TestObjectStore.xml,
> maven-test.txt, org.apache.hadoop.hive.metastore.TestObjectStore-output.txt,
> screenshot-builds.apache.org-2019.03.30-12-38-32.png,
> surefire_derby_stacktrace.log
>
>
> found in HIVE-21396
> TestObjectStore contains 24 tests, but 14 of them failed, the same ones,
> twice in a row
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16744/testReport]
> [https://builds.apache.org/job/PreCommit-HIVE-Build/16774/testReport]
> {code:java}
> org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
> (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=230)
> org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
> (batchId=230)
> {code}
> all of the tests fail while initializing (see [^maven-test.txt]), dropping
> all objects (TestObjectStore.setUp:141->dropAllStoreObjects:776)
> {code:java}
> SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MPartition' AS
> NUCLEUS_TYPE,A0.CREATE_TIME,A0.LAST_ACCESS_TIME,A0.PART_NAME,A0.WRITE_ID,A0.PART_ID,A0.PART_NAME
> AS NUCORDER0 FROM PARTITIONS A0 LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID =
> B0.TBL_ID LEFT OUTER JOIN DBS C0 ON B0.DB_ID = C0.DB_ID WHERE B0.TBL_NAME = ?
> AND C0."NAME" = ? AND C0.CTLG_NAME = ? ORDER BY NUCORDER0 FETCH NEXT 100 ROWS
> ONLY
> {code}
> seems like a deadlock or stuff, all the tests are failed in 2min0sec, so an
> increased timeout wouldn't help here i think
> {code:java}
> javax.jdo.JDODataStoreException: Error executing SQL query "select
> "PARTITIONS"."PART_ID" from "PARTITIONS" inner join "TBLS" on
> "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ? inner join
> "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID" and "DBS"."NAME" = ? where
> "DBS"."CTLG_NAME" = ? order by "PART_NAME" asc". at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:267)
> ~[datanucleus-api-jdo-4.2.4.jar:?] at
> org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray(MetastoreDirectSqlUtils.java:61)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1882)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionIdsViaSqlFilter(MetaStoreDirectSql.java:759)
> [classes/:?] at
> org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitions(MetaStoreDirectSql.java:673)
> [classes/:?] at
>
[jira] [Updated] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-20801: Attachment: HIVE-20801.3.patch > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal V >Assignee: Laszlo Bodor >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch, > HIVE-20801.2.patch, HIVE-20801.3.patch, HIVE-20801.3.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor reassigned HIVE-20801: --- Assignee: Gopal V (was: Laszlo Bodor) > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch, > HIVE-20801.2.patch, HIVE-20801.3.patch, HIVE-20801.3.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (HIVE-20801) ACID: Allow DbTxnManager to ignore non-ACID table locking
[ https://issues.apache.org/jira/browse/HIVE-20801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor reassigned HIVE-20801: --- Assignee: Laszlo Bodor (was: Gopal V) > ACID: Allow DbTxnManager to ignore non-ACID table locking > - > > Key: HIVE-20801 > URL: https://issues.apache.org/jira/browse/HIVE-20801 > Project: Hive > Issue Type: Bug > Components: Locking, Transactions >Affects Versions: 4.0.0 >Reporter: Gopal V >Assignee: Laszlo Bodor >Priority: Major > Labels: Branch3Candidate, TODOC > Attachments: HIVE-20801.1.patch, HIVE-20801.2.patch, > HIVE-20801.2.patch, HIVE-20801.3.patch, HIVE-20801.3.patch > > > Enabling ACIDv1 on a cluster produces a central locking bottleneck for all > table types, which is not always the intention. > The Hive locking for non-acid tables are advisory (i.e a client can > write/read without locking), which means that the implementation does not > offer strong consistency despite the lock manager consuming resources > centrally. > Disabling this lock acquisition would improve the performance of non-ACID > tables co-existing with a globally configured DbTxnManager implementation. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-16690) Configure Tez cartesian product edge based on LLAP cluster size
[ https://issues.apache.org/jira/browse/HIVE-16690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-16690: Attachment: HIVE-16690.2.patch > Configure Tez cartesian product edge based on LLAP cluster size > --- > > Key: HIVE-16690 > URL: https://issues.apache.org/jira/browse/HIVE-16690 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-16690.1.patch, HIVE-16690.2.patch, > HIVE-16690.2.patch, HIVE-16690.2.patch, HIVE-16690.2.patch, > HIVE-16690.2.patch, HIVE-16690.2.patch, HIVE-16690.addendum.patch > > > In HIVE-14731 we are using default value for target parallelism of fair > cartesian product edge. Ideally this should be set according to cluster size. > In case of LLAP it's pretty easy to get cluster size, i.e., number of > executors. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (HIVE-16690) Configure Tez cartesian product edge based on LLAP cluster size
[ https://issues.apache.org/jira/browse/HIVE-16690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor reassigned HIVE-16690: --- Assignee: Laszlo Bodor (was: Zhiyuan Yang) > Configure Tez cartesian product edge based on LLAP cluster size > --- > > Key: HIVE-16690 > URL: https://issues.apache.org/jira/browse/HIVE-16690 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-16690.1.patch, HIVE-16690.2.patch, > HIVE-16690.2.patch, HIVE-16690.2.patch, HIVE-16690.2.patch, > HIVE-16690.2.patch, HIVE-16690.addendum.patch > > > In HIVE-14731 we are using default value for target parallelism of fair > cartesian product edge. Ideally this should be set according to cluster size. > In case of LLAP it's pretty easy to get cluster size, i.e., number of > executors. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21944) Remove unused methods, fields and variables from Vectorizer
[
https://issues.apache.org/jira/browse/HIVE-21944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896565#comment-16896565
]
Laszlo Bodor commented on HIVE-21944:
-
[~isuller]: I've just reviewed your another patch and found 1 line which is
contradictory to that.
In
[HIVE-21962.2.patch|https://issues.apache.org/jira/secure/attachment/12975370/HIVE-21962.2.patch#file-6]
you refactored something in vectorizer which can avoid a needless shallow copy
(as far as I can understand):
from
{code}
mapWork.setVectorizationEnabledConditionsMet(new
ArrayList(enabledConditionsMetSet));
{code}
to
{code}
mapWork.setVectorizationEnabledConditionsMet(enabledConditionsMetSet);
{code}
however here the last patch seems to touch the same code part, but doesn't care
about that
https://issues.apache.org/jira/secure/attachment/12975371/HIVE-21944.1.patch
Apart from that, this looks good to me.
> Remove unused methods, fields and variables from Vectorizer
> ---
>
> Key: HIVE-21944
> URL: https://issues.apache.org/jira/browse/HIVE-21944
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Ivan Suller
>Assignee: Ivan Suller
>Priority: Trivial
> Attachments: HIVE-21944.1.patch, HIVE-21944.1.patch,
> HIVE-21944.1.patch, HIVE-21944.1.patch
>
>
> It seems there are many unused fields, variables and methods in
> org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer class. Removing them
> would make the code easier to understand.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HIVE-21962) Replacing ArrayList params with List in and around PlanUtils and MapWork
[ https://issues.apache.org/jira/browse/HIVE-21962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896558#comment-16896558 ] Laszlo Bodor commented on HIVE-21962: - +1 > Replacing ArrayList params with List in and around PlanUtils and MapWork > > > Key: HIVE-21962 > URL: https://issues.apache.org/jira/browse/HIVE-21962 > Project: Hive > Issue Type: Improvement >Reporter: Ivan Suller >Assignee: Ivan Suller >Priority: Minor > Attachments: HIVE-21962.1.patch, HIVE-21962.1.patch, > HIVE-21962.2.patch, HIVE-21962.2.patch > > > Using the implementing class is usually a bad practice. OO suggest to use the > least restrictive interface instead. ArrayList is used in many-many methods > as a parameter - this is just a tiny part of this work. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-21173) Upgrade Apache Thrift to 0.9.3-1
[ https://issues.apache.org/jira/browse/HIVE-21173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21173: Fix Version/s: 4.0.0 > Upgrade Apache Thrift to 0.9.3-1 > > > Key: HIVE-21173 > URL: https://issues.apache.org/jira/browse/HIVE-21173 > Project: Hive > Issue Type: Bug > Components: Thrift API >Reporter: James E. King III >Assignee: David Lavati >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21173.01.patch > > Time Spent: 40m > Remaining Estimate: 0h > > The project currently depends on libthrift-0.9.3, however thrift released > 0.12.0 on 2019-JAN-04. This release includes a security fix for THRIFT-4506 > (CVE-2018-1320). Updating thrift to the latest version will remove that > vulnerability. > Also note the Apache Thrift project does not publish "libfb303" any longer. > fb303 is contributed code (in '/contrib') and it has not been maintained. > > Ps.: 0.9.3.1 also addresses the CVE, see THRIFT-4506 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
