[jira] [Comment Edited] (HIVE-18263) Ptest execution are multiple times slower sometimes due to dying executor slaves
[
https://issues.apache.org/jira/browse/HIVE-18263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289286#comment-16289286
]
Barna Zsombor Klara edited comment on HIVE-18263 at 12/13/17 2:00 PM:
--
Based on the logs I tend to agree with [~szita] and the proposed patch is small
with a very limited impact. Even if we misunderstood something here the
possibility of regression is small.
So +1
was (Author: zsombor.klara):
Based on the logs I tend to agree with [~szita] and the proposed patch is small
with a very limited impact. Even if we misunderstood something here the
possibility of regression is small.
> Ptest execution are multiple times slower sometimes due to dying executor
> slaves
>
>
> Key: HIVE-18263
> URL: https://issues.apache.org/jira/browse/HIVE-18263
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: HIVE-18263.0.patch
>
>
> PreCommit-HIVE-Build job has been seen running very long from time to time.
> Usually it should take about 1.5 hours, but in some cases it took over 4-5
> hours.
> Looking in the logs of one such execution I've seen that some commands that
> were sent to test executing slaves returned 255. Here this typically means
> that there is unknown return code for the remote call since hiveptest-server
> can't reach these slaves anymore.
> In the hiveptest-server logs it is seen that some slaves were killed while
> running the job normally, and here is why:
> * Hive's ptest-server checks periodically in every 60 minutes the status of
> slaves. It also keeps track of slaves that were terminated.
> ** If upon such check it is found that a slave that was already killed
> ([mTerminatedHosts
> map|https://github.com/apache/hive/blob/master/testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/CloudExecutionContextProvider.java#L93]
> contains its IP) is still running, it will try and terminate it again.
> * The server also maintains a file on its local FS that contains the IP of
> hosts that were used before. (This probably for resilience reasons)
> ** This file is read when tomcat server starts and if any of the IPs in the
> file are seen as running slaves, ptest will terminate these first so it can
> begin with a fresh start
> ** The IPs of these terminated instances already make their way into
> {{mTerminatedHosts}} upon initialization...
> * The cloud provider may reuse some older IPs, so it is not too rare that the
> same IP that belonged to a terminated host is assigned to a new one.
> This is problematic: Hive ptest's slave caretaker thread kicks in every 60
> minutes and might see a running host that has the same IP as an old slave had
> which was terminated at startup. It will think that this host should be
> terminated since it already tried 60 minutes ago as its IP is in
> {{mTerminatedHosts}}
> We have to fix this by making sure that if a new slave is created, we check
> the contents of {{mTerminatedHosts}} and remove this IP from it if it is
> there.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18263) Ptest execution are multiple times slower sometimes due to dying executor slaves
[
https://issues.apache.org/jira/browse/HIVE-18263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289286#comment-16289286
]
Barna Zsombor Klara commented on HIVE-18263:
Based on the logs I tend to agree with [~szita] and the proposed patch is small
with a very limited impact. Even if we misunderstood something here the
possibility of regression is small.
> Ptest execution are multiple times slower sometimes due to dying executor
> slaves
>
>
> Key: HIVE-18263
> URL: https://issues.apache.org/jira/browse/HIVE-18263
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: HIVE-18263.0.patch
>
>
> PreCommit-HIVE-Build job has been seen running very long from time to time.
> Usually it should take about 1.5 hours, but in some cases it took over 4-5
> hours.
> Looking in the logs of one such execution I've seen that some commands that
> were sent to test executing slaves returned 255. Here this typically means
> that there is unknown return code for the remote call since hiveptest-server
> can't reach these slaves anymore.
> In the hiveptest-server logs it is seen that some slaves were killed while
> running the job normally, and here is why:
> * Hive's ptest-server checks periodically in every 60 minutes the status of
> slaves. It also keeps track of slaves that were terminated.
> ** If upon such check it is found that a slave that was already killed
> ([mTerminatedHosts
> map|https://github.com/apache/hive/blob/master/testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/CloudExecutionContextProvider.java#L93]
> contains its IP) is still running, it will try and terminate it again.
> * The server also maintains a file on its local FS that contains the IP of
> hosts that were used before. (This probably for resilience reasons)
> ** This file is read when tomcat server starts and if any of the IPs in the
> file are seen as running slaves, ptest will terminate these first so it can
> begin with a fresh start
> ** The IPs of these terminated instances already make their way into
> {{mTerminatedHosts}} upon initialization...
> * The cloud provider may reuse some older IPs, so it is not too rare that the
> same IP that belonged to a terminated host is assigned to a new one.
> This is problematic: Hive ptest's slave caretaker thread kicks in every 60
> minutes and might see a running host that has the same IP as an old slave had
> which was terminated at startup. It will think that this host should be
> terminated since it already tried 60 minutes ago as its IP is in
> {{mTerminatedHosts}}
> We have to fix this by making sure that if a new slave is created, we check
> the contents of {{mTerminatedHosts}} and remove this IP from it if it is
> there.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-18263) Ptest execution are multiple times slower sometimes due to dying executor slaves
[
https://issues.apache.org/jira/browse/HIVE-18263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16287976#comment-16287976
]
Barna Zsombor Klara edited comment on HIVE-18263 at 12/12/17 6:03 PM:
--
Thank you for the patch Adam and for the detailed analysis. Just one minor
question:
Any reason we gather all the addresses into one set then iterate over the set
instead of iterating over the nodes and iterating in a nested manner over the
addresses to remove them from the failed hosts collection?
Not a big issue, I'm just curious.
Otherwise +1.
[~spena] based on the linked Jira it seems you came to a different conclusion,
that the ips cannot clash between the killed and the live hosts. Would you
please help in clarifying what is/can be going on here? I'm confused.
was (Author: zsombor.klara):
Thank you for the patch Adam and for the detailed analysis. Just one minor
question:
Any reason we gather all the addresses into one set then iterate over the set
instead of iterating over the nodes and iterating in a nested manner over the
addresses to remove them from the failed hosts collection?
Not a big issue, I'm just curious.
Otherwise +1.
> Ptest execution are multiple times slower sometimes due to dying executor
> slaves
>
>
> Key: HIVE-18263
> URL: https://issues.apache.org/jira/browse/HIVE-18263
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: HIVE-18263.0.patch
>
>
> PreCommit-HIVE-Build job has been seen running very long from time to time.
> Usually it should take about 1.5 hours, but in some cases it took over 4-5
> hours.
> Looking in the logs of one such execution I've seen that some commands that
> were sent to test executing slaves returned 255. Here this typically means
> that there is unknown return code for the remote call since hiveptest-server
> can't reach these slaves anymore.
> In the hiveptest-server logs it is seen that some slaves were killed while
> running the job normally, and here is why:
> * Hive's ptest-server checks periodically in every 60 minutes the status of
> slaves. It also keeps track of slaves that were terminated.
> ** If upon such check it is found that a slave that was already killed
> ([mTerminatedHosts
> map|https://github.com/apache/hive/blob/master/testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/CloudExecutionContextProvider.java#L93]
> contains its IP) is still running, it will try and terminate it again.
> * The server also maintains a file on its local FS that contains the IP of
> hosts that were used before. (This probably for resilience reasons)
> ** This file is read when tomcat server starts and if any of the IPs in the
> file are seen as running slaves, ptest will terminate these first so it can
> begin with a fresh start
> ** The IPs of these terminated instances already make their way into
> {{mTerminatedHosts}} upon initialization...
> * The cloud provider may reuse some older IPs, so it is not too rare that the
> same IP that belonged to a terminated host is assigned to a new one.
> This is problematic: Hive ptest's slave caretaker thread kicks in every 60
> minutes and might see a running host that has the same IP as an old slave had
> which was terminated at startup. It will think that this host should be
> terminated since it already tried 60 minutes ago as its IP is in
> {{mTerminatedHosts}}
> We have to fix this by making sure that if a new slave is created, we check
> the contents of {{mTerminatedHosts}} and remove this IP from it if it is
> there.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18263) Ptest execution are multiple times slower sometimes due to dying executor slaves
[
https://issues.apache.org/jira/browse/HIVE-18263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16287976#comment-16287976
]
Barna Zsombor Klara commented on HIVE-18263:
Thank you for the patch Adam and for the detailed analysis. Just one minor
question:
Any reason we gather all the addresses into one set then iterate over the set
instead of iterating over the nodes and iterating in a nested manner over the
addresses to remove them from the failed hosts collection?
Not a big issue, I'm just curious.
Otherwise +1.
> Ptest execution are multiple times slower sometimes due to dying executor
> slaves
>
>
> Key: HIVE-18263
> URL: https://issues.apache.org/jira/browse/HIVE-18263
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: HIVE-18263.0.patch
>
>
> PreCommit-HIVE-Build job has been seen running very long from time to time.
> Usually it should take about 1.5 hours, but in some cases it took over 4-5
> hours.
> Looking in the logs of one such execution I've seen that some commands that
> were sent to test executing slaves returned 255. Here this typically means
> that there is unknown return code for the remote call since hiveptest-server
> can't reach these slaves anymore.
> In the hiveptest-server logs it is seen that some slaves were killed while
> running the job normally, and here is why:
> * Hive's ptest-server checks periodically in every 60 minutes the status of
> slaves. It also keeps track of slaves that were terminated.
> ** If upon such check it is found that a slave that was already killed
> ([mTerminatedHosts
> map|https://github.com/apache/hive/blob/master/testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/context/CloudExecutionContextProvider.java#L93]
> contains its IP) is still running, it will try and terminate it again.
> * The server also maintains a file on its local FS that contains the IP of
> hosts that were used before. (This probably for resilience reasons)
> ** This file is read when tomcat server starts and if any of the IPs in the
> file are seen as running slaves, ptest will terminate these first so it can
> begin with a fresh start
> ** The IPs of these terminated instances already make their way into
> {{mTerminatedHosts}} upon initialization...
> * The cloud provider may reuse some older IPs, so it is not too rare that the
> same IP that belonged to a terminated host is assigned to a new one.
> This is problematic: Hive ptest's slave caretaker thread kicks in every 60
> minutes and might see a running host that has the same IP as an old slave had
> which was terminated at startup. It will think that this host should be
> terminated since it already tried 60 minutes ago as its IP is in
> {{mTerminatedHosts}}
> We have to fix this by making sure that if a new slave is created, we check
> the contents of {{mTerminatedHosts}} and remove this IP from it if it is
> there.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17785) Encription tests are not running
[
https://issues.apache.org/jira/browse/HIVE-17785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235576#comment-16235576
]
Barna Zsombor Klara commented on HIVE-17785:
+1
> Encription tests are not running
>
>
> Key: HIVE-17785
> URL: https://issues.apache.org/jira/browse/HIVE-17785
> Project: Hive
> Issue Type: Bug
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-17785.patch
>
>
> The testconfiguration.properties contains multiple tests in
> {{encrypted.query.files}}.
> There is no comma at the end of the list, so the tests are not running
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17300) WebUI query plan graphs
[ https://issues.apache.org/jira/browse/HIVE-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225256#comment-16225256 ] Barna Zsombor Klara commented on HIVE-17300: +1 > WebUI query plan graphs > --- > > Key: HIVE-17300 > URL: https://issues.apache.org/jira/browse/HIVE-17300 > Project: Hive > Issue Type: Improvement > Components: Web UI >Reporter: Karen Coppage >Assignee: Karen Coppage > Attachments: HIVE-17300.3.patch, HIVE-17300.4.patch, > HIVE-17300.5.patch, HIVE-17300.patch, complete_success.png, > full_mapred_stats.png, graph_with_mapred_stats.png, last_stage_error.png, > last_stage_running.png, non_mapred_task_selected.png > > > Hi all, > I’m working on a feature of the Hive WebUI Query Plan tab that would provide > the option to display the query plan as a nice graph (scroll down for > screenshots). If you click on one of the graph’s stages, the plan for that > stage appears as text below. > Stages are color-coded if they have a status (Success, Error, Running), and > the rest are grayed out. Coloring is based on status already available in the > WebUI, under the Stages tab. > There is an additional option to display stats for MapReduce tasks. This > includes the job’s ID, tracking URL (where the logs are found), and mapper > and reducer numbers/progress, among other info. > The library I’m using for the graph is called vis.js (http://visjs.org/). It > has an Apache license, and the only necessary file to be included from this > library is about 700 KB. > I tried to keep server-side changes minimal, and graph generation is taken > care of by the client. Plans with more than a given number of stages > (default: 25) won't be displayed in order to preserve resources. > I’d love to hear any and all input from the community about this feature: do > you think it’s useful, and is there anything important I’m missing? > Thanks, > Karen Coppage > Review request: https://reviews.apache.org/r/61663/ > Any input is welcome! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17800) input_part6.q wants to test partition pruning, but tests expression evaluation
[
https://issues.apache.org/jira/browse/HIVE-17800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207864#comment-16207864
]
Barna Zsombor Klara commented on HIVE-17800:
+1
> input_part6.q wants to test partition pruning, but tests expression evaluation
> --
>
> Key: HIVE-17800
> URL: https://issues.apache.org/jira/browse/HIVE-17800
> Project: Hive
> Issue Type: Bug
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-17800.patch
>
>
> input_part6.q looks like this:
> {code}
> EXPLAIN
> SELECT x.* FROM SRCPART x WHERE x.ds = 2008-04-08 LIMIT 10;
> {code}
> The intended test most probably is this:
> {code}
> EXPLAIN
> SELECT x.* FROM SRCPART x WHERE x.ds = "2008-04-08" LIMIT 10;
> {code}
> Currently we evaluete 2008-4-8 to 1996:
> {code}
> predicate: (UDFToDouble(ds) = 1996.0) (type: boolean)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17706) Add a possibility to run the BeeLine tests on the default database
[ https://issues.apache.org/jira/browse/HIVE-17706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207059#comment-16207059 ] Barna Zsombor Klara commented on HIVE-17706: My mistake [~leftylev], sorry for not commenting on the Jira. +1 here as well. > Add a possibility to run the BeeLine tests on the default database > -- > > Key: HIVE-17706 > URL: https://issues.apache.org/jira/browse/HIVE-17706 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-17706.2.patch, HIVE-17706.3.patch, > HIVE-17706.4.patch, HIVE-17706.patch > > > Currently it is possible to run the BeeLine tests sequentially but it still > relies on cleaning up after the tests by cleaning up the database. Some of > the tests could be run only against the default database. We need a cleanup > mechanism between the tests -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194597#comment-16194597
]
Barna Zsombor Klara commented on HIVE-17078:
Pushed to master thanks [~Yibing] for the patch and [~pvary] for the review.
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Barna Zsombor Klara
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HIVE-17078.1.patch, HIVE-17078.2.patch,
> HIVE-17078.3.patch, HIVE-17078.4.PATCH, HIVE-17078.5.PATCH,
> HIVE-17078.6.patch, HIVE-17078.7.patch
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barna Zsombor Klara updated HIVE-17078:
---
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Barna Zsombor Klara
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HIVE-17078.1.patch, HIVE-17078.2.patch,
> HIVE-17078.3.patch, HIVE-17078.4.PATCH, HIVE-17078.5.PATCH,
> HIVE-17078.6.patch, HIVE-17078.7.patch
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17633) Make it possible to override the query results directory in TestBeeLineDriver
[ https://issues.apache.org/jira/browse/HIVE-17633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194374#comment-16194374 ] Barna Zsombor Klara commented on HIVE-17633: +1 > Make it possible to override the query results directory in TestBeeLineDriver > - > > Key: HIVE-17633 > URL: https://issues.apache.org/jira/browse/HIVE-17633 > Project: Hive > Issue Type: Bug > Components: Test >Affects Versions: 3.0.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-17633.patch > > > It would be good to have the possibility to override where the > TestBeeLineDriver looks for the golden files -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barna Zsombor Klara reassigned HIVE-17078:
--
Assignee: Barna Zsombor Klara (was: Yibing Shi)
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Barna Zsombor Klara
>Priority: Minor
> Attachments: HIVE-17078.1.patch, HIVE-17078.2.patch,
> HIVE-17078.3.patch, HIVE-17078.4.PATCH, HIVE-17078.5.PATCH, HIVE-17078.6.patch
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barna Zsombor Klara updated HIVE-17078:
---
Attachment: HIVE-17078.7.patch
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Barna Zsombor Klara
>Priority: Minor
> Attachments: HIVE-17078.1.patch, HIVE-17078.2.patch,
> HIVE-17078.3.patch, HIVE-17078.4.PATCH, HIVE-17078.5.PATCH,
> HIVE-17078.6.patch, HIVE-17078.7.patch
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16191003#comment-16191003
]
Barna Zsombor Klara commented on HIVE-17078:
Failures should not be related.
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Minor
> Attachments: HIVE-17078.1.patch, HIVE-17078.2.patch,
> HIVE-17078.3.patch, HIVE-17078.4.PATCH, HIVE-17078.5.PATCH, HIVE-17078.6.patch
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barna Zsombor Klara reassigned HIVE-17078:
--
Assignee: Yibing Shi (was: Barna Zsombor Klara)
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Minor
> Attachments: HIVE-17078.1.patch, HIVE-17078.2.patch,
> HIVE-17078.3.patch, HIVE-17078.4.PATCH, HIVE-17078.5.PATCH, HIVE-17078.6.patch
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barna Zsombor Klara updated HIVE-17078:
---
Attachment: HIVE-17078.6.patch
Added a common redirector for spark and local mr.
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Barna Zsombor Klara
>Priority: Minor
> Attachments: HIVE-17078.1.patch, HIVE-17078.2.patch,
> HIVE-17078.3.patch, HIVE-17078.4.PATCH, HIVE-17078.5.PATCH, HIVE-17078.6.patch
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Barna Zsombor Klara reassigned HIVE-17078:
--
Assignee: Barna Zsombor Klara (was: Yibing Shi)
Assigning to myself temporarily to upload a new version of the patch.
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Barna Zsombor Klara
>Priority: Minor
> Attachments: HIVE-17078.1.patch, HIVE-17078.2.patch,
> HIVE-17078.3.patch, HIVE-17078.4.PATCH, HIVE-17078.5.PATCH
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17317) Make Dbcp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171491#comment-16171491 ] Barna Zsombor Klara commented on HIVE-17317: Failing tests should not be related. Failed tests with an age of 1 are passing locally. > Make Dbcp configurable using hive properties in hive-site.xml > - > > Key: HIVE-17317 > URL: https://issues.apache.org/jira/browse/HIVE-17317 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17317.01.patch, HIVE-17317.02.patch, > HIVE-17317.03.patch, HIVE-17317.04.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17078) Add more logs to MapredLocalTask
[
https://issues.apache.org/jira/browse/HIVE-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171261#comment-16171261
]
Barna Zsombor Klara commented on HIVE-17078:
Hi [~Yibing],
do you have the free time to do the refactoring suggested by Peter? If not then
I'd be happy to add the final touch. The patch looks good, so it would be a
shame to not commit it.
> Add more logs to MapredLocalTask
>
>
> Key: HIVE-17078
> URL: https://issues.apache.org/jira/browse/HIVE-17078
> Project: Hive
> Issue Type: Improvement
>Reporter: Yibing Shi
>Assignee: Yibing Shi
>Priority: Minor
> Attachments: HIVE-17078.1.patch, HIVE-17078.2.patch,
> HIVE-17078.3.patch, HIVE-17078.4.PATCH, HIVE-17078.5.PATCH
>
>
> By default, {{MapredLocalTask}} is executed in a child process of Hive, in
> case the local task uses too much resources that may affect Hive. Currently,
> the stdout and stderr information of the child process is printed in Hive's
> stdout/stderr log, which doesn't have a timestamp information, and is
> separated from Hive service logs. This makes it hard to troubleshoot problems
> in MapredLocalTasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17317) Make Dbcp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17317: --- Attachment: HIVE-17317.04.patch > Make Dbcp configurable using hive properties in hive-site.xml > - > > Key: HIVE-17317 > URL: https://issues.apache.org/jira/browse/HIVE-17317 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17317.01.patch, HIVE-17317.02.patch, > HIVE-17317.03.patch, HIVE-17317.04.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17317) Make Dbcp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17317: --- Attachment: (was: HIVE-17317.03.patch) > Make Dbcp configurable using hive properties in hive-site.xml > - > > Key: HIVE-17317 > URL: https://issues.apache.org/jira/browse/HIVE-17317 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17317.01.patch, HIVE-17317.02.patch, > HIVE-17317.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17317) Make Dbcp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17317: --- Attachment: HIVE-17317.03.patch Rebased the patch. > Make Dbcp configurable using hive properties in hive-site.xml > - > > Key: HIVE-17317 > URL: https://issues.apache.org/jira/browse/HIVE-17317 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17317.01.patch, HIVE-17317.02.patch, > HIVE-17317.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17317) Make Dbcp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17317: --- Attachment: HIVE-17317.03.patch Reuploading same patch. > Make Dbcp configurable using hive properties in hive-site.xml > - > > Key: HIVE-17317 > URL: https://issues.apache.org/jira/browse/HIVE-17317 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17317.01.patch, HIVE-17317.02.patch, > HIVE-17317.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17317) Make Dbcp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166148#comment-16166148 ] Barna Zsombor Klara commented on HIVE-17317: I ran all the failures locally and they passed. I can reupload the patch to see if it is failing again, but I think it is just flakyness. > Make Dbcp configurable using hive properties in hive-site.xml > - > > Key: HIVE-17317 > URL: https://issues.apache.org/jira/browse/HIVE-17317 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17317.01.patch, HIVE-17317.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17317) Make Dbcp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17317: --- Attachment: HIVE-17317.02.patch Fixing unit tests. > Make Dbcp configurable using hive properties in hive-site.xml > - > > Key: HIVE-17317 > URL: https://issues.apache.org/jira/browse/HIVE-17317 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17317.01.patch, HIVE-17317.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17317) Make Dbcp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17317: --- Status: Patch Available (was: Open) > Make Dbcp configurable using hive properties in hive-site.xml > - > > Key: HIVE-17317 > URL: https://issues.apache.org/jira/browse/HIVE-17317 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17317.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17317) Make Dbcp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17317: --- Attachment: HIVE-17317.01.patch > Make Dbcp configurable using hive properties in hive-site.xml > - > > Key: HIVE-17317 > URL: https://issues.apache.org/jira/browse/HIVE-17317 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17317.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17317) Make Dbcp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara reassigned HIVE-17317: -- Assignee: Barna Zsombor Klara > Make Dbcp configurable using hive properties in hive-site.xml > - > > Key: HIVE-17317 > URL: https://issues.apache.org/jira/browse/HIVE-17317 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17382) Change startsWith relation introduced in HIVE-17316
[ https://issues.apache.org/jira/browse/HIVE-17382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17382: --- Attachment: HIVE-17382.03.patch Fixed two failing unit tests. > Change startsWith relation introduced in HIVE-17316 > --- > > Key: HIVE-17382 > URL: https://issues.apache.org/jira/browse/HIVE-17382 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Fix For: 3.0.0 > > Attachments: HIVE-17382.01.patch, HIVE-17382.02.patch, > HIVE-17382.03.patch > > > In HiveConf the new name should be checked if it starts with a > restricted/hidden variable prefix and not vice-versa. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17382) Change startsWith relation introduced in HIVE-17316
[ https://issues.apache.org/jira/browse/HIVE-17382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17382: --- Attachment: HIVE-17382.02.patch Fixed BeeLine tests. > Change startsWith relation introduced in HIVE-17316 > --- > > Key: HIVE-17382 > URL: https://issues.apache.org/jira/browse/HIVE-17382 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Fix For: 3.0.0 > > Attachments: HIVE-17382.01.patch, HIVE-17382.02.patch > > > In HiveConf the new name should be checked if it starts with a > restricted/hidden variable prefix and not vice-versa. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17318) Make Hikari CP configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145485#comment-16145485 ] Barna Zsombor Klara commented on HIVE-17318: Thank you for the review [~pvary] Configuration wikipage has been updated. Thanks for reminding me [~leftylev]! > Make Hikari CP configurable using hive properties in hive-site.xml > -- > > Key: HIVE-17318 > URL: https://issues.apache.org/jira/browse/HIVE-17318 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-17318.01.patch, HIVE-17318.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17389) Yetus is always failing on rat checks
[ https://issues.apache.org/jira/browse/HIVE-17389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17389: --- Status: Patch Available (was: Open) > Yetus is always failing on rat checks > - > > Key: HIVE-17389 > URL: https://issues.apache.org/jira/browse/HIVE-17389 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Trivial > Attachments: HIVE-17389.01.patch > > > Rat checks are failing on metastore_db/dblock and files under patchprocess > created by Yetus itself. > Both directories should be excluded from rat checks. > CC: [~pvary] [~kgyrtkirk] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17389) Yetus is always failing on rat checks
[ https://issues.apache.org/jira/browse/HIVE-17389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17389: --- Attachment: HIVE-17389.01.patch > Yetus is always failing on rat checks > - > > Key: HIVE-17389 > URL: https://issues.apache.org/jira/browse/HIVE-17389 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Trivial > Attachments: HIVE-17389.01.patch > > > Rat checks are failing on metastore_db/dblock and files under patchprocess > created by Yetus itself. > Both directories should be excluded from rat checks. > CC: [~pvary] [~kgyrtkirk] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17389) Yetus is always failing on rat checks
[ https://issues.apache.org/jira/browse/HIVE-17389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara reassigned HIVE-17389: -- > Yetus is always failing on rat checks > - > > Key: HIVE-17389 > URL: https://issues.apache.org/jira/browse/HIVE-17389 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Trivial > > Rat checks are failing on metastore_db/dblock and files under patchprocess > created by Yetus itself. > Both directories should be excluded from rat checks. > CC: [~pvary] [~kgyrtkirk] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17107) Upgrade Yetus to 0.5.0
[ https://issues.apache.org/jira/browse/HIVE-17107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17107: --- Status: Patch Available (was: Open) > Upgrade Yetus to 0.5.0 > -- > > Key: HIVE-17107 > URL: https://issues.apache.org/jira/browse/HIVE-17107 > Project: Hive > Issue Type: Task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Barna Zsombor Klara > Attachments: HIVE-17107.01.patch > > > [Yetus 0.5.0|https://yetus.apache.org/documentation/0.5.0/RELEASENOTES/] is > released, and it contains our fixes. > We should upgrade and remove our extra patched files. > CC: [~kgyrtkirk] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17107) Upgrade Yetus to 0.5.0
[ https://issues.apache.org/jira/browse/HIVE-17107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara reassigned HIVE-17107: -- Assignee: Barna Zsombor Klara (was: Peter Vary) > Upgrade Yetus to 0.5.0 > -- > > Key: HIVE-17107 > URL: https://issues.apache.org/jira/browse/HIVE-17107 > Project: Hive > Issue Type: Task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Barna Zsombor Klara > Attachments: HIVE-17107.01.patch > > > [Yetus 0.5.0|https://yetus.apache.org/documentation/0.5.0/RELEASENOTES/] is > released, and it contains our fixes. > We should upgrade and remove our extra patched files. > CC: [~kgyrtkirk] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17107) Upgrade Yetus to 0.5.0
[ https://issues.apache.org/jira/browse/HIVE-17107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17107: --- Attachment: HIVE-17107.01.patch Uploading patch. > Upgrade Yetus to 0.5.0 > -- > > Key: HIVE-17107 > URL: https://issues.apache.org/jira/browse/HIVE-17107 > Project: Hive > Issue Type: Task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Barna Zsombor Klara > Attachments: HIVE-17107.01.patch > > > [Yetus 0.5.0|https://yetus.apache.org/documentation/0.5.0/RELEASENOTES/] is > released, and it contains our fixes. > We should upgrade and remove our extra patched files. > CC: [~kgyrtkirk] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17382) Change startsWith relation introduced in HIVE-17316
[ https://issues.apache.org/jira/browse/HIVE-17382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17382: --- Status: Patch Available (was: Open) > Change startsWith relation introduced in HIVE-17316 > --- > > Key: HIVE-17382 > URL: https://issues.apache.org/jira/browse/HIVE-17382 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Fix For: 3.0.0 > > Attachments: HIVE-17382.01.patch > > > In HiveConf the new name should be checked if it starts with a > restricted/hidden variable prefix and not vice-versa. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17382) Change startsWith relation introduced in HIVE-17316
[ https://issues.apache.org/jira/browse/HIVE-17382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17382: --- Attachment: HIVE-17382.01.patch > Change startsWith relation introduced in HIVE-17316 > --- > > Key: HIVE-17382 > URL: https://issues.apache.org/jira/browse/HIVE-17382 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Fix For: 3.0.0 > > Attachments: HIVE-17382.01.patch > > > In HiveConf the new name should be checked if it starts with a > restricted/hidden variable prefix and not vice-versa. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17382) Change startsWith relation introduced in HIVE-17316
[ https://issues.apache.org/jira/browse/HIVE-17382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara reassigned HIVE-17382: -- > Change startsWith relation introduced in HIVE-17316 > --- > > Key: HIVE-17382 > URL: https://issues.apache.org/jira/browse/HIVE-17382 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Fix For: 3.0.0 > > > In HiveConf the new name should be checked if it starts with a > restricted/hidden variable prefix and not vice-versa. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17318) Make Hikari CP configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17318: --- Attachment: HIVE-17318.02.patch Updated the patch based on review board comments. > Make Hikari CP configurable using hive properties in hive-site.xml > -- > > Key: HIVE-17318 > URL: https://issues.apache.org/jira/browse/HIVE-17318 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17318.01.patch, HIVE-17318.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17318) Make Hikari CP configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17318: --- Status: Patch Available (was: Open) > Make Hikari CP configurable using hive properties in hive-site.xml > -- > > Key: HIVE-17318 > URL: https://issues.apache.org/jira/browse/HIVE-17318 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17318.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17318) Make Hikari CP configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17318: --- Attachment: HIVE-17318.01.patch Uploading first patch for HikariCP. Should be similar to the patch for BoneCP. > Make Hikari CP configurable using hive properties in hive-site.xml > -- > > Key: HIVE-17318 > URL: https://issues.apache.org/jira/browse/HIVE-17318 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17318.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17318) Make Hikari CP configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17318: --- Summary: Make Hikari CP configurable using hive properties in hive-site.xml (was: Make Hikari configurable using hive properties in hive-site.xml) > Make Hikari CP configurable using hive properties in hive-site.xml > -- > > Key: HIVE-17318 > URL: https://issues.apache.org/jira/browse/HIVE-17318 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17318) Make Hikari configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara reassigned HIVE-17318: -- Assignee: Barna Zsombor Klara > Make Hikari configurable using hive properties in hive-site.xml > --- > > Key: HIVE-17318 > URL: https://issues.apache.org/jira/browse/HIVE-17318 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17319) Make BoneCp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138166#comment-16138166 ] Barna Zsombor Klara commented on HIVE-17319: Tests failures should not be related. > Make BoneCp configurable using hive properties in hive-site.xml > --- > > Key: HIVE-17319 > URL: https://issues.apache.org/jira/browse/HIVE-17319 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17319.01.patch, HIVE-17319.02.patch, > HIVE-17319.03.patch, HIVE-17319.draft.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17319) Make BoneCp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17319: --- Attachment: HIVE-17319.03.patch Minor change: get the connection timeout property from the configuration object so we don't need to parse the long value from the string ourselves. > Make BoneCp configurable using hive properties in hive-site.xml > --- > > Key: HIVE-17319 > URL: https://issues.apache.org/jira/browse/HIVE-17319 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17319.01.patch, HIVE-17319.02.patch, > HIVE-17319.03.patch, HIVE-17319.draft.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17319) Make BoneCp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17319: --- Attachment: HIVE-17319.02.patch Rebased patch after the removal of Shims from HMS. > Make BoneCp configurable using hive properties in hive-site.xml > --- > > Key: HIVE-17319 > URL: https://issues.apache.org/jira/browse/HIVE-17319 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17319.01.patch, HIVE-17319.02.patch, > HIVE-17319.draft.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17356) Missing ASF headers 3 classes
[ https://issues.apache.org/jira/browse/HIVE-17356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17356: --- Status: Patch Available (was: Open) > Missing ASF headers 3 classes > - > > Key: HIVE-17356 > URL: https://issues.apache.org/jira/browse/HIVE-17356 > Project: Hive > Issue Type: Bug >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Trivial > Attachments: HIVE-17356.01.patch > > > JSONAddNotNullConstraintMessage.java, BucketCodec.java, TaskTrackerTest.java > are missing the ASF header that should be added. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17356) Missing ASF headers 3 classes
[ https://issues.apache.org/jira/browse/HIVE-17356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17356: --- Attachment: HIVE-17356.01.patch [~daijy], [~ekoifman], [~anishek] While running a Yetus check for a different patch I ran into some files missing the ASF headers authored and/or committed by you. If you don't mind I would add the headers, please -1 in case you would like to retain ownership of the files. > Missing ASF headers 3 classes > - > > Key: HIVE-17356 > URL: https://issues.apache.org/jira/browse/HIVE-17356 > Project: Hive > Issue Type: Bug >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Trivial > Attachments: HIVE-17356.01.patch > > > JSONAddNotNullConstraintMessage.java, BucketCodec.java, TaskTrackerTest.java > are missing the ASF header that should be added. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17356) Missing ASF headers 3 classes
[ https://issues.apache.org/jira/browse/HIVE-17356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara reassigned HIVE-17356: -- > Missing ASF headers 3 classes > - > > Key: HIVE-17356 > URL: https://issues.apache.org/jira/browse/HIVE-17356 > Project: Hive > Issue Type: Bug >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Trivial > > JSONAddNotNullConstraintMessage.java, BucketCodec.java, TaskTrackerTest.java > are missing the ASF header that should be added. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17319) Make BoneCp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17319: --- Attachment: HIVE-17319.01.patch > Make BoneCp configurable using hive properties in hive-site.xml > --- > > Key: HIVE-17319 > URL: https://issues.apache.org/jira/browse/HIVE-17319 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17319.01.patch, HIVE-17319.draft.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17319) Make BoneCp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17319: --- Status: Patch Available (was: In Progress) > Make BoneCp configurable using hive properties in hive-site.xml > --- > > Key: HIVE-17319 > URL: https://issues.apache.org/jira/browse/HIVE-17319 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17319.01.patch, HIVE-17319.draft.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17277) HiveMetastoreClient Log name is wrong
[ https://issues.apache.org/jira/browse/HIVE-17277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara reassigned HIVE-17277: -- Assignee: Zac Zhou (was: Barna Zsombor Klara) > HiveMetastoreClient Log name is wrong > - > > Key: HIVE-17277 > URL: https://issues.apache.org/jira/browse/HIVE-17277 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Zac Zhou >Assignee: Zac Zhou >Priority: Minor > Attachments: HIVE-17277.2.patch, HIVE-17277.3.patch, HIVE-17277.patch > > > The name of Log for HiveMetastoreClient is "hive.metastore". It's confused > for users to trace hive log -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17277) HiveMetastoreClient Log name is wrong
[ https://issues.apache.org/jira/browse/HIVE-17277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17277: --- Attachment: HIVE-17277.3.patch [~yuan_zac] The PTest framework was restarted while your tests were running so the results will not be updated in this Jira. I reuploaded the same patch (HIVE-17277.2.patch) to retrigger the testing for you. Sorry for the inconvenience. > HiveMetastoreClient Log name is wrong > - > > Key: HIVE-17277 > URL: https://issues.apache.org/jira/browse/HIVE-17277 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Zac Zhou >Assignee: Barna Zsombor Klara >Priority: Minor > Attachments: HIVE-17277.2.patch, HIVE-17277.3.patch, HIVE-17277.patch > > > The name of Log for HiveMetastoreClient is "hive.metastore". It's confused > for users to trace hive log -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17277) HiveMetastoreClient Log name is wrong
[ https://issues.apache.org/jira/browse/HIVE-17277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara reassigned HIVE-17277: -- Assignee: Barna Zsombor Klara (was: Zac Zhou) > HiveMetastoreClient Log name is wrong > - > > Key: HIVE-17277 > URL: https://issues.apache.org/jira/browse/HIVE-17277 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Zac Zhou >Assignee: Barna Zsombor Klara >Priority: Minor > Attachments: HIVE-17277.2.patch, HIVE-17277.patch > > > The name of Log for HiveMetastoreClient is "hive.metastore". It's confused > for users to trace hive log -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17319) Make BoneCp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17319: --- Attachment: HIVE-17319.draft.patch Attaching the first draft of the patch for BoneCP. [~thejas], [~ekoifman], [~pvary] I hope this will clarify a bit what I had in mind. If you have the time please feel free to comment on the patch here or on reviewboard. > Make BoneCp configurable using hive properties in hive-site.xml > --- > > Key: HIVE-17319 > URL: https://issues.apache.org/jira/browse/HIVE-17319 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17319.draft.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17319) Make BoneCp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara reassigned HIVE-17319: -- Assignee: Barna Zsombor Klara > Make BoneCp configurable using hive properties in hive-site.xml > --- > > Key: HIVE-17319 > URL: https://issues.apache.org/jira/browse/HIVE-17319 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Work started] (HIVE-17319) Make BoneCp configurable using hive properties in hive-site.xml
[ https://issues.apache.org/jira/browse/HIVE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-17319 started by Barna Zsombor Klara. -- > Make BoneCp configurable using hive properties in hive-site.xml > --- > > Key: HIVE-17319 > URL: https://issues.apache.org/jira/browse/HIVE-17319 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-14747) Remove JAVA paths from profiles by sending them from ptest-client
[ https://issues.apache.org/jira/browse/HIVE-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-14747: --- Attachment: HIVE-14747.06.patch > Remove JAVA paths from profiles by sending them from ptest-client > - > > Key: HIVE-14747 > URL: https://issues.apache.org/jira/browse/HIVE-14747 > Project: Hive > Issue Type: Sub-task > Components: Hive, Testing Infrastructure >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-14747.01.patch, HIVE-14747.02.patch, > HIVE-14747.03.patch, HIVE-14747.04.patch, HIVE-14747.05.patch, > HIVE-14747.06.patch > > > Hive ptest uses some properties files per branch that contain information > about how to execute the tests. > This profile includes JAVA paths to build and execute the tests. We should > get rid of these by passing such information from Jenkins to the > ptest-server. In case a profile needs a different java version, then we can > create a specific Jenkins job for that. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17316) Use String.startsWith for the hidden configuration variables
[ https://issues.apache.org/jira/browse/HIVE-17316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17316: --- Description: Currently HiveConf variables which should not be displayed to the user need to be enumerated. We should enhance this to be able to hide configuration variables by string prefix not just full equality. (was: Currently HiveConf variables which should not be displayed to the user need to be enumerated. We should enhance this to be able to hide configuration variables by substring not just full equality.) > Use String.startsWith for the hidden configuration variables > > > Key: HIVE-17316 > URL: https://issues.apache.org/jira/browse/HIVE-17316 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17316.01.patch, HIVE-17316.02.patch, > HIVE-17316.03.patch > > > Currently HiveConf variables which should not be displayed to the user need > to be enumerated. We should enhance this to be able to hide configuration > variables by string prefix not just full equality. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17316) Use String.startsWith for the hidden configuration variables
[ https://issues.apache.org/jira/browse/HIVE-17316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17316: --- Summary: Use String.startsWith for the hidden configuration variables (was: Use String.contains for the hidden configuration variables) > Use String.startsWith for the hidden configuration variables > > > Key: HIVE-17316 > URL: https://issues.apache.org/jira/browse/HIVE-17316 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17316.01.patch, HIVE-17316.02.patch, > HIVE-17316.03.patch > > > Currently HiveConf variables which should not be displayed to the user need > to be enumerated. We should enhance this to be able to hide configuration > variables by substring not just full equality. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17322) Serialise BeeLine qtest execution to prevent flakyness
[ https://issues.apache.org/jira/browse/HIVE-17322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129117#comment-16129117 ] Barna Zsombor Klara commented on HIVE-17322: [~pvary] I agree. I raised and linked the followup Jira and resolved the other two. > Serialise BeeLine qtest execution to prevent flakyness > -- > > Key: HIVE-17322 > URL: https://issues.apache.org/jira/browse/HIVE-17322 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Minor > Fix For: 3.0.0 > > Attachments: HIVE-17322.01.patch, HIVE-17322.02.patch, > HIVE-17322.03.patch, HIVE-17322.04.patch, HIVE-17322.05.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HIVE-16959) Flaky Test : TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
[ https://issues.apache.org/jira/browse/HIVE-16959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara resolved HIVE-16959. Resolution: Fixed Flakiness should be fixed by HIVE-17322. Please reopen if test starts failing again. > Flaky Test : > TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1] > > > Key: HIVE-16959 > URL: https://issues.apache.org/jira/browse/HIVE-16959 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Barna Zsombor Klara > > Test failed on the pre-commit but runs locally. > Error Message > Client result comparison failed with error code = 1 while executing > fname=insert_overwrite_local_directory_1 > 1172d1171 > < k21=v21#k22=v22#k31=v31:foo2 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HIVE-16796) Flaky Test : TestBeeLineDriver.testCliDriver[create_merge_compressed]
[ https://issues.apache.org/jira/browse/HIVE-16796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara resolved HIVE-16796. Resolution: Fixed Flakiness should be fixed by HIVE-17322. Please reopen if test starts failing again. > Flaky Test : TestBeeLineDriver.testCliDriver[create_merge_compressed] > - > > Key: HIVE-16796 > URL: https://issues.apache.org/jira/browse/HIVE-16796 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar > > Has been failing in many of the recent builds. > https://builds.apache.org/job/PreCommit-HIVE-Build/5481/ > https://builds.apache.org/job/PreCommit-HIVE-Build/5482/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17322) Serialise BeeLine qtest execution to prevent flakyness
[ https://issues.apache.org/jira/browse/HIVE-17322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128859#comment-16128859 ] Barna Zsombor Klara commented on HIVE-17322: [~pvary] BeeLineTest failures seem to have disappeared. I think we can safely assume that the parallelism was the source of the flakiness. > Serialise BeeLine qtest execution to prevent flakyness > -- > > Key: HIVE-17322 > URL: https://issues.apache.org/jira/browse/HIVE-17322 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Minor > Attachments: HIVE-17322.01.patch, HIVE-17322.02.patch, > HIVE-17322.03.patch, HIVE-17322.04.patch, HIVE-17322.05.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17325) Clean up intermittently failing uni tests
[ https://issues.apache.org/jira/browse/HIVE-17325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128660#comment-16128660 ] Barna Zsombor Klara commented on HIVE-17325: Just a small clarification HIVE-17305 is fixing 3 tests: -TestBlobstoreCliDriver.testCliDriver.insert_overwrite_dynamic_partitions_merge_move -TestBlobstoreCliDriver.testCliDriver.insert_overwrite_dynamic_partitions_merge_only -TestBlobstoreCliDriver.testCliDriver.insert_overwrite_dynamic_partitions_move_only These 3 are not failing anymore, sorry that it took so long to fix. Unfortunately it will *not* fix TestMiniSparkOnYarnCliDriver.testCliDriver.spark_dynamic_partition_pruning. > Clean up intermittently failing uni tests > - > > Key: HIVE-17325 > URL: https://issues.apache.org/jira/browse/HIVE-17325 > Project: Hive > Issue Type: Test > Components: Tests >Reporter: Alan Gates >Assignee: Alan Gates > > We have a number of intermittently failing tests. I propose to disable these > so that we can get clean (or at least cleaner) CI runs. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17316) Use String.contains for the hidden configuration variables
[ https://issues.apache.org/jira/browse/HIVE-17316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128658#comment-16128658 ] Barna Zsombor Klara commented on HIVE-17316: Failures should not be related. > Use String.contains for the hidden configuration variables > -- > > Key: HIVE-17316 > URL: https://issues.apache.org/jira/browse/HIVE-17316 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17316.01.patch, HIVE-17316.02.patch, > HIVE-17316.03.patch > > > Currently HiveConf variables which should not be displayed to the user need > to be enumerated. We should enhance this to be able to hide configuration > variables by substring not just full equality. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17316) Use String.contains for the hidden configuration variables
[ https://issues.apache.org/jira/browse/HIVE-17316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17316: --- Attachment: HIVE-17316.03.patch Regenerated the negative qtests with the new error message. > Use String.contains for the hidden configuration variables > -- > > Key: HIVE-17316 > URL: https://issues.apache.org/jira/browse/HIVE-17316 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17316.01.patch, HIVE-17316.02.patch, > HIVE-17316.03.patch > > > Currently HiveConf variables which should not be displayed to the user need > to be enumerated. We should enhance this to be able to hide configuration > variables by substring not just full equality. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17322) Serialise BeeLine qtest execution to prevent flakyness
[ https://issues.apache.org/jira/browse/HIVE-17322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17322: --- Attachment: HIVE-17322.05.patch > Serialise BeeLine qtest execution to prevent flakyness > -- > > Key: HIVE-17322 > URL: https://issues.apache.org/jira/browse/HIVE-17322 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Minor > Attachments: HIVE-17322.01.patch, HIVE-17322.02.patch, > HIVE-17322.03.patch, HIVE-17322.04.patch, HIVE-17322.05.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17316) Use String.contains for the hidden configuration variables
[ https://issues.apache.org/jira/browse/HIVE-17316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17316: --- Attachment: HIVE-17316.02.patch Made some small change. Instead of checking String.contains I would use String.startsWith to reduce the number of accidental parameter restrictions. Also fixed failing unit and q tests. > Use String.contains for the hidden configuration variables > -- > > Key: HIVE-17316 > URL: https://issues.apache.org/jira/browse/HIVE-17316 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17316.01.patch, HIVE-17316.02.patch > > > Currently HiveConf variables which should not be displayed to the user need > to be enumerated. We should enhance this to be able to hide configuration > variables by substring not just full equality. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17322) Serialise BeeLine qtest execution to prevent flakyness
[ https://issues.apache.org/jira/browse/HIVE-17322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17322: --- Attachment: HIVE-17322.04.patch > Serialise BeeLine qtest execution to prevent flakyness > -- > > Key: HIVE-17322 > URL: https://issues.apache.org/jira/browse/HIVE-17322 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Minor > Attachments: HIVE-17322.01.patch, HIVE-17322.02.patch, > HIVE-17322.03.patch, HIVE-17322.04.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17322) Serialise BeeLine qtest execution to prevent flakyness
[ https://issues.apache.org/jira/browse/HIVE-17322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17322: --- Attachment: HIVE-17322.03.patch > Serialise BeeLine qtest execution to prevent flakyness > -- > > Key: HIVE-17322 > URL: https://issues.apache.org/jira/browse/HIVE-17322 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Minor > Attachments: HIVE-17322.01.patch, HIVE-17322.02.patch, > HIVE-17322.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17316) Use String.contains for the hidden configuration variables
[ https://issues.apache.org/jira/browse/HIVE-17316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17316: --- Description: Currently HiveConf variables which should not be displayed to the user need to be enumerated. We should enhance this to be able to hide configuration variables by substring not just full equality. (was: Currently HiveConf variables which should not be displayed to the user need to be enumerated. We should enhance this to be able to set regular expressions and any variable matching it should be hidden.) > Use String.contains for the hidden configuration variables > -- > > Key: HIVE-17316 > URL: https://issues.apache.org/jira/browse/HIVE-17316 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17316.01.patch > > > Currently HiveConf variables which should not be displayed to the user need > to be enumerated. We should enhance this to be able to hide configuration > variables by substring not just full equality. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17316) Use String.contains for the hidden configuration variables
[ https://issues.apache.org/jira/browse/HIVE-17316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17316: --- Summary: Use String.contains for the hidden configuration variables (was: Use regular expressions for the hidden configuration variables) > Use String.contains for the hidden configuration variables > -- > > Key: HIVE-17316 > URL: https://issues.apache.org/jira/browse/HIVE-17316 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17316.01.patch > > > Currently HiveConf variables which should not be displayed to the user need > to be enumerated. We should enhance this to be able to set regular > expressions and any variable matching it should be hidden. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17316) Use regular expressions for the hidden configuration variables
[ https://issues.apache.org/jira/browse/HIVE-17316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127216#comment-16127216 ] Barna Zsombor Klara edited comment on HIVE-17316 at 8/15/17 1:43 PM: - First patch draft created. If we want to support the setting of properties used by the connection pool implementations we should make sure these cannot be set by the user (e.g. connection timeout/idle time should not be set by a user) and probably safer if those cannot be viewed by the user either. Even if viewing is not that dangerous, the connection pool properties should be less useful/meaningful to an average Hive user than standard hive properties. was (Author: zsombor.klara): First patch draft created. If we want to support the setting of properties used by the connection pool implementations we should make sure these cannot be set by the user (pool size, connection timeout should not be set) and probably safer if those cannot be viewed by the user either. Even if viewing is not that dangerous, the connection pool properties should be less useful/meaningful to an average Hive user than standard hive properties. > Use regular expressions for the hidden configuration variables > -- > > Key: HIVE-17316 > URL: https://issues.apache.org/jira/browse/HIVE-17316 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17316.01.patch > > > Currently HiveConf variables which should not be displayed to the user need > to be enumerated. We should enhance this to be able to set regular > expressions and any variable matching it should be hidden. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17316) Use regular expressions for the hidden configuration variables
[ https://issues.apache.org/jira/browse/HIVE-17316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127216#comment-16127216 ] Barna Zsombor Klara commented on HIVE-17316: First patch draft created. If we want to support the setting of properties used by the connection pool implementations we should make sure these cannot be set by the user (pool size, connection timeout should not be set) and probably safer if those cannot be viewed by the user either. Even if viewing is not that dangerous, the connection pool properties should be less useful/meaningful to an average Hive user than standard hive properties. > Use regular expressions for the hidden configuration variables > -- > > Key: HIVE-17316 > URL: https://issues.apache.org/jira/browse/HIVE-17316 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17316.01.patch > > > Currently HiveConf variables which should not be displayed to the user need > to be enumerated. We should enhance this to be able to set regular > expressions and any variable matching it should be hidden. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17316) Use regular expressions for the hidden configuration variables
[ https://issues.apache.org/jira/browse/HIVE-17316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17316: --- Status: Patch Available (was: In Progress) > Use regular expressions for the hidden configuration variables > -- > > Key: HIVE-17316 > URL: https://issues.apache.org/jira/browse/HIVE-17316 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17316.01.patch > > > Currently HiveConf variables which should not be displayed to the user need > to be enumerated. We should enhance this to be able to set regular > expressions and any variable matching it should be hidden. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17316) Use regular expressions for the hidden configuration variables
[ https://issues.apache.org/jira/browse/HIVE-17316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17316: --- Attachment: HIVE-17316.01.patch > Use regular expressions for the hidden configuration variables > -- > > Key: HIVE-17316 > URL: https://issues.apache.org/jira/browse/HIVE-17316 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17316.01.patch > > > Currently HiveConf variables which should not be displayed to the user need > to be enumerated. We should enhance this to be able to set regular > expressions and any variable matching it should be hidden. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17322) Serialise BeeLine qtest execution to prevent flakyness
[ https://issues.apache.org/jira/browse/HIVE-17322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17322: --- Attachment: HIVE-17322.02.patch > Serialise BeeLine qtest execution to prevent flakyness > -- > > Key: HIVE-17322 > URL: https://issues.apache.org/jira/browse/HIVE-17322 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Minor > Attachments: HIVE-17322.01.patch, HIVE-17322.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17322) Serialise BeeLine qtest execution to prevent flakyness
[ https://issues.apache.org/jira/browse/HIVE-17322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17322: --- Status: Patch Available (was: Open) > Serialise BeeLine qtest execution to prevent flakyness > -- > > Key: HIVE-17322 > URL: https://issues.apache.org/jira/browse/HIVE-17322 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Minor > Attachments: HIVE-17322.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17322) Serialise BeeLine qtest execution to prevent flakyness
[ https://issues.apache.org/jira/browse/HIVE-17322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17322: --- Attachment: HIVE-17322.01.patch > Serialise BeeLine qtest execution to prevent flakyness > -- > > Key: HIVE-17322 > URL: https://issues.apache.org/jira/browse/HIVE-17322 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Minor > Attachments: HIVE-17322.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17322) Serialise BeeLine qtest execution to prevent flakyness
[ https://issues.apache.org/jira/browse/HIVE-17322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17322: --- Summary: Serialise BeeLine qtest execution to prevent flakyness (was: Execute BeeLine qtests in a serial manner to prevent flakyness) > Serialise BeeLine qtest execution to prevent flakyness > -- > > Key: HIVE-17322 > URL: https://issues.apache.org/jira/browse/HIVE-17322 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Minor > Attachments: HIVE-17322.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17322) Execute BeeLine qtests in a serial manner to prevent flakyness
[ https://issues.apache.org/jira/browse/HIVE-17322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara reassigned HIVE-17322: -- > Execute BeeLine qtests in a serial manner to prevent flakyness > -- > > Key: HIVE-17322 > URL: https://issues.apache.org/jira/browse/HIVE-17322 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17315) Make the DataSource used by the DataNucleus in the HMS configurable using Hive properties
[ https://issues.apache.org/jira/browse/HIVE-17315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126381#comment-16126381 ] Barna Zsombor Klara commented on HIVE-17315: [~thejas] Yes, you are correct. We currently only set the connection pool size property on the datasource, but at least for BoneCp I think (though I haven't tested it myself) that other BoneCp specific properties can also be set by adding a bonecp-config.xml to the classpath. Look at the constructors for the config object for reference: [BoneCPConfig|http://grepcode.com/file/repo1.maven.org/maven2/com.jolbox/bonecp/0.7.1.RELEASE/com/jolbox/bonecp/BoneCPConfig.java#BoneCPConfig.%3Cinit%3E%28%29] I think that if the xml configuration file is present it will be loaded by BoneCp, so it also could be loaded by the BoneCp datasource backing the DataNucleus PersistenceManagerFactory. HikariConfig is using a java system property "hikaricp.configurationFile" that can point to a properties file with default values, and I don't know about dbcp, but I assume they also have something like this. I'm not sure if any of these "backdoor configurations" are used by anyone, but it should be possible. And yes you are also correct about my intention. I would like to be able to set any property supported by bonecp (or hikari, or dbcp) on the underlying datasource not just the connectionpool size. And while you are right that password was a bad example I would still like to have these properties hidden. Even if the current properties are safe, I would be afraid of what could be added by future versions of the connection pool implementations, since we would be exposing these settings the moment we upgrade the library. I don't want to add every property explicitly to HiveConf, I would like to propagate anything prefixed with bonecp/hikari/dbcp to the DataSource implementation and let it use/ignore it. [~ekoifman] Exactly. I want to be able to set bonecp/hikari/dbcp properties on the datasource. For example in TxnHandler the getConnectionTimeoutMs for every implementation and partition count for bonecp is hardcoded. We should be able to set these values in the hive-site.xml with a property like bonecp.partitionCount or hikari.getConnectionTimeoutMs. > Make the DataSource used by the DataNucleus in the HMS configurable using > Hive properties > - > > Key: HIVE-17315 > URL: https://issues.apache.org/jira/browse/HIVE-17315 > Project: Hive > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > > Currently we may use several connection pool implementations in the backend > (hikari, dbCp, boneCp) but these can only be configured using proprietary xml > files and not through hive-site.xml like DataNucleus. > We should make them configurable just like DataNucleus, by allowing Hive > properties prefix by hikari, dbcp, bonecp to be set in the hive-site.xml. > However since these configurations may contain sensitive information > (passwords) these properties should not be displayable or manually settable. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Work started] (HIVE-17316) Use regular expressions for the hidden configuration variables
[ https://issues.apache.org/jira/browse/HIVE-17316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-17316 started by Barna Zsombor Klara. -- > Use regular expressions for the hidden configuration variables > -- > > Key: HIVE-17316 > URL: https://issues.apache.org/jira/browse/HIVE-17316 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > > Currently HiveConf variables which should not be displayed to the user need > to be enumerated. We should enhance this to be able to set regular > expressions and any variable matching it should be hidden. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17316) Use regular expressions for the hidden configuration variables
[ https://issues.apache.org/jira/browse/HIVE-17316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara reassigned HIVE-17316: -- > Use regular expressions for the hidden configuration variables > -- > > Key: HIVE-17316 > URL: https://issues.apache.org/jira/browse/HIVE-17316 > Project: Hive > Issue Type: Sub-task >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > > Currently HiveConf variables which should not be displayed to the user need > to be enumerated. We should enhance this to be able to set regular > expressions and any variable matching it should be hidden. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17315) Make the DataSource used by the DataNucleus in the HMS configurable using Hive properties
[ https://issues.apache.org/jira/browse/HIVE-17315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara reassigned HIVE-17315: -- > Make the DataSource used by the DataNucleus in the HMS configurable using > Hive properties > - > > Key: HIVE-17315 > URL: https://issues.apache.org/jira/browse/HIVE-17315 > Project: Hive > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > > Currently we may use several connection pool implementations in the backend > (hikari, dbCp, boneCp) but these can only be configured using proprietary xml > files and not through hive-site.xml like DataNucleus. > We should make them configurable just like DataNucleus, by allowing Hive > properties prefix by hikari, dbcp, bonecp to be set in the hive-site.xml. > However since these configurations may contain sensitive information > (passwords) these properties should not be displayable or manually settable. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17305) New insert overwrite dynamic partitions qtest need to have the golden file regenerated
[ https://issues.apache.org/jira/browse/HIVE-17305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17305: --- Status: Patch Available (was: Open) > New insert overwrite dynamic partitions qtest need to have the golden file > regenerated > -- > > Key: HIVE-17305 > URL: https://issues.apache.org/jira/browse/HIVE-17305 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Trivial > Attachments: HIVE-17305.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17305) New insert overwrite dynamic partitions qtest need to have the golden file regenerated
[ https://issues.apache.org/jira/browse/HIVE-17305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124522#comment-16124522 ] Barna Zsombor Klara commented on HIVE-17305: [~pvary] could you please review? > New insert overwrite dynamic partitions qtest need to have the golden file > regenerated > -- > > Key: HIVE-17305 > URL: https://issues.apache.org/jira/browse/HIVE-17305 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Trivial > Attachments: HIVE-17305.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17305) New insert overwrite dynamic partitions qtest need to have the golden file regenerated
[ https://issues.apache.org/jira/browse/HIVE-17305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124521#comment-16124521 ] Barna Zsombor Klara commented on HIVE-17305: Only one qtest had to have its golden file regenerated. The other two were failing because of a regression. LineageState should retain information on paths even after optimisation or already captured lineage information will not be displayed. > New insert overwrite dynamic partitions qtest need to have the golden file > regenerated > -- > > Key: HIVE-17305 > URL: https://issues.apache.org/jira/browse/HIVE-17305 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Trivial > Attachments: HIVE-17305.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17305) New insert overwrite dynamic partitions qtest need to have the golden file regenerated
[ https://issues.apache.org/jira/browse/HIVE-17305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17305: --- Attachment: HIVE-17305.01.patch > New insert overwrite dynamic partitions qtest need to have the golden file > regenerated > -- > > Key: HIVE-17305 > URL: https://issues.apache.org/jira/browse/HIVE-17305 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Trivial > Attachments: HIVE-17305.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17267) Make HMS Notification Listeners typesafe
[ https://issues.apache.org/jira/browse/HIVE-17267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124491#comment-16124491 ] Barna Zsombor Klara commented on HIVE-17267: Unit test failures should not be related. > Make HMS Notification Listeners typesafe > > > Key: HIVE-17267 > URL: https://issues.apache.org/jira/browse/HIVE-17267 > Project: Hive > Issue Type: Bug >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17267.01.patch, HIVE-17267.02.patch, > HIVE-17267.03.patch > > > Currently in the HMS we support two types of notification listeners, > transactional and non-transactional ones. Transactional listeners will only > be invoked if the jdbc transaction finished successfully while > non-transactional ones are supposed to be resilient and will be invoked in > any case, even for failures. > Having the same type for these two is a source of confusion and opens the > door for misconfigurations. We should try to fix this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17305) New insert overwrite dynamic partitions qtest need to have the golden file regenerated
[ https://issues.apache.org/jira/browse/HIVE-17305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara reassigned HIVE-17305: -- > New insert overwrite dynamic partitions qtest need to have the golden file > regenerated > -- > > Key: HIVE-17305 > URL: https://issues.apache.org/jira/browse/HIVE-17305 > Project: Hive > Issue Type: Bug > Components: Tests >Affects Versions: 3.0.0 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Trivial > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-14746) Remove branch from profiles by sending them from ptest-client
[ https://issues.apache.org/jira/browse/HIVE-14746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-14746: --- Attachment: HIVE-14746.03.patch > Remove branch from profiles by sending them from ptest-client > - > > Key: HIVE-14746 > URL: https://issues.apache.org/jira/browse/HIVE-14746 > Project: Hive > Issue Type: Sub-task > Components: Hive, Testing Infrastructure >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-14746.01.patch, HIVE-14746.02.patch, > HIVE-14746.03.patch > > > Hive ptest uses some properties files per branch that contain information > about how to execute the tests. > This profile includes the branch name used to fetch the branch code. We > should get rid of this by detecting the branch from the > jenkins-execute-build.sh script, and send the information directly to > ptest-server as command line parameters. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-14747) Remove JAVA paths from profiles by sending them from ptest-client
[ https://issues.apache.org/jira/browse/HIVE-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123226#comment-16123226 ] Barna Zsombor Klara commented on HIVE-14747: [~spena] the last comment was generated by an instance of PTest which contained this patch. I ran it to validate my change. Do you have any other comments or suggestions? > Remove JAVA paths from profiles by sending them from ptest-client > - > > Key: HIVE-14747 > URL: https://issues.apache.org/jira/browse/HIVE-14747 > Project: Hive > Issue Type: Sub-task > Components: Hive, Testing Infrastructure >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-14747.01.patch, HIVE-14747.02.patch, > HIVE-14747.03.patch, HIVE-14747.04.patch, HIVE-14747.05.patch > > > Hive ptest uses some properties files per branch that contain information > about how to execute the tests. > This profile includes JAVA paths to build and execute the tests. We should > get rid of these by passing such information from Jenkins to the > ptest-server. In case a profile needs a different java version, then we can > create a specific Jenkins job for that. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17267) Make HMS Notification Listeners typesafe
[ https://issues.apache.org/jira/browse/HIVE-17267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17267: --- Attachment: HIVE-17267.03.patch > Make HMS Notification Listeners typesafe > > > Key: HIVE-17267 > URL: https://issues.apache.org/jira/browse/HIVE-17267 > Project: Hive > Issue Type: Bug >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17267.01.patch, HIVE-17267.02.patch, > HIVE-17267.03.patch > > > Currently in the HMS we support two types of notification listeners, > transactional and non-transactional ones. Transactional listeners will only > be invoked if the jdbc transaction finished successfully while > non-transactional ones are supposed to be resilient and will be invoked in > any case, even for failures. > Having the same type for these two is a source of confusion and opens the > door for misconfigurations. We should try to fix this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-14747) Remove JAVA paths from profiles by sending them from ptest-client
[ https://issues.apache.org/jira/browse/HIVE-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-14747: --- Attachment: HIVE-14747.05.patch Renamed new constant string to conform to existing naming standards. > Remove JAVA paths from profiles by sending them from ptest-client > - > > Key: HIVE-14747 > URL: https://issues.apache.org/jira/browse/HIVE-14747 > Project: Hive > Issue Type: Sub-task > Components: Hive, Testing Infrastructure >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-14747.01.patch, HIVE-14747.02.patch, > HIVE-14747.03.patch, HIVE-14747.04.patch, HIVE-14747.05.patch > > > Hive ptest uses some properties files per branch that contain information > about how to execute the tests. > This profile includes JAVA paths to build and execute the tests. We should > get rid of these by passing such information from Jenkins to the > ptest-server. In case a profile needs a different java version, then we can > create a specific Jenkins job for that. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17267) Make HMS Notification Listeners typesafe
[ https://issues.apache.org/jira/browse/HIVE-17267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-17267: --- Attachment: HIVE-17267.02.patch Addressed review board comments. > Make HMS Notification Listeners typesafe > > > Key: HIVE-17267 > URL: https://issues.apache.org/jira/browse/HIVE-17267 > Project: Hive > Issue Type: Bug >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Attachments: HIVE-17267.01.patch, HIVE-17267.02.patch > > > Currently in the HMS we support two types of notification listeners, > transactional and non-transactional ones. Transactional listeners will only > be invoked if the jdbc transaction finished successfully while > non-transactional ones are supposed to be resilient and will be invoked in > any case, even for failures. > Having the same type for these two is a source of confusion and opens the > door for misconfigurations. We should try to fix this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-14747) Remove JAVA paths from profiles by sending them from ptest-client
[ https://issues.apache.org/jira/browse/HIVE-14747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Barna Zsombor Klara updated HIVE-14747: --- Attachment: HIVE-14747.04.patch > Remove JAVA paths from profiles by sending them from ptest-client > - > > Key: HIVE-14747 > URL: https://issues.apache.org/jira/browse/HIVE-14747 > Project: Hive > Issue Type: Sub-task > Components: Hive, Testing Infrastructure >Reporter: Sergio Peña >Assignee: Barna Zsombor Klara > Attachments: HIVE-14747.01.patch, HIVE-14747.02.patch, > HIVE-14747.03.patch, HIVE-14747.04.patch > > > Hive ptest uses some properties files per branch that contain information > about how to execute the tests. > This profile includes JAVA paths to build and execute the tests. We should > get rid of these by passing such information from Jenkins to the > ptest-server. In case a profile needs a different java version, then we can > create a specific Jenkins job for that. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
