[jira] [Commented] (HIVE-15631) Optimize for hive client logs , you can filter the log for each session itself.
[ https://issues.apache.org/jira/browse/HIVE-15631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333979#comment-16333979 ] tartarus commented on HIVE-15631: - [~prasanth_j] I can create a new pull request to add it, print sessionid throuth console. > Optimize for hive client logs , you can filter the log for each session > itself. > --- > > Key: HIVE-15631 > URL: https://issues.apache.org/jira/browse/HIVE-15631 > Project: Hive > Issue Type: Improvement > Components: CLI, Clients, Hive >Reporter: tartarus >Assignee: tartarus >Priority: Major > Attachments: HIVE_15631.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > We have several hadoop cluster, about 15 thousand nodes. Every day we use > hive to submit above 100 thousand jobs. > So we have a large file of hive logs on every client host every day, but i > don not know the logs of my session submitted was which line. > So i hope to print the hive.session.id on every line of logs, and then i > could use grep to find the logs of my session submitted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15631) Optimize for hive client logs , you can filter the log for each session itself.
[ https://issues.apache.org/jira/browse/HIVE-15631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333977#comment-16333977 ] tartarus commented on HIVE-15631: - [~prasanth_j] I looked at the new version of the code, sessionid is set to ThreadName, so you can print in the log. What I hope is that sessionid can be printed through the console when starting up the Hive Client , so that I can associate with the backend's logs. We are in the production environment through the scheduling system to schedule Hive tasks, scheduling system can only get the console log. > Optimize for hive client logs , you can filter the log for each session > itself. > --- > > Key: HIVE-15631 > URL: https://issues.apache.org/jira/browse/HIVE-15631 > Project: Hive > Issue Type: Improvement > Components: CLI, Clients, Hive >Reporter: tartarus >Assignee: tartarus >Priority: Major > Attachments: HIVE_15631.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > We have several hadoop cluster, about 15 thousand nodes. Every day we use > hive to submit above 100 thousand jobs. > So we have a large file of hive logs on every client host every day, but i > don not know the logs of my session submitted was which line. > So i hope to print the hive.session.id on every line of logs, and then i > could use grep to find the logs of my session submitted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18051) qfiles: dataset support
[
https://issues.apache.org/jira/browse/HIVE-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333965#comment-16333965
]
Zoltan Haindrich commented on HIVE-18051:
-
I've gone thru the changes, so here are my comments :)
* {{test.src.tables}} has to be set to the the joined version of {{srcTables}}
to enable the table protector ; please add 2 negative tests to check that
dropping a dataset fails; if its not there; then the table "protector" will not
prevent changes to it (EnforceReadOnlyTables.java)
* I now think that it would be better to enable it for all the q tests; at
least parse the files all the time and look for the dataset pattern; it's too
defensive this way - there won't be any problems :)
* why is the sample dataset setting hive.stats.dbclass to fs? it was like
that? we should probably set it in hiveconf.java / hive-site.xml ; doesn't seem
to be something which belong to the dataset itself...
other minor notices:
* data/files/kv1.txt has the same content as data/files/testdataset.txt
* I feel that we shouldn't be loading all datasets upfront; just load what's
needed to run the actual test...that could enable to remove the dataset from
the cliconfig interface ; and move it to somewhere when the test is being
executed; in the current design it would only work for the {{CoreCliDriver}}
family tests - this probably come handy later; last week I've created a junit
rule which is able to run driver tests from ide; incorporating into that later
might make it more convinient
* 1 dataset = 1 table ; I think the Dataset interface should show the
contract; for now I don't think an interface is neccessary, because theres only
1 implementation
it's great that you are working on this; I'm looking forward to start using it!
> qfiles: dataset support
> ---
>
> Key: HIVE-18051
> URL: https://issues.apache.org/jira/browse/HIVE-18051
> Project: Hive
> Issue Type: Improvement
> Components: Testing Infrastructure
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18051.01.patch, HIVE-18051.02.patch,
> HIVE-18051.03.patch, HIVE-18051.04.patch, HIVE-18051.05.patch,
> HIVE-18051.06.patch
>
>
> it would be great to have some kind of test dataset support; currently there
> is the {{q_test_init.sql}} which is quite large; and I'm often override it
> with an invalid string; because I write independent qtests most of the time -
> and the load of {{src}} and other tables are just a waste of time for me ;
> not to mention that the loading of those tables may also trigger breakpoints
> - which is a bit annoying.
> Most of the tests are "only" using the {{src}} table and possibly 2 others;
> however the main init script contains a bunch of tables - meanwhile there are
> quite few other tests which could possibly also benefit from a more general
> feature; for example the creation of {{bucket_small}} is present in 20 q
> files.
> the proposal would be to enable the qfiles to be annotated with metadata like
> datasets:
> {code}
> --! qt:dataset:src,bucket_small
> {code}
> proposal for storing a dataset:
> * the loader script would be at: {{data/datasets/__NAME__/load.hive.sql}}
> * the table data could be stored under that location
> a draft about this; and other qfiles related ideas:
> https://docs.google.com/document/d/1KtcIx8ggL9LxDintFuJo8NQuvNWkmtvv_ekbWrTLNGc/edit?usp=sharing
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18192) Introduce WriteID per table rather than using global transaction ID
[ https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18192: Attachment: HIVE-18192.04.patch > Introduce WriteID per table rather than using global transaction ID > --- > > Key: HIVE-18192 > URL: https://issues.apache.org/jira/browse/HIVE-18192 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18192.01.patch, HIVE-18192.02.patch, > HIVE-18192.03.patch, HIVE-18192.04.patch > > > To support ACID replication, we will be introducing a per table write Id > which will replace the transaction id in the primary key for each row in a > ACID table. > The current primary key is determined via > > which will move to > > For each table modified by the given transaction will have a table level > write ID allocated and a persisted map of global txn id -> to table -> write > id for that table has to be maintained to allow Snapshot isolation. > Readers should use the combination of ValidTxnList and > ValidWriteIdList(Table) for snapshot isolation. > > [Hive Replication - ACID > Tables.pdf|https://issues.apache.org/jira/secure/attachment/12903157/Hive%20Replication-%20ACID%20Tables.pdf] > has a section "Per Table Sequences (Write-Id)" with more detials -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18192) Introduce WriteID per table rather than using global transaction ID
[ https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18192: Status: Patch Available (was: Open) > Introduce WriteID per table rather than using global transaction ID > --- > > Key: HIVE-18192 > URL: https://issues.apache.org/jira/browse/HIVE-18192 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18192.01.patch, HIVE-18192.02.patch, > HIVE-18192.03.patch, HIVE-18192.04.patch > > > To support ACID replication, we will be introducing a per table write Id > which will replace the transaction id in the primary key for each row in a > ACID table. > The current primary key is determined via > > which will move to > > For each table modified by the given transaction will have a table level > write ID allocated and a persisted map of global txn id -> to table -> write > id for that table has to be maintained to allow Snapshot isolation. > Readers should use the combination of ValidTxnList and > ValidWriteIdList(Table) for snapshot isolation. > > [Hive Replication - ACID > Tables.pdf|https://issues.apache.org/jira/secure/attachment/12903157/Hive%20Replication-%20ACID%20Tables.pdf] > has a section "Per Table Sequences (Write-Id)" with more detials -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18192) Introduce WriteID per table rather than using global transaction ID
[ https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333960#comment-16333960 ] Sankar Hariappan commented on HIVE-18192: - Attached 04.patch after rebasing with master. > Introduce WriteID per table rather than using global transaction ID > --- > > Key: HIVE-18192 > URL: https://issues.apache.org/jira/browse/HIVE-18192 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18192.01.patch, HIVE-18192.02.patch, > HIVE-18192.03.patch > > > To support ACID replication, we will be introducing a per table write Id > which will replace the transaction id in the primary key for each row in a > ACID table. > The current primary key is determined via > > which will move to > > For each table modified by the given transaction will have a table level > write ID allocated and a persisted map of global txn id -> to table -> write > id for that table has to be maintained to allow Snapshot isolation. > Readers should use the combination of ValidTxnList and > ValidWriteIdList(Table) for snapshot isolation. > > [Hive Replication - ACID > Tables.pdf|https://issues.apache.org/jira/secure/attachment/12903157/Hive%20Replication-%20ACID%20Tables.pdf] > has a section "Per Table Sequences (Write-Id)" with more detials -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde
[
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17580:
---
Attachment: HIVE-17580.04-standalone-metastore.patch
> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch,
> HIVE-17580.04-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}}
> class to access the fields metadata for the cases where it is stored along
> with the data files (avro tables). The problem is Deserializer classes is
> defined in hive-serde module and in order to make metastore independent of
> Hive we will have to remove this dependency (atleast we should change it to
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this
> functionality.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18192) Introduce WriteID per table rather than using global transaction ID
[ https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18192: Status: Open (was: Patch Available) > Introduce WriteID per table rather than using global transaction ID > --- > > Key: HIVE-18192 > URL: https://issues.apache.org/jira/browse/HIVE-18192 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18192.01.patch, HIVE-18192.02.patch, > HIVE-18192.03.patch > > > To support ACID replication, we will be introducing a per table write Id > which will replace the transaction id in the primary key for each row in a > ACID table. > The current primary key is determined via > > which will move to > > For each table modified by the given transaction will have a table level > write ID allocated and a persisted map of global txn id -> to table -> write > id for that table has to be maintained to allow Snapshot isolation. > Readers should use the combination of ValidTxnList and > ValidWriteIdList(Table) for snapshot isolation. > > [Hive Replication - ACID > Tables.pdf|https://issues.apache.org/jira/secure/attachment/12903157/Hive%20Replication-%20ACID%20Tables.pdf] > has a section "Per Table Sequences (Write-Id)" with more detials -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde
[ https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333955#comment-16333955 ] ASF GitHub Bot commented on HIVE-17580: --- GitHub user vihangk1 opened a pull request: https://github.com/apache/hive/pull/294 HIVE-17580 Remove dependency of get_fields_with_environment_context API to serde This is an alternative approach to the solve the dependencies with serdes for get_fields HMS API. The earlier attempt for HIVE-17580 was very disruptive since it attempted to move TypeInfo, and various Type implementations to storage-api and also created another module called serde-api. This patch is a lot more cleaner and less disruptive. Instead of moving TypeInfo, it creates similar classes in standalone-metastore. The PR is broken into multiple commits with descriptive commit messages. You can merge this pull request into a Git repository by running: $ git pull https://github.com/vihangk1/hive vihangk1_HIVE-17580v2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hive/pull/294.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #294 commit 708443af3f6356ab73133e271cf00e3418ced8ef Author: Vihang Karajgaonkar Date: 2018-01-21T23:54:04Z Added MetastoreTypeInfo similar to TypeInfo This patch adds classes similar to TypeInfo called MetastoreTypeInfo in standalone-metastore. Ideally, we should move TypeInfo to standalone-metastore since they store the information about types. However, moving TypeInfo to standalone-metastore is non-trivial effort primarily because of the below reasons: 1. TypeInfo is annotated as Public API. This means we can only alter/move these classes in a compatible way. 2. Directly moving these classes is not straight-forward because TypeInfo uses PrimitiveEntry class which internally maps the TypeInfo to Type implementations. Ideally metastore should not use Type implementation which makes it harder to move the TypeInfo directly. However, if we are ready to break compatibility, then TypeInfo broken such that it doesn't use PrimitiveEntry directly. In such a world TypeInfo will store just what it needs to store. Metadata of Types i.e the type category, its qualified name, whether its a parameterized type or not and if yes, how do we validate the parameters. I am assuming that breaking TypeInfo is a no-go and hence I am copying the relevant code from TypeInfo to Metastore and calling it MetastoreTypeInfo. MetastoreTypeInfo and its sub-classes are used by TypeInfoParser (also copied) to parse the column type strings into TypeInfos. commit 6ec0efa59408c355cfa9aec7fd9dd59d3545aff2 Author: Vihang Karajgaonkar Date: 2018-01-03T19:45:32Z Add avro storeage schema reader This commit adds a AvroStorageSchemaReader which reads the Avro schema files both for external schema and regular avro tables. Most of the util methods are in AvroSchemaUtils class which has methods copied from AvroSerDeUtils. Some of the needed classes like SchemaResolutionProblem, InstanceCache, SchemaToTypeInfo, TypeInfoToSchema are also copied from Hive. The constants defined in AvroSerde are copied in AvroSerdeConstants. The class AvroFieldSchemaGenerator converts the AvroSchema into List of FieldSchema which is returned by the AvroStorageSchemaReader Avro schema reader uses MetastoreTypeInfo and MetastoreTypeInfoParser introduced earlier commit b0f6d1df1ddb627e0f3c1cff3a164c9397337be0 Author: Vihang Karajgaonkar Date: 2018-01-04T01:02:40Z Introduce default storage schema reader This change introduces a default storage schema reader which copies the common code from serdes initialization method and uses it to parse the column name, type and comments from the table properties. For custom storage schema reades like Avro we will have to add more schema readers as and when required commit 5ae977a0bf3fd54389671bed86322d3d4652bc20 Author: Vihang Karajgaonkar Date: 2018-01-04T19:18:03Z Integrates the avro schema reader into the DefaultStorageaSchemaReader commit 2074b16e12c1bdc7ef3781f50e01ab4dd4c71890 Author: Vihang Karajgaonkar Date: 2018-01-05T02:38:28Z Added a test for getFields method in standalone-metastore commit 4159b5ee9852b41a64489274040e79dbddad54f1 Author: Vihang Karajgaonkar Date: 2018-01-22T07:16:13Z HIVE-18508 : Port schema changes from HIVE-14498 to standalone-metastore > Remove dependency of get_fields_with_environment_context API to serde > - > > Key: HIVE-17580 > URL: https://issues.apache.org/
[jira] [Commented] (HIVE-18489) Automatically migration s3n URIs to s3a URIs
[
https://issues.apache.org/jira/browse/HIVE-18489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333948#comment-16333948
]
Hive QA commented on HIVE-18489:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12907043/HIVE-18489.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 11630 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_rollup_empty]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_literals]
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby_rollup_empty]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_authorization_create_no_grant]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_1]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=121)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=254)
org.apache.hive.hcatalog.common.TestHiveClientCache.testCloseAllClients
(batchId=198)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=232)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=232)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8746/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8746/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8746/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 21 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12907043 - PreCommit-HIVE-Build
> Automatically migration s3n URIs to s3a URIs
>
>
> Key: HIVE-18489
> URL: https://issues.apache.org/jira/browse/HIVE-18489
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18489.1.patch, HIVE-18489.2.patch,
> HIVE-18489.3.patch, HIVE-18489.4.patch
>
>
> {{s3n}} has been removed from Hadoop 3.x, we should auto-migrate tables with
> {{s3n}} URIs to the {{s3a}} URIs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18508) Port schema changes from HIVE-14498 to standalone-metastore
[ https://issues.apache.org/jira/browse/HIVE-18508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar reassigned HIVE-18508: -- > Port schema changes from HIVE-14498 to standalone-metastore > --- > > Key: HIVE-18508 > URL: https://issues.apache.org/jira/browse/HIVE-18508 > Project: Hive > Issue Type: Task > Components: Standalone Metastore >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Critical > > HIVE-14498 introduced a number of schema changes which are missing from the > standalone-metastore *.sql files. Due to this queries are erroring out using > standalone-metastore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18489) Automatically migration s3n URIs to s3a URIs
[
https://issues.apache.org/jira/browse/HIVE-18489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333907#comment-16333907
]
Hive QA commented on HIVE-18489:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 0m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a59cb88 |
| modules | C: metastore U: metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8746/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Automatically migration s3n URIs to s3a URIs
>
>
> Key: HIVE-18489
> URL: https://issues.apache.org/jira/browse/HIVE-18489
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18489.1.patch, HIVE-18489.2.patch,
> HIVE-18489.3.patch, HIVE-18489.4.patch
>
>
> {{s3n}} has been removed from Hadoop 3.x, we should auto-migrate tables with
> {{s3n}} URIs to the {{s3a}} URIs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18373) Make it easier to search for column name in a table
[
https://issues.apache.org/jira/browse/HIVE-18373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16331785#comment-16331785
]
Madhudeep Petwal edited comment on HIVE-18373 at 1/22/18 6:11 AM:
--
Ok I looked up Hive.g grammar file. It looks like hive supports both
{code:java}
hive> show tables '*abc*';{code}
{code:java}
hive> show tables like '*abc*';{code}
as we can see grammar rule for show tables
{code:java}
KW_SHOW KW_TABLES ((KW_FROM|KW_IN) db_name=identifier)? (KW_LIKE
showStmtIdentifier|showStmtIdentifier)? -> ^(TOK_SHOWTABLES (TOK_FROM
$db_name)? showStmtIdentifier?){code}
I will keep the syntax similar to show tables then.
This is my proposed grammar rule for show columns. Please review [~thejas]
{code:java}
KW_SHOW KW_COLUMNS (KW_FROM|KW_IN) tableName ((KW_FROM|KW_IN)
db_name=identifier)? (KW_LIKE showStmtIdentifier|showStmtIdentifier)?
-> ^(TOK_SHOWCOLUMNS tableName (TOK_FROM $db_name)? showStmtIdentifier?){code}
was (Author: minions):
Ok I looked up Hive.g grammar file. It looks like hive supports both
{code:java}
hive> show tables '*abc*';{code}
{code:java}
hive> show tables like '*abc*';{code}
as we can see grammar rule for show tables
{code:java}
KW_SHOW KW_TABLES ((KW_FROM|KW_IN) db_name=identifier)? (KW_LIKE
showStmtIdentifier|showStmtIdentifier)? -> ^(TOK_SHOWTABLES (TOK_FROM
$db_name)? showStmtIdentifier?){code}
I will keep the syntax similar to show tables then.
This is my proposed grammar rule for show columns. Please review [~thejas]
{code:java}
KW_SHOW KW_COLUMNS (KW_FROM|KW_IN) tableName ((KW_FROM|KW_IN)
db_name=identifier)? (KW_LIKE showStmtIdentifier|showStmtIdentifier)?
-> ^(TOK_SHOWCOLUMNS tableName $db_name? showStmtIdentifier?){code}
> Make it easier to search for column name in a table
> ---
>
> Key: HIVE-18373
> URL: https://issues.apache.org/jira/browse/HIVE-18373
> Project: Hive
> Issue Type: New Feature
>Reporter: Siddhant Saraf
>Assignee: Madhudeep Petwal
>Priority: Minor
>
> Within a database, to filter for tables with the string 'abc' in its name, I
> can use something like:
> {code:java}
> hive> use my_database;
> hive> show tables '*abc*';
> {code}
> It would be great if I can do something similar to search within the list of
> columns in a table.
> I have a table with around 3200 columns. Searching for the column of interest
> is an onerous task after doing a {{describe}} on it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18504) Hive is throwing InvalidObjectException(message:Invalid column type name is too long.
[
https://issues.apache.org/jira/browse/HIVE-18504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimson K James updated HIVE-18504:
--
Description:
Hive 2.6.3 is still throwing InvalidObjectException(message:Invalid column type
name is too long.
Please find attached the create table query. For more details please refer to
HIVE-15249
{code:java}
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. InvalidObjectException(message:Invalid
column type name length 2980 exceeds max allowed length 2000, type
struct,entities:struct,text:string>>,symbols:array...
{code}
{code:java}
[root@sandbox-hdp hive-json]# hive --version
Hive 1.2.1000.2.6.3.0-235
Subversion
git://ctr-e134-1499953498516-254436-01-04.hwx.site/grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive
-r 5f360bda08bb5489fbb3189b5aeaaf58029ed4b5
Compiled by jenkins on Mon Oct 30 02:48:31 UTC 2017
>From source with checksum 94298cc1f5f5bf0f3470f3ea2e92d646
[root@sandbox-hdp hive-json]# beeline
Beeline version 1.2.1000.2.6.3.0-235 by Apache Hive
beeline> !connect
jdbc:hive2://sandbox-hdp.hortonworks.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
Connecting to
jdbc:hive2://sandbox-hdp.hortonworks.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
Enter username for
jdbc:hive2://sandbox-hdp.hortonworks.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2:
hive
Enter password for
jdbc:hive2://sandbox-hdp.hortonworks.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2:
Connected to: Apache Hive (version 1.2.1000.2.6.3.0-235)
Driver: Hive JDBC (version 1.2.1000.2.6.3.0-235)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://sandbox-hdp.hortonworks.com:2>
{code}
was:
Hive 2.1.0 is still throwing InvalidObjectException(message:Invalid column type
name is too long.
Please find attached the create table query. For more details please refer to
HIVE-15249
{code:java}
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. InvalidObjectException(message:Invalid
column type name length 2980 exceeds max allowed length 2000, type
struct,entities:struct,text:string>>,symbols:array...
{code}
{code:java}
[root@sandbox-hdp hive-json]# hive --version
Hive 1.2.1000.2.6.3.0-235
Subversion
git://ctr-e134-1499953498516-254436-01-04.hwx.site/grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive
-r 5f360bda08bb5489fbb3189b5aeaaf58029ed4b5
Compiled by jenkins on Mon Oct 30 02:48:31 UTC 2017
>From source with checksum 94298cc1f5f5bf0f3470f3ea2e92d646
[root@sandbox-hdp hive-json]# beeline
Beeline version 1.2.1000.2.6.3.0-235 by Apache Hive
beeline> !connect
jdbc:hive2://sandbox-hdp.hortonworks.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
Connecting to
jdbc:hive2://sandbox-hdp.hortonworks.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
Enter username for
jdbc:hive2://sandbox-hdp.hortonworks.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2:
hive
Enter password for
jdbc:hive2://sandbox-hdp.hortonworks.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2:
Connected to: Apache Hive (version 1.2.1000.2.6.3.0-235)
Driver: Hive JDBC (version 1.2.1000.2.6.3.0-235)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://sandbox-hdp.hortonworks.com:2>
{code}
> Hive is throwing InvalidObjectException(message:Invalid column type name is
> too long.
> -
>
> Key: HIVE-18504
> URL: https://issues.apache.org/jira/browse/HIVE-18504
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Jimson K James
>Assignee: Naveen Gangam
>Priority: Major
> Fix For: 2.3.0, 3.0.0
>
> Attachments: tweets.sql
>
>
> Hive 2.6.3 is still throwing InvalidObjectException(message:Invalid column
> type name is too long.
> Please find attached the create table query. For more details please refer to
> HIVE-15249
> {code:java}
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask.
> InvalidObjectException(message:Invalid column type name length 2980 exceeds
> max allowed length 2000, type
> struct,entities:struct,text:string>>,symbols:array...
> {code}
>
> {code:java}
> [root@sandbox-hdp hive-json]# hive --version
> Hive 1.2.1000.2.6.3.0-235
> Subversion
> git://ctr-e134-1499953498516-254436-01-04.hwx.site/grid/0/jenkins/workspace/HDP-parallel-centos6/SOURCES/hive
> -r 5f360bda08bb5489fbb3189b5aeaaf58029ed4b5
> Compiled by jenkins on Mon Oct 30 02:48:31 UTC 2017
> From source with checksum 94298cc1f5f5bf0f3470f3ea2e92d646
> [root@sandbox-hdp hive-json]# beeline
> Beeline version 1.
[jira] [Commented] (HIVE-18489) Automatically migration s3n URIs to s3a URIs
[
https://issues.apache.org/jira/browse/HIVE-18489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333896#comment-16333896
]
Sahil Takiar commented on HIVE-18489:
-
[~aihuaxu] yeah its probably safer that way. Attached an updated patch that
matches on {{s3n://%}} so if the string starts with {{s3n://}} it will replace
{{s3n}} with {{s3a}} (I didn't used the SQL REPLACE method intentionally to
avoid replacing any other occurrences of {{s3n}} in the URI; by using
{{SUBSTRING}} and the {{CONCAT}} operator the update only updates the start of
the string).
> Automatically migration s3n URIs to s3a URIs
>
>
> Key: HIVE-18489
> URL: https://issues.apache.org/jira/browse/HIVE-18489
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18489.1.patch, HIVE-18489.2.patch,
> HIVE-18489.3.patch, HIVE-18489.4.patch
>
>
> {{s3n}} has been removed from Hadoop 3.x, we should auto-migrate tables with
> {{s3n}} URIs to the {{s3a}} URIs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18489) Automatically migration s3n URIs to s3a URIs
[
https://issues.apache.org/jira/browse/HIVE-18489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-18489:
Attachment: HIVE-18489.4.patch
> Automatically migration s3n URIs to s3a URIs
>
>
> Key: HIVE-18489
> URL: https://issues.apache.org/jira/browse/HIVE-18489
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18489.1.patch, HIVE-18489.2.patch,
> HIVE-18489.3.patch, HIVE-18489.4.patch
>
>
> {{s3n}} has been removed from Hadoop 3.x, we should auto-migrate tables with
> {{s3n}} URIs to the {{s3a}} URIs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18490) Query with EXISTS and NOT EXISTS with non-equi predicate can produce wrong result
[
https://issues.apache.org/jira/browse/HIVE-18490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-18490:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Query with EXISTS and NOT EXISTS with non-equi predicate can produce wrong
> result
> -
>
> Key: HIVE-18490
> URL: https://issues.apache.org/jira/browse/HIVE-18490
> Project: Hive
> Issue Type: Bug
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Critical
> Attachments: HIVE-18490.1.patch, HIVE-18490.2.patch
>
>
> Queries such as following can produce wrong result
> {code:sql}
> select
>count(ws_order_number)
> from
>web_sales ws1
> where
> and exists (select *
> from web_sales ws2
> where ws1.ws_order_number = ws2.ws_order_number
> and ws1.ws_warehouse_sk <> ws2.ws_warehouse_sk)
> and not exists(select *
>from web_returns wr1
>where ws1.ws_order_number = wr1.wr_order_number);
> {code}
> This query is simplified version of tpcds query 94. Such queries are
> rewritten into LEFT SEMI JOIN and LEFT OUTER JOIN with residual
> predicate/filter (non-equi join key). Problem is that these joins are being
> merged, we shouldn't be merging since semi join has non-equi join filter.
> Basically the underlying issue is that if a query has multiple join with LEFT
> SEMI JOIN with non-equi join key it is being merged with other joins. Merge
> logic should check such cases and avoid merging.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18490) Query with EXISTS and NOT EXISTS with non-equi predicate can produce wrong result
[
https://issues.apache.org/jira/browse/HIVE-18490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333885#comment-16333885
]
Vineet Garg commented on HIVE-18490:
Thanks [~jcamachorodriguez]. Pushed this to mater.
> Query with EXISTS and NOT EXISTS with non-equi predicate can produce wrong
> result
> -
>
> Key: HIVE-18490
> URL: https://issues.apache.org/jira/browse/HIVE-18490
> Project: Hive
> Issue Type: Bug
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Critical
> Attachments: HIVE-18490.1.patch, HIVE-18490.2.patch
>
>
> Queries such as following can produce wrong result
> {code:sql}
> select
>count(ws_order_number)
> from
>web_sales ws1
> where
> and exists (select *
> from web_sales ws2
> where ws1.ws_order_number = ws2.ws_order_number
> and ws1.ws_warehouse_sk <> ws2.ws_warehouse_sk)
> and not exists(select *
>from web_returns wr1
>where ws1.ws_order_number = wr1.wr_order_number);
> {code}
> This query is simplified version of tpcds query 94. Such queries are
> rewritten into LEFT SEMI JOIN and LEFT OUTER JOIN with residual
> predicate/filter (non-equi join key). Problem is that these joins are being
> merged, we shouldn't be merging since semi join has non-equi join filter.
> Basically the underlying issue is that if a query has multiple join with LEFT
> SEMI JOIN with non-equi join key it is being merged with other joins. Merge
> logic should check such cases and avoid merging.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18462) Explain formatted for queries with map join has columnExprMap with unformatted column name
[
https://issues.apache.org/jira/browse/HIVE-18462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333883#comment-16333883
]
Vineet Garg commented on HIVE-18462:
Thanks for pointing it out [~anishek]. I have reverted and re-commited with
appropriate JIRA number.
> Explain formatted for queries with map join has columnExprMap with
> unformatted column name
> --
>
> Key: HIVE-18462
> URL: https://issues.apache.org/jira/browse/HIVE-18462
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-18462.1.patch, HIVE-18462.2.patch
>
>
> e.g.
> {code:sql}
> "columnExprMap:":{
> "_col0":"0:Column[_col0]",
> "_col1":"0:Column[_col1]",
> "_col2":"1:Column[_col0]",
> "_col3":"1:Column[_col1]"
> }
> {code}
> It is better formatted as:
> {code:sql}
> "columnExprMap:":{
> "_col0":"0:_col0",
> "_col1":"0:_col1",
> "_col2":"1:_col0",
> "_col3":"1:_col1"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18489) Automatically migration s3n URIs to s3a URIs
[
https://issues.apache.org/jira/browse/HIVE-18489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333876#comment-16333876
]
Aihua Xu commented on HIVE-18489:
-
[~stakiar] One question: do you think if it's better to replace s3n:// with
s3a:// instead of s3n with s3a?
> Automatically migration s3n URIs to s3a URIs
>
>
> Key: HIVE-18489
> URL: https://issues.apache.org/jira/browse/HIVE-18489
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18489.1.patch, HIVE-18489.2.patch,
> HIVE-18489.3.patch
>
>
> {{s3n}} has been removed from Hadoop 3.x, we should auto-migrate tables with
> {{s3n}} URIs to the {{s3a}} URIs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18462) Explain formatted for queries with map join has columnExprMap with unformatted column name
[
https://issues.apache.org/jira/browse/HIVE-18462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333873#comment-16333873
]
anishek commented on HIVE-18462:
[~vgarg]/[~ashutoshc] looks like the bug number for this is not correct in
apache master shows HIVE-186462, can you please have a look.
> Explain formatted for queries with map join has columnExprMap with
> unformatted column name
> --
>
> Key: HIVE-18462
> URL: https://issues.apache.org/jira/browse/HIVE-18462
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-18462.1.patch, HIVE-18462.2.patch
>
>
> e.g.
> {code:sql}
> "columnExprMap:":{
> "_col0":"0:Column[_col0]",
> "_col1":"0:Column[_col1]",
> "_col2":"1:Column[_col0]",
> "_col3":"1:Column[_col1]"
> }
> {code}
> It is better formatted as:
> {code:sql}
> "columnExprMap:":{
> "_col0":"0:_col0",
> "_col1":"0:_col1",
> "_col2":"1:_col0",
> "_col3":"1:_col1"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-8436) Modify SparkWork to split works with multiple child works [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333870#comment-16333870
]
liyunzhang commented on HIVE-8436:
--
[~csun]:
can you spend some time to explain why need add
[MapInput::CopyFunction|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java#L72]?
the input is Tuple2, the output is
Tuple2, why need to add HadoopRDD->CopyFunction?
{code:java}
private static class CopyFunction implements
PairFunction,
WritableComparable, Writable> {
private transient Configuration conf;
@Override
public Tuple2
call(Tuple2 tuple) throws Exception {
if (conf == null) {
conf = new Configuration();
}
return new Tuple2(tuple._1(),
WritableUtils.clone(tuple._2(), conf));
}
}
{code}
> Modify SparkWork to split works with multiple child works [Spark Branch]
>
>
> Key: HIVE-8436
> URL: https://issues.apache.org/jira/browse/HIVE-8436
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Xuefu Zhang
>Assignee: Chao Sun
>Priority: Major
> Fix For: 1.1.0
>
> Attachments: HIVE-8436.1-spark.patch, HIVE-8436.10-spark.patch,
> HIVE-8436.11-spark.patch, HIVE-8436.2-spark.patch, HIVE-8436.3-spark.patch,
> HIVE-8436.4-spark.patch, HIVE-8436.5-spark.patch, HIVE-8436.6-spark.patch,
> HIVE-8436.7-spark.patch, HIVE-8436.8-spark.patch, HIVE-8436.9-spark.patch
>
>
> Based on the design doc, we need to split the operator tree of a work in
> SparkWork if the work is connected to multiple child works. The way splitting
> the operator tree is performed by cloning the original work and removing
> unwanted branches in the operator tree. Please refer to the design doc for
> details.
> This process should be done right before we generate SparkPlan. We should
> have a utility method that takes the orignal SparkWork and return a modified
> SparkWork.
> This process should also keep the information about the original work and its
> clones. Such information will be needed during SparkPlan generation
> (HIVE-8437).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15631) Optimize for hive client logs , you can filter the log for each session itself.
[ https://issues.apache.org/jira/browse/HIVE-15631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333802#comment-16333802 ] tartarus commented on HIVE-15631: - [~ashutoshc] [~prasanth_j] Thank you for your attention. MDC is a good idea. Users can decide on their own log on a file or divided into multiple files. We also considered the case of a log file per session, so that our files would be very heavy every day. > Optimize for hive client logs , you can filter the log for each session > itself. > --- > > Key: HIVE-15631 > URL: https://issues.apache.org/jira/browse/HIVE-15631 > Project: Hive > Issue Type: Improvement > Components: CLI, Clients, Hive >Reporter: tartarus >Assignee: tartarus >Priority: Major > Attachments: HIVE_15631.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > We have several hadoop cluster, about 15 thousand nodes. Every day we use > hive to submit above 100 thousand jobs. > So we have a large file of hive logs on every client host every day, but i > don not know the logs of my session submitted was which line. > So i hope to print the hive.session.id on every line of logs, and then i > could use grep to find the logs of my session submitted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18489) Automatically migration s3n URIs to s3a URIs
[
https://issues.apache.org/jira/browse/HIVE-18489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333776#comment-16333776
]
Aihua Xu commented on HIVE-18489:
-
The patch looks good. +1.
> Automatically migration s3n URIs to s3a URIs
>
>
> Key: HIVE-18489
> URL: https://issues.apache.org/jira/browse/HIVE-18489
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18489.1.patch, HIVE-18489.2.patch,
> HIVE-18489.3.patch
>
>
> {{s3n}} has been removed from Hadoop 3.x, we should auto-migrate tables with
> {{s3n}} URIs to the {{s3a}} URIs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18489) Automatically migration s3n URIs to s3a URIs
[
https://issues.apache.org/jira/browse/HIVE-18489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333752#comment-16333752
]
Sahil Takiar commented on HIVE-18489:
-
[~aihuaxu] can you review? I did some additional testing on a MySQL instance
and tested the syntax for Oracle, MsSQL, and Postgres using
http://rextester.com/
> Automatically migration s3n URIs to s3a URIs
>
>
> Key: HIVE-18489
> URL: https://issues.apache.org/jira/browse/HIVE-18489
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18489.1.patch, HIVE-18489.2.patch,
> HIVE-18489.3.patch
>
>
> {{s3n}} has been removed from Hadoop 3.x, we should auto-migrate tables with
> {{s3n}} URIs to the {{s3a}} URIs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18492) Wrong argument in the WorkloadManager.resetAndQueryKill()
[
https://issues.apache.org/jira/browse/HIVE-18492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333737#comment-16333737
]
Hive QA commented on HIVE-18492:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12907025/HIVE-18492.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8745/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8745/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8745/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-01-21 23:30:53.926
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-8745/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-01-21 23:30:53.929
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 90d236a HIVE-18231 : validate resource plan - part 2 - validate
action and trigger expressions (Harish Jaiprakash, reviewed by Sergey Shelukhin)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 90d236a HIVE-18231 : validate resource plan - part 2 - validate
action and trigger expressions (Harish Jaiprakash, reviewed by Sergey Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-01-21 23:30:54.459
+ rm -rf ../yetus
+ mkdir ../yetus
+ git gc
+ cp -R . ../yetus
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-8745/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
fatal: corrupt patch at line 28
fatal: corrupt patch at line 28
fatal: corrupt patch at line 28
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12907025 - PreCommit-HIVE-Build
> Wrong argument in the WorkloadManager.resetAndQueryKill()
> -
>
> Key: HIVE-18492
> URL: https://issues.apache.org/jira/browse/HIVE-18492
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Assignee: Oleg Danilov
>Priority: Trivial
> Attachments: HIVE-18492.2.patch, HIVE-18492.patch
>
>
> Caused by HIVE-18088, [~prasanth_j], could you please check this?
> {code:java}
> private void resetAndQueueKill(Map
> toKillQuery,
> KillQueryContext killQueryContext, Map toReuse) {
> WmTezSession toKill = killQueryContext.session;
> ...
> if (poolState != null) {
> poolState.getSessions().remove(toKill);
> poolState.getInitializingSessions().remove(toKill);
> ...
> {code}
> getInitializingSessions() returns List of SessionInitContext, so toKill
> definitely can't be in this list and therefore no needs to remove it.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18507) AccumuloIndexedOutputFormat.AccumuloRecordWriter.close() - typo in the condition
[
https://issues.apache.org/jira/browse/HIVE-18507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333736#comment-16333736
]
Hive QA commented on HIVE-18507:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12907024/HIVE-18507.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8744/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8744/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8744/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-01-21 23:20:55.954
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-8744/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-01-21 23:20:55.957
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 90d236a HIVE-18231 : validate resource plan - part 2 - validate
action and trigger expressions (Harish Jaiprakash, reviewed by Sergey Shelukhin)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 90d236a HIVE-18231 : validate resource plan - part 2 - validate
action and trigger expressions (Harish Jaiprakash, reviewed by Sergey Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-01-21 23:20:56.491
+ rm -rf ../yetus
+ mkdir ../yetus
+ git gc
+ cp -R . ../yetus
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-8744/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/mr/AccumuloIndexedOutputFormat.java:54
error: repository lacks the necessary blob to fall back on 3-way merge.
error:
accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/mr/AccumuloIndexedOutputFormat.java:
patch does not apply
error:
src/java/org/apache/hadoop/hive/accumulo/mr/AccumuloIndexedOutputFormat.java:
does not exist in index
error:
java/org/apache/hadoop/hive/accumulo/mr/AccumuloIndexedOutputFormat.java: does
not exist in index
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12907024 - PreCommit-HIVE-Build
> AccumuloIndexedOutputFormat.AccumuloRecordWriter.close() - typo in the
> condition
>
>
> Key: HIVE-18507
> URL: https://issues.apache.org/jira/browse/HIVE-18507
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Priority: Trivial
> Attachments: HIVE-18507.patch
>
>
> This condition always true:
> {code:java}
> if(var7.getAuthorizationFailuresMap().size() >= 0) {
> {code}
> Obviously, size() always >= 0 :), so should > instead of.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18492) Wrong argument in the WorkloadManager.resetAndQueryKill()
[
https://issues.apache.org/jira/browse/HIVE-18492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Danilov updated HIVE-18492:
Attachment: HIVE-18492.2.patch
> Wrong argument in the WorkloadManager.resetAndQueryKill()
> -
>
> Key: HIVE-18492
> URL: https://issues.apache.org/jira/browse/HIVE-18492
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Assignee: Oleg Danilov
>Priority: Trivial
> Attachments: HIVE-18492.2.patch, HIVE-18492.patch
>
>
> Caused by HIVE-18088, [~prasanth_j], could you please check this?
> {code:java}
> private void resetAndQueueKill(Map
> toKillQuery,
> KillQueryContext killQueryContext, Map toReuse) {
> WmTezSession toKill = killQueryContext.session;
> ...
> if (poolState != null) {
> poolState.getSessions().remove(toKill);
> poolState.getInitializingSessions().remove(toKill);
> ...
> {code}
> getInitializingSessions() returns List of SessionInitContext, so toKill
> definitely can't be in this list and therefore no needs to remove it.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18507) AccumuloIndexedOutputFormat.AccumuloRecordWriter.close() - typo in the condition
[
https://issues.apache.org/jira/browse/HIVE-18507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Danilov updated HIVE-18507:
Status: Patch Available (was: Open)
> AccumuloIndexedOutputFormat.AccumuloRecordWriter.close() - typo in the
> condition
>
>
> Key: HIVE-18507
> URL: https://issues.apache.org/jira/browse/HIVE-18507
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Priority: Trivial
> Attachments: HIVE-18507.patch
>
>
> This condition always true:
> {code:java}
> if(var7.getAuthorizationFailuresMap().size() >= 0) {
> {code}
> Obviously, size() always >= 0 :), so should > instead of.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18507) AccumuloIndexedOutputFormat.AccumuloRecordWriter.close() - typo in the condition
[
https://issues.apache.org/jira/browse/HIVE-18507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Danilov updated HIVE-18507:
Attachment: HIVE-18507.patch
> AccumuloIndexedOutputFormat.AccumuloRecordWriter.close() - typo in the
> condition
>
>
> Key: HIVE-18507
> URL: https://issues.apache.org/jira/browse/HIVE-18507
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Priority: Trivial
> Attachments: HIVE-18507.patch
>
>
> This condition always true:
> {code:java}
> if(var7.getAuthorizationFailuresMap().size() >= 0) {
> {code}
> Obviously, size() always >= 0 :), so should > instead of.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18506) LlapBaseInputFormat - negative array index
[
https://issues.apache.org/jira/browse/HIVE-18506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333728#comment-16333728
]
Hive QA commented on HIVE-18506:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12907023/HIVE-18506.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8743/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8743/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8743/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-01-21 22:56:14.521
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-8743/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-01-21 22:56:14.524
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 90d236a HIVE-18231 : validate resource plan - part 2 - validate
action and trigger expressions (Harish Jaiprakash, reviewed by Sergey Shelukhin)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 90d236a HIVE-18231 : validate resource plan - part 2 - validate
action and trigger expressions (Harish Jaiprakash, reviewed by Sergey Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-01-21 22:56:15.231
+ rm -rf ../yetus
+ mkdir ../yetus
+ git gc
+ cp -R . ../yetus
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-8743/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java:373
error: repository lacks the necessary blob to fall back on 3-way merge.
error:
llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java:
patch does not apply
error: src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java: does not
exist in index
error: java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java: does not
exist in index
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12907023 - PreCommit-HIVE-Build
> LlapBaseInputFormat - negative array index
> --
>
> Key: HIVE-18506
> URL: https://issues.apache.org/jira/browse/HIVE-18506
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Assignee: Oleg Danilov
>Priority: Minor
> Attachments: HIVE-18506.patch
>
>
>
> "nextInt % value" may generate negative values:
> {code:java}
> int randIdx = rand.nextInt() % allInstances.size();
> serviceInstance = allInstances.toArray(serviceInstanceArray)[randIdx];
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18489) Automatically migration s3n URIs to s3a URIs
[
https://issues.apache.org/jira/browse/HIVE-18489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333723#comment-16333723
]
Hive QA commented on HIVE-18489:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12907022/HIVE-18489.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 11630 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_rollup_empty]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby_rollup_empty]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=121)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=254)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=232)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=232)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8742/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8742/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8742/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12907022 - PreCommit-HIVE-Build
> Automatically migration s3n URIs to s3a URIs
>
>
> Key: HIVE-18489
> URL: https://issues.apache.org/jira/browse/HIVE-18489
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18489.1.patch, HIVE-18489.2.patch,
> HIVE-18489.3.patch
>
>
> {{s3n}} has been removed from Hadoop 3.x, we should auto-migrate tables with
> {{s3n}} URIs to the {{s3a}} URIs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18506) LlapBaseInputFormat - negative array index
[
https://issues.apache.org/jira/browse/HIVE-18506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V reassigned HIVE-18506:
--
Assignee: Oleg Danilov
> LlapBaseInputFormat - negative array index
> --
>
> Key: HIVE-18506
> URL: https://issues.apache.org/jira/browse/HIVE-18506
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Assignee: Oleg Danilov
>Priority: Minor
> Attachments: HIVE-18506.patch
>
>
>
> "nextInt % value" may generate negative values:
> {code:java}
> int randIdx = rand.nextInt() % allInstances.size();
> serviceInstance = allInstances.toArray(serviceInstanceArray)[randIdx];
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18506) LlapBaseInputFormat - negative array index
[
https://issues.apache.org/jira/browse/HIVE-18506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Danilov updated HIVE-18506:
Attachment: HIVE-18506.patch
> LlapBaseInputFormat - negative array index
> --
>
> Key: HIVE-18506
> URL: https://issues.apache.org/jira/browse/HIVE-18506
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Priority: Minor
> Attachments: HIVE-18506.patch
>
>
>
> "nextInt % value" may generate negative values:
> {code:java}
> int randIdx = rand.nextInt() % allInstances.size();
> serviceInstance = allInstances.toArray(serviceInstanceArray)[randIdx];
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18506) LlapBaseInputFormat - negative array index
[
https://issues.apache.org/jira/browse/HIVE-18506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Danilov updated HIVE-18506:
Status: Patch Available (was: Open)
> LlapBaseInputFormat - negative array index
> --
>
> Key: HIVE-18506
> URL: https://issues.apache.org/jira/browse/HIVE-18506
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Priority: Minor
> Attachments: HIVE-18506.patch
>
>
>
> "nextInt % value" may generate negative values:
> {code:java}
> int randIdx = rand.nextInt() % allInstances.size();
> serviceInstance = allInstances.toArray(serviceInstanceArray)[randIdx];
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18489) Automatically migration s3n URIs to s3a URIs
[
https://issues.apache.org/jira/browse/HIVE-18489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333694#comment-16333694
]
Hive QA commented on HIVE-18489:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
45s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 90d236a |
| modules | C: metastore U: metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8742/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Automatically migration s3n URIs to s3a URIs
>
>
> Key: HIVE-18489
> URL: https://issues.apache.org/jira/browse/HIVE-18489
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18489.1.patch, HIVE-18489.2.patch,
> HIVE-18489.3.patch
>
>
> {{s3n}} has been removed from Hadoop 3.x, we should auto-migrate tables with
> {{s3n}} URIs to the {{s3a}} URIs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18489) Automatically migration s3n URIs to s3a URIs
[
https://issues.apache.org/jira/browse/HIVE-18489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-18489:
Attachment: HIVE-18489.3.patch
> Automatically migration s3n URIs to s3a URIs
>
>
> Key: HIVE-18489
> URL: https://issues.apache.org/jira/browse/HIVE-18489
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18489.1.patch, HIVE-18489.2.patch,
> HIVE-18489.3.patch
>
>
> {{s3n}} has been removed from Hadoop 3.x, we should auto-migrate tables with
> {{s3n}} URIs to the {{s3a}} URIs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17331) Path must be used as key type of the pathToAlises
[
https://issues.apache.org/jira/browse/HIVE-17331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333688#comment-16333688
]
Hive QA commented on HIVE-17331:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12907021/HIVE-17331.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8741/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8741/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8741/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-01-21 21:22:10.191
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-8741/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-01-21 21:22:10.194
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 90d236a HIVE-18231 : validate resource plan - part 2 - validate
action and trigger expressions (Harish Jaiprakash, reviewed by Sergey Shelukhin)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 90d236a HIVE-18231 : validate resource plan - part 2 - validate
action and trigger expressions (Harish Jaiprakash, reviewed by Sergey Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-01-21 21:22:14.394
+ rm -rf ../yetus
+ mkdir ../yetus
+ git gc
+ cp -R . ../yetus
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-8741/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:243
error: repository lacks the necessary blob to fall back on 3-way merge.
error: ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java:
patch does not apply
error: src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java: does
not exist in index
error: java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java: does not
exist in index
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12907021 - PreCommit-HIVE-Build
> Path must be used as key type of the pathToAlises
> -
>
> Key: HIVE-17331
> URL: https://issues.apache.org/jira/browse/HIVE-17331
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Assignee: Oleg Danilov
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-17331.2.patch, HIVE-17331.3.patch, HIVE-17331.patch
>
>
> This code uses String instead of Path as key type of the pathToAliases map,
> so seems like get(String) always null.
> +*GenMapRedUtils.java*+
> {code:java}
> for (int pos = 0; pos < size; pos++) {
> String taskTmpDir = taskTmpDirLst.get(pos);
> TableDesc tt_desc = tt_descLst.get(pos);
> MapWork mWork = plan.getMapWork();
> if (mWork.getPathToAliases().get(taskTmpDir) == null) {
> taskTmpDir = taskTmpDir.intern();
> Path taskTmpDirPath =
> StringInternUtils.internUriStringsInPath(new Path(taskTmpDir));
> mWork.removePathToAlias(taskTmpDirPath);
> mWork.addPathToAlias(taskTmpDirPath, taskTmpDir);
> mWork.addPathToPartitionInfo(taskTmpDirPath, new
> PartitionDesc(tt_desc, null));
> mWork.getAliasToWork().put(taskTmpDir, topOperators.get(pos));
>
[jira] [Commented] (HIVE-18485) Add more unit tests for hive.strict.checks.* properties
[
https://issues.apache.org/jira/browse/HIVE-18485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333686#comment-16333686
]
Hive QA commented on HIVE-18485:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12907018/HIVE-18485.3.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 11637 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_rollup_empty]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby_rollup_empty]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=121)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=254)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=232)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=232)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8740/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8740/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8740/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12907018 - PreCommit-HIVE-Build
> Add more unit tests for hive.strict.checks.* properties
> ---
>
> Key: HIVE-18485
> URL: https://issues.apache.org/jira/browse/HIVE-18485
> Project: Hive
> Issue Type: Test
> Components: Test
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18485.1.patch, HIVE-18485.2.patch,
> HIVE-18485.3.patch
>
>
> We should add some more negative tests for {{hive.strict.checks.\*}}
> properties that explicitly check the {{hive.strict.checks.*}} properties -
> right now they all rely on {{hive.mapred.mode=strict}} which is deprecated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17331) Path must be used as key type of the pathToAlises
[
https://issues.apache.org/jira/browse/HIVE-17331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Danilov updated HIVE-17331:
Attachment: HIVE-17331.3.patch
> Path must be used as key type of the pathToAlises
> -
>
> Key: HIVE-17331
> URL: https://issues.apache.org/jira/browse/HIVE-17331
> Project: Hive
> Issue Type: Bug
>Reporter: Oleg Danilov
>Assignee: Oleg Danilov
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-17331.2.patch, HIVE-17331.3.patch, HIVE-17331.patch
>
>
> This code uses String instead of Path as key type of the pathToAliases map,
> so seems like get(String) always null.
> +*GenMapRedUtils.java*+
> {code:java}
> for (int pos = 0; pos < size; pos++) {
> String taskTmpDir = taskTmpDirLst.get(pos);
> TableDesc tt_desc = tt_descLst.get(pos);
> MapWork mWork = plan.getMapWork();
> if (mWork.getPathToAliases().get(taskTmpDir) == null) {
> taskTmpDir = taskTmpDir.intern();
> Path taskTmpDirPath =
> StringInternUtils.internUriStringsInPath(new Path(taskTmpDir));
> mWork.removePathToAlias(taskTmpDirPath);
> mWork.addPathToAlias(taskTmpDirPath, taskTmpDir);
> mWork.addPathToPartitionInfo(taskTmpDirPath, new
> PartitionDesc(tt_desc, null));
> mWork.getAliasToWork().put(taskTmpDir, topOperators.get(pos));
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18485) Add more unit tests for hive.strict.checks.* properties
[
https://issues.apache.org/jira/browse/HIVE-18485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333658#comment-16333658
]
Hive QA commented on HIVE-18485:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 7 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 90d236a |
| Default Java | 1.8.0_111 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8740/yetus/whitespace-eol.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8740/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add more unit tests for hive.strict.checks.* properties
> ---
>
> Key: HIVE-18485
> URL: https://issues.apache.org/jira/browse/HIVE-18485
> Project: Hive
> Issue Type: Test
> Components: Test
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18485.1.patch, HIVE-18485.2.patch,
> HIVE-18485.3.patch
>
>
> We should add some more negative tests for {{hive.strict.checks.\*}}
> properties that explicitly check the {{hive.strict.checks.*}} properties -
> right now they all rely on {{hive.mapred.mode=strict}} which is deprecated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18489) Automatically migration s3n URIs to s3a URIs
[
https://issues.apache.org/jira/browse/HIVE-18489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333651#comment-16333651
]
Hive QA commented on HIVE-18489:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12907019/HIVE-18489.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 11630 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_rollup_empty]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby_rollup_empty]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=121)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query39]
(batchId=248)
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
(batchId=299)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=254)
org.apache.hive.beeline.TestSchemaTool.testMetastoreDbPropertiesAfterUpgrade
(batchId=229)
org.apache.hive.beeline.TestSchemaTool.testSchemaUpgrade (batchId=229)
org.apache.hive.beeline.TestSchemaTool.testValidateSchemaTables (batchId=229)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=232)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=232)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=232)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8739/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8739/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8739/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 23 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12907019 - PreCommit-HIVE-Build
> Automatically migration s3n URIs to s3a URIs
>
>
> Key: HIVE-18489
> URL: https://issues.apache.org/jira/browse/HIVE-18489
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18489.1.patch, HIVE-18489.2.patch
>
>
> {{s3n}} has been removed from Hadoop 3.x, we should auto-migrate tables with
> {{s3n}} URIs to the {{s3a}} URIs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18489) Automatically migration s3n URIs to s3a URIs
[
https://issues.apache.org/jira/browse/HIVE-18489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333629#comment-16333629
]
Hive QA commented on HIVE-18489:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 0m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 90d236a |
| modules | C: metastore U: metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8739/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Automatically migration s3n URIs to s3a URIs
>
>
> Key: HIVE-18489
> URL: https://issues.apache.org/jira/browse/HIVE-18489
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18489.1.patch, HIVE-18489.2.patch
>
>
> {{s3n}} has been removed from Hadoop 3.x, we should auto-migrate tables with
> {{s3n}} URIs to the {{s3a}} URIs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18489) Automatically migration s3n URIs to s3a URIs
[
https://issues.apache.org/jira/browse/HIVE-18489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-18489:
Attachment: HIVE-18489.2.patch
> Automatically migration s3n URIs to s3a URIs
>
>
> Key: HIVE-18489
> URL: https://issues.apache.org/jira/browse/HIVE-18489
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18489.1.patch, HIVE-18489.2.patch
>
>
> {{s3n}} has been removed from Hadoop 3.x, we should auto-migrate tables with
> {{s3n}} URIs to the {{s3a}} URIs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18485) Add more unit tests for hive.strict.checks.* properties
[
https://issues.apache.org/jira/browse/HIVE-18485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-18485:
Attachment: HIVE-18485.3.patch
> Add more unit tests for hive.strict.checks.* properties
> ---
>
> Key: HIVE-18485
> URL: https://issues.apache.org/jira/browse/HIVE-18485
> Project: Hive
> Issue Type: Test
> Components: Test
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18485.1.patch, HIVE-18485.2.patch,
> HIVE-18485.3.patch
>
>
> We should add some more negative tests for {{hive.strict.checks.\*}}
> properties that explicitly check the {{hive.strict.checks.*}} properties -
> right now they all rely on {{hive.mapred.mode=strict}} which is deprecated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18421) Vectorized execution does not handle integer overflows
[
https://issues.apache.org/jira/browse/HIVE-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333621#comment-16333621
]
Vihang Karajgaonkar commented on HIVE-18421:
I think one easy way to solve this is to cast the values in the long vector to
the outputType. Based on my testing it works (atleast for the arithmetic
expressions for which I tested). Currently, I am compiling a list of
expressions which are affected by this issue. Any thoughts on the cast
operator? I can do some benchmarks if there are automated tests in the source
code and verify how much it affects the performance. Here is the snippet code
which if I add solves the problem. For example, if the outputType is {{int}}
you add the following code after the expression is evaluated on the
LongColumnVector.
{code}
+ //int
+ if (v.isRepeating) {
+v.vector[0] = (int) v.vector[0];
+ } else if (selectedInUse) {
+for (int j = 0; j != n; j++) {
+ int i = sel[j];
+ v.vector[i] = (int) v.vector[i];
+}
+ } else {
+for (int i = 0; i != n; i++) {
+ v.vector[i] = (int) v.vector[i];
+}
+ }
{code}
I think the good news here is AFAIK only the supportedGenericUDFs in the
vectorizer will be affected because the rest will use {{VectorUDFAdapter}}
which should not be affected by this issue. That reduces the scope of the
problem than earlier thought. However, we will have to be careful while adding
to new UDFs to the supported list.
[~gopalv] [~mmccline] any thoughts on the down casting the values in the Column
vector like the snippet is doing above?
> Vectorized execution does not handle integer overflows
> --
>
> Key: HIVE-18421
> URL: https://issues.apache.org/jira/browse/HIVE-18421
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 2.1.1, 2.2.0, 3.0.0, 2.3.2
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>
> In vectorized execution arithmetic operations which cause integer overflows
> can give wrong results. Issue is reproducible in both Orc and parquet.
> Simple test case to reproduce this issue
> {noformat}
> set hive.vectorized.execution.enabled=true;
> create table parquettable (t1 tinyint, t2 tinyint) stored as parquet;
> insert into parquettable values (-104, 25), (-112, 24), (54, 9);
> select t1, t2, (t1-t2) as diff from parquettable where (t1-t2) < 50 order by
> diff desc;
> +---+-+---+
> | t1 | t2 | diff |
> +---+-+---+
> | -104 | 25 | 127 |
> | -112 | 24 | 120 |
> | 54| 9 | 45|
> +---+-+---+
> {noformat}
> When vectorization is turned off the same query produces only one row.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15355) Concurrency issues during parallel moveFile due to HDFSUtils.setFullFileStatus
[
https://issues.apache.org/jira/browse/HIVE-15355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333619#comment-16333619
]
Vihang Karajgaonkar commented on HIVE-15355:
Hi [~sershe] Do you mean {{if (sourceStatus.getAclEntries() != null) }} check?
If you look at the implementation of {{HadoopFileStatus}} its the aclEntries is
derived from {{aclStatus}} which is a final member of the object and it returns
a unmodifiable collection. Can you give an example of the race condition? Also,
interestingly enough, I don't see the code which calls this method in parallel
now. Not sure if some patch changed the parallel implementation later.
> Concurrency issues during parallel moveFile due to HDFSUtils.setFullFileStatus
> --
>
> Key: HIVE-15355
> URL: https://issues.apache.org/jira/browse/HIVE-15355
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Fix For: 2.2.0
>
> Attachments: HIVE-15355.01.patch, HIVE-15355.02.patch
>
>
> It is possible to run into concurrency issues during multi-threaded moveFile
> issued when processing queries like {{INSERT OVERWRITE TABLE ... SELECT ..}}
> when there are multiple files in the staging directory which is a
> subdirectory of the target directory. The issue is hard to reproduce but
> following stacktrace is one such example:
> {noformat}
> INFO : Loading data to table
> functional_text_gzip.alltypesaggmultifilesnopart from
> hdfs://localhost:20500/test-warehouse/alltypesaggmultifilesnopart_text_gzip/.hive-staging_hive_2016-12-01_19-58-21_712_8968735301422943318-1/-ext-1
> ERROR : Failed with exception java.lang.ArrayIndexOutOfBoundsException
> org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ArrayIndexOutOfBoundsException
> at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2858)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:3124)
> at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1701)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:313)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1976)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1689)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1421)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1205)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1200)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:293)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
> Getting log thread is interrupted, since query is done!
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:306)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at java.util.ArrayList.removeRange(ArrayList.java:616)
> at java.util.ArrayList$SubList.removeRange(ArrayList.java:1021)
> at java.util.AbstractList.clear(AbstractList.java:234)
> at
> com.google.common.collect.Iterables.removeIfFromRandomAccessList(Iterables.java:213)
> at com.google.common.collect.Iterables.removeIf(Iterables.java:184)
> at
> org.apache.hadoop.hive.shims.Hadoop23Shims.removeBaseAclEntries(Hadoop23Shims.java:865)
> at
> org.apache.hadoop.hive.shims.Hadoop23Shims.setFullFileStatus(Hadoop23Shims.java:757)
> at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:2835)
> at org.apache.hadoop.hive.ql.metadata.Hive$3.call(Hive.java:2828)
> ... 4 more
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask
>
[jira] [Commented] (HIVE-18393) Error returned when some other type is read as string from parquet tables
[
https://issues.apache.org/jira/browse/HIVE-18393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333612#comment-16333612
]
Vihang Karajgaonkar commented on HIVE-18393:
Hi [~janulatha] Can you confirm if test failures are related. Some of the tests
failures are not among the regular ones. Esp. {{tez_bmj_schema_evolution}}
> Error returned when some other type is read as string from parquet tables
> -
>
> Key: HIVE-18393
> URL: https://issues.apache.org/jira/browse/HIVE-18393
> Project: Hive
> Issue Type: Bug
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18393.1.patch, HIVE-18393.2.patch,
> HIVE-18393.3.patch, HIVE-18393.4.patch, HIVE-18393.5.patch
>
>
> TimeStamp, Decimal, Double, Float, BigInt, Int, SmallInt, Tinyint and Boolean
> when read as String, Varchar or Char should return the correct data. Now
> this results in error for parquet tables.
> Test Case:
> {code}
> drop table if exists testAltCol;
> create table testAltCol
> (cId TINYINT,
> cTimeStamp TIMESTAMP,
> cDecimal DECIMAL(38,18),
> cDoubleDOUBLE,
> cFloat FLOAT,
> cBigIntBIGINT,
> cInt INT,
> cSmallInt SMALLINT,
> cTinyint TINYINT,
> cBoolean BOOLEAN);
> insert into testAltCol values
> (1,
> '2017-11-07 09:02:49.9',
> 12345678901234567890.123456789012345678,
> 1.79e308,
> 3.4e38,
> 1234567890123456789,
> 1234567890,
> 12345,
> 123,
> TRUE);
> insert into testAltCol values
> (2,
> '1400-01-01 01:01:01.1',
> 1.1,
> 2.2,
> 3.3,
> 1,
> 2,
> 3,
> 4,
> FALSE);
> insert into testAltCol values
> (3,
> '1400-01-01 01:01:01.1',
> 10.1,
> 20.2,
> 30.3,
> 1234567890123456789,
> 1234567890,
> 12345,
> 123,
> TRUE);
> select cId, cTimeStamp from testAltCol order by cId;
> select cId, cDecimal, cDouble, cFloat from testAltCol order by cId;
> select cId, cBigInt, cInt, cSmallInt, cTinyint from testAltCol order by cId;
> select cId, cBoolean from testAltCol order by cId;
> drop table if exists testAltColP;
> create table testAltColP stored as parquet as select * from testAltCol;
> select cId, cTimeStamp from testAltColP order by cId;
> select cId, cDecimal, cDouble, cFloat from testAltColP order by cId;
> select cId, cBigInt, cInt, cSmallInt, cTinyint from testAltColP order by cId;
> select cId, cBoolean from testAltColP order by cId;
> alter table testAltColP replace columns
> (cId TINYINT,
> cTimeStamp STRING,
> cDecimal STRING,
> cDoubleSTRING,
> cFloat STRING,
> cBigIntSTRING,
> cInt STRING,
> cSmallInt STRING,
> cTinyint STRING,
> cBoolean STRING);
> select cId, cTimeStamp from testAltColP order by cId;
> select cId, cDecimal, cDouble, cFloat from testAltColP order by cId;
> select cId, cBigInt, cInt, cSmallInt, cTinyint from testAltColP order by cId;
> select cId, cBoolean from testAltColP order by cId;
> alter table testAltColP replace columns
> (cId TINYINT,
> cTimeStamp VARCHAR(100),
> cDecimal VARCHAR(100),
> cDoubleVARCHAR(100),
> cFloat VARCHAR(100),
> cBigIntVARCHAR(100),
> cInt VARCHAR(100),
> cSmallInt VARCHAR(100),
> cTinyint VARCHAR(100),
> cBoolean VARCHAR(100));
> select cId, cTimeStamp from testAltColP order by cId;
> select cId, cDecimal, cDouble, cFloat from testAltColP order by cId;
> select cId, cBigInt, cInt, cSmallInt, cTinyint from testAltColP order by cId;
> select cId, cBoolean from testAltColP order by cId;
> alter table testAltColP replace columns
> (cId TINYINT,
> cTimeStamp CHAR(100),
> cDecimal CHAR(100),
> cDoubleCHAR(100),
> cFloat CHAR(100),
> cBigIntCHAR(100),
> cInt CHAR(100),
> cSmallInt CHAR(100),
> cTinyint CHAR(100),
> cBoolean CHAR(100));
> select cId, cTimeStamp from testAltColP order by cId;
> select cId, cDecimal, cDouble, cFloat from testAltColP order by cId;
> select cId, cBigInt, cInt, cSmallInt, cTinyint from testAltColP order by cId;
> select cId, cBoolean from testAltColP order by cId;
> drop table if exists testAltColP;
> {code}
> {code}
> Error:
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> Excerpt for log:
> 2018-01-05T15:54:05,756 ERROR [LocalJobRunner Map Task Executor #0]
> mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row [Error getting row data with exception
> java.lang.UnsupportedOperationException: Cannot inspect
> org.apache.hadoop.hive.serde2.io.TimestampWritable
> at
> org.apache.hadoop.hive.ql.io.parquet.serde.primitive.ParquetStringInspector.getPrimitiveJavaObject(ParquetStringInspect
[jira] [Commented] (HIVE-18505) Added external hive configuration to prepDb in TxnDbUtil
[ https://issues.apache.org/jira/browse/HIVE-18505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333565#comment-16333565 ] ASF GitHub Bot commented on HIVE-18505: --- Github user chandulal closed the pull request at: https://github.com/apache/hive/pull/293 > Added external hive configuration to prepDb in TxnDbUtil > > > Key: HIVE-18505 > URL: https://issues.apache.org/jira/browse/HIVE-18505 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Chandu Kavar >Assignee: Chandu Kavar >Priority: Minor > Labels: pull-request-available > > In Hive Metastore, We have TxtDbUtil.java and it contains few utils required > for tests. > There is prepDb() method, it is creating connection and execute some system > queries in order to prepare db. While creating connection it's create new > HiveConf object and not taking configs from outside. > TxtDbUtil.java should also contains prepDb method that can accept external > hive configs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18192) Introduce WriteID per table rather than using global transaction ID
[
https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333553#comment-16333553
]
Hive QA commented on HIVE-18192:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12907005/HIVE-18192.03.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8738/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8738/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8738/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-01-21 15:04:14.083
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-8738/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-01-21 15:04:14.086
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 90d236a HIVE-18231 : validate resource plan - part 2 - validate
action and trigger expressions (Harish Jaiprakash, reviewed by Sergey Shelukhin)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 90d236a HIVE-18231 : validate resource plan - part 2 - validate
action and trigger expressions (Harish Jaiprakash, reviewed by Sergey Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-01-21 15:04:14.603
+ rm -rf ../yetus
+ mkdir ../yetus
+ git gc
+ cp -R . ../yetus
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-8738/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
fatal: git apply: bad git-diff - inconsistent old filename on line 48126
error: patch failed:
standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp:1240
Falling back to three-way merge...
Applied patch to
'standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp' with
conflicts.
error: patch failed:
standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp:23884
Falling back to three-way merge...
Applied patch to
'standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp' with
conflicts.
error: patch failed:
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java:34145
Falling back to three-way merge...
Applied patch to
'standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java'
with conflicts.
error: patch failed:
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/WMGetTriggersForResourePlanResponse.java:346
Falling back to three-way merge...
Applied patch to
'standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/WMGetTriggersForResourePlanResponse.java'
with conflicts.
error: patch failed:
standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/WMValidateResourcePlanResponse.java:376
Falling back to three-way merge...
Applied patch to
'standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/WMValidateResourcePlanResponse.java'
with conflicts.
error: patch failed:
standalone-metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php:12984
Falling back to three-way merge...
Applied patch to
'standalone-metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php'
with conflicts.
error: patch failed:
standalone-metastore/src/gen/thrift/gen-php/metastore/Types.php:23418
Falling back to three-way merge...
Applied patch to
'standalone-metastore/src/gen/thrift
[jira] [Updated] (HIVE-18192) Introduce WriteID per table rather than using global transaction ID
[ https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18192: Status: Patch Available (was: Open) > Introduce WriteID per table rather than using global transaction ID > --- > > Key: HIVE-18192 > URL: https://issues.apache.org/jira/browse/HIVE-18192 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18192.01.patch, HIVE-18192.02.patch, > HIVE-18192.03.patch > > > To support ACID replication, we will be introducing a per table write Id > which will replace the transaction id in the primary key for each row in a > ACID table. > The current primary key is determined via > > which will move to > > For each table modified by the given transaction will have a table level > write ID allocated and a persisted map of global txn id -> to table -> write > id for that table has to be maintained to allow Snapshot isolation. > Readers should use the combination of ValidTxnList and > ValidWriteIdList(Table) for snapshot isolation. > > [Hive Replication - ACID > Tables.pdf|https://issues.apache.org/jira/secure/attachment/12903157/Hive%20Replication-%20ACID%20Tables.pdf] > has a section "Per Table Sequences (Write-Id)" with more detials -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18192) Introduce WriteID per table rather than using global transaction ID
[ https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18192: Attachment: HIVE-18192.03.patch > Introduce WriteID per table rather than using global transaction ID > --- > > Key: HIVE-18192 > URL: https://issues.apache.org/jira/browse/HIVE-18192 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18192.01.patch, HIVE-18192.02.patch, > HIVE-18192.03.patch > > > To support ACID replication, we will be introducing a per table write Id > which will replace the transaction id in the primary key for each row in a > ACID table. > The current primary key is determined via > > which will move to > > For each table modified by the given transaction will have a table level > write ID allocated and a persisted map of global txn id -> to table -> write > id for that table has to be maintained to allow Snapshot isolation. > Readers should use the combination of ValidTxnList and > ValidWriteIdList(Table) for snapshot isolation. > > [Hive Replication - ACID > Tables.pdf|https://issues.apache.org/jira/secure/attachment/12903157/Hive%20Replication-%20ACID%20Tables.pdf] > has a section "Per Table Sequences (Write-Id)" with more detials -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18192) Introduce WriteID per table rather than using global transaction ID
[ https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18192: Attachment: (was: HIVE-18192.03.patch) > Introduce WriteID per table rather than using global transaction ID > --- > > Key: HIVE-18192 > URL: https://issues.apache.org/jira/browse/HIVE-18192 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18192.01.patch, HIVE-18192.02.patch > > > To support ACID replication, we will be introducing a per table write Id > which will replace the transaction id in the primary key for each row in a > ACID table. > The current primary key is determined via > > which will move to > > For each table modified by the given transaction will have a table level > write ID allocated and a persisted map of global txn id -> to table -> write > id for that table has to be maintained to allow Snapshot isolation. > Readers should use the combination of ValidTxnList and > ValidWriteIdList(Table) for snapshot isolation. > > [Hive Replication - ACID > Tables.pdf|https://issues.apache.org/jira/secure/attachment/12903157/Hive%20Replication-%20ACID%20Tables.pdf] > has a section "Per Table Sequences (Write-Id)" with more detials -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18192) Introduce WriteID per table rather than using global transaction ID
[ https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18192: Status: Open (was: Patch Available) > Introduce WriteID per table rather than using global transaction ID > --- > > Key: HIVE-18192 > URL: https://issues.apache.org/jira/browse/HIVE-18192 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18192.01.patch, HIVE-18192.02.patch > > > To support ACID replication, we will be introducing a per table write Id > which will replace the transaction id in the primary key for each row in a > ACID table. > The current primary key is determined via > > which will move to > > For each table modified by the given transaction will have a table level > write ID allocated and a persisted map of global txn id -> to table -> write > id for that table has to be maintained to allow Snapshot isolation. > Readers should use the combination of ValidTxnList and > ValidWriteIdList(Table) for snapshot isolation. > > [Hive Replication - ACID > Tables.pdf|https://issues.apache.org/jira/secure/attachment/12903157/Hive%20Replication-%20ACID%20Tables.pdf] > has a section "Per Table Sequences (Write-Id)" with more detials -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18505) Added external hive configuration to prepDb in TxnDbUtil
[ https://issues.apache.org/jira/browse/HIVE-18505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333504#comment-16333504 ] Chandu Kavar commented on HIVE-18505: - I think this issue is no longer valid. It's created by mistake. > Added external hive configuration to prepDb in TxnDbUtil > > > Key: HIVE-18505 > URL: https://issues.apache.org/jira/browse/HIVE-18505 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Chandu Kavar >Assignee: Chandu Kavar >Priority: Minor > Labels: pull-request-available > > In Hive Metastore, We have TxtDbUtil.java and it contains few utils required > for tests. > There is prepDb() method, it is creating connection and execute some system > queries in order to prepare db. While creating connection it's create new > HiveConf object and not taking configs from outside. > TxtDbUtil.java should also contains prepDb method that can accept external > hive configs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18505) Added external hive configuration to prepDb in TxnDbUtil
[ https://issues.apache.org/jira/browse/HIVE-18505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16333504#comment-16333504 ] Chandu Kavar edited comment on HIVE-18505 at 1/21/18 1:05 PM: -- This issue is no longer valid. It's created by mistake. was (Author: cckavar): I think this issue is no longer valid. It's created by mistake. > Added external hive configuration to prepDb in TxnDbUtil > > > Key: HIVE-18505 > URL: https://issues.apache.org/jira/browse/HIVE-18505 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Chandu Kavar >Assignee: Chandu Kavar >Priority: Minor > Labels: pull-request-available > > In Hive Metastore, We have TxtDbUtil.java and it contains few utils required > for tests. > There is prepDb() method, it is creating connection and execute some system > queries in order to prepare db. While creating connection it's create new > HiveConf object and not taking configs from outside. > TxtDbUtil.java should also contains prepDb method that can accept external > hive configs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
