[jira] [Commented] (HIVE-19279) remove magic directory skipping from CopyTask
[
https://issues.apache.org/jira/browse/HIVE-19279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448935#comment-16448935
]
Thejas M Nair commented on HIVE-19279:
--
Does any feature rely on this behavior ?
> remove magic directory skipping from CopyTask
> -
>
> Key: HIVE-19279
> URL: https://issues.apache.org/jira/browse/HIVE-19279
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Major
>
> Follow up from HIVE-17657.
> Code exists in copytask that copies files (fancy that); however, when listing
> the files, if a single directory exists at the source with no other files, it
> will skip the directory and copy the files inside instead.
> This directory in various tests is either the "data" directory from export,
> or some random partition directory ("foo=bar") that if not skipped makes it
> into the real partition directory at the destination.
> The directory is not skipped if it's not by itself, i.e. any other files or
> directories are present.
> This seems brittle. Caller of the CopyTask should specify exactly what it
> wants copied instead of relying on this behavior.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19279) remove magic directory skipping from CopyTask
[
https://issues.apache.org/jira/browse/HIVE-19279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448935#comment-16448935
]
Thejas M Nair edited comment on HIVE-19279 at 4/23/18 10:08 PM:
Does any feature rely on this existing behavior ?
was (Author: thejas):
Does any feature rely on this behavior ?
> remove magic directory skipping from CopyTask
> -
>
> Key: HIVE-19279
> URL: https://issues.apache.org/jira/browse/HIVE-19279
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Major
>
> Follow up from HIVE-17657.
> Code exists in copytask that copies files (fancy that); however, when listing
> the files, if a single directory exists at the source with no other files, it
> will skip the directory and copy the files inside instead.
> This directory in various tests is either the "data" directory from export,
> or some random partition directory ("foo=bar") that if not skipped makes it
> into the real partition directory at the destination.
> The directory is not skipped if it's not by itself, i.e. any other files or
> directories are present.
> This seems brittle. Caller of the CopyTask should specify exactly what it
> wants copied instead of relying on this behavior.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19273) Fix TestBeeLineWithArgs.testQueryProgressParallel
[ https://issues.apache.org/jira/browse/HIVE-19273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair reassigned HIVE-19273: Assignee: Vaibhav Gumashta (was: Thejas M Nair) > Fix TestBeeLineWithArgs.testQueryProgressParallel > - > > Key: HIVE-19273 > URL: https://issues.apache.org/jira/browse/HIVE-19273 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Vaibhav Gumashta >Priority: Major > > seems to be failing from time-to-time: > https://builds.apache.org/job/PreCommit-HIVE-Build/10429/testReport/org.apache.hive.beeline/TestBeeLineWithArgs/testQueryProgressParallel/history/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19273) Fix TestBeeLineWithArgs.testQueryProgressParallel
[ https://issues.apache.org/jira/browse/HIVE-19273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449214#comment-16449214 ] Thejas M Nair commented on HIVE-19273: -- [~vgumashta] Can you please take a look ? > Fix TestBeeLineWithArgs.testQueryProgressParallel > - > > Key: HIVE-19273 > URL: https://issues.apache.org/jira/browse/HIVE-19273 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Vaibhav Gumashta >Priority: Major > > seems to be failing from time-to-time: > https://builds.apache.org/job/PreCommit-HIVE-Build/10429/testReport/org.apache.hive.beeline/TestBeeLineWithArgs/testQueryProgressParallel/history/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19310) Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might need to be run only in test env
[
https://issues.apache.org/jira/browse/HIVE-19310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453088#comment-16453088
]
Thejas M Nair commented on HIVE-19310:
--
I think we we are skipping the init, we should skip all the initQueries, as I
see all of them serving the same purpose.
> Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might
> need to be run only in test env
> -
>
> Key: HIVE-19310
> URL: https://issues.apache.org/jira/browse/HIVE-19310
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-19310.1.patch
>
>
> MetaStoreDirectSql.ensureDbInit has the following 2 calls which we have
> observed taking a long time in our testing:
> {code}
> initQueries.add(pm.newQuery(MNotificationLog.class, "dbName == ''"));
> initQueries.add(pm.newQuery(MNotificationNextId.class, "nextEventId < -1"));
> {code}
> In a production environment, these tables should be initialized using
> schematool, however in a test environment, these calls might be needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19310) Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might need to be run only in test env
[
https://issues.apache.org/jira/browse/HIVE-19310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453091#comment-16453091
]
Thejas M Nair commented on HIVE-19310:
--
The whole initialization done via ObjectStore.setConf also needs to be
re-examined. But that can be a follow up jira.
> Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might
> need to be run only in test env
> -
>
> Key: HIVE-19310
> URL: https://issues.apache.org/jira/browse/HIVE-19310
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-19310.1.patch
>
>
> MetaStoreDirectSql.ensureDbInit has the following 2 calls which we have
> observed taking a long time in our testing:
> {code}
> initQueries.add(pm.newQuery(MNotificationLog.class, "dbName == ''"));
> initQueries.add(pm.newQuery(MNotificationNextId.class, "nextEventId < -1"));
> {code}
> In a production environment, these tables should be initialized using
> schematool, however in a test environment, these calls might be needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19135) Need tool to allow admins to create catalogs and move existing dbs to catalog during upgrade
[
https://issues.apache.org/jira/browse/HIVE-19135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453155#comment-16453155
]
Thejas M Nair commented on HIVE-19135:
--
[~alangates]
It would be good to have a 'create if not exists' semantics for the create
catalog. That way ambari can just run it in an idempotent fashion.
Also, should we do a "select .. for update" to get a lock on the table, so that
two concurrently running commands don't end up failing because they use same
CTLG_ID.
rs.getInt("CTLG_ID") would be more readable than rs.getInt(1)
getConnectionToMetastore(false) - pass true instead for useful debug info ?
> Need tool to allow admins to create catalogs and move existing dbs to catalog
> during upgrade
>
>
> Key: HIVE-19135
> URL: https://issues.apache.org/jira/browse/HIVE-19135
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 3.1.0
>
> Attachments: HIVE19135.patch
>
>
> As part of upgrading to Hive 3 admins may wish to create new catalogs and
> move some existing databases into those catalogs. We can do this by adding
> options to schematool. This guarantees that only admins can do these
> operations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19331) Repl load config in "with" clause not pass to Context.getStagingDir
[
https://issues.apache.org/jira/browse/HIVE-19331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16455577#comment-16455577
]
Thejas M Nair commented on HIVE-19331:
--
+1 pending tests
> Repl load config in "with" clause not pass to Context.getStagingDir
> ---
>
> Key: HIVE-19331
> URL: https://issues.apache.org/jira/browse/HIVE-19331
> Project: Hive
> Issue Type: Bug
> Components: repl
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-19331.1.patch
>
>
> Another failure similar to HIVE-18626, causing exception when s3 credentials
> are in "REPL LOAD" with clause.
> {code}
> Caused by: java.lang.IllegalStateException: Error getting FileSystem for
> s3a://nat-yc-r7-nmys-beacon-cloud-s3-2/hive_incremental_testing.db/hive_incremental_testing_new_tabl...:
> org.apache.hadoop.fs.s3a.AWSClientIOException: doesBucketExist on
> nat-yc-r7-nmys-beacon-cloud-s3-2: com.amazonaws.AmazonClientException: No AWS
> Credentials provided by BasicAWSCredentialsProvider
> EnvironmentVariableCredentialsProvider
> SharedInstanceProfileCredentialsProvider :
> com.amazonaws.AmazonClientException: Unable to load credentials from Amazon
> EC2 metadata service: No AWS Credentials provided by
> BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider
> SharedInstanceProfileCredentialsProvider :
> com.amazonaws.AmazonClientException: Unable to load credentials from Amazon
> EC2 metadata service
> at org.apache.hadoop.hive.ql.Context.getStagingDir(Context.java:359)
> at
> org.apache.hadoop.hive.ql.Context.getExternalScratchDir(Context.java:487)
> at
> org.apache.hadoop.hive.ql.Context.getExternalTmpPath(Context.java:565)
> at
> org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.loadTable(ImportSemanticAnalyzer.java:370)
> at
> org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.createReplImportTasks(ImportSemanticAnalyzer.java:926)
> at
> org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.prepareImport(ImportSemanticAnalyzer.java:329)
> at
> org.apache.hadoop.hive.ql.parse.repl.load.message.TableHandler.handle(TableHandler.java:43)
> ... 24 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19135) Need tool to allow admins to create catalogs and move existing dbs to catalog during upgrade
[ https://issues.apache.org/jira/browse/HIVE-19135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16456824#comment-16456824 ] Thejas M Nair commented on HIVE-19135: -- +1 > Need tool to allow admins to create catalogs and move existing dbs to catalog > during upgrade > > > Key: HIVE-19135 > URL: https://issues.apache.org/jira/browse/HIVE-19135 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Blocker > Fix For: 3.1.0 > > Attachments: HIVE-19135.2.patch, HIVE-19135.3.patch, HIVE19135.patch > > > As part of upgrading to Hive 3 admins may wish to create new catalogs and > move some existing databases into those catalogs. We can do this by adding > options to schematool. This guarantees that only admins can do these > operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19054) Function replication shall use "hive.repl.replica.functions.root.dir" as root
[ https://issues.apache.org/jira/browse/HIVE-19054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16457166#comment-16457166 ] Thejas M Nair commented on HIVE-19054: -- +1 pending tests > Function replication shall use "hive.repl.replica.functions.root.dir" as root > - > > Key: HIVE-19054 > URL: https://issues.apache.org/jira/browse/HIVE-19054 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19054.1.patch, HIVE-19054.2.patch, > HIVE-19054.3.patch, HIVE-19054.4.patch > > > It's wrongly use fs.defaultFS as the root, ignore > "hive.repl.replica.functions.root.dir" definition, thus prevent replicating > to cloud destination. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18988) Support bootstrap replication of ACID tables
[ https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16458838#comment-16458838 ] Thejas M Nair commented on HIVE-18988: -- +1 pending tests > Support bootstrap replication of ACID tables > > > Key: HIVE-18988 > URL: https://issues.apache.org/jira/browse/HIVE-18988 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-18988.01.patch, HIVE-18988.02.patch, > HIVE-18988.03.patch, HIVE-18988.04.patch, HIVE-18988.05.patch, > HIVE-18988.06.patch > > > Bootstrapping of ACID tables, need special handling to replicate a stable > state of data. > - If ACID feature enables, then perform bootstrap dump for ACID tables with > in read txn. > -> Dump table/partition metadata. > -> Get the list of valid data files for a table using same logic as read txn > do. > -> Dump latest ValidWriteIdList as per current read txn. > - Set the valid last replication state such that it doesn't miss any open > txn started after triggering bootstrap dump. > - If any txns on-going which was opened before triggering bootstrap dump, > then it is not guaranteed that if open_txn event captured for these txns. > Also, if these txns are opened for streaming ingest case, then dumped ACID > table data may include data of open txns which impact snapshot isolation at > target. To avoid that, bootstrap dump should wait for timeout (new > configuration: hive.repl.bootstrap.dump.open.txn.timeout). After timeout, > just force abort those txns and continue. > - If any txns force aborted belongs to a streaming ingest case, then dumped > ACID table data may have aborted data too. So, it is necessary to replicate > the aborted write ids to target to mark those data invalid for any readers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19135) Need tool to allow admins to create catalogs and move existing dbs to catalog during upgrade
[ https://issues.apache.org/jira/browse/HIVE-19135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459325#comment-16459325 ] Thejas M Nair commented on HIVE-19135: -- +1 pending tests However, I thought that connection/statement close would automatically rollback the txn. > Need tool to allow admins to create catalogs and move existing dbs to catalog > during upgrade > > > Key: HIVE-19135 > URL: https://issues.apache.org/jira/browse/HIVE-19135 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Blocker > Fix For: 3.1.0 > > Attachments: HIVE-19135.2.patch, HIVE-19135.3.patch, > HIVE-19135.4.patch, HIVE19135.patch > > > As part of upgrading to Hive 3 admins may wish to create new catalogs and > move some existing databases into those catalogs. We can do this by adding > options to schematool. This guarantees that only admins can do these > operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19120) catalog not properly set for some tables in SQL upgrade scripts
[ https://issues.apache.org/jira/browse/HIVE-19120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459326#comment-16459326 ] Thejas M Nair commented on HIVE-19120: -- +1 > catalog not properly set for some tables in SQL upgrade scripts > --- > > Key: HIVE-19120 > URL: https://issues.apache.org/jira/browse/HIVE-19120 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Blocker > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-19120.patch > > > A catalog column is added to the PARTITION_EVENTS and NOTIFICATION_LOG but > the upgrade scripts do not include an UPDATE statement to set this to the > default value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19381) Function replication in cloud fail when download resource from AWS
[ https://issues.apache.org/jira/browse/HIVE-19381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460493#comment-16460493 ] Thejas M Nair commented on HIVE-19381: -- +1 pending tests > Function replication in cloud fail when download resource from AWS > -- > > Key: HIVE-19381 > URL: https://issues.apache.org/jira/browse/HIVE-19381 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-19381.1.patch > > > Another case replication shall use the config in with clause. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19161) Add authorizations to information schema
[ https://issues.apache.org/jira/browse/HIVE-19161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461565#comment-16461565 ] Thejas M Nair commented on HIVE-19161: -- +1 pending tests > Add authorizations to information schema > > > Key: HIVE-19161 > URL: https://issues.apache.org/jira/browse/HIVE-19161 > Project: Hive > Issue Type: Improvement >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-19161.1.patch, HIVE-19161.10.patch, > HIVE-19161.11.patch, HIVE-19161.12.patch, HIVE-19161.2.patch, > HIVE-19161.3.patch, HIVE-19161.4.patch, HIVE-19161.5.patch, > HIVE-19161.6.patch, HIVE-19161.7.patch, HIVE-19161.8.patch, HIVE-19161.9.patch > > > We need to control the access of information schema so user can only query > the information authorized to. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19430) ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending events
[ https://issues.apache.org/jira/browse/HIVE-19430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair reassigned HIVE-19430: > ObjectStore.cleanNotificationEvents OutOfMemory on large number of pending > events > - > > Key: HIVE-19430 > URL: https://issues.apache.org/jira/browse/HIVE-19430 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Major > > If there are large number of events that haven't been cleaned up for some > reason, then ObjectStore.cleanNotificationEvents() can run out of memory > while it loads all the events to be deleted. > It should fetch events in batches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19248) REPL LOAD couldn't copy file from source CM path and also doesn't throw error if file copy fails.
[
https://issues.apache.org/jira/browse/HIVE-19248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16467568#comment-16467568
]
Thejas M Nair commented on HIVE-19248:
--
+1
> REPL LOAD couldn't copy file from source CM path and also doesn't throw error
> if file copy fails.
> -
>
> Key: HIVE-19248
> URL: https://issues.apache.org/jira/browse/HIVE-19248
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Blocker
> Labels: DR, pull-request-available, replication
> Fix For: 3.0.0, 3.1.0
>
> Attachments: HIVE-19248.01.patch, HIVE-19248.02.patch
>
>
> Hive replication uses Hadoop distcp to copy files from primary to replica
> warehouse. If the HDFS block size is different across clusters, it cause file
> copy failures.
> {code:java}
> 2018-04-09 14:32:06,690 ERROR [main]
> org.apache.hadoop.tools.mapred.CopyMapper: Failure in copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> java.io.IOException: File copy failed:
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> -->
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:299)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:266)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:52)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
> Caused by: java.io.IOException: Couldn't run retriable-command: Copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:296)
> ... 10 more
> Caused by: java.io.IOException: Check-sum mismatch between
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> and
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/.distcp.tmp.attempt_1522833620762_4416_m_00_0.
> Source and target differ in block-size. Use -pb to preserve block-sizes
> during copy. Alternatively, skip checksum-checks altogether, using -skipCrc.
> (NOTE: By skipping checksums, one runs the risk of masking data-corruption
> during file-transfer.)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:212)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:130)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> ... 11 more
> {code}
> Distcp failed as the CM path for the file doesn't point to source file
> system. So, it is needed to get the qualified cm root URI as part of files
> listed in dump.
> Also, REPL LOAD returns success even if distcp jobs failed.
> CopyUtils.doCopyRetry doesn't throw error if copy failed even after maximum
> attempts.
> So, need to perform 2 things.
> # If copy of multiple files fail for some reason, then retry with same set
> of files again but need to set CM path if original source file is missing or
> modified based on checksum. Let distcp to skip the properly copied files.
> FileUtil.copy will always overwrite the files.
> # If source path is moved to CM path, then delete the incorrectly copied
> files.
> # If copy fails for maximum attempt, then throw error.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19248) REPL LOAD couldn't copy file from source CM path and also doesn't throw error if file copy fails.
[
https://issues.apache.org/jira/browse/HIVE-19248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-19248:
-
Priority: Critical (was: Blocker)
> REPL LOAD couldn't copy file from source CM path and also doesn't throw error
> if file copy fails.
> -
>
> Key: HIVE-19248
> URL: https://issues.apache.org/jira/browse/HIVE-19248
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Critical
> Labels: DR, pull-request-available, replication
> Fix For: 3.0.0, 3.1.0
>
> Attachments: HIVE-19248.01-branch-3.patch, HIVE-19248.01.patch,
> HIVE-19248.02.patch
>
>
> Hive replication uses Hadoop distcp to copy files from primary to replica
> warehouse. If the HDFS block size is different across clusters, it cause file
> copy failures.
> {code:java}
> 2018-04-09 14:32:06,690 ERROR [main]
> org.apache.hadoop.tools.mapred.CopyMapper: Failure in copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> java.io.IOException: File copy failed:
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> -->
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:299)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:266)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:52)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
> Caused by: java.io.IOException: Couldn't run retriable-command: Copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:296)
> ... 10 more
> Caused by: java.io.IOException: Check-sum mismatch between
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> and
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/.distcp.tmp.attempt_1522833620762_4416_m_00_0.
> Source and target differ in block-size. Use -pb to preserve block-sizes
> during copy. Alternatively, skip checksum-checks altogether, using -skipCrc.
> (NOTE: By skipping checksums, one runs the risk of masking data-corruption
> during file-transfer.)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:212)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:130)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> ... 11 more
> {code}
> Distcp failed as the CM path for the file doesn't point to source file
> system. So, it is needed to get the qualified cm root URI as part of files
> listed in dump.
> Also, REPL LOAD returns success even if distcp jobs failed.
> CopyUtils.doCopyRetry doesn't throw error if copy failed even after maximum
> attempts.
> So, need to perform 2 things.
> # If copy of multiple files fail for some reason, then retry with same set
> of files again but need to set CM path if original source file is missing or
> modified based on checksum. Let distcp to skip the properly copied files.
> FileUtil.copy will always overwrite the files.
> # If source path is moved to CM path, then delete the incorrectly copied
> files.
> # If copy fails for maximum attempt, then throw error.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19248) REPL LOAD couldn't copy file from source CM path and also doesn't throw error if file copy fails.
[
https://issues.apache.org/jira/browse/HIVE-19248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16469223#comment-16469223
]
Thejas M Nair commented on HIVE-19248:
--
[~sankarh]
Lets not block the 3.0.0 release for this one. If test results look good, we
can get it in.
Otherwise, it can go into a 3.0.1 release.
> REPL LOAD couldn't copy file from source CM path and also doesn't throw error
> if file copy fails.
> -
>
> Key: HIVE-19248
> URL: https://issues.apache.org/jira/browse/HIVE-19248
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Critical
> Labels: DR, pull-request-available, replication
> Fix For: 3.0.0, 3.1.0
>
> Attachments: HIVE-19248.01-branch-3.patch, HIVE-19248.01.patch,
> HIVE-19248.02.patch
>
>
> Hive replication uses Hadoop distcp to copy files from primary to replica
> warehouse. If the HDFS block size is different across clusters, it cause file
> copy failures.
> {code:java}
> 2018-04-09 14:32:06,690 ERROR [main]
> org.apache.hadoop.tools.mapred.CopyMapper: Failure in copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> java.io.IOException: File copy failed:
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> -->
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:299)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:266)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:52)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
> Caused by: java.io.IOException: Couldn't run retriable-command: Copying
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0 to
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/000259_0
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:296)
> ... 10 more
> Caused by: java.io.IOException: Check-sum mismatch between
> hdfs://chelsea/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/000259_0
> and
> hdfs://marilyn/apps/hive/warehouse/tpch_flat_orc_1000.db/customer/.hive-staging_hive_2018-04-09_14-30-45_723_7153496419225102220-2/-ext-10001/.distcp.tmp.attempt_1522833620762_4416_m_00_0.
> Source and target differ in block-size. Use -pb to preserve block-sizes
> during copy. Alternatively, skip checksum-checks altogether, using -skipCrc.
> (NOTE: By skipping checksums, one runs the risk of masking data-corruption
> during file-transfer.)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:212)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:130)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> ... 11 more
> {code}
> Distcp failed as the CM path for the file doesn't point to source file
> system. So, it is needed to get the qualified cm root URI as part of files
> listed in dump.
> Also, REPL LOAD returns success even if distcp jobs failed.
> CopyUtils.doCopyRetry doesn't throw error if copy failed even after maximum
> attempts.
> So, need to perform 2 things.
> # If copy of multiple files fail for some reason, then retry with same set
> of files again but need to set CM path if original source file is missing or
> modified based on checksum. Let distcp to skip the properly copied files.
> FileUtil.copy will always overwrite the files.
> # If source path is moved to CM path, then delete the incorrectly
[jira] [Updated] (HIVE-19440) Make StorageBasedAuthorizer work with information schema
[ https://issues.apache.org/jira/browse/HIVE-19440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-19440: - Fix Version/s: 3.0.0 > Make StorageBasedAuthorizer work with information schema > > > Key: HIVE-19440 > URL: https://issues.apache.org/jira/browse/HIVE-19440 > Project: Hive > Issue Type: Improvement >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Blocker > Fix For: 3.0.0 > > > With HIVE-19161, Hive information schema works with external authorizer (such > as ranger). However, we also need to make StorageBasedAuthorizer > synchronization work as it is also widely use. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19440) Make StorageBasedAuthorizer work with information schema
[ https://issues.apache.org/jira/browse/HIVE-19440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16469269#comment-16469269 ] Thejas M Nair commented on HIVE-19440: -- [~vgarg] This needs a DB change, so I think we should get this into 3.0 release (instead of a 3.0.1) release. We should have a patch very soon. > Make StorageBasedAuthorizer work with information schema > > > Key: HIVE-19440 > URL: https://issues.apache.org/jira/browse/HIVE-19440 > Project: Hive > Issue Type: Improvement >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Blocker > Fix For: 3.0.0 > > > With HIVE-19161, Hive information schema works with external authorizer (such > as ranger). However, we also need to make StorageBasedAuthorizer > synchronization work as it is also widely use. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18193) Migrate existing ACID tables to use write id per table rather than global transaction id
[ https://issues.apache.org/jira/browse/HIVE-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16469277#comment-16469277 ] Thejas M Nair commented on HIVE-18193: -- +1 > Migrate existing ACID tables to use write id per table rather than global > transaction id > > > Key: HIVE-18193 > URL: https://issues.apache.org/jira/browse/HIVE-18193 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Blocker > Labels: ACID, Upgrade > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-18193.01.patch, HIVE-18193.02.patch > > > dependent upon HIVE-18192 > For existing ACID Tables we need to update the table level write id > metatables/sequences so any new operations on these tables works seamlessly > without any conflicting data in existing base/delta files. > 1. Need to create metadata tables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID. > 2. Add entries for each ACID/MM tables into NEXT_WRITE_ID where NWI_NEXT is > set to current value of NEXT_TXN_ID.NTXN_NEXT. > 3. All current open/abort transactions to have an entry in TXN_TO_WRITE_ID > such that T2W_TXNID=T2W_WRITEID=Open/AbortedTxnId. > 4. Added new column TC_WRITEID in TXN_COMPONENTS and CTC_WRITEID in > COMPLETED_TXN_COMPONENTS to store the write id which should be set as > respective values of TC_TXNID and CTC_TXNID from the same row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19435) Incremental replication cause data loss if a table is dropped followed by create and insert-into with different partition type.
[ https://issues.apache.org/jira/browse/HIVE-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472165#comment-16472165 ] Thejas M Nair commented on HIVE-19435: -- +1 > Incremental replication cause data loss if a table is dropped followed by > create and insert-into with different partition type. > --- > > Key: HIVE-19435 > URL: https://issues.apache.org/jira/browse/HIVE-19435 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-19435.01.patch, HIVE-19435.02.patch, > HIVE-19435.03.patch > > > If the incremental dump have drop of partitioned table followed by > create/insert on non-partitioned table with same name, doesn't replicate the > data. Explained below. > Let's say we have a partitioned table T1 which was already replicated to > target. > DROP_TABLE(T1)->CREATE_TABLE(T1) (Non-partitioned) -> INSERT(T1)(10) > After REPL LOAD, T1 doesn't have any data. > Same is valid for non-partitioned to partitioned and partition spec mismatch > case as well. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19528) Beeline: When beeline-site.xml is present and the default named url is incorrect, throw an exception instead of relying on resolution via hive-site.xml/beeline-hs2-conn
[ https://issues.apache.org/jira/browse/HIVE-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16479847#comment-16479847 ] Thejas M Nair commented on HIVE-19528: -- bq. miniHS2.getBaseJdbcURL() + "default;user=hive;password=hive;transportMode=http;httpPath=cliservice"; This could use the constants used to set configs in HS2 startup (HS2_HTTP_ENDPOINT, HS2_HTTP_MODE). Can you add a negative test also, since this test is improving the error handling ? > Beeline: When beeline-site.xml is present and the default named url is > incorrect, throw an exception instead of relying on resolution via > hive-site.xml/beeline-hs2-connection.xml > --- > > Key: HIVE-19528 > URL: https://issues.apache.org/jira/browse/HIVE-19528 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 3.0.0, 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-19528.1.patch, HIVE-19528.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19609) pointless callstacks in the logs as usual
[ https://issues.apache.org/jira/browse/HIVE-19609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16481208#comment-16481208 ] Thejas M Nair commented on HIVE-19609: -- +1 Thanks! This is a patch I wished for many many times, finally granted by Sergey! :) > pointless callstacks in the logs as usual > - > > Key: HIVE-19609 > URL: https://issues.apache.org/jira/browse/HIVE-19609 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19609.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19632) Remove webapps directory from standalone jar
[ https://issues.apache.org/jira/browse/HIVE-19632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16483100#comment-16483100 ] Thejas M Nair commented on HIVE-19632: -- +1 > Remove webapps directory from standalone jar > > > Key: HIVE-19632 > URL: https://issues.apache.org/jira/browse/HIVE-19632 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19632.1.patch, HIVE-19632.2.patch > > > JDBC standalone jar contains webapps static files which just adds to the jar > size and are not required by the clients. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19639) a transactional Hive table cannot be imported as an external table
[ https://issues.apache.org/jira/browse/HIVE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16483236#comment-16483236 ] Thejas M Nair commented on HIVE-19639: -- +1 > a transactional Hive table cannot be imported as an external table > -- > > Key: HIVE-19639 > URL: https://issues.apache.org/jira/browse/HIVE-19639 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19639.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19639) a transactional Hive table cannot be imported as an external table
[ https://issues.apache.org/jira/browse/HIVE-19639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-19639: - Description: When transactional table is imported to a external table, the table should be imported as a non transactional table, as external tables cannot be transactional. > a transactional Hive table cannot be imported as an external table > -- > > Key: HIVE-19639 > URL: https://issues.apache.org/jira/browse/HIVE-19639 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19639.patch > > > When transactional table is imported to a external table, the table should be > imported as a non transactional table, as external tables cannot be > transactional. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19531) TransactionalValidationListener is getting catalog name from conf instead of table object.
[ https://issues.apache.org/jira/browse/HIVE-19531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16484333#comment-16484333 ] Thejas M Nair commented on HIVE-19531: -- +1 > TransactionalValidationListener is getting catalog name from conf instead of > table object. > -- > > Key: HIVE-19531 > URL: https://issues.apache.org/jira/browse/HIVE-19531 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Blocker > Fix For: 3.0.1 > > Attachments: HIVE-19531.1again.patch, HIVE-19531.1take3.patch, > HIVE-19531.1take4.patch, HIVE-19531.patch > > > TransactionalValidationListener.validateTableStructure get the catalog from > the conf file rather than taking it from the passed in table structure. This > causes createTable operations to fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19631) reduce epic locking in AbstractService
[ https://issues.apache.org/jira/browse/HIVE-19631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16484453#comment-16484453 ] Thejas M Nair commented on HIVE-19631: -- AbstractService.getHiveConf has been synchronized since the very first commit of HS2. But I don't see any reason why needs to be synchronized. I think we can just remove the synchronization of it and register/unregister. These methods on the service are not called concurrently from different threads. getDelegationTokenFromMetaStore can also be made not synchronous. Its making a remote api call with thread local Hive object. > reduce epic locking in AbstractService > -- > > Key: HIVE-19631 > URL: https://issues.apache.org/jira/browse/HIVE-19631 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19631.patch > > > Some services have synchronized method that do lengthy remote calls, and > these block everyone trying to e.g. getHiveConf. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19666) SQL standard auth for create fn may make an impossible privilege check (branch-2)
[ https://issues.apache.org/jira/browse/HIVE-19666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16489814#comment-16489814 ] Thejas M Nair commented on HIVE-19666: -- yeah, copying the jar to test temp dir might do the trick. The actual code changes look good to me. > SQL standard auth for create fn may make an impossible privilege check > (branch-2) > - > > Key: HIVE-19666 > URL: https://issues.apache.org/jira/browse/HIVE-19666 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19666.01-branch-2.patch, HIVE-19666.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19631) reduce epic locking in AbstractService
[ https://issues.apache.org/jira/browse/HIVE-19631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16489818#comment-16489818 ] Thejas M Nair commented on HIVE-19631: -- +1 > reduce epic locking in AbstractService > -- > > Key: HIVE-19631 > URL: https://issues.apache.org/jira/browse/HIVE-19631 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19631.01.patch, HIVE-19631.patch > > > Some services have synchronized method that do lengthy remote calls, and > these block everyone trying to e.g. getHiveConf. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19666) SQL standard auth for create fn may make an impossible privilege check (branch-2)
[ https://issues.apache.org/jira/browse/HIVE-19666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16489901#comment-16489901 ] Thejas M Nair commented on HIVE-19666: -- +1 pending tests > SQL standard auth for create fn may make an impossible privilege check > (branch-2) > - > > Key: HIVE-19666 > URL: https://issues.apache.org/jira/browse/HIVE-19666 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19666.01-branch-2.patch, > HIVE-19666.02-branch-2.patch, HIVE-19666.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18940) Hive notifications serialize all write DDL operations
[
https://issues.apache.org/jira/browse/HIVE-18940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16405643#comment-16405643
]
Thejas M Nair commented on HIVE-18940:
--
[~vihangk1]'s approach of accumulating the events and only getting the lock
towards end seems like a reasonable way to reduce the duration lock is held to
a very small time, for metastore calls that lead to several events. Other parts
of the transactions can go ahead in parallel.
It doesn't have to use same commitID for all events, getting new values should
be OK, as long as the lock on NOTIFICATION_SEQUENCE is obtained at the end.
> Hive notifications serialize all write DDL operations
> -
>
> Key: HIVE-18940
> URL: https://issues.apache.org/jira/browse/HIVE-18940
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alexander Kolbasov
>Priority: Major
>
> The implementation of DbNotificationListener uses a single row to store
> current notification ID and uses {{SELECT FOR UPDATE}} to lock the row. This
> serializes all write DDL operations which isn't good.
> We should consider using database auto-increment for notification ID instead.
> Especially on mMySQL/innoDb it is supported natively with relatively
> light-weight locking.
> This creates potential issue for consumers though because such IDs may have
> holes. There are two types of holes - transient hole for a transaction which
> have not committed yet and will be committed shortly and permanent holes for
> transactions that fail. Consumers need to deal with it. It may be useful to
> add DB-generated timestamp as well to assist in recovery from holes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19012) Support ARM and PPC arch
[ https://issues.apache.org/jira/browse/HIVE-19012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair reassigned HIVE-19012: Assignee: Vi On > Support ARM and PPC arch > > > Key: HIVE-19012 > URL: https://issues.apache.org/jira/browse/HIVE-19012 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Vi On >Assignee: Vi On >Priority: Major > Attachments: HIVE-19012.patch > > > Hive standalone metastore uses protoc-jar-maven-plugin 3.5.1.1 which supports > downloading from maven repo. Artifact download should be supported for ARM > and PPC architecture since some protobuf versions do not exist in ARM/PPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19012) Support builds for ARM and PPC arch
[ https://issues.apache.org/jira/browse/HIVE-19012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-19012: - Status: Patch Available (was: Open) > Support builds for ARM and PPC arch > --- > > Key: HIVE-19012 > URL: https://issues.apache.org/jira/browse/HIVE-19012 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Vi On >Assignee: Thejas M Nair >Priority: Major > Attachments: HIVE-19012.patch > > > Hive standalone metastore uses protoc-jar-maven-plugin 3.5.1.1 which supports > downloading from maven repo. Artifact download should be supported for ARM > and PPC architecture since some protobuf versions do not exist in ARM/PPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19012) Support builds for ARM and PPC arch
[ https://issues.apache.org/jira/browse/HIVE-19012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408561#comment-16408561 ] Thejas M Nair commented on HIVE-19012: -- +1 This change enables download from local maven repo if available there. > Support builds for ARM and PPC arch > --- > > Key: HIVE-19012 > URL: https://issues.apache.org/jira/browse/HIVE-19012 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Vi On >Assignee: Vi On >Priority: Major > Attachments: HIVE-19012.patch > > > Hive standalone metastore uses protoc-jar-maven-plugin 3.5.1.1 which supports > downloading from maven repo. Artifact download should be supported for ARM > and PPC architecture since some protobuf versions do not exist in ARM/PPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19012) Support builds for ARM and PPC arch
[ https://issues.apache.org/jira/browse/HIVE-19012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair reassigned HIVE-19012: Assignee: Thejas M Nair (was: Vi On) > Support builds for ARM and PPC arch > --- > > Key: HIVE-19012 > URL: https://issues.apache.org/jira/browse/HIVE-19012 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Vi On >Assignee: Thejas M Nair >Priority: Major > Attachments: HIVE-19012.patch > > > Hive standalone metastore uses protoc-jar-maven-plugin 3.5.1.1 which supports > downloading from maven repo. Artifact download should be supported for ARM > and PPC architecture since some protobuf versions do not exist in ARM/PPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19012) Support builds for ARM and PPC arch
[ https://issues.apache.org/jira/browse/HIVE-19012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-19012: - Summary: Support builds for ARM and PPC arch (was: Support ARM and PPC arch) > Support builds for ARM and PPC arch > --- > > Key: HIVE-19012 > URL: https://issues.apache.org/jira/browse/HIVE-19012 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Vi On >Assignee: Vi On >Priority: Major > Attachments: HIVE-19012.patch > > > Hive standalone metastore uses protoc-jar-maven-plugin 3.5.1.1 which supports > downloading from maven repo. Artifact download should be supported for ARM > and PPC architecture since some protobuf versions do not exist in ARM/PPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19012) Support builds for ARM and PPC arch
[ https://issues.apache.org/jira/browse/HIVE-19012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair reassigned HIVE-19012: Assignee: Vi On (was: Thejas M Nair) > Support builds for ARM and PPC arch > --- > > Key: HIVE-19012 > URL: https://issues.apache.org/jira/browse/HIVE-19012 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Vi On >Assignee: Vi On >Priority: Major > Attachments: HIVE-19012.patch > > > Hive standalone metastore uses protoc-jar-maven-plugin 3.5.1.1 which supports > downloading from maven repo. Artifact download should be supported for ARM > and PPC architecture since some protobuf versions do not exist in ARM/PPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19012) Support builds for ARM and PPC arch
[ https://issues.apache.org/jira/browse/HIVE-19012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408561#comment-16408561 ] Thejas M Nair edited comment on HIVE-19012 at 3/21/18 8:47 PM: --- +1 pending tests This change enables download from local maven repo if available there. was (Author: thejas): +1 This change enables download from local maven repo if available there. > Support builds for ARM and PPC arch > --- > > Key: HIVE-19012 > URL: https://issues.apache.org/jira/browse/HIVE-19012 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Vi On >Assignee: Vi On >Priority: Major > Attachments: HIVE-19012.patch > > > Hive standalone metastore uses protoc-jar-maven-plugin 3.5.1.1 which supports > downloading from maven repo. Artifact download should be supported for ARM > and PPC architecture since some protobuf versions do not exist in ARM/PPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18967) Standalone metastore SQL upgrade scripts do not properly set schema version
[ https://issues.apache.org/jira/browse/HIVE-18967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409970#comment-16409970 ] Thejas M Nair commented on HIVE-18967: -- +1 > Standalone metastore SQL upgrade scripts do not properly set schema version > --- > > Key: HIVE-18967 > URL: https://issues.apache.org/jira/browse/HIVE-18967 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18967.patch > > > The new combined upgrade scripts for Hive 2.3 to 3.0 transition do not > properly set the schema version after they have completed the upgrade. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19007) Support REPL LOAD from primary using replica connection configurations received through WITH clause.
[ https://issues.apache.org/jira/browse/HIVE-19007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410843#comment-16410843 ] Thejas M Nair commented on HIVE-19007: -- +1 > Support REPL LOAD from primary using replica connection configurations > received through WITH clause. > > > Key: HIVE-19007 > URL: https://issues.apache.org/jira/browse/HIVE-19007 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 3.0.0 > > Attachments: HIVE-19007.01.patch > > > Need to support running REPL LOAD command from primary for different > use-cases such as Cloud replication (for efficient use of cloud resources) or > workload management. > To achieve this, WITH clause of REPL LOAD lets user to pass Hive configs such > as hive.metastore.warehouse.dir, hive.metastore.uris, > hive.repl.replica.functions.root.dir etc, which can be used to establish > connection with replica warehouse. > The configs received from WITH clause of REPL LOAD are not set properly (due > to changes by HIVE-18716) to the tasks created. It is also required to re-get > the Hive db object if the configs are changed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18885) DbNotificationListener has a deadlock between Java and DB locks (2.x line)
[
https://issues.apache.org/jira/browse/HIVE-18885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-18885:
-
Summary: DbNotificationListener has a deadlock between Java and DB locks
(2.x line) (was: DbNotificationListener has a deadlock between Java and DB
locks)
> DbNotificationListener has a deadlock between Java and DB locks (2.x line)
> --
>
> Key: HIVE-18885
> URL: https://issues.apache.org/jira/browse/HIVE-18885
> Project: Hive
> Issue Type: Bug
> Components: Hive, Metastore
>Affects Versions: 2.3.2
>Reporter: Alexander Kolbasov
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-18885.01.branch-2.patch
>
>
> You can see the problem from looking at the code, but it actually created
> severe problems for real life Hive user.
> When {{alter table}} has {{cascade}} option it does the following:
> {code:java}
> msdb.openTransaction()
> ...
> List parts = msdb.getPartitions(dbname, name, -1);
> for (Partition part : parts) {
> List oldCols = part.getSd().getCols();
> part.getSd().setCols(newt.getSd().getCols());
> String oldPartName =
> Warehouse.makePartName(oldt.getPartitionKeys(), part.getValues());
> updatePartColumnStatsForAlterColumns(msdb, part, oldPartName,
> part.getValues(), oldCols, part);
> msdb.alterPartition(dbname, name, part.getValues(), part);
> }
> {code}
> So it walks all partitions (and this may be huge list) and does some
> non-trivial operations in one single uber-transaction.
> When DbNotificationListener is enabled, it adds an event for each partition,
> all while
> holding a row lock on NOTIFICATION_SEQUENCE table. As a result, while this is
> happening no other write DDL can proceed. This can sometimes cause DB lock
> timeouts which cause HMS level operation retries which make things even worse.
> In one particular case this pretty much made HMS unusable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs
[ https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16412282#comment-16412282 ] Thejas M Nair commented on HIVE-18755: -- On the lines of what Alexander said, I think it makes sense to create a GetCatalogReq object and DropCatalogReq objects for use in get_catalog and drop_catalog methods respectively. You might have arguments that are specific to drop_catalog in future (eg "cascade=true"). Similarly even for create_catalog, having a CreateCatalogReq object would help to keep it future proof. Now thinking on those lines, I think its simpler are more consistent to use *Req and *Response objects as far as possible. > Modifications to the metastore for catalogs > --- > > Key: HIVE-18755 > URL: https://issues.apache.org/jira/browse/HIVE-18755 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18755.2.patch, HIVE-18755.nothrift, HIVE-18755.patch > > > Step 1 of adding catalogs is to add support in the metastore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs
[ https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413236#comment-16413236 ] Thejas M Nair commented on HIVE-18755: -- [~vgumashta] [~daijy] Note that this removes use of ConcurrentMaps with non concurrent ones in SharedCache. But with the read/write locking that is already in place, that seems to be fine. > Modifications to the metastore for catalogs > --- > > Key: HIVE-18755 > URL: https://issues.apache.org/jira/browse/HIVE-18755 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18755.2.patch, HIVE-18755.nothrift, HIVE-18755.patch > > > Step 1 of adding catalogs is to add support in the metastore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs
[ https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413437#comment-16413437 ] Thejas M Nair commented on HIVE-18755: -- I have added some comments in github pull request. > Modifications to the metastore for catalogs > --- > > Key: HIVE-18755 > URL: https://issues.apache.org/jira/browse/HIVE-18755 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18755.2.patch, HIVE-18755.nothrift, HIVE-18755.patch > > > Step 1 of adding catalogs is to add support in the metastore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19012) Support builds for ARM and PPC arch
[ https://issues.apache.org/jira/browse/HIVE-19012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair reassigned HIVE-19012: Assignee: Vi On (was: Thejas M Nair) > Support builds for ARM and PPC arch > --- > > Key: HIVE-19012 > URL: https://issues.apache.org/jira/browse/HIVE-19012 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Vi On >Assignee: Vi On >Priority: Major > Attachments: HIVE-19012.patch, HIVE-19012.patch > > > Hive standalone metastore uses protoc-jar-maven-plugin 3.5.1.1 which supports > downloading from maven repo. Artifact download should be supported for ARM > and PPC architecture since some protobuf versions do not exist in ARM/PPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19012) Support builds for ARM and PPC arch
[ https://issues.apache.org/jira/browse/HIVE-19012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-19012: - Attachment: HIVE-19012.patch > Support builds for ARM and PPC arch > --- > > Key: HIVE-19012 > URL: https://issues.apache.org/jira/browse/HIVE-19012 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Vi On >Assignee: Thejas M Nair >Priority: Major > Attachments: HIVE-19012.patch, HIVE-19012.patch > > > Hive standalone metastore uses protoc-jar-maven-plugin 3.5.1.1 which supports > downloading from maven repo. Artifact download should be supported for ARM > and PPC architecture since some protobuf versions do not exist in ARM/PPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19012) Support builds for ARM and PPC arch
[ https://issues.apache.org/jira/browse/HIVE-19012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair reassigned HIVE-19012: Assignee: Thejas M Nair (was: Vi On) > Support builds for ARM and PPC arch > --- > > Key: HIVE-19012 > URL: https://issues.apache.org/jira/browse/HIVE-19012 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Vi On >Assignee: Thejas M Nair >Priority: Major > Attachments: HIVE-19012.patch, HIVE-19012.patch > > > Hive standalone metastore uses protoc-jar-maven-plugin 3.5.1.1 which supports > downloading from maven repo. Artifact download should be supported for ARM > and PPC architecture since some protobuf versions do not exist in ARM/PPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19012) Support builds for ARM and PPC arch
[ https://issues.apache.org/jira/browse/HIVE-19012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415946#comment-16415946 ] Thejas M Nair commented on HIVE-19012: -- Attaching file again to kick off tests > Support builds for ARM and PPC arch > --- > > Key: HIVE-19012 > URL: https://issues.apache.org/jira/browse/HIVE-19012 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Vi On >Assignee: Vi On >Priority: Major > Attachments: HIVE-19012.patch, HIVE-19012.patch > > > Hive standalone metastore uses protoc-jar-maven-plugin 3.5.1.1 which supports > downloading from maven repo. Artifact download should be supported for ARM > and PPC architecture since some protobuf versions do not exist in ARM/PPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19054) Function replication shall use "hive.repl.replica.functions.root.dir" as root
[ https://issues.apache.org/jira/browse/HIVE-19054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16416833#comment-16416833 ] Thejas M Nair commented on HIVE-19054: -- +1 > Function replication shall use "hive.repl.replica.functions.root.dir" as root > - > > Key: HIVE-19054 > URL: https://issues.apache.org/jira/browse/HIVE-19054 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-19054.1.patch > > > It's wrongly use fs.defaultFS as the root, ignore > "hive.repl.replica.functions.root.dir" definition, thus prevent replicating > to cloud destination. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19054) Function replication shall use "hive.repl.replica.functions.root.dir" as root
[ https://issues.apache.org/jira/browse/HIVE-19054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair reassigned HIVE-19054: Assignee: Thejas M Nair (was: Daniel Dai) > Function replication shall use "hive.repl.replica.functions.root.dir" as root > - > > Key: HIVE-19054 > URL: https://issues.apache.org/jira/browse/HIVE-19054 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Thejas M Nair >Priority: Major > Attachments: HIVE-19054.1.patch > > > It's wrongly use fs.defaultFS as the root, ignore > "hive.repl.replica.functions.root.dir" definition, thus prevent replicating > to cloud destination. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19054) Function replication shall use "hive.repl.replica.functions.root.dir" as root
[ https://issues.apache.org/jira/browse/HIVE-19054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-19054: - Assignee: Daniel Dai (was: Thejas M Nair) Status: Patch Available (was: Open) > Function replication shall use "hive.repl.replica.functions.root.dir" as root > - > > Key: HIVE-19054 > URL: https://issues.apache.org/jira/browse/HIVE-19054 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-19054.1.patch > > > It's wrongly use fs.defaultFS as the root, ignore > "hive.repl.replica.functions.root.dir" definition, thus prevent replicating > to cloud destination. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19065) Metastore client compatibility check should include syncMetaStoreClient
[ https://issues.apache.org/jira/browse/HIVE-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16418165#comment-16418165 ] Thejas M Nair commented on HIVE-19065: -- +1 > Metastore client compatibility check should include syncMetaStoreClient > --- > > Key: HIVE-19065 > URL: https://issues.apache.org/jira/browse/HIVE-19065 > Project: Hive > Issue Type: Bug >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-19065.1.patch > > > I saw a case Hive.get(HiveConf c) reuse syncMetaStoreClient with different > config (in my case, hive.metastore.uris is different), which makes > syncMetaStoreClient connect to wrong metastore server. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19033) Provide an option to purge LLAP IO cache
[ https://issues.apache.org/jira/browse/HIVE-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16418214#comment-16418214 ] Thejas M Nair commented on HIVE-19033: -- Security aspect looks good to me. Ranger implements the HiveAuthorizer class and it would need additional change to handle the two new operation types. > Provide an option to purge LLAP IO cache > > > Key: HIVE-19033 > URL: https://issues.apache.org/jira/browse/HIVE-19033 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19033.1.patch, HIVE-19033.2.patch, > HIVE-19033.3.patch, HIVE-19033.4.patch > > > Provide an API endpoint that will trigger purging of LLAP IO cache. Also CLI > tool to invoke the endpoint of all LLAP daemons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19012) Support builds for ARM and PPC arch
[ https://issues.apache.org/jira/browse/HIVE-19012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-19012: - Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Thanks for the patch [~vilamon]! Committed to master. > Support builds for ARM and PPC arch > --- > > Key: HIVE-19012 > URL: https://issues.apache.org/jira/browse/HIVE-19012 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Vi On >Assignee: Vi On >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19012.patch, HIVE-19012.patch > > > Hive standalone metastore uses protoc-jar-maven-plugin 3.5.1.1 which supports > downloading from maven repo. Artifact download should be supported for ARM > and PPC architecture since some protobuf versions do not exist in ARM/PPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18841) Support authorization of UDF usage in hive
[ https://issues.apache.org/jira/browse/HIVE-18841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-18841: - Attachment: HIVE-18841.1.patch > Support authorization of UDF usage in hive > -- > > Key: HIVE-18841 > URL: https://issues.apache.org/jira/browse/HIVE-18841 > Project: Hive > Issue Type: New Feature >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Critical > Attachments: HIVE-18841.1.patch > > > It should be possible to create authorization policies on UDF usage. > ie, it should be possible to control who can use certain UDF in their queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-18841) Support authorization of UDF usage in hive
[ https://issues.apache.org/jira/browse/HIVE-18841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-18841 started by Thejas M Nair. > Support authorization of UDF usage in hive > -- > > Key: HIVE-18841 > URL: https://issues.apache.org/jira/browse/HIVE-18841 > Project: Hive > Issue Type: New Feature >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Critical > Attachments: HIVE-18841.1.patch > > > It should be possible to create authorization policies on UDF usage. > ie, it should be possible to control who can use certain UDF in their queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work stopped] (HIVE-18841) Support authorization of UDF usage in hive
[ https://issues.apache.org/jira/browse/HIVE-18841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-18841 stopped by Thejas M Nair. > Support authorization of UDF usage in hive > -- > > Key: HIVE-18841 > URL: https://issues.apache.org/jira/browse/HIVE-18841 > Project: Hive > Issue Type: New Feature >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Critical > Attachments: HIVE-18841.1.patch > > > It should be possible to create authorization policies on UDF usage. > ie, it should be possible to control who can use certain UDF in their queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18841) Support authorization of UDF usage in hive
[ https://issues.apache.org/jira/browse/HIVE-18841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-18841: - Status: Patch Available (was: Open) > Support authorization of UDF usage in hive > -- > > Key: HIVE-18841 > URL: https://issues.apache.org/jira/browse/HIVE-18841 > Project: Hive > Issue Type: New Feature >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Critical > Attachments: HIVE-18841.1.patch > > > It should be possible to create authorization policies on UDF usage. > ie, it should be possible to control who can use certain UDF in their queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18841) Support authorization of UDF usage in hive
[ https://issues.apache.org/jira/browse/HIVE-18841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419636#comment-16419636 ] Thejas M Nair commented on HIVE-18841: -- Note that only permanent UDF are passed with this patch for authorization. That would address the primary use cases for this feature (encryption related UDFs provided by 3rd party vendors/cluster admins). It doesn't make sense to do usage authorization for temporary UDF as the user already has to have permissions to create them (you have to create it every time to use it). In case of built-in UDF, a lot of these UDFs are implicitly used (eg. cast one type to another), so it doesn't make sense to authorize their usage, as it would be very confusing for users. > Support authorization of UDF usage in hive > -- > > Key: HIVE-18841 > URL: https://issues.apache.org/jira/browse/HIVE-18841 > Project: Hive > Issue Type: New Feature >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Critical > Attachments: HIVE-18841.1.patch > > > It should be possible to create authorization policies on UDF usage. > ie, it should be possible to control who can use certain UDF in their queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18841) Support authorization of UDF usage in hive
[ https://issues.apache.org/jira/browse/HIVE-18841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419994#comment-16419994 ] Thejas M Nair commented on HIVE-18841: -- I think we can add that as well in follow up patch, if we see asks for this. The current ask from users that we have seen is to be able to allow/deny access to permanent functions (eg encryption related ones), by name of the functions. > Support authorization of UDF usage in hive > -- > > Key: HIVE-18841 > URL: https://issues.apache.org/jira/browse/HIVE-18841 > Project: Hive > Issue Type: New Feature >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Critical > Attachments: HIVE-18841.1.patch > > > It should be possible to create authorization policies on UDF usage. > ie, it should be possible to control who can use certain UDF in their queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs
[ https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16420877#comment-16420877 ] Thejas M Nair commented on HIVE-18755: -- +1 pending tests It looks like the metastore client getTableMeta is not getting filtered any more by the authorization hook. filterNames method was being used to filter that, but that has been removed. There is a new filterHook.filterTableMetas method being used, but that is not implemented in AuthorizationMetaStoreFilterHook. We can take that as follow up jira since this is a very large patch that can go out of sync again soon. > Modifications to the metastore for catalogs > --- > > Key: HIVE-18755 > URL: https://issues.apache.org/jira/browse/HIVE-18755 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18755.2.patch, HIVE-18755.3.patch, > HIVE-18755.4.patch, HIVE-18755.nothrift, HIVE-18755.patch > > > Step 1 of adding catalogs is to add support in the metastore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18755) Modifications to the metastore for catalogs
[ https://issues.apache.org/jira/browse/HIVE-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16420929#comment-16420929 ] Thejas M Nair commented on HIVE-18755: -- [~akolb] The github pull request is https://github.com/apache/hive/pull/320 > Modifications to the metastore for catalogs > --- > > Key: HIVE-18755 > URL: https://issues.apache.org/jira/browse/HIVE-18755 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18755.2.patch, HIVE-18755.3.patch, > HIVE-18755.4.patch, HIVE-18755.nothrift, HIVE-18755.patch > > > Step 1 of adding catalogs is to add support in the metastore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18963) JDBC: Provide an option to simplify beeline usage by supporting default and named URL for beeline
[
https://issues.apache.org/jira/browse/HIVE-18963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16421953#comment-16421953
]
Thejas M Nair commented on HIVE-18963:
--
+1 pending tests
> JDBC: Provide an option to simplify beeline usage by supporting default and
> named URL for beeline
> -
>
> Key: HIVE-18963
> URL: https://issues.apache.org/jira/browse/HIVE-18963
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-18963.1.patch, HIVE-18963.2.patch,
> HIVE-18963.3.patch
>
>
> Currently, after opening Beeline CLI, the user needs to supply a connection
> string to use the HS2 instance and set up the jdbc driver. Since we plan to
> replace Hive CLI with Beeline in future (HIVE-10511), it will help the
> usability if the user can simply type {{beeline}} and get start the hive
> session. The jdbc url can be specified in a beeline-site.xml (which can
> contain other named jdbc urls as well, and they can be accessed by something
> like: {{beeline -c namedUrl}}. The use of beeline-site.xml can also be
> potentially expanded later if needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19033) Provide an option to purge LLAP IO cache
[ https://issues.apache.org/jira/browse/HIVE-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16422703#comment-16422703 ] Thejas M Nair commented on HIVE-19033: -- This test failure seems related - https://builds.apache.org/job/PreCommit-HIVE-Build/9956/testReport/org.apache.hadoop.hive.ql.processors/TestCommandProcessorFactory/testAvailableCommands/ > Provide an option to purge LLAP IO cache > > > Key: HIVE-19033 > URL: https://issues.apache.org/jira/browse/HIVE-19033 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19033.1.patch, HIVE-19033.2.patch, > HIVE-19033.3.patch, HIVE-19033.4.patch, HIVE-19033.5.patch, HIVE-19033.6.patch > > > Provide an API endpoint that will trigger purging of LLAP IO cache. Also CLI > tool to invoke the endpoint of all LLAP daemons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18963) JDBC: Provide an option to simplify beeline usage by supporting default and named URL for beeline
[
https://issues.apache.org/jira/browse/HIVE-18963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16422705#comment-16422705
]
Thejas M Nair commented on HIVE-18963:
--
https://builds.apache.org/job/PreCommit-HIVE-Build/9956/testReport/org.apache.hadoop.hive.ql.processors/TestCommandProcessorFactory/testAvailableCommands/
test failure seems to be related to another patch - HIVE-19033
> JDBC: Provide an option to simplify beeline usage by supporting default and
> named URL for beeline
> -
>
> Key: HIVE-18963
> URL: https://issues.apache.org/jira/browse/HIVE-18963
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-18963.1.patch, HIVE-18963.2.patch,
> HIVE-18963.3.patch
>
>
> Currently, after opening Beeline CLI, the user needs to supply a connection
> string to use the HS2 instance and set up the jdbc driver. Since we plan to
> replace Hive CLI with Beeline in future (HIVE-10511), it will help the
> usability if the user can simply type {{beeline}} and get start the hive
> session. The jdbc url can be specified in a beeline-site.xml (which can
> contain other named jdbc urls as well, and they can be accessed by something
> like: {{beeline -c namedUrl}}. The use of beeline-site.xml can also be
> potentially expanded later if needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18963) JDBC: Provide an option to simplify beeline usage by supporting default and named URL for beeline
[
https://issues.apache.org/jira/browse/HIVE-18963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16422707#comment-16422707
]
Thejas M Nair commented on HIVE-18963:
--
[~vgumashta]
Two other failures are in jdbc/beeline (with test age=1), can you please check
if they could be related ?
> JDBC: Provide an option to simplify beeline usage by supporting default and
> named URL for beeline
> -
>
> Key: HIVE-18963
> URL: https://issues.apache.org/jira/browse/HIVE-18963
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-18963.1.patch, HIVE-18963.2.patch,
> HIVE-18963.3.patch
>
>
> Currently, after opening Beeline CLI, the user needs to supply a connection
> string to use the HS2 instance and set up the jdbc driver. Since we plan to
> replace Hive CLI with Beeline in future (HIVE-10511), it will help the
> usability if the user can simply type {{beeline}} and get start the hive
> session. The jdbc url can be specified in a beeline-site.xml (which can
> contain other named jdbc urls as well, and they can be accessed by something
> like: {{beeline -c namedUrl}}. The use of beeline-site.xml can also be
> potentially expanded later if needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18841) Support authorization of UDF usage in hive
[ https://issues.apache.org/jira/browse/HIVE-18841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-18841: - Attachment: HIVE-18841.1.patch > Support authorization of UDF usage in hive > -- > > Key: HIVE-18841 > URL: https://issues.apache.org/jira/browse/HIVE-18841 > Project: Hive > Issue Type: New Feature >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Critical > Attachments: HIVE-18841.1.patch, HIVE-18841.1.patch > > > It should be possible to create authorization policies on UDF usage. > ie, it should be possible to control who can use certain UDF in their queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18841) Support authorization of UDF usage in hive
[ https://issues.apache.org/jira/browse/HIVE-18841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16422998#comment-16422998 ] Thejas M Nair commented on HIVE-18841: -- Attaching again for test run. > Support authorization of UDF usage in hive > -- > > Key: HIVE-18841 > URL: https://issues.apache.org/jira/browse/HIVE-18841 > Project: Hive > Issue Type: New Feature >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Critical > Attachments: HIVE-18841.1.patch, HIVE-18841.1.patch > > > It should be possible to create authorization policies on UDF usage. > ie, it should be possible to control who can use certain UDF in their queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19014) utilize YARN-8028 (queue ACL check) in Hive Tez session pool
[ https://issues.apache.org/jira/browse/HIVE-19014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16423295#comment-16423295 ] Thejas M Nair commented on HIVE-19014: -- bq. any comment on the SessionState.getUserFromAuthenticator() vs SessionState.getUserName() That needs some cleanup. As of now, hive code is using both in different places. The main difference is that SessionState.getUserFromAuthenticator() lets you configure something like SessionStateConfigUserAuthenticator as the authenticator, which can be used to switch usernames within a .q file. When the cleanup is done, we should be able to replace all uses of the api calls with one of them. Usage in this patch is fine. > utilize YARN-8028 (queue ACL check) in Hive Tez session pool > > > Key: HIVE-19014 > URL: https://issues.apache.org/jira/browse/HIVE-19014 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19014.01.patch, HIVE-19014.02.patch, > HIVE-19014.03.patch, HIVE-19014.04.patch, HIVE-19014.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-16949) Leak of threads from Get-Input-Paths and Get-Input-Summary thread pool
[
https://issues.apache.org/jira/browse/HIVE-16949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16424458#comment-16424458
]
Thejas M Nair commented on HIVE-16949:
--
[~stakiar]
This is missing the fix version, can you please add it ?
> Leak of threads from Get-Input-Paths and Get-Input-Summary thread pool
> --
>
> Key: HIVE-16949
> URL: https://issues.apache.org/jira/browse/HIVE-16949
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Birger Brunswiek
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-16949.1.patch
>
>
> The commit
> [20210de|https://github.com/apache/hive/commit/20210dec94148c9b529132b1545df3dd7be083c3]
> which was part of HIVE-15546 [introduced a thread
> pool|https://github.com/apache/hive/blob/824b9c80b443dc4e2b9ad35214a23ac756e75234/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L3109]
> which is not shutdown upon completion of its threads. This leads to a leak
> of threads for each query which uses more than 1 partition. They are not
> removed automatically. When queries spanning multiple partitions are made the
> number of threads increases and is never reduced. On my machine hiveserver2
> starts to get slower and slower once 10k threads are reached.
> Thread pools only shutdown automatically in special circumstances (see
> [documentation section
> _Finalization_|https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ThreadPoolExecutor.html]).
> This is not currently the case for the Get-Input-Paths thread pool. I would
> add a _pool.shutdown()_ in a finally block just before returning the result
> to make sure the threads are really shutdown.
> My current workaround is to set {{hive.exec.input.listing.max.threads = 1}}.
> This prevents the the thread pool from being spawned
> [\[1\]|https://github.com/apache/hive/blob/824b9c80b443dc4e2b9ad35214a23ac756e75234/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2118]
>
> [\[2\]|https://github.com/apache/hive/blob/824b9c80b443dc4e2b9ad35214a23ac756e75234/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L3107].
> The same issue probably also applies to the [Get-Input-Summary thread
> pool|https://github.com/apache/hive/blob/824b9c80b443dc4e2b9ad35214a23ac756e75234/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2193].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18841) Support authorization of UDF usage in hive
[ https://issues.apache.org/jira/browse/HIVE-18841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16426013#comment-16426013 ] Thejas M Nair commented on HIVE-18841: -- 2.patch - Updated q.out files, fixed checkstyle issues > Support authorization of UDF usage in hive > -- > > Key: HIVE-18841 > URL: https://issues.apache.org/jira/browse/HIVE-18841 > Project: Hive > Issue Type: New Feature >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Critical > Attachments: HIVE-18841.1.patch, HIVE-18841.1.patch, > HIVE-18841.2.patch > > > It should be possible to create authorization policies on UDF usage. > ie, it should be possible to control who can use certain UDF in their queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18841) Support authorization of UDF usage in hive
[ https://issues.apache.org/jira/browse/HIVE-18841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-18841: - Attachment: HIVE-18841.2.patch > Support authorization of UDF usage in hive > -- > > Key: HIVE-18841 > URL: https://issues.apache.org/jira/browse/HIVE-18841 > Project: Hive > Issue Type: New Feature >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Critical > Attachments: HIVE-18841.1.patch, HIVE-18841.1.patch, > HIVE-18841.2.patch > > > It should be possible to create authorization policies on UDF usage. > ie, it should be possible to control who can use certain UDF in their queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18841) Support authorization of UDF usage in hive
[ https://issues.apache.org/jira/browse/HIVE-18841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-18841: - Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the review [~daijy]! > Support authorization of UDF usage in hive > -- > > Key: HIVE-18841 > URL: https://issues.apache.org/jira/browse/HIVE-18841 > Project: Hive > Issue Type: New Feature >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-18841.1.patch, HIVE-18841.1.patch, > HIVE-18841.2.patch > > > It should be possible to create authorization policies on UDF usage. > ie, it should be possible to control who can use certain UDF in their queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19031) Mark duplicate configs in HiveConf as deprecated
[ https://issues.apache.org/jira/browse/HIVE-19031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16426095#comment-16426095 ] Thejas M Nair commented on HIVE-19031: -- +1 > Mark duplicate configs in HiveConf as deprecated > > > Key: HIVE-19031 > URL: https://issues.apache.org/jira/browse/HIVE-19031 > Project: Hive > Issue Type: Sub-task > Components: Configuration, Standalone Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Blocker > Attachments: HIVE-19031.2.patch, HIVE-19031.patch > > > There are a number of configuration values that were copied from HiveConf to > MetastoreConf. They have been left in HiveConf for backwards compatibility. > But they need to be marked as deprecated so that users know to use the new > values in MetastoreConf. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19126) CachedStore: Use memory estimation to limit cache size during prewarm
[ https://issues.apache.org/jira/browse/HIVE-19126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16428648#comment-16428648 ] Thejas M Nair commented on HIVE-19126: -- Thanks for the patch. Can you please add a reviewboard link or github pull request ? Some initial comments - * Can you also add SizeValidator to standalone metastore (similar to standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/conf/TimeValidator) and use it for this config ? That way size can be specified in GB. * Many files have only changes in import sequence, can you please revert those changes ? > CachedStore: Use memory estimation to limit cache size during prewarm > - > > Key: HIVE-19126 > URL: https://issues.apache.org/jira/browse/HIVE-19126 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-19126.1.patch > > > We can rely on > https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java > to estimate memory of SharedCache. This jira addresses the size estimation > during prewarm, so that we can stop when we hit the memory limit. In a > follow-up jira, we will work on estimation/eviction after prewarm is > complete, so that we can keep the frequently used tables and their partitions > in cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19105) HIVE-18781 broke WarehouseInstance
[
https://issues.apache.org/jira/browse/HIVE-19105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430960#comment-16430960
]
Thejas M Nair commented on HIVE-19105:
--
[~sankarh]
Can you please review ?
cc [~maheshk114]
I see these tests were consistently failing to run in HIVE-18781.
> HIVE-18781 broke WarehouseInstance
> --
>
> Key: HIVE-19105
> URL: https://issues.apache.org/jira/browse/HIVE-19105
> Project: Hive
> Issue Type: Bug
> Components: repl, Tests
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19105.patch
>
>
> HIVE-18781 removed the line
> {{hiveConf.set(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY.varname, "false");}}
> from WarehouseInstance. This causes tests such as
> TestReplicationOnHDFSEncryptedZones and TestExportImport to hang as they are
> trying to connect to a non-existent ZooKeeper instance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19105) HIVE-18781 broke WarehouseInstance
[
https://issues.apache.org/jira/browse/HIVE-19105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430961#comment-16430961
]
Thejas M Nair commented on HIVE-19105:
--
+1
I see that these tests are no longer hanging after this change.
> HIVE-18781 broke WarehouseInstance
> --
>
> Key: HIVE-19105
> URL: https://issues.apache.org/jira/browse/HIVE-19105
> Project: Hive
> Issue Type: Bug
> Components: repl, Tests
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19105.patch
>
>
> HIVE-18781 removed the line
> {{hiveConf.set(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY.varname, "false");}}
> from WarehouseInstance. This causes tests such as
> TestReplicationOnHDFSEncryptedZones and TestExportImport to hang as they are
> trying to connect to a non-existent ZooKeeper instance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19126) CachedStore: Use memory estimation to limit cache size during prewarm
[ https://issues.apache.org/jira/browse/HIVE-19126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16431560#comment-16431560 ] Thejas M Nair commented on HIVE-19126: -- +1 > CachedStore: Use memory estimation to limit cache size during prewarm > - > > Key: HIVE-19126 > URL: https://issues.apache.org/jira/browse/HIVE-19126 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-19126.1.patch, HIVE-19126.2.patch, > HIVE-19126.3.patch, HIVE-19126.4.patch > > > We can rely on > https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java > to estimate memory of SharedCache. This jira addresses the size estimation > during prewarm, so that we can stop when we hit the memory limit. In a > follow-up jira, we will work on estimation/eviction after prewarm is > complete, so that we can keep the frequently used tables and their partitions > in cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19130) NPE is thrown when REPL LOAD applied drop partition event.
[
https://issues.apache.org/jira/browse/HIVE-19130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-19130:
-
Description:
During incremental replication, if we split the events batch as follows, then
the REPL LOAD on second batch throws NPE.
Batch-1: CREATE_TABLE(t1) -> ADD_PARTITION(t1.p1) -> DROP_PARTITION (t1.p1)
Batch-2: DROP_TABLE(t1) -> CREATE_TABLE(t1) -> ADD_PARTITION(t1.p1) ->
DROP_PARTITION (t1.p1)
{code}
2018-04-05 16:20:36,531 ERROR [HiveServer2-Background-Pool: Thread-107044]:
metadata.Hive (Hive.java:getTable(1219)) - Table catalog_sales_new not found:
new5_tpcds_real_bin_partitioned_orc_1000.catalog_sales_new table not found
2018-04-05 16:20:36,538 ERROR [HiveServer2-Background-Pool: Thread-107044]:
exec.DDLTask (DDLTask.java:failed(540)) -
org.apache.hadoop.hive.ql.metadata.HiveException
at
org.apache.hadoop.hive.ql.exec.DDLTask.dropPartitions(DDLTask.java:4016)
at
org.apache.hadoop.hive.ql.exec.DDLTask.dropTableOrPartitions(DDLTask.java:3983)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:341)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:162)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1765)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1506)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1303)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1170)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1165)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
at
org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
at
org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
at
org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartitionsByExpr(Hive.java:2613)
at
org.apache.hadoop.hive.ql.exec.DDLTask.dropPartitions(DDLTask.java:4008)
... 23 more
{code}
was:
During incremental replication, if we split the events batch as follows, then
the REPL LOAD on second batch throws NPE.
Batch-1: CREATE_TABLE(t1) -> ADD_PARTITION(t1.p1) -> DROP_PARTITION (t1.p1)
Batch-2: DROP_TABLE(t1) -> CREATE_TABLE(t1) -> ADD_PARTITION(t1.p1) ->
DROP_PARTITION (t1.p1)
> NPE is thrown when REPL LOAD applied drop partition event.
> --
>
> Key: HIVE-19130
> URL: https://issues.apache.org/jira/browse/HIVE-19130
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Fix For: 3.1.0
>
> Attachments: HIVE-19130.01.patch
>
>
> During incremental replication, if we split the events batch as follows, then
> the REPL LOAD on second batch throws NPE.
> Batch-1: CREATE_TABLE(t1) -> ADD_PARTITION(t1.p1) -> DROP_PARTITION (t1.p1)
> Batch-2: DROP_TABLE(t1) -> CREATE_TABLE(t1) -> ADD_PARTITION(t1.p1) ->
> DROP_PARTITION (t1.p1)
> {code}
> 2018-04-05 16:20:36,531 ERROR [HiveServer2-Background-Pool: Thread-107044]:
> metadata.Hive (Hive.java:getTable(1219)) - Table catalog_sales_new not found:
> new5_tpcds_real_bin_partitioned_orc_1000.catalog_sales_new table not found
> 2018-04-05 16:20:36,538 ERROR [HiveServer2-Background-Pool: Thread-107044]:
> exec.DDLTask (DDLTask.java:failed(540)) -
> org.apache.hadoop.hive.ql.metadata.HiveException
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.dropPartitions(DDLTask.java:4016)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.dropTableOrPartitions(DDLTask.java:3983)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:341)
> at org.apache.hadoop.hiv
[jira] [Commented] (HIVE-19130) NPE is thrown when REPL LOAD applied drop partition event.
[
https://issues.apache.org/jira/browse/HIVE-19130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16433426#comment-16433426
]
Thejas M Nair commented on HIVE-19130:
--
[~maheshk114]
Can you also please review this patch ?
> NPE is thrown when REPL LOAD applied drop partition event.
> --
>
> Key: HIVE-19130
> URL: https://issues.apache.org/jira/browse/HIVE-19130
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Fix For: 3.1.0
>
> Attachments: HIVE-19130.01.patch
>
>
> During incremental replication, if we split the events batch as follows, then
> the REPL LOAD on second batch throws NPE.
> Batch-1: CREATE_TABLE(t1) -> ADD_PARTITION(t1.p1) -> DROP_PARTITION (t1.p1)
> Batch-2: DROP_TABLE(t1) -> CREATE_TABLE(t1) -> ADD_PARTITION(t1.p1) ->
> DROP_PARTITION (t1.p1)
> {code}
> 2018-04-05 16:20:36,531 ERROR [HiveServer2-Background-Pool: Thread-107044]:
> metadata.Hive (Hive.java:getTable(1219)) - Table catalog_sales_new not found:
> new5_tpcds_real_bin_partitioned_orc_1000.catalog_sales_new table not found
> 2018-04-05 16:20:36,538 ERROR [HiveServer2-Background-Pool: Thread-107044]:
> exec.DDLTask (DDLTask.java:failed(540)) -
> org.apache.hadoop.hive.ql.metadata.HiveException
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.dropPartitions(DDLTask.java:4016)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.dropTableOrPartitions(DDLTask.java:3983)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:341)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:162)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1765)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1506)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1303)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1170)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1165)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getPartitionsByExpr(Hive.java:2613)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.dropPartitions(DDLTask.java:4008)
> ... 23 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19130) NPE is thrown when REPL LOAD applied drop partition event.
[
https://issues.apache.org/jira/browse/HIVE-19130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16434282#comment-16434282
]
Thejas M Nair commented on HIVE-19130:
--
+1
> NPE is thrown when REPL LOAD applied drop partition event.
> --
>
> Key: HIVE-19130
> URL: https://issues.apache.org/jira/browse/HIVE-19130
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, repl
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, Replication, pull-request-available
> Fix For: 3.1.0
>
> Attachments: HIVE-19130.01.patch
>
>
> During incremental replication, if we split the events batch as follows, then
> the REPL LOAD on second batch throws NPE.
> Batch-1: CREATE_TABLE(t1) -> ADD_PARTITION(t1.p1) -> DROP_PARTITION (t1.p1)
> Batch-2: DROP_TABLE(t1) -> CREATE_TABLE(t1) -> ADD_PARTITION(t1.p1) ->
> DROP_PARTITION (t1.p1)
> {code}
> 2018-04-05 16:20:36,531 ERROR [HiveServer2-Background-Pool: Thread-107044]:
> metadata.Hive (Hive.java:getTable(1219)) - Table catalog_sales_new not found:
> new5_tpcds_real_bin_partitioned_orc_1000.catalog_sales_new table not found
> 2018-04-05 16:20:36,538 ERROR [HiveServer2-Background-Pool: Thread-107044]:
> exec.DDLTask (DDLTask.java:failed(540)) -
> org.apache.hadoop.hive.ql.metadata.HiveException
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.dropPartitions(DDLTask.java:4016)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.dropTableOrPartitions(DDLTask.java:3983)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:341)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:162)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1765)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1506)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1303)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1170)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1165)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getPartitionsByExpr(Hive.java:2613)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.dropPartitions(DDLTask.java:4008)
> ... 23 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19181) Remove BreakableService (unused class)
[ https://issues.apache.org/jira/browse/HIVE-19181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-19181: - Labels: newbie trivial (was: ) > Remove BreakableService (unused class) > -- > > Key: HIVE-19181 > URL: https://issues.apache.org/jira/browse/HIVE-19181 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0, 2.3.2 >Reporter: Thejas M Nair >Priority: Major > Labels: newbie, trivial > > BreakableService.java is not used anywhere -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19161) Add authorizations to information schema
[ https://issues.apache.org/jira/browse/HIVE-19161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16434613#comment-16434613 ] Thejas M Nair commented on HIVE-19161: -- [~daijy] Can you please include a review board link or pull request ? > Add authorizations to information schema > > > Key: HIVE-19161 > URL: https://issues.apache.org/jira/browse/HIVE-19161 > Project: Hive > Issue Type: Improvement >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-19161.1.patch, HIVE-19161.2.patch > > > We need to control the access of information schema so user can only query > the information authorized to. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19126) CachedStore: Use memory estimation to limit cache size during prewarm
[ https://issues.apache.org/jira/browse/HIVE-19126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436278#comment-16436278 ] Thejas M Nair commented on HIVE-19126: -- +1 pending tests > CachedStore: Use memory estimation to limit cache size during prewarm > - > > Key: HIVE-19126 > URL: https://issues.apache.org/jira/browse/HIVE-19126 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-19126.1.patch, HIVE-19126.2.patch, > HIVE-19126.3.patch, HIVE-19126.4.patch, HIVE-19126.5.patch > > > We can rely on > https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java > to estimate memory of SharedCache. This jira addresses the size estimation > during prewarm, so that we can stop when we hit the memory limit. In a > follow-up jira, we will work on estimation/eviction after prewarm is > complete, so that we can keep the frequently used tables and their partitions > in cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19000) Fix TestNegativeCliDriver
[ https://issues.apache.org/jira/browse/HIVE-19000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair reassigned HIVE-19000: Assignee: Thejas M Nair > Fix TestNegativeCliDriver > - > > Key: HIVE-19000 > URL: https://issues.apache.org/jira/browse/HIVE-19000 > Project: Hive > Issue Type: Sub-task >Reporter: Vihang Karajgaonkar >Assignee: Thejas M Nair >Priority: Major > > TestNegativeCliDriver is failing since a while. We should investigate and fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-19149) Vulnerability CVE-2018-1284, CVE-2018-1282, CVE-2018-1315
[ https://issues.apache.org/jira/browse/HIVE-19149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair resolved HIVE-19149. -- Resolution: Invalid > Vulnerability CVE-2018-1284, CVE-2018-1282, CVE-2018-1315 > - > > Key: HIVE-19149 > URL: https://issues.apache.org/jira/browse/HIVE-19149 > Project: Hive > Issue Type: Bug >Reporter: Rohit Persai >Priority: Major > > Need a fix for below Vulnerabilities for Hive > CVE-2018-1284, > CVE-2018-1282, > CVE-2018-1315 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19197) TestReplicationScenarios is flaky
[
https://issues.apache.org/jira/browse/HIVE-19197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440024#comment-16440024
]
Thejas M Nair commented on HIVE-19197:
--
[~maheshk114] Can you please review ?
> TestReplicationScenarios is flaky
> -
>
> Key: HIVE-19197
> URL: https://issues.apache.org/jira/browse/HIVE-19197
> Project: Hive
> Issue Type: Sub-task
> Components: repl, Test
>Reporter: Ashutosh Chauhan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-19197.01.patch
>
>
> Fails once in a while.
> {code}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.verifyResults(TestReplicationScenarios.java:3629)
> at
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.verifyRun(TestReplicationScenarios.java:3711)
> at
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.verifyRun(TestReplicationScenarios.java:3706)
> at
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.verifyAndReturnTblReplStatus(TestReplicationScenarios.java:3600)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18778) Needs to capture input/output entities in explain
[
https://issues.apache.org/jira/browse/HIVE-18778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441492#comment-16441492
]
Thejas M Nair commented on HIVE-18778:
--
[~ngangam]
Daniel is away for few days, so I doubt if he is going to get back on this soon.
> Needs to capture input/output entities in explain
> -
>
> Key: HIVE-18778
> URL: https://issues.apache.org/jira/browse/HIVE-18778
> Project: Hive
> Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-18778-SparkPositive.patch, HIVE-18778.1.patch,
> HIVE-18778.2.patch, HIVE-18778.3.patch, HIVE-18778_TestCliDriver.patch,
> HIVE-18788_SparkNegative.patch, HIVE-18788_SparkPerf.patch
>
>
> With Sentry enabled, commands like explain drop table foo fail with {{explain
> drop table foo;}}
> {code}
> Error: Error while compiling statement: FAILED: SemanticException No valid
> privileges
> Required privilege( Table) not available in input privileges
> The required privileges: (state=42000,code=4)
> {code}
> Sentry fails to authorize because the ExplainSemanticAnalyzer uses an
> instance of DDLSemanticAnalyzer to analyze the explain query.
> {code}
> BaseSemanticAnalyzer sem = SemanticAnalyzerFactory.get(conf, input);
> sem.analyze(input, ctx);
> sem.validate()
> {code}
> The inputs/outputs entities for this query are set in the above code.
> However, these are never set on the instance of ExplainSemanticAnalyzer
> itself and thus is not propagated into the HookContext in the calling Driver
> code.
> {code}
> sem.analyze(tree, ctx); --> this results in calling the above code that uses
> DDLSA
> hookCtx.update(sem); --> sem is an instance of ExplainSemanticAnalyzer, this
> code attempts to update the HookContext with the input/output info from ESA
> which is never set.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19197) TestReplicationScenarios is flaky
[
https://issues.apache.org/jira/browse/HIVE-19197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16442860#comment-16442860
]
Thejas M Nair commented on HIVE-19197:
--
+1
> TestReplicationScenarios is flaky
> -
>
> Key: HIVE-19197
> URL: https://issues.apache.org/jira/browse/HIVE-19197
> Project: Hive
> Issue Type: Sub-task
> Components: repl, Test
>Affects Versions: 3.0.0
>Reporter: Ashutosh Chauhan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.1.0
>
> Attachments: HIVE-19197.01.patch
>
>
> Fails once in a while.
> {code}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.verifyResults(TestReplicationScenarios.java:3629)
> at
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.verifyRun(TestReplicationScenarios.java:3711)
> at
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.verifyRun(TestReplicationScenarios.java:3706)
> at
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.verifyAndReturnTblReplStatus(TestReplicationScenarios.java:3600)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19219) Incremental REPL DUMP should throw error if requested events are cleaned-up.
[ https://issues.apache.org/jira/browse/HIVE-19219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16442862#comment-16442862 ] Thejas M Nair commented on HIVE-19219: -- +1 > Incremental REPL DUMP should throw error if requested events are cleaned-up. > > > Key: HIVE-19219 > URL: https://issues.apache.org/jira/browse/HIVE-19219 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-19219.01.patch, HIVE-19219.02.patch, > HIVE-19219.03.patch > > > This is the case where the events were deleted on source because of old event > purging and hence min(source event id) > target event id (last replicated > event id). > Repl dump should fail in this case so that user can drop the database and > bootstrap again. > Cleaner thread is concurrently removing the expired events from > NOTIFICATION_LOG table. So, it is necessary to check if the current dump > missed any event while dumping. After fetching events in batches, we shall > check if it is fetched in contiguous sequence of event id. If it is not in > contiguous sequence, then likely some events missed in the dump and hence > throw error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19198) Few flaky hcatalog tests
[ https://issues.apache.org/jira/browse/HIVE-19198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair reassigned HIVE-19198: Assignee: Thejas M Nair (was: Daniel Dai) > Few flaky hcatalog tests > > > Key: HIVE-19198 > URL: https://issues.apache.org/jira/browse/HIVE-19198 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Chauhan >Assignee: Thejas M Nair >Priority: Major > > TestPermsGrp : Consider removing this since hcat cli is not widely used. > TestHCatPartitionPublish.testPartitionPublish > TestHCatMultiOutputFormat.testOutputFormat -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19249) Replication: WITH clause is not passing the configuration to Task correctly in all cases
[
https://issues.apache.org/jira/browse/HIVE-19249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16444858#comment-16444858
]
Thejas M Nair commented on HIVE-19249:
--
+1 pending tests.
I noticed that in master branch because of changes in HIVE-18238, every query
already has a new HiveConf object. So we no longer need to create a new copy in
ReplicationSemanticAnalyzer. This could be part of a follow up change (its not
introduced by this patch).
> Replication: WITH clause is not passing the configuration to Task correctly
> in all cases
>
>
> Key: HIVE-19249
> URL: https://issues.apache.org/jira/browse/HIVE-19249
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-19249.1.patch, HIVE-19249.2.patch
>
>
> When running repl load like following:
> {code}
> REPL LOAD `repldb_kms207` FROM
> 'hdfs://url:8020/apps/hive/repl/f8b057a7-c3f2-43bd-8baa-f7408a9008fc' WITH
> ('hive.exec.parallel'='true','hive.distcp.privileged.doAs'='beacon','hive.metastore.uris'='thrift://metastore-url:9083','hive.metastore.warehouse.dir'='s3a://s3-warehouse','hive.warehouse.subdir.inherit.perms'='false','hive.repl.replica.functions.root.dir'='s3a://s3-warehouse','fs.s3a.bucket.ss-datasets.endpoint'='s3-bucket-endpoint','fs.s3a.impl.disable.cache'='true','fs.s3a.server-side-encryption-algorithm'='SSE-KMS','fs.s3a.server-side-encryption.key'='encr-key','distcp.options.pp'='','distcp.options.pg'='','distcp.options.pu'='');
> {code}
> the task that get created need to use the configs that are passed in the
> USING clause. However, in some cases the wrong config object gets used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19251) ObjectStore.getNextNotification with LIMIT should use less memory
[ https://issues.apache.org/jira/browse/HIVE-19251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16444960#comment-16444960 ] Thejas M Nair commented on HIVE-19251: -- +1 > ObjectStore.getNextNotification with LIMIT should use less memory > - > > Key: HIVE-19251 > URL: https://issues.apache.org/jira/browse/HIVE-19251 > Project: Hive > Issue Type: Bug > Components: repl, Standalone Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-19251.1.patch > > > Experience OOM when Hive metastore try to retrieve huge amount of > notification logs even there's limit clause. Hive shall only retrieve > necessary rows. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19198) Few flaky hcatalog tests
[ https://issues.apache.org/jira/browse/HIVE-19198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair reassigned HIVE-19198: Assignee: Daniel Dai (was: Thejas M Nair) > Few flaky hcatalog tests > > > Key: HIVE-19198 > URL: https://issues.apache.org/jira/browse/HIVE-19198 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Chauhan >Assignee: Daniel Dai >Priority: Major > > TestPermsGrp : Consider removing this since hcat cli is not widely used. > TestHCatPartitionPublish.testPartitionPublish > TestHCatMultiOutputFormat.testOutputFormat -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19252) TestJdbcWithMiniKdcCookie.testCookieNegative is failing consistently
[ https://issues.apache.org/jira/browse/HIVE-19252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair reassigned HIVE-19252: Assignee: Vaibhav Gumashta (was: Thejas M Nair) > TestJdbcWithMiniKdcCookie.testCookieNegative is failing consistently > > > Key: HIVE-19252 > URL: https://issues.apache.org/jira/browse/HIVE-19252 > Project: Hive > Issue Type: Sub-task > Components: Test >Reporter: Ashutosh Chauhan >Assignee: Vaibhav Gumashta >Priority: Major > > For last 8 builds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18626) Repl load "with" clause does not pass config to tasks
[ https://issues.apache.org/jira/browse/HIVE-18626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354787#comment-16354787 ] Thejas M Nair commented on HIVE-18626: -- Changes look good. Can you please take a look at checkstyle warnings (seems like an indentation issue), and also verify that TestMarkPartition.testMarkingPartitionSet is not related to this issue ? (only test failure that has age=1 in jenkins) > Repl load "with" clause does not pass config to tasks > - > > Key: HIVE-18626 > URL: https://issues.apache.org/jira/browse/HIVE-18626 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-18626.1.patch > > > The "with" clause in repl load suppose to pass custom hive config entries to > replication. However, the config is only effective in > BootstrapEventsIterator, but not the generated tasks (such as MoveTask, > DDLTask). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18626) Repl load "with" clause does not pass config to tasks
[ https://issues.apache.org/jira/browse/HIVE-18626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354985#comment-16354985 ] Thejas M Nair commented on HIVE-18626: -- +1 pending above comments > Repl load "with" clause does not pass config to tasks > - > > Key: HIVE-18626 > URL: https://issues.apache.org/jira/browse/HIVE-18626 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-18626.1.patch > > > The "with" clause in repl load suppose to pass custom hive config entries to > replication. However, the config is only effective in > BootstrapEventsIterator, but not the generated tasks (such as MoveTask, > DDLTask). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
