[jira] [Commented] (HIVE-4838) Refactor MapJoin HashMap code to improve testability and readability
[ https://issues.apache.org/jira/browse/HIVE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735785#comment-13735785 ] Ashutosh Chauhan commented on HIVE-4838: [~brocknoland] Lets get this in, before the patch gets stale. Refactor MapJoin HashMap code to improve testability and readability Key: HIVE-4838 URL: https://issues.apache.org/jira/browse/HIVE-4838 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-4838.patch, HIVE-4838.patch, HIVE-4838.patch, HIVE-4838.patch, HIVE-4838.patch MapJoin is an essential component for high performance joins in Hive and the current code has done great service for many years. However, the code is showing it's age and currently suffers from the following issues: * Uses static state via the MapJoinMetaData class to pass serialization metadata to the Key, Row classes. * The api of a logical Table Container is not defined and therefore it's unclear what apis HashMapWrapper needs to publicize. Additionally HashMapWrapper has many used public methods. * HashMapWrapper contains logic to serialize, test memory bounds, and implement the table container. Ideally these logical units could be seperated * HashTableSinkObjectCtx has unused fields and unused methods * CommonJoinOperator and children use ArrayList on left hand side when only List is required * There are unused classes MRU, DCLLItemm and classes which duplicate functionality MapJoinSingleKey and MapJoinDoubleKeys -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4985) refactor/clean up partition name pruning to be usable inside metastore server
[
https://issues.apache.org/jira/browse/HIVE-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735788#comment-13735788
]
Hive QA commented on HIVE-4985:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12597226/HIVE-4985.D11961.4.patch
{color:red}ERROR:{color} -1 due to 39 failed/errored test(s), 2774 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_09_part_spec_nonoverlap
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppr_pushdown2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonmr_fetch
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_dependency2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_date2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_19_00_part_external_location
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_07_all_part_over_nonoverlap
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input42
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_bucket_mapjoin_wrong_table_metadata_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_coltype
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_dependency
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppr_pushdown3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat15
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_04_all_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_escape2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_15_external_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_05_some_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_vc
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/377/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/377/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 39 tests failed
{noformat}
This message is automatically generated.
refactor/clean up partition name pruning to be usable inside metastore server
--
Key: HIVE-4985
URL: https://issues.apache.org/jira/browse/HIVE-4985
Project: Hive
Issue Type: Improvement
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: HIVE-4985.D11961.1.patch, HIVE-4985.D11961.2.patch,
HIVE-4985.D11961.3.patch, HIVE-4985.D11961.4.patch
Preliminary for HIVE-4914.
The patch is going to be large already, so some refactoring and dead code
removal that is non-controversial can be done in advance in a separate patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5028) Some tests with fail OutOfMemoryError PermGen Space on Hadoop2
[
https://issues.apache.org/jira/browse/HIVE-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-5028:
---
Resolution: Fixed
Fix Version/s: 0.12.0
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks, Brock!
Some tests with fail OutOfMemoryError PermGen Space on Hadoop2
--
Key: HIVE-5028
URL: https://issues.apache.org/jira/browse/HIVE-5028
Project: Hive
Issue Type: Sub-task
Reporter: Brock Noland
Assignee: Brock Noland
Fix For: 0.12.0
Attachments: HIVE-5028.patch
{noformat}
java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:181)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:430)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:349)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:978)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:890)
at org.apache.hcatalog.cli.HCatDriver.run(HCatDriver.java:43)
at
org.apache.hcatalog.hbase.TestHBaseBulkOutputFormat.bulkModeAbortTest(TestHBaseBulkOutputFormat.java:540)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-5052) Set parallelism when generating the tez tasks
Gunther Hagleitner created HIVE-5052: Summary: Set parallelism when generating the tez tasks Key: HIVE-5052 URL: https://issues.apache.org/jira/browse/HIVE-5052 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner In GenTezTask any intermediate task has parallelism set to 1. This needs to be fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-5053) Let user override the parallelism of each tez task
Gunther Hagleitner created HIVE-5053: Summary: Let user override the parallelism of each tez task Key: HIVE-5053 URL: https://issues.apache.org/jira/browse/HIVE-5053 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner We need to come up with a way to let the user choose the parallelism for each vertex in the graph. We're numbering the vertices in the graph so we could use that to let the user specify the parallelism. Another way would be to introduce hints in the sql query itself. But that's a lot more complicated for little added value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5052) Set parallelism when generating the tez tasks
[ https://issues.apache.org/jira/browse/HIVE-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-5052: - Fix Version/s: tez-branch Set parallelism when generating the tez tasks - Key: HIVE-5052 URL: https://issues.apache.org/jira/browse/HIVE-5052 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Fix For: tez-branch In GenTezTask any intermediate task has parallelism set to 1. This needs to be fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5053) Let user override the parallelism of each tez task
[ https://issues.apache.org/jira/browse/HIVE-5053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-5053: - Fix Version/s: tez-branch Let user override the parallelism of each tez task --- Key: HIVE-5053 URL: https://issues.apache.org/jira/browse/HIVE-5053 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Fix For: tez-branch We need to come up with a way to let the user choose the parallelism for each vertex in the graph. We're numbering the vertices in the graph so we could use that to let the user specify the parallelism. Another way would be to introduce hints in the sql query itself. But that's a lot more complicated for little added value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4999) Shim class HiveHarFileSystem does not have a hadoop2 counterpart
[ https://issues.apache.org/jira/browse/HIVE-4999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735805#comment-13735805 ] Ashutosh Chauhan commented on HIVE-4999: Is there any test which fails because of this ? Shim class HiveHarFileSystem does not have a hadoop2 counterpart Key: HIVE-4999 URL: https://issues.apache.org/jira/browse/HIVE-4999 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-4999.patch HiveHarFileSystem only exists in the 0.20 shim. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5051) StorageBasedAuthorizationProvider masks lower level exception with IllegalStateException
[
https://issues.apache.org/jira/browse/HIVE-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735814#comment-13735814
]
Hive QA commented on HIVE-5051:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12597229/HIVE-5051.patch
{color:green}SUCCESS:{color} +1 2774 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/378/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/378/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
StorageBasedAuthorizationProvider masks lower level exception with
IllegalStateException
Key: HIVE-5051
URL: https://issues.apache.org/jira/browse/HIVE-5051
Project: Hive
Issue Type: Bug
Components: Authorization
Affects Versions: 0.12.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman
Fix For: 0.12.0
Attachments: HIVE-5051.patch
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
has the following 2 methods
private HiveException hiveException(Exception e) {
HiveException ex = new HiveException(e);
ex.initCause(e);
return ex;
}
private AuthorizationException authorizationException(Exception e) {
AuthorizationException ex = new AuthorizationException(e);
ex.initCause(e);
return ex;
}
both, when called, will cause
throw new IllegalStateException(Can't overwrite cause); to be thrown and
the original Exception is lost.
See
http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/lang/Throwable.java#Throwable.initCause%28java.lang.Throwable%29
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5019) Use StringBuffer instead of += (issue 1)
[
https://issues.apache.org/jira/browse/HIVE-5019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735815#comment-13735815
]
Hive QA commented on HIVE-5019:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12597243/HIVE-5019.2.patch.txt
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/379/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/379/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests failed with: NonZeroExitCodeException: Command 'bash
/data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-379/source-prep.txt
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted
'ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java'
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf build hcatalog/build hcatalog/core/build
hcatalog/storage-handlers/hbase/build hcatalog/server-extensions/build
hcatalog/webhcat/svr/build hcatalog/webhcat/java-client/build
hcatalog/hcatalog-pig-adapter/build common/src/gen
+ svn update
Uhcatalog/build.properties
Uhcatalog/storage-handlers/hbase/build.xml
Uhcatalog/build-support/ant/test.xml
Fetching external item into 'hcatalog/src/test/e2e/harness'
Updated external to revision 1512583.
Updated to revision 1512583.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
Going to apply patch with: patch -p1
patching file
hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hcatalog/utils/SumNumbers.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java
patching file ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenMRSkewJoinProcessor.java
patching file ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLike.java
patching file
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSentences.java
patching file
shims/src/common-secure/java/org/apache/hadoop/hive/thrift/ZooKeeperTokenStore.java
+ [[ true == \t\r\u\e ]]
+ rm -rf /data/hive-ptest/working/ivy /data/hive-ptest/working/maven
+ mkdir /data/hive-ptest/working/ivy /data/hive-ptest/working/maven
+ ant -Dtest.continue.on.failure=true -Dtest.silent=false
-Divy.default.ivy.user.dir=/data/hive-ptest/working/ivy
-Dmvn.local.repo=/data/hive-ptest/working/maven clean package test
-Dtestcase=nothing
Buildfile: /data/hive-ptest/working/apache-svn-trunk-source/build.xml
clean:
[echo] Project: hive
clean:
[echo] Project: anttasks
clean:
[echo] Project: shims
clean:
[echo] Project: common
clean:
[echo] Project: serde
clean:
[echo] Project: metastore
clean:
[echo] Project: ql
clean:
[echo] Project: contrib
clean:
[echo] Project: service
clean:
[echo] Project: cli
clean:
[echo] Project: jdbc
clean:
[echo] Project: beeline
clean:
[echo] Project: hwi
clean:
[echo] Project: hbase-handler
clean:
[echo] Project: testutils
clean:
[echo] hcatalog
clean:
[echo] hcatalog-core
clean:
[echo] hcatalog-pig-adapter
clean:
[echo] hcatalog-server-extensions
clean:
[echo] webhcat
clean:
[echo] webhcat-java-client
clean:
clean:
[echo] shims
clean:
[echo] Project: odbc
[exec] rm -rf /data/hive-ptest/working/apache-svn-trunk-source/build/odbc
/data/hive-ptest/working/apache-svn-trunk-source/build/service/objs
/data/hive-ptest/working/apache-svn-trunk-source/build/ql/objs
/data/hive-ptest/working/apache-svn-trunk-source/build/metastore/objs
clean-online:
[echo] Project: hive
[jira] [Updated] (HIVE-4913) Put deterministic ordering in the top-K ngrams output of UDF context_ngrams()
[ https://issues.apache.org/jira/browse/HIVE-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-4913: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Xuefu! Put deterministic ordering in the top-K ngrams output of UDF context_ngrams() - Key: HIVE-4913 URL: https://issues.apache.org/jira/browse/HIVE-4913 Project: Hive Issue Type: Improvement Components: UDF Affects Versions: 0.10.0, 0.11.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4913.patch, HIVE-4913.patch Currently UDF context_ngrams() output top-K ngrams in the order of descending frequency. When there are ties, i.e., phrases with same ngram value, the order is indeterministic. Tie breaker is needed to break the ties, so that the output is deterministic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-1577) Add configuration property hive.exec.local.scratchdir
[
https://issues.apache.org/jira/browse/HIVE-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan resolved HIVE-1577.
Resolution: Fixed
Such a config property exists on current task.
Add configuration property hive.exec.local.scratchdir
-
Key: HIVE-1577
URL: https://issues.apache.org/jira/browse/HIVE-1577
Project: Hive
Issue Type: New Feature
Components: Configuration
Reporter: Carl Steinbach
When Hive is run in local mode it uses the hardcoded local directory
{{/${java.io.tmpdir}/${user.name}}} for temporary files. This path should be
configurable via the property {{hive.exec.local.scratchdir}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5054) Remove unused property submitviachild
[ https://issues.apache.org/jira/browse/HIVE-5054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5054: --- Attachment: HIVE-5054.patch Remove unused property submitviachild - Key: HIVE-5054 URL: https://issues.apache.org/jira/browse/HIVE-5054 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Ashutosh Chauhan Attachments: HIVE-5054.patch This property only exist in HiveConf and is always set to false. Lets get rid of dead code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-5054) Remove unused property submitviachild
Ashutosh Chauhan created HIVE-5054: -- Summary: Remove unused property submitviachild Key: HIVE-5054 URL: https://issues.apache.org/jira/browse/HIVE-5054 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Ashutosh Chauhan Attachments: HIVE-5054.patch This property only exist in HiveConf and is always set to false. Lets get rid of dead code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5054) Remove unused property submitviachild
[ https://issues.apache.org/jira/browse/HIVE-5054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-5054: --- Assignee: Ashutosh Chauhan Status: Patch Available (was: Open) Simple patch. Remove unused property submitviachild - Key: HIVE-5054 URL: https://issues.apache.org/jira/browse/HIVE-5054 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-5054.patch This property only exist in HiveConf and is always set to false. Lets get rid of dead code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HIVE-1577) Add configuration property hive.exec.local.scratchdir
[
https://issues.apache.org/jira/browse/HIVE-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735830#comment-13735830
]
Ashutosh Chauhan edited comment on HIVE-1577 at 8/10/13 10:31 AM:
--
Such a config property exists on current trunk.
was (Author: ashutoshc):
Such a config property exists on current task.
Add configuration property hive.exec.local.scratchdir
-
Key: HIVE-1577
URL: https://issues.apache.org/jira/browse/HIVE-1577
Project: Hive
Issue Type: New Feature
Components: Configuration
Reporter: Carl Steinbach
When Hive is run in local mode it uses the hardcoded local directory
{{/${java.io.tmpdir}/${user.name}}} for temporary files. This path should be
configurable via the property {{hive.exec.local.scratchdir}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4863) Fix parallel order by on hadoop2
[
https://issues.apache.org/jira/browse/HIVE-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735850#comment-13735850
]
Hive QA commented on HIVE-4863:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12597249/HIVE-4863.2.patch.txt
{color:green}SUCCESS:{color} +1 2774 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/380/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/380/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
Fix parallel order by on hadoop2
Key: HIVE-4863
URL: https://issues.apache.org/jira/browse/HIVE-4863
Project: Hive
Issue Type: Bug
Reporter: Edward Capriolo
Assignee: Edward Capriolo
Attachments: HIVE-4863.1.patch.txt, HIVE-4863.2.patch.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3772) Fix a concurrency bug in LazyBinaryUtils due to a static field (patch by Reynold Xin)
[
https://issues.apache.org/jira/browse/HIVE-3772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735894#comment-13735894
]
Hive QA commented on HIVE-3772:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12597250/HIVE-3772.1.patch.txt
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 2774 tests executed
*Failed tests:*
{noformat}
org.apache.hcatalog.mapreduce.TestHCatExternalDynamicPartitioned.testHCatDynamicPartitionedTable
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/381/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/381/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
Fix a concurrency bug in LazyBinaryUtils due to a static field (patch by
Reynold Xin)
-
Key: HIVE-3772
URL: https://issues.apache.org/jira/browse/HIVE-3772
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Affects Versions: 0.9.0
Reporter: Mikhail Bautin
Attachments: D7155.1.patch, D7155.2.patch, HIVE-3772.1.patch.txt,
HIVE-3772-2012-12-04.patch
Creating a JIRA for [~rxin]'s patch needed by the Shark project.
https://github.com/amplab/hive/commit/17e1c3dd2f6d8eca767115dc46d5a880aed8c765
writeVLong should not use a static field due to concurrency concerns.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5027) Upgrade Ivy to 2.3
[ https://issues.apache.org/jira/browse/HIVE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735919#comment-13735919 ] Hudson commented on HIVE-5027: -- SUCCESS: Integrated in Hive-trunk-h0.21 #2258 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2258/]) HIVE-5027 - Upgrade Ivy to 2.3 (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512309) * /hive/trunk/build.properties Upgrade Ivy to 2.3 -- Key: HIVE-5027 URL: https://issues.apache.org/jira/browse/HIVE-5027 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Fix For: 0.12.0 Attachments: HIVE-5027.patch Ivy version 2.1 doesn't include classifiers when creating pom files. Therefore our generated pom's are not correct. Version 2.3 fixes this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5006) Re-factor HiveServer2 JDBC PreparedStatement to avoid duplicate code
[ https://issues.apache.org/jira/browse/HIVE-5006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735918#comment-13735918 ] Hudson commented on HIVE-5006: -- SUCCESS: Integrated in Hive-trunk-h0.21 #2258 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2258/]) HIVE-5006: Re-factor HiveServer2 JDBC PreparedStatement to avoid duplicate code (Prasad Mujumdar via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512506) * /hive/trunk/jdbc/src/java/org/apache/hive/jdbc/HivePreparedStatement.java * /hive/trunk/jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java Re-factor HiveServer2 JDBC PreparedStatement to avoid duplicate code Key: HIVE-5006 URL: https://issues.apache.org/jira/browse/HIVE-5006 Project: Hive Issue Type: Improvement Components: HiveServer2, JDBC Affects Versions: 0.11.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.12.0 Attachments: HIVE-5006.1.patch, HIVE-5006.3.patch The JDBC PreparedStatement interface extends Statement interface. The Hive JDBC driver implementations of HivePreparedStatement interface is disjoint from the HiveStatement implementation. As a result there's some duplicate code in these two classes for executing query. Also patches like implicit close on re-execute etc. are not there for PreparedStatement. The HivePreparedStatement should extend the HiveStatement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-494) Select columns by index instead of name
[ https://issues.apache.org/jira/browse/HIVE-494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735928#comment-13735928 ] Xuefu Zhang commented on HIVE-494: -- Pig supports this, though using $1, $2 syntax, which is useful and convenient in some sense. However, I didn't find it's in standard SQL. One downside of supporting this is that ordering starts to matter now in the select list. If I do select a, b, c from T, the output is deterministic regardless T's schema (as long it has a, b, and c). On the other hand, if I do select $1, $2, $3 from T and if later on the table's schema is changed as (a, b, d, c), then my query will return a different data set. So, projecting by numbers is different from they just get translated into numbers anyway. Adding columns is quite common in hadoop data. Of course, one can argue that columns should always add at the end, which doesn't happen that way always. Select columns by index instead of name --- Key: HIVE-494 URL: https://issues.apache.org/jira/browse/HIVE-494 Project: Hive Issue Type: Wish Components: Clients, Query Processor Reporter: Adam Kramer Priority: Minor Labels: SQL Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-494.D1641.1.patch SELECT mytable[0], mytable[2] FROM some_table_name mytable; ...should return the first and third columns, respectively, from mytable regardless of their column names. The need for names specifically is kind of silly when they just get translated into numbers anyway. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-494) Select columns by index instead of name
[ https://issues.apache.org/jira/browse/HIVE-494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735932#comment-13735932 ] Edward Capriolo commented on HIVE-494: -- I think any user will realize that '$1' can change. In the end i think hive should be more dynamic somewhat like pig. Imagine something like this: create table x stored by dynamichandler; select $1 , $2 from x (inputformat=textinputformat, inpath=/x/y/z); We are close to this now because Navis added the ability to specify per query table properties. What is, or what is not in the SQL spec should not be our metric, we can already do amazing things that SQL can't so I want to keep innovating. As long as something does not produce an ambiguity in the language I see no harm in it. Select columns by index instead of name --- Key: HIVE-494 URL: https://issues.apache.org/jira/browse/HIVE-494 Project: Hive Issue Type: Wish Components: Clients, Query Processor Reporter: Adam Kramer Priority: Minor Labels: SQL Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-494.D1641.1.patch SELECT mytable[0], mytable[2] FROM some_table_name mytable; ...should return the first and third columns, respectively, from mytable regardless of their column names. The need for names specifically is kind of silly when they just get translated into numbers anyway. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1662) Add file pruning into Hive.
[
https://issues.apache.org/jira/browse/HIVE-1662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735933#comment-13735933
]
Hive QA commented on HIVE-1662:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12597252/HIVE-1662.8.patch.txt
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 2775 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/382/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/382/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
Add file pruning into Hive.
---
Key: HIVE-1662
URL: https://issues.apache.org/jira/browse/HIVE-1662
Project: Hive
Issue Type: New Feature
Reporter: He Yongqiang
Assignee: Navis
Attachments: HIVE-1662.8.patch.txt, HIVE-1662.D8391.1.patch,
HIVE-1662.D8391.2.patch, HIVE-1662.D8391.3.patch, HIVE-1662.D8391.4.patch,
HIVE-1662.D8391.5.patch, HIVE-1662.D8391.6.patch, HIVE-1662.D8391.7.patch
now hive support filename virtual column.
if a file name filter presents in a query, hive should be able to only add
files which passed the filter to input paths.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3772) Fix a concurrency bug in LazyBinaryUtils due to a static field
[ https://issues.apache.org/jira/browse/HIVE-3772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo updated HIVE-3772: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed. Thanks all. Fix a concurrency bug in LazyBinaryUtils due to a static field -- Key: HIVE-3772 URL: https://issues.apache.org/jira/browse/HIVE-3772 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 0.9.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Fix For: 0.12.0 Attachments: D7155.1.patch, D7155.2.patch, HIVE-3772.1.patch.txt, HIVE-3772-2012-12-04.patch Creating a JIRA for [~rxin]'s patch needed by the Shark project. https://github.com/amplab/hive/commit/17e1c3dd2f6d8eca767115dc46d5a880aed8c765 writeVLong should not use a static field due to concurrency concerns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3772) Fix a concurrency bug in LazyBinaryUtils due to a static field
[ https://issues.apache.org/jira/browse/HIVE-3772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo updated HIVE-3772: -- Fix Version/s: 0.12.0 Assignee: Mikhail Bautin Summary: Fix a concurrency bug in LazyBinaryUtils due to a static field (was: Fix a concurrency bug in LazyBinaryUtils due to a static field (patch by Reynold Xin)) Fix a concurrency bug in LazyBinaryUtils due to a static field -- Key: HIVE-3772 URL: https://issues.apache.org/jira/browse/HIVE-3772 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 0.9.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Fix For: 0.12.0 Attachments: D7155.1.patch, D7155.2.patch, HIVE-3772.1.patch.txt, HIVE-3772-2012-12-04.patch Creating a JIRA for [~rxin]'s patch needed by the Shark project. https://github.com/amplab/hive/commit/17e1c3dd2f6d8eca767115dc46d5a880aed8c765 writeVLong should not use a static field due to concurrency concerns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-494) Select columns by index instead of name
[ https://issues.apache.org/jira/browse/HIVE-494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735943#comment-13735943 ] Edward Capriolo commented on HIVE-494: -- I think we should also support negative numbers to query from the right end like awk's $NF Select columns by index instead of name --- Key: HIVE-494 URL: https://issues.apache.org/jira/browse/HIVE-494 Project: Hive Issue Type: Wish Components: Clients, Query Processor Reporter: Adam Kramer Priority: Minor Labels: SQL Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-494.D1641.1.patch SELECT mytable[0], mytable[2] FROM some_table_name mytable; ...should return the first and third columns, respectively, from mytable regardless of their column names. The need for names specifically is kind of silly when they just get translated into numbers anyway. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5054) Remove unused property submitviachild
[ https://issues.apache.org/jira/browse/HIVE-5054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735944#comment-13735944 ] Edward Capriolo commented on HIVE-5054: --- +1 . Thought I have often wondered aboutthis. What is the code design for ? Debugging? Remove unused property submitviachild - Key: HIVE-5054 URL: https://issues.apache.org/jira/browse/HIVE-5054 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-5054.patch This property only exist in HiveConf and is always set to false. Lets get rid of dead code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5054) Remove unused property submitviachild
[ https://issues.apache.org/jira/browse/HIVE-5054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735945#comment-13735945 ] Edward Capriolo commented on HIVE-5054: --- Could it be useful in hive server 2 type scenarios? hive.exec.submitviachildDetermines whether the map/reduce jobs should be submitted through a separate jvm in the non local mode.false - By default jobs are submitted through the same jvm as the compiler Remove unused property submitviachild - Key: HIVE-5054 URL: https://issues.apache.org/jira/browse/HIVE-5054 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-5054.patch This property only exist in HiveConf and is always set to false. Lets get rid of dead code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4999) Shim class HiveHarFileSystem does not have a hadoop2 counterpart
[ https://issues.apache.org/jira/browse/HIVE-4999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4999: --- Priority: Minor (was: Major) Shim class HiveHarFileSystem does not have a hadoop2 counterpart Key: HIVE-4999 URL: https://issues.apache.org/jira/browse/HIVE-4999 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Attachments: HIVE-4999.patch HiveHarFileSystem only exists in the 0.20 shim. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4999) Shim class HiveHarFileSystem does not have a hadoop2 counterpart
[ https://issues.apache.org/jira/browse/HIVE-4999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4999: --- Issue Type: Task (was: Sub-task) Parent: (was: HIVE-3949) Shim class HiveHarFileSystem does not have a hadoop2 counterpart Key: HIVE-4999 URL: https://issues.apache.org/jira/browse/HIVE-4999 Project: Hive Issue Type: Task Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Attachments: HIVE-4999.patch HiveHarFileSystem only exists in the 0.20 shim. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4999) Shim class HiveHarFileSystem does not have a hadoop2 counterpart
[ https://issues.apache.org/jira/browse/HIVE-4999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735949#comment-13735949 ] Brock Noland commented on HIVE-4999: Hey good call, I am not sure why I make this a subtask of the Hadoop2 test failure JIRA. No tests fail because of this...I just think since in practice it's shared so we should move it to the shared directory. Shim class HiveHarFileSystem does not have a hadoop2 counterpart Key: HIVE-4999 URL: https://issues.apache.org/jira/browse/HIVE-4999 Project: Hive Issue Type: Task Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Attachments: HIVE-4999.patch HiveHarFileSystem only exists in the 0.20 shim. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4838) Refactor MapJoin HashMap code to improve testability and readability
[ https://issues.apache.org/jira/browse/HIVE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735950#comment-13735950 ] Brock Noland commented on HIVE-4838: Good call, I will make the change tonight and update a new patch. Refactor MapJoin HashMap code to improve testability and readability Key: HIVE-4838 URL: https://issues.apache.org/jira/browse/HIVE-4838 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-4838.patch, HIVE-4838.patch, HIVE-4838.patch, HIVE-4838.patch, HIVE-4838.patch MapJoin is an essential component for high performance joins in Hive and the current code has done great service for many years. However, the code is showing it's age and currently suffers from the following issues: * Uses static state via the MapJoinMetaData class to pass serialization metadata to the Key, Row classes. * The api of a logical Table Container is not defined and therefore it's unclear what apis HashMapWrapper needs to publicize. Additionally HashMapWrapper has many used public methods. * HashMapWrapper contains logic to serialize, test memory bounds, and implement the table container. Ideally these logical units could be seperated * HashTableSinkObjectCtx has unused fields and unused methods * CommonJoinOperator and children use ArrayList on left hand side when only List is required * There are unused classes MRU, DCLLItemm and classes which duplicate functionality MapJoinSingleKey and MapJoinDoubleKeys -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4999) Shim class HiveHarFileSystem does not have a hadoop2 counterpart
[ https://issues.apache.org/jira/browse/HIVE-4999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735951#comment-13735951 ] Edward Capriolo commented on HIVE-4999: --- Ashutosh? Are you reviewing? I am +1 for this. It can not break anything existing. [~brocknoland] There must be some tests for HAR functionality that are excluding 23 we should change those as well? Shim class HiveHarFileSystem does not have a hadoop2 counterpart Key: HIVE-4999 URL: https://issues.apache.org/jira/browse/HIVE-4999 Project: Hive Issue Type: Task Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Attachments: HIVE-4999.patch HiveHarFileSystem only exists in the 0.20 shim. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5019) Use StringBuffer instead of += (issue 1)
[
https://issues.apache.org/jira/browse/HIVE-5019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edward Capriolo updated HIVE-5019:
--
Status: Open (was: Patch Available)
patch has a compile error
{quote}
[javac] Compiling 14 source files to
/data/hive-ptest/working/apache-svn-trunk-source/build/shims/classes
[javac]
/data/hive-ptest/working/apache-svn-trunk-source/shims/src/common-secure/java/org/apache/hadoop/hive/thrift/ZooKeeperTokenStore.java:168:
cannot find symbol
[javac] symbol : method
create(java.lang.StringBuffer,byte[],java.util.Listorg.apache.zookeeper.data.ACL,org.apache.zookeeper.CreateMode)
[javac] location: class org.apache.zookeeper.ZooKeeper
[javac] String node = zk.create(currentPath, new byte[0], acl,
{quote}
Use StringBuffer instead of += (issue 1)
Key: HIVE-5019
URL: https://issues.apache.org/jira/browse/HIVE-5019
Project: Hive
Issue Type: Sub-task
Reporter: Benjamin Jakobus
Assignee: Benjamin Jakobus
Fix For: 0.12.0
Attachments: HIVE-5019.1.patch.txt, HIVE-5019.2.patch.txt
Issue 1 (use of StringBuffer over +=)
java/org/apache/hadoop/hive/ql/optimizer/physical/GenMRSkewJoinProcessor.java
java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java
java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java
java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java
java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
java/org/apache/hadoop/hive/ql/security/authorization/BitSetCheckedAuthorizationProvider.java
java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsUtils.java
java/org/apache/hadoop/hive/ql/udf/UDFLike.java
java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSentences.java
java/org/apache/hadoop/hive/ql/udf/generic/NumDistinctValueEstimator.java
java/org/apache/hadoop/hive/ql/udf/ptf/NPath.java
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request 11029: HIVE-4513 - disable hivehistory logs by default
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11029/ --- (Updated Aug. 10, 2013, 4:24 p.m.) Review request for hive. Changes --- HIVE-4513.6.patch - addresses review comments. Fixes race condition that was causing the TestHiveServerSessions.testSessionVars test failure. Bugs: HIVE-4513 https://issues.apache.org/jira/browse/HIVE-4513 Repository: hive-git Description --- HiveHistory log files (hive_job_log_hive_*.txt files) store information about hive query such as query string, plan , counters and MR job progress information. There is no mechanism to delete these files and as a result they get accumulated over time, using up lot of disk space. I don't think this is used by most people, so I think it would better to turn this off by default. Jobtracker logs already capture most of this information, though it is not as structured as history logs. The change : A new config parameter hive.session.history.enabled controls if the history-log is enabled. By default it is set to false. SessionState initializes the HiveHIstory object. When this config is set to false, it creates a Proxy object that does not do anything. I did this instead of having SessionState return null, because that would add null checks in too many places. This keeps the code cleaner and avoids possibility of NPE. As the proxy only works against interfaces, i created a HiveHistory interface, moved the implementation to HiveHistoryImpl. static functions were moved to HiveHistoryUtil . Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 83f337b conf/hive-default.xml.template 0a6e433 hbase-handler/src/test/templates/TestHBaseCliDriver.vm c59e882 ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java 97436c5 ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryImpl.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryProxyHandler.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryUtil.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryViewer.java fdd56db ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java ab369f0 ql/src/test/org/apache/hadoop/hive/ql/history/TestHiveHistory.java a783303 ql/src/test/templates/TestCliDriver.vm a6ae6c3 Diff: https://reviews.apache.org/r/11029/diff/ Testing --- Thanks, Thejas Nair
Re: Review Request 11029: HIVE-4513 - disable hivehistory logs by default
On June 3, 2013, 9:03 p.m., Ashutosh Chauhan wrote: ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryImpl.java, line 86 https://reviews.apache.org/r/11029/diff/2/?file=290954#file290954line86 In case of incorrect config, should this throw an exception instead of silent return, otherwise there will be errors later when something is tried to be written in history file. Errors will not be there later, as it does not write if the histStream has not been initialized. I don't think we should fail the query just because hive history logging failed. This is also current behavior in hive. - Thejas --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11029/#review21352 --- On Aug. 10, 2013, 4:24 p.m., Thejas Nair wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11029/ --- (Updated Aug. 10, 2013, 4:24 p.m.) Review request for hive. Bugs: HIVE-4513 https://issues.apache.org/jira/browse/HIVE-4513 Repository: hive-git Description --- HiveHistory log files (hive_job_log_hive_*.txt files) store information about hive query such as query string, plan , counters and MR job progress information. There is no mechanism to delete these files and as a result they get accumulated over time, using up lot of disk space. I don't think this is used by most people, so I think it would better to turn this off by default. Jobtracker logs already capture most of this information, though it is not as structured as history logs. The change : A new config parameter hive.session.history.enabled controls if the history-log is enabled. By default it is set to false. SessionState initializes the HiveHIstory object. When this config is set to false, it creates a Proxy object that does not do anything. I did this instead of having SessionState return null, because that would add null checks in too many places. This keeps the code cleaner and avoids possibility of NPE. As the proxy only works against interfaces, i created a HiveHistory interface, moved the implementation to HiveHistoryImpl. static functions were moved to HiveHistoryUtil . Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 83f337b conf/hive-default.xml.template 0a6e433 hbase-handler/src/test/templates/TestHBaseCliDriver.vm c59e882 ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistory.java 97436c5 ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryImpl.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryProxyHandler.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryUtil.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryViewer.java fdd56db ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java ab369f0 ql/src/test/org/apache/hadoop/hive/ql/history/TestHiveHistory.java a783303 ql/src/test/templates/TestCliDriver.vm a6ae6c3 Diff: https://reviews.apache.org/r/11029/diff/ Testing --- Thanks, Thejas Nair
[jira] [Updated] (HIVE-4513) disable hivehistory logs by default
[ https://issues.apache.org/jira/browse/HIVE-4513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-4513: Attachment: HIVE-4513.6.patch HIVE-4513.6.patch - addresses review comments. Also fixes race condition the race condition that was causing the TestHiveServerSessions.testSessionVars test failure. This race condition gets exposed when hivehistory is disabled, because when hive history is enabled it attempts to create the same dir this way, but on failure it just logs a warning. disable hivehistory logs by default --- Key: HIVE-4513 URL: https://issues.apache.org/jira/browse/HIVE-4513 Project: Hive Issue Type: Bug Components: Configuration, Logging Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-4513.1.patch, HIVE-4513.2.patch, HIVE-4513.3.patch, HIVE-4513.4.patch, HIVE-4513.5.patch, HIVE-4513.6.patch HiveHistory log files (hive_job_log_hive_*.txt files) store information about hive query such as query string, plan , counters and MR job progress information. There is no mechanism to delete these files and as a result they get accumulated over time, using up lot of disk space. I don't think this is used by most people, so I think it would better to turn this off by default. Jobtracker logs already capture most of this information, though it is not as structured as history logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5054) Remove unused property submitviachild
[ https://issues.apache.org/jira/browse/HIVE-5054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735962#comment-13735962 ] Edward Capriolo commented on HIVE-5054: --- WE have an open issue where we are trying to avoid hive-server crashes caused by bad udfs in local mode maybe this remains an answer. Remove unused property submitviachild - Key: HIVE-5054 URL: https://issues.apache.org/jira/browse/HIVE-5054 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-5054.patch This property only exist in HiveConf and is always set to false. Lets get rid of dead code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5039) Support autoReconnect at JDBC
[
https://issues.apache.org/jira/browse/HIVE-5039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735963#comment-13735963
]
Hive QA commented on HIVE-5039:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12597258/HIVE-5039.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 2775 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/383/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/383/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
Support autoReconnect at JDBC
--
Key: HIVE-5039
URL: https://issues.apache.org/jira/browse/HIVE-5039
Project: Hive
Issue Type: New Feature
Components: JDBC
Affects Versions: 0.11.0
Reporter: Azrael Park
Assignee: Azrael Park
Priority: Trivial
Attachments: HIVE-5039.patch
If hiveServer2 is shutdown, connection is broken. Let the connection can
reconnect automatically after hiveServer2 re-started.
{noformat}
jdbc:hive2://localhost:1/default?autoReconnect=true
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-5055) SessionState temp file gets created in history file directory
Thejas M Nair created HIVE-5055: --- Summary: SessionState temp file gets created in history file directory Key: HIVE-5055 URL: https://issues.apache.org/jira/browse/HIVE-5055 Project: Hive Issue Type: Bug Affects Versions: 0.11.0 Reporter: Thejas M Nair SessionState.start creates a temp file for temp results, but this file is created in hive.querylog.location, which supposed to be used only for hive history log files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4513) disable hivehistory logs by default
[ https://issues.apache.org/jira/browse/HIVE-4513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735964#comment-13735964 ] Thejas M Nair commented on HIVE-4513: - I found another issue while making these changes, - SessionState temp file gets created in history file directory, created HIVE-5055 to track that. disable hivehistory logs by default --- Key: HIVE-4513 URL: https://issues.apache.org/jira/browse/HIVE-4513 Project: Hive Issue Type: Bug Components: Configuration, Logging Reporter: Thejas M Nair Assignee: Thejas M Nair Attachments: HIVE-4513.1.patch, HIVE-4513.2.patch, HIVE-4513.3.patch, HIVE-4513.4.patch, HIVE-4513.5.patch, HIVE-4513.6.patch HiveHistory log files (hive_job_log_hive_*.txt files) store information about hive query such as query string, plan , counters and MR job progress information. There is no mechanism to delete these files and as a result they get accumulated over time, using up lot of disk space. I don't think this is used by most people, so I think it would better to turn this off by default. Jobtracker logs already capture most of this information, though it is not as structured as history logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Key components of developer guide are blank!
Those sections have been blank since 2011 (beginning of Page History): https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27820469. On Sat, Aug 10, 2013 at 11:43 AM, Edward Capriolo edlinuxg...@gmail.comwrote: IF you editing this page recently please take a look. https://cwiki.apache.org/Hive/developerguide.html -- Lefty
Re: Key components of developer guide are blank!
I mean to say in my firefox browser I see this: Running Hive Without a Hadoop Cluster From Thejas: Then you can run 'build/dist/bin/hive' and it will work against your local file system. The section which lists the commands are empty. On Sat, Aug 10, 2013 at 12:56 PM, Lefty Leverenz leftylever...@gmail.comwrote: Those sections have been blank since 2011 (beginning of Page History): https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27820469. On Sat, Aug 10, 2013 at 11:43 AM, Edward Capriolo edlinuxg...@gmail.com wrote: IF you editing this page recently please take a look. https://cwiki.apache.org/Hive/developerguide.html -- Lefty
Discuss: End of static, thread local
I just committed https://issues.apache.org/jira/browse/HIVE-3772. For hive-server2 Carl and others did a lot of work to clean up un thread safe things from hive. Hive was originally build as a fat client so it is not surprising that many such constructs exist. Now since we have retrofitted multi-threaded-ness onto the project we have a number of edge case bugs. My suggestions here would be for that the next release 0.13 we make a push to remove all possible non thread safe code and explicitly pass context objects or serialized structures everywhere thread safety is needed. I can see this would start with something like the Function Registry, this would be a per session object passed around rather then a global object with static hashmap instances in it. I know that this probably will not be as simple as removing all static members from our codebase, but does anyone know of specific challenges that will be intrinsically hard to solve? Please comment.
[jira] [Commented] (HIVE-3772) Fix a concurrency bug in LazyBinaryUtils due to a static field
[ https://issues.apache.org/jira/browse/HIVE-3772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735980#comment-13735980 ] Konstantin Boudnik commented on HIVE-3772: -- Thank you so much, Edward. Fix a concurrency bug in LazyBinaryUtils due to a static field -- Key: HIVE-3772 URL: https://issues.apache.org/jira/browse/HIVE-3772 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 0.9.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Fix For: 0.12.0 Attachments: D7155.1.patch, D7155.2.patch, HIVE-3772.1.patch.txt, HIVE-3772-2012-12-04.patch Creating a JIRA for [~rxin]'s patch needed by the Shark project. https://github.com/amplab/hive/commit/17e1c3dd2f6d8eca767115dc46d5a880aed8c765 writeVLong should not use a static field due to concurrency concerns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3772) Fix a concurrency bug in LazyBinaryUtils due to a static field
[ https://issues.apache.org/jira/browse/HIVE-3772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735981#comment-13735981 ] Konstantin Boudnik commented on HIVE-3772: -- Edward, any chance it can be also backported into 0.11.1 ? Fix a concurrency bug in LazyBinaryUtils due to a static field -- Key: HIVE-3772 URL: https://issues.apache.org/jira/browse/HIVE-3772 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 0.9.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Fix For: 0.12.0 Attachments: D7155.1.patch, D7155.2.patch, HIVE-3772.1.patch.txt, HIVE-3772-2012-12-04.patch Creating a JIRA for [~rxin]'s patch needed by the Shark project. https://github.com/amplab/hive/commit/17e1c3dd2f6d8eca767115dc46d5a880aed8c765 writeVLong should not use a static field due to concurrency concerns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5054) Remove unused property submitviachild
[
https://issues.apache.org/jira/browse/HIVE-5054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735996#comment-13735996
]
Hive QA commented on HIVE-5054:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12597274/HIVE-5054.patch
{color:red}ERROR:{color} -1 due to 535 failed/errored test(s), 2774 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonreserved_keywords_insert_into1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_error
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_map_keys
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_quote2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_covar_pop
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input18
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_date_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_multipartitioning
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input11_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_find_in_set
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semicolon
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sort_merge_join_desc_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lateral_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_degrees
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union15
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_xpath_int
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_view_inputs
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_case_thrift
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rcfile_createas1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join27
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input_part5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_second
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_compression
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby6_noskew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ctas_date
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver_udf_row_sequence
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_vs_table_metadata
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_unix_timestamp
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver_fileformat_base64
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_to_unix_timestamp
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppr_pushdown2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_field
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_fs
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_general_queries
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_filters
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_distinct_samekey
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_diff_part_cols
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_19
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ba_table_udfs
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_lateralview
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multiMapJoin2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_outer_join4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_unquote_or
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_into5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_navfn
org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver_serde_typedbytes
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_virtual_column
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_min
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part6
Re: Key components of developer guide are blank!
Now I see the problem. Well, in the Safari browser code samples appear
throughout the *Compiling and Running Hive* section but *only on the
display version* (https://cwiki.apache.org/*confluence/display*/Hive/...),
not on the wiki URL that you gave (https://cwiki.apache.org/Hive/):
- Looks fine:
https://cwiki.apache.org/confluence/display/Hive/DeveloperGuide#DeveloperGuide-CompilingandRunningHive
- Missing code samples:
https://cwiki.apache.org/Hive/developerguide.html#DeveloperGuide-CompilingandRunningHive
In other docs, if the code sample markup is {noformat} then the code
appears in both versions of the wiki, but if it's {code} then it only
appears in the display version. Safari shows {code} in a dashed-line box
and {noformat} in a solid-line box. Examples:
1. Joins doc uses {code}:
https://cwiki.apache.org/Hive/languagemanual-joins.html
2. HCat load/store doc uses {noformat}:
https://cwiki.apache.org/Hive/hcatalog-loadstore.html
And that reminds me, several weeks ago {code} was broken even in the
display version, so I changed some docs to {noformat}. It's in the edit
comments for the pages I changed, so I could find when that was fairly
easily. But then {code} started working again. Maybe it's half-broken now.
On Sat, Aug 10, 2013 at 1:17 PM, Edward Capriolo edlinuxg...@gmail.comwrote:
I mean to say in my firefox browser I see this:
Running Hive Without a Hadoop Cluster
From Thejas:
Then you can run 'build/dist/bin/hive' and it will work against your local
file system.
The section which lists the commands are empty.
On Sat, Aug 10, 2013 at 12:56 PM, Lefty Leverenz leftylever...@gmail.com
wrote:
Those sections have been blank since 2011 (beginning of Page History):
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27820469.
On Sat, Aug 10, 2013 at 11:43 AM, Edward Capriolo edlinuxg...@gmail.com
wrote:
IF you editing this page recently please take a look.
https://cwiki.apache.org/Hive/developerguide.html
-- Lefty
-- Lefty
Re: Key components of developer guide are blank!
More problems: In the non-display wiki the language manual's docs *Union*,
*Sub Queries*, *Sampling*, and *Virtual Columns* give 404 Not Found but
they're fine in the display wiki.
- Union https://cwiki.apache.org/Hive/languagemanual-union.html
- Sub Queries https://cwiki.apache.org/Hive/languagemanual-subqueries.html
- Sampling https://cwiki.apache.org/Hive/languagemanual-sampling.html
- Virtual
Columnshttps://cwiki.apache.org/Hive/languagemanual-virtualcolumns.html
Maybe more, I don't have time for a thorough check right now. -- Lefty
On Sat, Aug 10, 2013 at 3:52 PM, Lefty Leverenz leftylever...@gmail.comwrote:
Now I see the problem. Well, in the Safari browser code samples appear
throughout the *Compiling and Running Hive* section but *only on the
display version* (https://cwiki.apache.org/*confluence/display*/Hive/...),
not on the wiki URL that you gave (https://cwiki.apache.org/Hive/):
- Looks fine:
https://cwiki.apache.org/confluence/display/Hive/DeveloperGuide#DeveloperGuide-CompilingandRunningHive
- Missing code samples:
https://cwiki.apache.org/Hive/developerguide.html#DeveloperGuide-CompilingandRunningHive
In other docs, if the code sample markup is {noformat} then the code
appears in both versions of the wiki, but if it's {code} then it only
appears in the display version. Safari shows {code} in a dashed-line box
and {noformat} in a solid-line box. Examples:
1. Joins doc uses {code}:
https://cwiki.apache.org/Hive/languagemanual-joins.html
2. HCat load/store doc uses {noformat}:
https://cwiki.apache.org/Hive/hcatalog-loadstore.html
And that reminds me, several weeks ago {code} was broken even in the
display version, so I changed some docs to {noformat}. It's in the edit
comments for the pages I changed, so I could find when that was fairly
easily. But then {code} started working again. Maybe it's half-broken now.
On Sat, Aug 10, 2013 at 1:17 PM, Edward Capriolo edlinuxg...@gmail.comwrote:
I mean to say in my firefox browser I see this:
Running Hive Without a Hadoop Cluster
From Thejas:
Then you can run 'build/dist/bin/hive' and it will work against your local
file system.
The section which lists the commands are empty.
On Sat, Aug 10, 2013 at 12:56 PM, Lefty Leverenz leftylever...@gmail.com
wrote:
Those sections have been blank since 2011 (beginning of Page History):
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27820469
.
On Sat, Aug 10, 2013 at 11:43 AM, Edward Capriolo
edlinuxg...@gmail.com
wrote:
IF you editing this page recently please take a look.
https://cwiki.apache.org/Hive/developerguide.html
-- Lefty
-- Lefty
[jira] [Commented] (HIVE-4513) disable hivehistory logs by default
[
https://issues.apache.org/jira/browse/HIVE-4513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736022#comment-13736022
]
Hive QA commented on HIVE-4513:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12597289/HIVE-4513.6.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 2776 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/385/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/385/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
disable hivehistory logs by default
---
Key: HIVE-4513
URL: https://issues.apache.org/jira/browse/HIVE-4513
Project: Hive
Issue Type: Bug
Components: Configuration, Logging
Reporter: Thejas M Nair
Assignee: Thejas M Nair
Attachments: HIVE-4513.1.patch, HIVE-4513.2.patch, HIVE-4513.3.patch,
HIVE-4513.4.patch, HIVE-4513.5.patch, HIVE-4513.6.patch
HiveHistory log files (hive_job_log_hive_*.txt files) store information about
hive query such as query string, plan , counters and MR job progress
information.
There is no mechanism to delete these files and as a result they get
accumulated over time, using up lot of disk space.
I don't think this is used by most people, so I think it would better to turn
this off by default. Jobtracker logs already capture most of this
information, though it is not as structured as history logs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5028) Some tests with fail OutOfMemoryError PermGen Space on Hadoop2
[
https://issues.apache.org/jira/browse/HIVE-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736029#comment-13736029
]
Hudson commented on HIVE-5028:
--
FAILURE: Integrated in Hive-trunk-hadoop2 #349 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2/349/])
HIVE-5028 : Some tests with fail OutOfMemoryError PermGen Space on Hadoop2
(Brock Noland via Ashutosh Chauhan) (hashutosh:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512576)
* /hive/trunk/hcatalog/build-support/ant/test.xml
* /hive/trunk/hcatalog/build.properties
* /hive/trunk/hcatalog/storage-handlers/hbase/build.xml
Some tests with fail OutOfMemoryError PermGen Space on Hadoop2
--
Key: HIVE-5028
URL: https://issues.apache.org/jira/browse/HIVE-5028
Project: Hive
Issue Type: Sub-task
Reporter: Brock Noland
Assignee: Brock Noland
Fix For: 0.12.0
Attachments: HIVE-5028.patch
{noformat}
java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:181)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:430)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:349)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:978)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:890)
at org.apache.hcatalog.cli.HCatDriver.run(HCatDriver.java:43)
at
org.apache.hcatalog.hbase.TestHBaseBulkOutputFormat.bulkModeAbortTest(TestHBaseBulkOutputFormat.java:540)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4324) ORC Turn off dictionary encoding when number of distinct keys is greater than threshold
[ https://issues.apache.org/jira/browse/HIVE-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4324: -- Attachment: HIVE-4324.D12045.3.patch omalley updated the revision HIVE-4324 [jira] ORC Turn off dictionary encoding when number of distinct keys is greater than threshold. Removed debugging line from q file that was making it pass in my machine but fail in jenkins. Reviewers: ashutoshc, JIRA REVISION DETAIL https://reviews.facebook.net/D12045 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D12045?vs=37245id=37521#toc BRANCH h-4324 ARCANIST PROJECT hive AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java conf/hive-default.xml.template ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/OutStream.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/StringRedBlackTree.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestFileDump.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java ql/src/test/queries/clientpositive/orc_dictionary_threshold.q ql/src/test/resources/orc-file-dump-dictionary-threshold.out ql/src/test/results/clientpositive/orc_dictionary_threshold.q.out To: JIRA, ashutoshc, omalley ORC Turn off dictionary encoding when number of distinct keys is greater than threshold --- Key: HIVE-4324 URL: https://issues.apache.org/jira/browse/HIVE-4324 Project: Hive Issue Type: Sub-task Components: File Formats Affects Versions: 0.11.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Fix For: 0.12.0 Attachments: HIVE-4324.1.patch.txt, HIVE-4324.D12045.1.patch, HIVE-4324.D12045.2.patch, HIVE-4324.D12045.2.patch, HIVE-4324.D12045.3.patch Add a configurable threshold so that if the number of distinct values in a string column is greater than that fraction of non-null values, dictionary encoding is turned off. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-5046) Hcatalog's bin/hcat script doesn't respect HIVE_HOME
[
https://issues.apache.org/jira/browse/HIVE-5046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-5046:
---
Resolution: Fixed
Fix Version/s: 0.12.0
Status: Resolved (was: Patch Available)
Committed to trunk! Thanks Mark for your contribution!
Hcatalog's bin/hcat script doesn't respect HIVE_HOME
Key: HIVE-5046
URL: https://issues.apache.org/jira/browse/HIVE-5046
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 0.11.0
Reporter: Mark Grover
Assignee: Mark Grover
Fix For: 0.12.0
Attachments: HIVE-5046.1.patch
https://github.com/apache/hive/blob/trunk/hcatalog/bin/hcat#L81
The quoted snippet (see below) intends to set HIVE_HOME if it's not set (i.e.
HIVE_HOME is currently null).
{code}
if [ -n ${HIVE_HOME} ]; then
{code}
However, {{-n}} checks if the variable is _not_ null. So, the above code ends
up setting HIVE_HOME to the default value if it is actually set already,
overriding the set value. This condition needs to be negated.
Moreover, {{-n}} checks requires the string being tested to be enclosed in
quotes.
Reference:
http://tldp.org/LDP/abs/html/comparison-ops.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4930) Classes of metastore should not be included MR-task
[ https://issues.apache.org/jira/browse/HIVE-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736090#comment-13736090 ] Hudson commented on HIVE-4930: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/]) HIVE-4930 : Classes of metastore should not be included MR-task (Navis via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511541) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ListBucketingCtx.java Classes of metastore should not be included MR-task --- Key: HIVE-4930 URL: https://issues.apache.org/jira/browse/HIVE-4930 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Navis Assignee: Navis Fix For: 0.12.0 Attachments: HIVE-4930.D11817.1.patch metastore.api.SkewedValueList is included in FileSinkOperator, which should not be. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4992) add ability to skip javadoc during build
[ https://issues.apache.org/jira/browse/HIVE-4992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736087#comment-13736087 ] Hudson commented on HIVE-4992: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/]) HIVE-4992 : add ability to skip javadoc during build (Sergey Shelukhin via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512138) * /hive/trunk/hcatalog/webhcat/svr/build.xml HIVE-4992 : add ability to skip javadoc during build (Sergey Shelukhin via Ashutosh h Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511374) * /hive/trunk/build.xml * /hive/trunk/hcatalog/build.xml add ability to skip javadoc during build Key: HIVE-4992 URL: https://issues.apache.org/jira/browse/HIVE-4992 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Trivial Fix For: 0.12.0 Attachments: HIVE-4992-addendum.patch, HIVE-4992.D11967.1.patch, HIVE-4992.D11967.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5033) Test result of ppd_vc.q is not udpated
[ https://issues.apache.org/jira/browse/HIVE-5033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736072#comment-13736072 ] Hudson commented on HIVE-5033: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/]) HIVE-5033: Test result of ppd_vc.q is not udpated (Navis Ryu via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512155) * /hive/trunk/ql/src/test/results/clientpositive/ppd_vc.q.out Test result of ppd_vc.q is not udpated --- Key: HIVE-5033 URL: https://issues.apache.org/jira/browse/HIVE-5033 Project: Hive Issue Type: Test Components: Tests Reporter: Navis Assignee: Navis Priority: Trivial Attachments: HIVE-5033.D12123.1.patch Also due to HIVE-3926. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4987) Javadoc can generate argument list too long error
[ https://issues.apache.org/jira/browse/HIVE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736083#comment-13736083 ] Hudson commented on HIVE-4987: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/]) HIVE-4987 : Javadoc can generate argument list too long error (Brock Noland via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511375) * /hive/trunk/build.xml * /hive/trunk/hcatalog/webhcat/svr/build.xml Javadoc can generate argument list too long error - Key: HIVE-4987 URL: https://issues.apache.org/jira/browse/HIVE-4987 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4987.patch We just to add to useexternalfile=yes to the javadoc statements. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3191) timestamp - timestamp causes null pointer exception
[ https://issues.apache.org/jira/browse/HIVE-3191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736075#comment-13736075 ] Hudson commented on HIVE-3191: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/]) HIVE-3191: timestamp - timestamp causes null pointer exception (Jason Dere via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512165) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/NumericOpMethodResolver.java * /hive/trunk/ql/src/test/queries/clientnegative/invalid_arithmetic_type.q * /hive/trunk/ql/src/test/results/clientnegative/invalid_arithmetic_type.q.out timestamp - timestamp causes null pointer exception --- Key: HIVE-3191 URL: https://issues.apache.org/jira/browse/HIVE-3191 Project: Hive Issue Type: Bug Affects Versions: 0.8.0 Reporter: N Campbell Assignee: Jason Dere Fix For: 0.12.0 Attachments: HIVE-3191.1.patch, HIVE-3191.2.patch, HIVE-3191.3.patch select tts.rnum, tts.cts - tts.cts from cert.tts tts Error: Query returned non-zero code: 12, cause: FAILED: Hive Internal Error: java.lang.NullPointerException(null) SQLState: 42000 ErrorCode: 12 create table if not exists CERT.TTS ( RNUM int , CTS timestamp) stored as sequencefile; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4911) Enable QOP configuration for Hive Server 2 thrift transport
[ https://issues.apache.org/jira/browse/HIVE-4911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736076#comment-13736076 ] Hudson commented on HIVE-4911: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/]) HIVE-4911 : Enable QOP configuration for Hive Server 2 thrift transport (Arup Malakar via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512010) * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/conf/hive-default.xml.template * /hive/trunk/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java * /hive/trunk/service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java * /hive/trunk/service/src/java/org/apache/hive/service/auth/KerberosSaslHelper.java * /hive/trunk/service/src/java/org/apache/hive/service/auth/SaslQOP.java * /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge20S.java * /hive/trunk/shims/src/common-secure/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java * /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge.java Enable QOP configuration for Hive Server 2 thrift transport --- Key: HIVE-4911 URL: https://issues.apache.org/jira/browse/HIVE-4911 Project: Hive Issue Type: New Feature Reporter: Arup Malakar Assignee: Arup Malakar Fix For: 0.12.0 Attachments: 20-build-temp-change-1.patch, 20-build-temp-change.patch, HIVE-4911-trunk-0.patch, HIVE-4911-trunk-1.patch, HIVE-4911-trunk-2.patch, HIVE-4911-trunk-3.patch The QoP for hive server 2 should be configurable to enable encryption. A new configuration should be exposed hive.server2.thrift.rpc.protection. This would give greater control configuring hive server 2 service. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4990) ORC seeks fails with non-zero offset or column projection
[ https://issues.apache.org/jira/browse/HIVE-4990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736088#comment-13736088 ] Hudson commented on HIVE-4990: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/]) HIVE-4990. ORC seeks fail with non-zero offset or column project (omalley) (omalley: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512062) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java ORC seeks fails with non-zero offset or column projection - Key: HIVE-4990 URL: https://issues.apache.org/jira/browse/HIVE-4990 Project: Hive Issue Type: Bug Components: File Formats Affects Versions: 0.11.0 Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.11.1, 0.12.0 Attachments: HIVE-4990.D12009.1.patch The ORC reader gets exceptions when seeking with non-zero offsets or column projection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5011) Dynamic partitioning in HCatalog broken on external tables
[
https://issues.apache.org/jira/browse/HIVE-5011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736079#comment-13736079
]
Hudson commented on HIVE-5011:
--
FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/])
HIVE-5011 Dynamic partitioning in HCatalog broken on external tables (Sushanth
Sowmyan, reviewed by Daniel Dai) (khorgath:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512031)
*
/hive/trunk/hcatalog/core/src/main/java/org/apache/hcatalog/mapreduce/FosterStorageHandler.java
*
/hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/mapreduce/HCatMapReduceTest.java
*
/hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/mapreduce/TestHCatExternalDynamicPartitioned.java
*
/hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/mapreduce/TestHCatExternalHCatNonPartitioned.java
*
/hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/mapreduce/TestHCatExternalPartitioned.java
Dynamic partitioning in HCatalog broken on external tables
--
Key: HIVE-5011
URL: https://issues.apache.org/jira/browse/HIVE-5011
Project: Hive
Issue Type: Bug
Components: HCatalog
Reporter: Sushanth Sowmyan
Assignee: Sushanth Sowmyan
Priority: Critical
Attachments: HIVE-5011.patch
Dynamic partitioning with HCatalog has been broken as a result of
HCATALOG-500 trying to support user-set paths for external tables.
The goal there was to be able to support other custom destinations apart from
the normal hive-style partitions. However, it is not currently possible for
users to set paths for dynamic ptn writes, since we don't support any way for
users to specify patterns(like, say $\{rootdir\}/$v1.$v2/) into which
writes happen, only locations, and the values for dyn. partitions are not
known ahead of time. Also, specifying a custom path messes with the way
dynamic ptn. code tries to determine what was written to where from the
output committer, which means that even if we supported patterned-writes
instead of location-writes, we still have to do some more deep diving into
the output committer code to support it.
Thus, my current proposal is that we honour writes to user-specified paths
for external tables *ONLY* for static partition writes - i.e., if we can
determine that the write is a dyn. ptn. write, we will ignore the user
specification. (Note that this does not mean we ignore the table's external
location - we honour that - we just don't honour any HCatStorer/etc provided
additional location - we stick to what metadata tells us the root location is.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5028) Some tests with fail OutOfMemoryError PermGen Space on Hadoop2
[
https://issues.apache.org/jira/browse/HIVE-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736078#comment-13736078
]
Hudson commented on HIVE-5028:
--
FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/])
HIVE-5028 : Some tests with fail OutOfMemoryError PermGen Space on Hadoop2
(Brock Noland via Ashutosh Chauhan) (hashutosh:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512576)
* /hive/trunk/hcatalog/build-support/ant/test.xml
* /hive/trunk/hcatalog/build.properties
* /hive/trunk/hcatalog/storage-handlers/hbase/build.xml
Some tests with fail OutOfMemoryError PermGen Space on Hadoop2
--
Key: HIVE-5028
URL: https://issues.apache.org/jira/browse/HIVE-5028
Project: Hive
Issue Type: Sub-task
Reporter: Brock Noland
Assignee: Brock Noland
Fix For: 0.12.0
Attachments: HIVE-5028.patch
{noformat}
java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:181)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:430)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:349)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:978)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:890)
at org.apache.hcatalog.cli.HCatDriver.run(HCatDriver.java:43)
at
org.apache.hcatalog.hbase.TestHBaseBulkOutputFormat.bulkModeAbortTest(TestHBaseBulkOutputFormat.java:540)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5017) DBTokenStore gives compiler warnings
[ https://issues.apache.org/jira/browse/HIVE-5017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736074#comment-13736074 ] Hudson commented on HIVE-5017: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/]) HIVE-5017 : DBTokenStore gives compiler warnings (Gunther Hagleitner via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512137) * /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/thrift/DBTokenStore.java DBTokenStore gives compiler warnings Key: HIVE-5017 URL: https://issues.apache.org/jira/browse/HIVE-5017 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: 0.12.0 Attachments: HIVE-5017.1.patch The Method.invoke call in 2 cases is done via (Object[])null but empty Object array will shut up the compiler. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5027) Upgrade Ivy to 2.3
[ https://issues.apache.org/jira/browse/HIVE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736080#comment-13736080 ] Hudson commented on HIVE-5027: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/]) HIVE-5027 - Upgrade Ivy to 2.3 (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512309) * /hive/trunk/build.properties Upgrade Ivy to 2.3 -- Key: HIVE-5027 URL: https://issues.apache.org/jira/browse/HIVE-5027 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Fix For: 0.12.0 Attachments: HIVE-5027.patch Ivy version 2.1 doesn't include classifiers when creating pom files. Therefore our generated pom's are not correct. Version 2.3 fixes this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5010) HCatalog maven integration doesn't override mvn.local.repo in two locations
[
https://issues.apache.org/jira/browse/HIVE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736084#comment-13736084
]
Hudson commented on HIVE-5010:
--
FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/])
HIVE-5010: HCatalog maven integration doesn't override mvn.local.repo in two
locations (brock:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511847)
* /hive/trunk/hcatalog/build-support/ant/deploy.xml
HCatalog maven integration doesn't override mvn.local.repo in two locations
---
Key: HIVE-5010
URL: https://issues.apache.org/jira/browse/HIVE-5010
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Brock Noland
Assignee: Brock Noland
Fix For: 0.12.0
Attachments: HIVE-5010.patch
Most locations in hcatalog/build-support/ant/deploy.xml specify:
{noformat}
localRepository path=${mvn.local.repo}/
{noformat}
but there are two locations which do not. This causes issues with ptest2
since those two locations end up using $HOME/.m2
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4913) Put deterministic ordering in the top-K ngrams output of UDF context_ngrams()
[ https://issues.apache.org/jira/browse/HIVE-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736073#comment-13736073 ] Hudson commented on HIVE-4913: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/]) HIVE-4913 : Put deterministic ordering in the top-K ngrams output of UDF context_ngrams() (Xuefu Zhang via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512590) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NGramEstimator.java * /hive/trunk/ql/src/test/results/clientpositive/udaf_context_ngrams.q.out Put deterministic ordering in the top-K ngrams output of UDF context_ngrams() - Key: HIVE-4913 URL: https://issues.apache.org/jira/browse/HIVE-4913 Project: Hive Issue Type: Improvement Components: UDF Affects Versions: 0.10.0, 0.11.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4913.patch, HIVE-4913.patch Currently UDF context_ngrams() output top-K ngrams in the order of descending frequency. When there are ties, i.e., phrases with same ngram value, the order is indeterministic. Tie breaker is needed to break the ties, so that the output is deterministic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4886) beeline code should have apache license headers
[ https://issues.apache.org/jira/browse/HIVE-4886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736086#comment-13736086 ] Hudson commented on HIVE-4886: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/]) HIVE-4886 Beeline code should have Apache license headers. (Thejas M Nair via omalley) (omalley: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511485) * /hive/trunk/beeline/src/java/org/apache/hive/beeline/AbstractCommandHandler.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/AbstractOutputFormat.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLine.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLineCommandCompletor.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLineCompletor.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLineSignalHandler.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/BufferedRows.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/ColorBuffer.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/CommandHandler.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/Commands.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/DatabaseConnection.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/DatabaseConnections.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/DriverInfo.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/IncrementalRows.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/OutputFile.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/OutputFormat.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/ReflectiveCommandHandler.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/Reflector.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/Rows.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/SQLCompletor.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/SeparatedValuesOutputFormat.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/SunSignalHandler.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/TableNameCompletor.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/TableOutputFormat.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/VerticalOutputFormat.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/XMLAttributeOutputFormat.java * /hive/trunk/beeline/src/java/org/apache/hive/beeline/XMLElementOutputFormat.java beeline code should have apache license headers --- Key: HIVE-4886 URL: https://issues.apache.org/jira/browse/HIVE-4886 Project: Hive Issue Type: Task Components: JDBC Reporter: Thejas M Nair Assignee: Thejas M Nair Fix For: 0.12.0 Attachments: HIVE-4886.2.patch, HIVE-4886.patch The beeline jdbc client added as part of hive server2 changes is based on SQLLine. As beeline is modified version of SQLLine and further modifications are also under apache license, the license headers of these files need to be replaced with apache license headers. We already have the license text of SQLLine in LICENSE file . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5006) Re-factor HiveServer2 JDBC PreparedStatement to avoid duplicate code
[ https://issues.apache.org/jira/browse/HIVE-5006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736071#comment-13736071 ] Hudson commented on HIVE-5006: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/]) HIVE-5006: Re-factor HiveServer2 JDBC PreparedStatement to avoid duplicate code (Prasad Mujumdar via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512506) * /hive/trunk/jdbc/src/java/org/apache/hive/jdbc/HivePreparedStatement.java * /hive/trunk/jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java Re-factor HiveServer2 JDBC PreparedStatement to avoid duplicate code Key: HIVE-5006 URL: https://issues.apache.org/jira/browse/HIVE-5006 Project: Hive Issue Type: Improvement Components: HiveServer2, JDBC Affects Versions: 0.11.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.12.0 Attachments: HIVE-5006.1.patch, HIVE-5006.3.patch The JDBC PreparedStatement interface extends Statement interface. The Hive JDBC driver implementations of HivePreparedStatement interface is disjoint from the HiveStatement implementation. As a result there's some duplicate code in these two classes for executing query. Also patches like implicit close on re-execute etc. are not there for PreparedStatement. The HivePreparedStatement should extend the HiveStatement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3772) Fix a concurrency bug in LazyBinaryUtils due to a static field
[ https://issues.apache.org/jira/browse/HIVE-3772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736089#comment-13736089 ] Hudson commented on HIVE-3772: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/]) HIVE-3772 Fix concurrency bug in LazyBinaryUtils due to a static field (Mikhail Bautin via egc) Submitted by:Mikhail Bautin and Reynold Xin Reviewed by: Edward Capriolo Approved by: Edward Capriolo (ecapriolo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512758) * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java Fix a concurrency bug in LazyBinaryUtils due to a static field -- Key: HIVE-3772 URL: https://issues.apache.org/jira/browse/HIVE-3772 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 0.9.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Fix For: 0.12.0 Attachments: D7155.1.patch, D7155.2.patch, HIVE-3772.1.patch.txt, HIVE-3772-2012-12-04.patch Creating a JIRA for [~rxin]'s patch needed by the Shark project. https://github.com/amplab/hive/commit/17e1c3dd2f6d8eca767115dc46d5a880aed8c765 writeVLong should not use a static field due to concurrency concerns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4233) The TGT gotten from class 'CLIService' should be renewed on time
[ https://issues.apache.org/jira/browse/HIVE-4233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736082#comment-13736082 ] Hudson commented on HIVE-4233: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/]) HIVE-4233: The TGT gotten from class 'CLIService' should be renewed on time (Thejas M Nair via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511574) * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java * /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java * /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java * /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java The TGT gotten from class 'CLIService' should be renewed on time - Key: HIVE-4233 URL: https://issues.apache.org/jira/browse/HIVE-4233 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 0.11.0 Environment: CentOS release 6.3 (Final) jdk1.6.0_31 HiveServer2 0.10.0-cdh4.2.0 Kerberos Security Reporter: Dongyong Wang Assignee: Thejas M Nair Priority: Critical Fix For: 0.12.0 Attachments: 0001-FIX-HIVE-4233.patch, HIVE-4233-2.patch, HIVE-4233-3.patch, HIVE-4233.4.patch, HIVE-4233.5.patch When the HIveServer2 have started more than 7 days, I use beeline shell to connect the HiveServer2,all operation failed. The log of HiveServer2 shows it was caused by the Kerberos auth failure,the exception stack trace is: 2013-03-26 11:55:20,932 ERROR hive.ql.metadata.Hive: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1084) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.init(RetryingMetaStoreClient.java:51) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:61) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2140) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2151) at org.apache.hadoop.hive.ql.metadata.Hive.getDelegationToken(Hive.java:2275) at org.apache.hive.service.cli.CLIService.getDelegationTokenFromMetaStore(CLIService.java:358) at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:127) at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1073) at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1058) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge20S.java:565) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedConstructorAccessor52.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1082) ... 16 more Caused by: java.lang.IllegalStateException: This ticket is no longer valid at javax.security.auth.kerberos.KerberosTicket.toString(KerberosTicket.java:601) at java.lang.String.valueOf(String.java:2826) at java.lang.StringBuilder.append(StringBuilder.java:115) at sun.security.jgss.krb5.SubjectComber.findAux(SubjectComber.java:120) at sun.security.jgss.krb5.SubjectComber.find(SubjectComber.java:41) at sun.security.jgss.krb5.Krb5Util.getTicket(Krb5Util.java:130) at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:328) at java.security.AccessController.doPrivileged(Native Method) at sun.security.jgss.krb5.Krb5InitCredential.getTgt(Krb5InitCredential.java:325) at
[jira] [Commented] (HIVE-3926) PPD on virtual column of partitioned table is not working
[
https://issues.apache.org/jira/browse/HIVE-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736085#comment-13736085
]
Hudson commented on HIVE-3926:
--
FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See
[https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/])
HIVE-5026: HIVE-3926 is committed in the state of not rebased to trunk (Navis
Ryu via Gunther Hagleitner) (gunther:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511977)
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java
HIVE-3926 PPD on virtual column of partitioned table is not working (Navis Ryu
and Gunther Hagleitner via egc) (ecapriolo:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511578)
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/VirtualColumn.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/AbstractBucketJoinProc.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/AbstractSMBJoinProc.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/AvgPartitionSizeBasedBigTableSelectorForAutoSMJ.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GlobalLimitOptimizer.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GroupByOptimizer.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/TableSizeBasedBigTableSelectorForAutoSMJ.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/listbucketingpruner/LBPartitionProcFactory.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcCtx.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrOpProcFactory.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartExprEvalUtils.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java
* /hive/trunk/ql/src/test/queries/clientpositive/ppd_vc.q
* /hive/trunk/ql/src/test/results/clientpositive/ppd_vc.q.out
PPD on virtual column of partitioned table is not working
-
Key: HIVE-3926
URL: https://issues.apache.org/jira/browse/HIVE-3926
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor
Fix For: 0.12.0
Attachments: HIVE-3926.6.patch, HIVE-3926.D8121.1.patch,
HIVE-3926.D8121.2.patch, HIVE-3926.D8121.3.patch, HIVE-3926.D8121.4.patch,
HIVE-3926.D8121.5.patch
{code}
select * from src where BLOCK__OFFSET__INSIDE__FILE100;
{code}
is working, but
{code}
select * from srcpart where BLOCK__OFFSET__INSIDE__FILE100;
{code}
throws SemanticException. Disabling PPD makes it work.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4967) Don't serialize unnecessary fields in query plan
[ https://issues.apache.org/jira/browse/HIVE-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736077#comment-13736077 ] Hudson commented on HIVE-4967: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/]) HIVE-4967 : Don't serialize unnecessary fields in query plan (Ashutosh Chauhan. Reviewed by Brock Noland) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511377) * /hive/trunk/contrib/src/java/org/apache/hadoop/hive/contrib/genericudf/example/GenericUDFDBOutput.java * /hive/trunk/contrib/src/java/org/apache/hadoop/hive/contrib/udtf/example/GenericUDTFExplode2.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFEWAHBitmapBop.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFReflect.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFAverage.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFContextNGrams.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCorrelation.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovariance.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEWAHBitmap.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFHistogramNumeric.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFLeadLag.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMax.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMin.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFNTile.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVariance.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFArray.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFArrayContains.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseCompare.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCase.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCoalesce.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcatWS.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEWAHBitmapEmpty.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFElt.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFField.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFormatNumber.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFromUtcTimestamp.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFHash.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIf.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIn.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInstr.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLocate.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMap.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMapKeys.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMapValues.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFNvl.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPAnd.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPNot.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPOr.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFPrintf.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFReflect2.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSentences.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSize.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java *
[jira] [Commented] (HIVE-5026) HIVE-3926 is committed in the state of not rebased to trunk
[ https://issues.apache.org/jira/browse/HIVE-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736081#comment-13736081 ] Hudson commented on HIVE-5026: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #51 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/51/]) HIVE-5026: HIVE-3926 is committed in the state of not rebased to trunk (Navis Ryu via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511977) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java HIVE-3926 is committed in the state of not rebased to trunk --- Key: HIVE-5026 URL: https://issues.apache.org/jira/browse/HIVE-5026 Project: Hive Issue Type: Bug Components: Tests Reporter: Navis Assignee: Navis Priority: Blocker Fix For: 0.12.0 Attachments: HIVE-5026.D12099.1.patch Current trunk build fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4999) Shim class HiveHarFileSystem does not have a hadoop2 counterpart
[ https://issues.apache.org/jira/browse/HIVE-4999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736092#comment-13736092 ] Brock Noland commented on HIVE-4999: Hey, There are test for har for 0.23 (archive_excludeHadoop20.q). I do believe these are failing for 0.23 (HIVE-3025) but not for because of the issue described here. There isn't any binary incompatibility problem here so the class compiled with 0.20 works fine for 0.23. It's just that the file is in the wrong directory. Shim class HiveHarFileSystem does not have a hadoop2 counterpart Key: HIVE-4999 URL: https://issues.apache.org/jira/browse/HIVE-4999 Project: Hive Issue Type: Task Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Attachments: HIVE-4999.patch HiveHarFileSystem only exists in the 0.20 shim. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4838) Refactor MapJoin HashMap code to improve testability and readability
[ https://issues.apache.org/jira/browse/HIVE-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4838: --- Attachment: HIVE-4838.patch The rebased patch is on trunk. I was thinking about our plan and I have a concern. Since we aren't allocating memory in large chunks when we do OOM it's likely to be a very slow process with the local task doing lots of GC before finally throwing an OOM. Therefore in the case where we fail with an OOM I think it could be a significant negatively impact on performance. How about we commit the patch as-is and then file a follow-on JIRA so that I or someone else can prove or disprove this theory. Refactor MapJoin HashMap code to improve testability and readability Key: HIVE-4838 URL: https://issues.apache.org/jira/browse/HIVE-4838 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland Attachments: HIVE-4838.patch, HIVE-4838.patch, HIVE-4838.patch, HIVE-4838.patch, HIVE-4838.patch, HIVE-4838.patch MapJoin is an essential component for high performance joins in Hive and the current code has done great service for many years. However, the code is showing it's age and currently suffers from the following issues: * Uses static state via the MapJoinMetaData class to pass serialization metadata to the Key, Row classes. * The api of a logical Table Container is not defined and therefore it's unclear what apis HashMapWrapper needs to publicize. Additionally HashMapWrapper has many used public methods. * HashMapWrapper contains logic to serialize, test memory bounds, and implement the table container. Ideally these logical units could be seperated * HashTableSinkObjectCtx has unused fields and unused methods * CommonJoinOperator and children use ArrayList on left hand side when only List is required * There are unused classes MRU, DCLLItemm and classes which duplicate functionality MapJoinSingleKey and MapJoinDoubleKeys -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Discuss: End of static, thread local
I would love to get rid of the static thread local stuff. It was required to make hive work in a server model but isn't the correct solution to this problem. I do think it will be a large amount of work so it'd be great to see whoever leads this effort to have a high level plan as opposed to an adhoc effort. On Sat, Aug 10, 2013 at 12:32 PM, Edward Capriolo edlinuxg...@gmail.comwrote: I just committed https://issues.apache.org/jira/browse/HIVE-3772. For hive-server2 Carl and others did a lot of work to clean up un thread safe things from hive. Hive was originally build as a fat client so it is not surprising that many such constructs exist. Now since we have retrofitted multi-threaded-ness onto the project we have a number of edge case bugs. My suggestions here would be for that the next release 0.13 we make a push to remove all possible non thread safe code and explicitly pass context objects or serialized structures everywhere thread safety is needed. I can see this would start with something like the Function Registry, this would be a per session object passed around rather then a global object with static hashmap instances in it. I know that this probably will not be as simple as removing all static members from our codebase, but does anyone know of specific challenges that will be intrinsically hard to solve? Please comment. -- Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
[jira] [Updated] (HIVE-4246) Implement predicate pushdown for ORC
[ https://issues.apache.org/jira/browse/HIVE-4246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-4246: -- Attachment: HIVE-4246.D11415.2.patch omalley updated the revision HIVE-4246 [jira] Implement predicate pushdown for ORC. Addressed Gunther's comments. Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D11415 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D11415?vs=34977id=37533#toc AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/BitFieldReader.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/InStream.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSerde.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java ql/src/java/org/apache/hadoop/hive/ql/io/orc/RunLengthByteReader.java ql/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestBitFieldReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInStream.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRecordReaderImpl.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRunLengthByteReader.java ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRunLengthIntegerReader.java ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestSearchArgumentImpl.java serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java To: JIRA, omalley Cc: hagleitn Implement predicate pushdown for ORC Key: HIVE-4246 URL: https://issues.apache.org/jira/browse/HIVE-4246 Project: Hive Issue Type: New Feature Components: File Formats Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: HIVE-4246.D11415.1.patch, HIVE-4246.D11415.2.patch By using the push down predicates from the table scan operator, ORC can skip over 10,000 rows at a time that won't satisfy the predicate. This will help a lot, especially if the file is sorted by the column that is used in the predicate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4123) The RLE encoding for ORC can be improved
[ https://issues.apache.org/jira/browse/HIVE-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736097#comment-13736097 ] Owen O'Malley commented on HIVE-4123: - +1, it looks good to me. The RLE encoding for ORC can be improved Key: HIVE-4123 URL: https://issues.apache.org/jira/browse/HIVE-4123 Project: Hive Issue Type: New Feature Components: File Formats Affects Versions: 0.12.0 Reporter: Owen O'Malley Assignee: Prasanth J Labels: orcfile Fix For: 0.12.0 Attachments: HIVE-4123.1.git.patch.txt, HIVE-4123.2.git.patch.txt, HIVE-4123.3.patch.txt, HIVE-4123.4.patch.txt, HIVE-4123.5.txt, HIVE-4123.6.txt, HIVE-4123.7.txt, HIVE-4123.8.txt, ORC-Compression-Ratio-Comparison.xlsx The run length encoding of integers can be improved: * tighter bit packing * allow delta encoding * allow longer runs -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3772) Fix a concurrency bug in LazyBinaryUtils due to a static field
[ https://issues.apache.org/jira/browse/HIVE-3772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736099#comment-13736099 ] Edward Capriolo commented on HIVE-3772: --- Generally in hive we do not back port we just move forward. There are not many .1 or .2 releases. Fix a concurrency bug in LazyBinaryUtils due to a static field -- Key: HIVE-3772 URL: https://issues.apache.org/jira/browse/HIVE-3772 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 0.9.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Fix For: 0.12.0 Attachments: D7155.1.patch, D7155.2.patch, HIVE-3772.1.patch.txt, HIVE-3772-2012-12-04.patch Creating a JIRA for [~rxin]'s patch needed by the Shark project. https://github.com/amplab/hive/commit/17e1c3dd2f6d8eca767115dc46d5a880aed8c765 writeVLong should not use a static field due to concurrency concerns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4579) Create a SARG interface for RecordReaders
[ https://issues.apache.org/jira/browse/HIVE-4579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736103#comment-13736103 ] Owen O'Malley commented on HIVE-4579: - All of the unit tests pass on my machine and those tests are unrelated to the code in this patch. Create a SARG interface for RecordReaders - Key: HIVE-4579 URL: https://issues.apache.org/jira/browse/HIVE-4579 Project: Hive Issue Type: Improvement Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: h-4579.patch, HIVE-4579.D11409.1.patch, HIVE-4579.D11409.2.patch, pushdown.pdf I think we should create a SARG (http://en.wikipedia.org/wiki/Sargable) interface for RecordReaders. For a first pass, I'll create an API that uses the value stored in hive.io.filter.expr.serialized. The desire is to define an simpler interface that the direct AST expression that is provided by hive.io.filter.expr.serialized so that the code to evaluate expressions can be generalized instead of put inside a particular RecordReader. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4999) Shim class HiveHarFileSystem does not have a hadoop2 counterpart
[ https://issues.apache.org/jira/browse/HIVE-4999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736104#comment-13736104 ] Edward Capriolo commented on HIVE-4999: --- I am +1. Will run move in 24 hours unless someone stops me. Shim class HiveHarFileSystem does not have a hadoop2 counterpart Key: HIVE-4999 URL: https://issues.apache.org/jira/browse/HIVE-4999 Project: Hive Issue Type: Task Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Attachments: HIVE-4999.patch HiveHarFileSystem only exists in the 0.20 shim. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3772) Fix a concurrency bug in LazyBinaryUtils due to a static field
[ https://issues.apache.org/jira/browse/HIVE-3772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736107#comment-13736107 ] Konstantin Boudnik commented on HIVE-3772: -- Got it, thanks for the explanation. Fix a concurrency bug in LazyBinaryUtils due to a static field -- Key: HIVE-3772 URL: https://issues.apache.org/jira/browse/HIVE-3772 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 0.9.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Fix For: 0.12.0 Attachments: D7155.1.patch, D7155.2.patch, HIVE-3772.1.patch.txt, HIVE-3772-2012-12-04.patch Creating a JIRA for [~rxin]'s patch needed by the Shark project. https://github.com/amplab/hive/commit/17e1c3dd2f6d8eca767115dc46d5a880aed8c765 writeVLong should not use a static field due to concurrency concerns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4579) Create a SARG interface for RecordReaders
[ https://issues.apache.org/jira/browse/HIVE-4579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736108#comment-13736108 ] Edward Capriolo commented on HIVE-4579: --- I think I roughly understand the interface, we are going to pass a Sarg into the conf of the RecordReader, then the record reader can apply these directly on the input row? Create a SARG interface for RecordReaders - Key: HIVE-4579 URL: https://issues.apache.org/jira/browse/HIVE-4579 Project: Hive Issue Type: Improvement Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: h-4579.patch, HIVE-4579.D11409.1.patch, HIVE-4579.D11409.2.patch, pushdown.pdf I think we should create a SARG (http://en.wikipedia.org/wiki/Sargable) interface for RecordReaders. For a first pass, I'll create an API that uses the value stored in hive.io.filter.expr.serialized. The desire is to define an simpler interface that the direct AST expression that is provided by hive.io.filter.expr.serialized so that the code to evaluate expressions can be generalized instead of put inside a particular RecordReader. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4579) Create a SARG interface for RecordReaders
[
https://issues.apache.org/jira/browse/HIVE-4579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736110#comment-13736110
]
Edward Capriolo commented on HIVE-4579:
---
{code}
+private final StackExpressionTree currentTree =
+new StackExpressionTree();
{code}
Can you use Deque and ArrayDeque here instead.
Create a SARG interface for RecordReaders
-
Key: HIVE-4579
URL: https://issues.apache.org/jira/browse/HIVE-4579
Project: Hive
Issue Type: Improvement
Reporter: Owen O'Malley
Assignee: Owen O'Malley
Attachments: h-4579.patch, HIVE-4579.D11409.1.patch,
HIVE-4579.D11409.2.patch, pushdown.pdf
I think we should create a SARG (http://en.wikipedia.org/wiki/Sargable)
interface for RecordReaders. For a first pass, I'll create an API that uses
the value stored in hive.io.filter.expr.serialized.
The desire is to define an simpler interface that the direct AST expression
that is provided by hive.io.filter.expr.serialized so that the code to
evaluate expressions can be generalized instead of put inside a particular
RecordReader.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4579) Create a SARG interface for RecordReaders
[ https://issues.apache.org/jira/browse/HIVE-4579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736112#comment-13736112 ] Edward Capriolo commented on HIVE-4579: --- Other then the Deque and ArrayDeque I am +1. Create a SARG interface for RecordReaders - Key: HIVE-4579 URL: https://issues.apache.org/jira/browse/HIVE-4579 Project: Hive Issue Type: Improvement Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: h-4579.patch, HIVE-4579.D11409.1.patch, HIVE-4579.D11409.2.patch, pushdown.pdf I think we should create a SARG (http://en.wikipedia.org/wiki/Sargable) interface for RecordReaders. For a first pass, I'll create an API that uses the value stored in hive.io.filter.expr.serialized. The desire is to define an simpler interface that the direct AST expression that is provided by hive.io.filter.expr.serialized so that the code to evaluate expressions can be generalized instead of put inside a particular RecordReader. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4579) Create a SARG interface for RecordReaders
[
https://issues.apache.org/jira/browse/HIVE-4579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736113#comment-13736113
]
Edward Capriolo commented on HIVE-4579:
---
One other q.
{quote}
+ public static enum Type {
+INTEGER, // all of the integer types
+FLOAT, // float and double
+STRING
+ }
{quote}
Should we call these Integral, Real instead of INTEGER, FLOAT? Or should we
call them LONG and DOUBLE? Because naming them the widest type might make more
sense?
Create a SARG interface for RecordReaders
-
Key: HIVE-4579
URL: https://issues.apache.org/jira/browse/HIVE-4579
Project: Hive
Issue Type: Improvement
Reporter: Owen O'Malley
Assignee: Owen O'Malley
Attachments: h-4579.patch, HIVE-4579.D11409.1.patch,
HIVE-4579.D11409.2.patch, pushdown.pdf
I think we should create a SARG (http://en.wikipedia.org/wiki/Sargable)
interface for RecordReaders. For a first pass, I'll create an API that uses
the value stored in hive.io.filter.expr.serialized.
The desire is to define an simpler interface that the direct AST expression
that is provided by hive.io.filter.expr.serialized so that the code to
evaluate expressions can be generalized instead of put inside a particular
RecordReader.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5046) Hcatalog's bin/hcat script doesn't respect HIVE_HOME
[
https://issues.apache.org/jira/browse/HIVE-5046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736134#comment-13736134
]
Hudson commented on HIVE-5046:
--
SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #122 (See
[https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/122/])
HIVE-5046: Hcatalog's bin/hcat script doesn't respect HIVE_HOME (Mark Grover
via Brock Noland) (brock:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512832)
* /hive/trunk/hcatalog/bin/hcat
Hcatalog's bin/hcat script doesn't respect HIVE_HOME
Key: HIVE-5046
URL: https://issues.apache.org/jira/browse/HIVE-5046
Project: Hive
Issue Type: Bug
Components: HCatalog
Affects Versions: 0.11.0
Reporter: Mark Grover
Assignee: Mark Grover
Fix For: 0.12.0
Attachments: HIVE-5046.1.patch
https://github.com/apache/hive/blob/trunk/hcatalog/bin/hcat#L81
The quoted snippet (see below) intends to set HIVE_HOME if it's not set (i.e.
HIVE_HOME is currently null).
{code}
if [ -n ${HIVE_HOME} ]; then
{code}
However, {{-n}} checks if the variable is _not_ null. So, the above code ends
up setting HIVE_HOME to the default value if it is actually set already,
overriding the set value. This condition needs to be negated.
Moreover, {{-n}} checks requires the string being tested to be enclosed in
quotes.
Reference:
http://tldp.org/LDP/abs/html/comparison-ops.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4913) Put deterministic ordering in the top-K ngrams output of UDF context_ngrams()
[ https://issues.apache.org/jira/browse/HIVE-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736124#comment-13736124 ] Hudson commented on HIVE-4913: -- SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #122 (See [https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/122/]) HIVE-4913 : Put deterministic ordering in the top-K ngrams output of UDF context_ngrams() (Xuefu Zhang via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512590) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NGramEstimator.java * /hive/trunk/ql/src/test/results/clientpositive/udaf_context_ngrams.q.out Put deterministic ordering in the top-K ngrams output of UDF context_ngrams() - Key: HIVE-4913 URL: https://issues.apache.org/jira/browse/HIVE-4913 Project: Hive Issue Type: Improvement Components: UDF Affects Versions: 0.10.0, 0.11.0 Reporter: Xuefu Zhang Assignee: Xuefu Zhang Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4913.patch, HIVE-4913.patch Currently UDF context_ngrams() output top-K ngrams in the order of descending frequency. When there are ties, i.e., phrases with same ngram value, the order is indeterministic. Tie breaker is needed to break the ties, so that the output is deterministic. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3772) Fix a concurrency bug in LazyBinaryUtils due to a static field
[ https://issues.apache.org/jira/browse/HIVE-3772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736138#comment-13736138 ] Hudson commented on HIVE-3772: -- SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #122 (See [https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/122/]) HIVE-3772 Fix concurrency bug in LazyBinaryUtils due to a static field (Mikhail Bautin via egc) Submitted by:Mikhail Bautin and Reynold Xin Reviewed by: Edward Capriolo Approved by: Edward Capriolo (ecapriolo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512758) * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java Fix a concurrency bug in LazyBinaryUtils due to a static field -- Key: HIVE-3772 URL: https://issues.apache.org/jira/browse/HIVE-3772 Project: Hive Issue Type: Bug Components: Serializers/Deserializers Affects Versions: 0.9.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Fix For: 0.12.0 Attachments: D7155.1.patch, D7155.2.patch, HIVE-3772.1.patch.txt, HIVE-3772-2012-12-04.patch Creating a JIRA for [~rxin]'s patch needed by the Shark project. https://github.com/amplab/hive/commit/17e1c3dd2f6d8eca767115dc46d5a880aed8c765 writeVLong should not use a static field due to concurrency concerns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5017) DBTokenStore gives compiler warnings
[ https://issues.apache.org/jira/browse/HIVE-5017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736125#comment-13736125 ] Hudson commented on HIVE-5017: -- SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #122 (See [https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/122/]) HIVE-5017 : DBTokenStore gives compiler warnings (Gunther Hagleitner via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512137) * /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/thrift/DBTokenStore.java DBTokenStore gives compiler warnings Key: HIVE-5017 URL: https://issues.apache.org/jira/browse/HIVE-5017 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: 0.12.0 Attachments: HIVE-5017.1.patch The Method.invoke call in 2 cases is done via (Object[])null but empty Object array will shut up the compiler. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4911) Enable QOP configuration for Hive Server 2 thrift transport
[ https://issues.apache.org/jira/browse/HIVE-4911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736126#comment-13736126 ] Hudson commented on HIVE-4911: -- SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #122 (See [https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/122/]) HIVE-4911 : Enable QOP configuration for Hive Server 2 thrift transport (Arup Malakar via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512010) * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/conf/hive-default.xml.template * /hive/trunk/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java * /hive/trunk/service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java * /hive/trunk/service/src/java/org/apache/hive/service/auth/KerberosSaslHelper.java * /hive/trunk/service/src/java/org/apache/hive/service/auth/SaslQOP.java * /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge20S.java * /hive/trunk/shims/src/common-secure/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java * /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge.java Enable QOP configuration for Hive Server 2 thrift transport --- Key: HIVE-4911 URL: https://issues.apache.org/jira/browse/HIVE-4911 Project: Hive Issue Type: New Feature Reporter: Arup Malakar Assignee: Arup Malakar Fix For: 0.12.0 Attachments: 20-build-temp-change-1.patch, 20-build-temp-change.patch, HIVE-4911-trunk-0.patch, HIVE-4911-trunk-1.patch, HIVE-4911-trunk-2.patch, HIVE-4911-trunk-3.patch The QoP for hive server 2 should be configurable to enable encryption. A new configuration should be exposed hive.server2.thrift.rpc.protection. This would give greater control configuring hive server 2 service. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4992) add ability to skip javadoc during build
[ https://issues.apache.org/jira/browse/HIVE-4992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736136#comment-13736136 ] Hudson commented on HIVE-4992: -- SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #122 (See [https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/122/]) HIVE-4992 : add ability to skip javadoc during build (Sergey Shelukhin via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512138) * /hive/trunk/hcatalog/webhcat/svr/build.xml add ability to skip javadoc during build Key: HIVE-4992 URL: https://issues.apache.org/jira/browse/HIVE-4992 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Trivial Fix For: 0.12.0 Attachments: HIVE-4992-addendum.patch, HIVE-4992.D11967.1.patch, HIVE-4992.D11967.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3926) PPD on virtual column of partitioned table is not working
[
https://issues.apache.org/jira/browse/HIVE-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736135#comment-13736135
]
Hudson commented on HIVE-3926:
--
SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #122 (See
[https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/122/])
HIVE-5026: HIVE-3926 is committed in the state of not rebased to trunk (Navis
Ryu via Gunther Hagleitner) (gunther:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511977)
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java
HIVE-3926 PPD on virtual column of partitioned table is not working (Navis Ryu
and Gunther Hagleitner via egc) (ecapriolo:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511578)
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/VirtualColumn.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/AbstractBucketJoinProc.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/AbstractSMBJoinProc.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/AvgPartitionSizeBasedBigTableSelectorForAutoSMJ.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GlobalLimitOptimizer.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GroupByOptimizer.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/TableSizeBasedBigTableSelectorForAutoSMJ.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/listbucketingpruner/LBPartitionProcFactory.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcCtx.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrOpProcFactory.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartExprEvalUtils.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java
* /hive/trunk/ql/src/test/queries/clientpositive/ppd_vc.q
* /hive/trunk/ql/src/test/results/clientpositive/ppd_vc.q.out
PPD on virtual column of partitioned table is not working
-
Key: HIVE-3926
URL: https://issues.apache.org/jira/browse/HIVE-3926
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor
Fix For: 0.12.0
Attachments: HIVE-3926.6.patch, HIVE-3926.D8121.1.patch,
HIVE-3926.D8121.2.patch, HIVE-3926.D8121.3.patch, HIVE-3926.D8121.4.patch,
HIVE-3926.D8121.5.patch
{code}
select * from src where BLOCK__OFFSET__INSIDE__FILE100;
{code}
is working, but
{code}
select * from srcpart where BLOCK__OFFSET__INSIDE__FILE100;
{code}
throws SemanticException. Disabling PPD makes it work.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5028) Some tests with fail OutOfMemoryError PermGen Space on Hadoop2
[
https://issues.apache.org/jira/browse/HIVE-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736128#comment-13736128
]
Hudson commented on HIVE-5028:
--
SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #122 (See
[https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/122/])
HIVE-5028 : Some tests with fail OutOfMemoryError PermGen Space on Hadoop2
(Brock Noland via Ashutosh Chauhan) (hashutosh:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512576)
* /hive/trunk/hcatalog/build-support/ant/test.xml
* /hive/trunk/hcatalog/build.properties
* /hive/trunk/hcatalog/storage-handlers/hbase/build.xml
Some tests with fail OutOfMemoryError PermGen Space on Hadoop2
--
Key: HIVE-5028
URL: https://issues.apache.org/jira/browse/HIVE-5028
Project: Hive
Issue Type: Sub-task
Reporter: Brock Noland
Assignee: Brock Noland
Fix For: 0.12.0
Attachments: HIVE-5028.patch
{noformat}
java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:181)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:430)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:349)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:978)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:890)
at org.apache.hcatalog.cli.HCatDriver.run(HCatDriver.java:43)
at
org.apache.hcatalog.hbase.TestHBaseBulkOutputFormat.bulkModeAbortTest(TestHBaseBulkOutputFormat.java:540)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5006) Re-factor HiveServer2 JDBC PreparedStatement to avoid duplicate code
[ https://issues.apache.org/jira/browse/HIVE-5006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736122#comment-13736122 ] Hudson commented on HIVE-5006: -- SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #122 (See [https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/122/]) HIVE-5006: Re-factor HiveServer2 JDBC PreparedStatement to avoid duplicate code (Prasad Mujumdar via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512506) * /hive/trunk/jdbc/src/java/org/apache/hive/jdbc/HivePreparedStatement.java * /hive/trunk/jdbc/src/test/org/apache/hive/jdbc/TestJdbcDriver2.java Re-factor HiveServer2 JDBC PreparedStatement to avoid duplicate code Key: HIVE-5006 URL: https://issues.apache.org/jira/browse/HIVE-5006 Project: Hive Issue Type: Improvement Components: HiveServer2, JDBC Affects Versions: 0.11.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.12.0 Attachments: HIVE-5006.1.patch, HIVE-5006.3.patch The JDBC PreparedStatement interface extends Statement interface. The Hive JDBC driver implementations of HivePreparedStatement interface is disjoint from the HiveStatement implementation. As a result there's some duplicate code in these two classes for executing query. Also patches like implicit close on re-execute etc. are not there for PreparedStatement. The HivePreparedStatement should extend the HiveStatement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5010) HCatalog maven integration doesn't override mvn.local.repo in two locations
[
https://issues.apache.org/jira/browse/HIVE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736133#comment-13736133
]
Hudson commented on HIVE-5010:
--
SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #122 (See
[https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/122/])
HIVE-5010: HCatalog maven integration doesn't override mvn.local.repo in two
locations (brock:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1511847)
* /hive/trunk/hcatalog/build-support/ant/deploy.xml
HCatalog maven integration doesn't override mvn.local.repo in two locations
---
Key: HIVE-5010
URL: https://issues.apache.org/jira/browse/HIVE-5010
Project: Hive
Issue Type: Bug
Affects Versions: 0.12.0
Reporter: Brock Noland
Assignee: Brock Noland
Fix For: 0.12.0
Attachments: HIVE-5010.patch
Most locations in hcatalog/build-support/ant/deploy.xml specify:
{noformat}
localRepository path=${mvn.local.repo}/
{noformat}
but there are two locations which do not. This causes issues with ptest2
since those two locations end up using $HOME/.m2
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5011) Dynamic partitioning in HCatalog broken on external tables
[
https://issues.apache.org/jira/browse/HIVE-5011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736129#comment-13736129
]
Hudson commented on HIVE-5011:
--
SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #122 (See
[https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/122/])
HIVE-5011 Dynamic partitioning in HCatalog broken on external tables (Sushanth
Sowmyan, reviewed by Daniel Dai) (khorgath:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512031)
*
/hive/trunk/hcatalog/core/src/main/java/org/apache/hcatalog/mapreduce/FosterStorageHandler.java
*
/hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/mapreduce/HCatMapReduceTest.java
*
/hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/mapreduce/TestHCatExternalDynamicPartitioned.java
*
/hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/mapreduce/TestHCatExternalHCatNonPartitioned.java
*
/hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/mapreduce/TestHCatExternalPartitioned.java
Dynamic partitioning in HCatalog broken on external tables
--
Key: HIVE-5011
URL: https://issues.apache.org/jira/browse/HIVE-5011
Project: Hive
Issue Type: Bug
Components: HCatalog
Reporter: Sushanth Sowmyan
Assignee: Sushanth Sowmyan
Priority: Critical
Attachments: HIVE-5011.patch
Dynamic partitioning with HCatalog has been broken as a result of
HCATALOG-500 trying to support user-set paths for external tables.
The goal there was to be able to support other custom destinations apart from
the normal hive-style partitions. However, it is not currently possible for
users to set paths for dynamic ptn writes, since we don't support any way for
users to specify patterns(like, say $\{rootdir\}/$v1.$v2/) into which
writes happen, only locations, and the values for dyn. partitions are not
known ahead of time. Also, specifying a custom path messes with the way
dynamic ptn. code tries to determine what was written to where from the
output committer, which means that even if we supported patterned-writes
instead of location-writes, we still have to do some more deep diving into
the output committer code to support it.
Thus, my current proposal is that we honour writes to user-specified paths
for external tables *ONLY* for static partition writes - i.e., if we can
determine that the write is a dyn. ptn. write, we will ignore the user
specification. (Note that this does not mean we ignore the table's external
location - we honour that - we just don't honour any HCatStorer/etc provided
additional location - we stick to what metadata tells us the root location is.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5033) Test result of ppd_vc.q is not udpated
[ https://issues.apache.org/jira/browse/HIVE-5033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736123#comment-13736123 ] Hudson commented on HIVE-5033: -- SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #122 (See [https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/122/]) HIVE-5033: Test result of ppd_vc.q is not udpated (Navis Ryu via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512155) * /hive/trunk/ql/src/test/results/clientpositive/ppd_vc.q.out Test result of ppd_vc.q is not udpated --- Key: HIVE-5033 URL: https://issues.apache.org/jira/browse/HIVE-5033 Project: Hive Issue Type: Test Components: Tests Reporter: Navis Assignee: Navis Priority: Trivial Attachments: HIVE-5033.D12123.1.patch Also due to HIVE-3926. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3191) timestamp - timestamp causes null pointer exception
[ https://issues.apache.org/jira/browse/HIVE-3191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736127#comment-13736127 ] Hudson commented on HIVE-3191: -- SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #122 (See [https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/122/]) HIVE-3191: timestamp - timestamp causes null pointer exception (Jason Dere via Gunther Hagleitner) (gunther: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512165) * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/NumericOpMethodResolver.java * /hive/trunk/ql/src/test/queries/clientnegative/invalid_arithmetic_type.q * /hive/trunk/ql/src/test/results/clientnegative/invalid_arithmetic_type.q.out timestamp - timestamp causes null pointer exception --- Key: HIVE-3191 URL: https://issues.apache.org/jira/browse/HIVE-3191 Project: Hive Issue Type: Bug Affects Versions: 0.8.0 Reporter: N Campbell Assignee: Jason Dere Fix For: 0.12.0 Attachments: HIVE-3191.1.patch, HIVE-3191.2.patch, HIVE-3191.3.patch select tts.rnum, tts.cts - tts.cts from cert.tts tts Error: Query returned non-zero code: 12, cause: FAILED: Hive Internal Error: java.lang.NullPointerException(null) SQLState: 42000 ErrorCode: 12 create table if not exists CERT.TTS ( RNUM int , CTS timestamp) stored as sequencefile; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5027) Upgrade Ivy to 2.3
[ https://issues.apache.org/jira/browse/HIVE-5027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13736130#comment-13736130 ] Hudson commented on HIVE-5027: -- SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #122 (See [https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/122/]) HIVE-5027 - Upgrade Ivy to 2.3 (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1512309) * /hive/trunk/build.properties Upgrade Ivy to 2.3 -- Key: HIVE-5027 URL: https://issues.apache.org/jira/browse/HIVE-5027 Project: Hive Issue Type: Improvement Components: Build Infrastructure Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Fix For: 0.12.0 Attachments: HIVE-5027.patch Ivy version 2.1 doesn't include classifiers when creating pom files. Therefore our generated pom's are not correct. Version 2.3 fixes this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
