[jira] [Commented] (HIVE-16589) Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and COMPLETE for AVG, VARIANCE
[ https://issues.apache.org/jira/browse/HIVE-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106334#comment-16106334 ] Lefty Leverenz commented on HIVE-16589: --- Doc note: This adds two configs (*hive.vectorized.complex.types.enabled* and *hive.vectorized.groupby.complex.types.enabled*) to HiveConf.java, so they need to be documented in the wiki. * [Configuration Properties -- Vectorization | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-Vectorization] Added a TODOC3.0 label. > Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and > COMPLETE for AVG, VARIANCE > --- > > Key: HIVE-16589 > URL: https://issues.apache.org/jira/browse/HIVE-16589 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-16589.01.patch, HIVE-16589.02.patch, > HIVE-16589.03.patch, HIVE-16589.04.patch, HIVE-16589.05.patch, > HIVE-16589.06.patch, HIVE-16589.07.patch, HIVE-16589.08.patch, > HIVE-16589.091.patch, HIVE-16589.092.patch, HIVE-16589.093.patch, > HIVE-16589.094.patch, HIVE-16589.095.patch, HIVE-16589.096.patch, > HIVE-16589.097.patch, HIVE-16589.098.patch, HIVE-16589.0991.patch, > HIVE-16589.0992.patch, HIVE-16589.0993.patch, HIVE-16589.0994.patch, > HIVE-16589.0995.patch, HIVE-16589.099.patch, HIVE-16589.09.patch > > > Allow Complex Types to be vectorized (since HIVE-16207: "Add support for > Complex Types in Fast SerDe" was committed). > Add more classes we vectorize AVG in preparation for fully supporting AVG > GroupBy. In particular, the PARTIAL2 and FINAL groupby modes that take in > the AVG struct as input. And, add the COMPLETE mode that takes in the > Original data and produces the Full Aggregation for completeness, so to speak. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16589) Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and COMPLETE for AVG, VARIANCE
[ https://issues.apache.org/jira/browse/HIVE-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-16589: -- Labels: TODOC3.0 (was: ) > Vectorization: Support Complex Types and GroupBy modes PARTIAL2, FINAL, and > COMPLETE for AVG, VARIANCE > --- > > Key: HIVE-16589 > URL: https://issues.apache.org/jira/browse/HIVE-16589 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Labels: TODOC3.0 > Fix For: 3.0.0 > > Attachments: HIVE-16589.01.patch, HIVE-16589.02.patch, > HIVE-16589.03.patch, HIVE-16589.04.patch, HIVE-16589.05.patch, > HIVE-16589.06.patch, HIVE-16589.07.patch, HIVE-16589.08.patch, > HIVE-16589.091.patch, HIVE-16589.092.patch, HIVE-16589.093.patch, > HIVE-16589.094.patch, HIVE-16589.095.patch, HIVE-16589.096.patch, > HIVE-16589.097.patch, HIVE-16589.098.patch, HIVE-16589.0991.patch, > HIVE-16589.0992.patch, HIVE-16589.0993.patch, HIVE-16589.0994.patch, > HIVE-16589.0995.patch, HIVE-16589.099.patch, HIVE-16589.09.patch > > > Allow Complex Types to be vectorized (since HIVE-16207: "Add support for > Complex Types in Fast SerDe" was committed). > Add more classes we vectorize AVG in preparation for fully supporting AVG > GroupBy. In particular, the PARTIAL2 and FINAL groupby modes that take in > the AVG struct as input. And, add the COMPLETE mode that takes in the > Original data and produces the Full Aggregation for completeness, so to speak. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17190) Schema changes for bitvectors for unpartitioned tables
[ https://issues.apache.org/jira/browse/HIVE-17190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106314#comment-16106314 ] Pengcheng Xiong commented on HIVE-17190: sounds like we did not have bitvectors for perfclidriver when we load tables... > Schema changes for bitvectors for unpartitioned tables > -- > > Key: HIVE-17190 > URL: https://issues.apache.org/jira/browse/HIVE-17190 > Project: Hive > Issue Type: Test > Components: Metastore, Statistics >Affects Versions: 3.0.0 >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Attachments: HIVE-17190.2.patch > > > Missed in HIVE-16997 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17167) Create metastore specific configuration tool
[
https://issues.apache.org/jira/browse/HIVE-17167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106210#comment-16106210
]
Vihang Karajgaonkar commented on HIVE-17167:
Hi [~alangates], left some comments on the github review. I am a bit concerned
that this might become error-prone when there are different values for the
metastore key and hive variant of that key. Either we should enforce the order
of lookup in the get() method or not allow conflicting values of different
variants of the keys. I can easily see the end-users setting some configuration
in metastore-site.xml but code is looking up the hive variant of the key and in
the end the configuration change is not really getting enabled in the code.
Instead of controlling how not to allow users doing
{{conf.set("metastore.myconfig.name", "x")}} via reviews, I think it would be
good to enforce this in code somehow.
> Create metastore specific configuration tool
>
>
> Key: HIVE-17167
> URL: https://issues.apache.org/jira/browse/HIVE-17167
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-17167.patch
>
>
> As part of making the metastore a separately releasable module we need
> configuration tools that are specific to that module. It cannot use or
> extend HiveConf as that is in hive common. But it must take a HiveConf
> object and be able to operate on it.
> The best way to achieve this is using Hadoop's Configuration object (which
> HiveConf extends) together with enums and static methods.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17195) Long chain of tasks created by REPL LOAD shouldn't cause stack corruption.
[
https://issues.apache.org/jira/browse/HIVE-17195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106193#comment-16106193
]
Hive QA commented on HIVE-17195:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12879492/HIVE-17195.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11017 tests
executed
*Failed tests:*
{noformat}
TestPerfCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=240)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=99)
org.apache.hadoop.hive.metastore.TestHiveMetaStoreStatsMerge.testStatsMerge
(batchId=206)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=179)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6190/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6190/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6190/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12879492 - PreCommit-HIVE-Build
> Long chain of tasks created by REPL LOAD shouldn't cause stack corruption.
> --
>
> Key: HIVE-17195
> URL: https://issues.apache.org/jira/browse/HIVE-17195
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
> Labels: DAG, DR, Executor, replication
> Fix For: 3.0.0
>
> Attachments: HIVE-17195.01.patch, HIVE-17195.02.patch
>
>
> Currently, long chain REPL LOAD tasks lead to huge recursive calls when try
> to traverse the DAG.
> For example, getMRTasks, getTezTasks, getSparkTasks and iterateTasks methods
> run recursively to traverse the DAG.
> Need to modify this traversal logic to reduce stack usage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16998) Add config to enable HoS DPP only for map-joins
[
https://issues.apache.org/jira/browse/HIVE-16998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106177#comment-16106177
]
Sahil Takiar commented on HIVE-16998:
-
+1
> Add config to enable HoS DPP only for map-joins
> ---
>
> Key: HIVE-16998
> URL: https://issues.apache.org/jira/browse/HIVE-16998
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer, Spark
>Reporter: Sahil Takiar
>Assignee: Janaki Lahorani
> Attachments: HIVE16998.1.patch, HIVE16998.2.patch, HIVE16998.3.patch,
> HIVE16998.4.patch, HIVE16998.5.patch
>
>
> HoS DPP will split a given operator tree in two under the following
> conditions: it has detected that the query can benefit from DPP, and the
> filter is not a map-join (see SplitOpTreeForDPP).
> This can hurt performance if the the non-partitioned side of the join
> involves a complex operator tree - e.g. the query {{select count(*) from
> srcpart where srcpart.ds in (select max(srcpart.ds) from srcpart union all
> select min(srcpart.ds) from srcpart)}} will require running the subquery
> twice, once in each Spark job.
> Queries with map-joins don't get split into two operator trees and thus don't
> suffer from this drawback. Thus, it would be nice to have a config key that
> just enables DPP on HoS for map-joins.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17205) add functional support
[
https://issues.apache.org/jira/browse/HIVE-17205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106145#comment-16106145
]
Hive QA commented on HIVE-17205:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12879463/HIVE-17205.01.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 11020 tests
executed
*Failed tests:*
{noformat}
TestPerfCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
TestTxnCommandsBase - did not produce a TEST-*.xml file (likely timed out)
(batchId=281)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_nonacid_from_acid]
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
(batchId=22)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=99)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[delete_non_acid_table]
(batchId=90)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[update_non_acid_table]
(batchId=90)
org.apache.hadoop.hive.metastore.TestHiveMetaStoreStatsMerge.testStatsMerge
(batchId=206)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderFooterSerializeWithDeltas
(batchId=264)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderNoFooterSerializeWithDeltas
(batchId=264)
org.apache.hadoop.hive.ql.io.orc.TestOrcRawRecordMerger.testNewBaseAndDelta
(batchId=264)
org.apache.hadoop.hive.ql.io.orc.TestOrcRawRecordMerger.testRecordReaderNewBaseAndDelta
(batchId=264)
org.apache.hadoop.hive.ql.io.orc.TestOrcRawRecordMerger.testRecordReaderOldBaseAndDelta
(batchId=264)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=179)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6188/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6188/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6188/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12879463 - PreCommit-HIVE-Build
> add functional support
> --
>
> Key: HIVE-17205
> URL: https://issues.apache.org/jira/browse/HIVE-17205
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-17205.01.patch
>
>
> make sure unbucketed tables can be marked transactional=true
> make insert/update/delete/compaction work
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16908) Failures in TestHcatClient due to HIVE-16844
[
https://issues.apache.org/jira/browse/HIVE-16908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106123#comment-16106123
]
Hive QA commented on HIVE-16908:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12879456/HIVE-16908.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 11017 tests
executed
*Failed tests:*
{noformat}
TestPerfCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=100)
org.apache.hadoop.hive.metastore.TestHiveMetaStoreStatsMerge.testStatsMerge
(batchId=206)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6187/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6187/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6187/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12879456 - PreCommit-HIVE-Build
> Failures in TestHcatClient due to HIVE-16844
>
>
> Key: HIVE-16908
> URL: https://issues.apache.org/jira/browse/HIVE-16908
> Project: Hive
> Issue Type: Bug
>Reporter: Sunitha Beeram
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-16908.1.patch, HIVE-16908.2.patch,
> HIVE-16908.3.patch
>
>
> Some of the tests in TestHCatClient.java, for ex:
> {noformat}
> org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
> (batchId=177)
> org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
> (batchId=177)
> org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
> (batchId=177)
> {noformat}
> are failing due to HIVE-16844. HIVE-16844 fixes a connection leak when a new
> configuration object is set on the ObjectStore. TestHCatClient fires up a
> second instance of metastore thread with a different conf object that results
> in the PersistenceMangaerFactory closure and hence tests fail.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17207) Hiveserver2 fails to start when hive.server2.authentication is set to KERBEROS
[
https://issues.apache.org/jira/browse/HIVE-17207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rey Rey Chang updated HIVE-17207:
-
Description:
The logic in getAuthTransFactory() is flawed, as it will always return
"Unsupported authentication type" error when authTypeStr is set to KERBEROS.
inside HiveAuthFactory class.
public TTransportFactory getAuthTransFactory()
throws LoginException
{
TTransportFactory transportFactory;
if (isSASLWithKerberizedHadoop())
{
try
{
serverTransportFactory =
this.saslServer.createSaslServerTransportFactory(
getSaslProperties());
}
catch (TTransportException e)
{
TSaslServerTransport.Factory serverTransportFactory;
throw new LoginException(e.getMessage());
}
TSaslServerTransport.Factory serverTransportFactory;
if (!this.authTypeStr.equalsIgnoreCase(AuthTypes.KERBEROS.getAuthName()))
{
if ((this.authTypeStr.equalsIgnoreCase(AuthTypes.NONE.getAuthName()))
||
(this.authTypeStr.equalsIgnoreCase(AuthTypes.LDAP.getAuthName())) ||
(this.authTypeStr.equalsIgnoreCase(AuthTypes.PAM.getAuthName())) ||
(this.authTypeStr.equalsIgnoreCase(AuthTypes.CUSTOM.getAuthName( {
try
{
serverTransportFactory.addServerDefinition("PLAIN",
this.authTypeStr, null, new HashMap(), new
PlainSaslHelper.PlainServerCallbackHandler(this.authTypeStr));
}
catch (AuthenticationException e)
{
throw new LoginException("Error setting callback handler" + e);
}
} else {
throw new LoginException("Unsupported authentication type " +
this.authTypeStr);
}
}
transportFactory =
this.saslServer.wrapTransportFactory(serverTransportFactory);
}
else
{
TTransportFactory transportFactory;
if ((this.authTypeStr.equalsIgnoreCase(AuthTypes.NONE.getAuthName())) ||
(this.authTypeStr.equalsIgnoreCase(AuthTypes.LDAP.getAuthName())) ||
(this.authTypeStr.equalsIgnoreCase(AuthTypes.PAM.getAuthName())) ||
(this.authTypeStr.equalsIgnoreCase(AuthTypes.CUSTOM.getAuthName(
{
transportFactory =
PlainSaslHelper.getPlainTransportFactory(this.authTypeStr);
}
else
{
TTransportFactory transportFactory;
if (this.authTypeStr.equalsIgnoreCase(AuthTypes.NOSASL.getAuthName())) {
transportFactory = new TTransportFactory();
} else {
throw new LoginException("Unsupported authentication type " +
this.authTypeStr);
}
}
}
TTransportFactory transportFactory;
return transportFactory;
}
was:
The logic in getAuthTransFactory() is flawed, as it will always "Unsupported
authentication type" error when authTypeStr is set to KERBEROS.
inside HiveAuthFactory class.
public TTransportFactory getAuthTransFactory()
throws LoginException
{
TTransportFactory transportFactory;
if (isSASLWithKerberizedHadoop())
{
try
{
serverTransportFactory =
this.saslServer.createSaslServerTransportFactory(
getSaslProperties());
}
catch (TTransportException e)
{
TSaslServerTransport.Factory serverTransportFactory;
throw new LoginException(e.getMessage());
}
TSaslServerTransport.Factory serverTransportFactory;
if (!this.authTypeStr.equalsIgnoreCase(AuthTypes.KERBEROS.getAuthName()))
{
if ((this.authTypeStr.equalsIgnoreCase(AuthTypes.NONE.getAuthName()))
||
(this.authTypeStr.equalsIgnoreCase(AuthTypes.LDAP.getAuthName())) ||
(this.authTypeStr.equalsIgnoreCase(AuthTypes.PAM.getAuthName())) ||
(this.authTypeStr.equalsIgnoreCase(AuthTypes.CUSTOM.getAuthName( {
try
{
serverTransportFactory.addServerDefinition("PLAIN",
this.authTypeStr, null, new HashMap(), new
PlainSaslHelper.PlainServerCallbackHandler(this.authTypeStr));
}
catch (AuthenticationException e)
{
throw new LoginException("Error setting callback handler" + e);
}
} else {
throw new LoginException("Unsupported authentication type " +
this.authTypeStr);
}
}
transportFactory =
this.saslServer.wrapTransportFactory(serverTransportFactory);
}
else
{
TTransportFactory transportFactory;
if ((this.authTypeStr.equalsIgnoreCase(AuthTypes.NONE.getAuthName())) ||
(this.authTypeStr.equalsIgnoreCase(AuthTypes.LDAP.getAuthName())) ||
(this.authTypeStr.equalsIgnoreCase(AuthTypes.PAM.getAuthName())) ||
(this.authTypeStr.equalsIgnoreCase(AuthTypes.CUSTOM.getAuthName(
{
transportFactory =
PlainSaslHelper.getPlainTransportFactory(this.authTypeStr);
}
else
{
TTransportFactory
[jira] [Commented] (HIVE-17202) Add InterfaceAudience and InterfaceStability annotations for HMS Listener APIs
[
https://issues.apache.org/jira/browse/HIVE-17202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106109#comment-16106109
]
Hive QA commented on HIVE-17202:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12879419/HIVE-17202.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 11011 tests
executed
*Failed tests:*
{noformat}
TestPerfCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=99)
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver
(batchId=242)
org.apache.hadoop.hive.metastore.TestHiveMetaStoreStatsMerge.testStatsMerge
(batchId=206)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=179)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6186/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6186/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6186/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12879419 - PreCommit-HIVE-Build
> Add InterfaceAudience and InterfaceStability annotations for HMS Listener APIs
> --
>
> Key: HIVE-17202
> URL: https://issues.apache.org/jira/browse/HIVE-17202
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17202.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17195) Long chain of tasks created by REPL LOAD shouldn't cause stack corruption.
[ https://issues.apache.org/jira/browse/HIVE-17195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-17195: Status: Patch Available (was: Open) > Long chain of tasks created by REPL LOAD shouldn't cause stack corruption. > -- > > Key: HIVE-17195 > URL: https://issues.apache.org/jira/browse/HIVE-17195 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DAG, DR, Executor, replication > Fix For: 3.0.0 > > Attachments: HIVE-17195.01.patch, HIVE-17195.02.patch > > > Currently, long chain REPL LOAD tasks lead to huge recursive calls when try > to traverse the DAG. > For example, getMRTasks, getTezTasks, getSparkTasks and iterateTasks methods > run recursively to traverse the DAG. > Need to modify this traversal logic to reduce stack usage. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17195) Long chain of tasks created by REPL LOAD shouldn't cause stack corruption.
[ https://issues.apache.org/jira/browse/HIVE-17195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-17195: Attachment: HIVE-17195.02.patch Added 02.patch with fixes for test failures. > Long chain of tasks created by REPL LOAD shouldn't cause stack corruption. > -- > > Key: HIVE-17195 > URL: https://issues.apache.org/jira/browse/HIVE-17195 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DAG, DR, Executor, replication > Fix For: 3.0.0 > > Attachments: HIVE-17195.01.patch, HIVE-17195.02.patch > > > Currently, long chain REPL LOAD tasks lead to huge recursive calls when try > to traverse the DAG. > For example, getMRTasks, getTezTasks, getSparkTasks and iterateTasks methods > run recursively to traverse the DAG. > Need to modify this traversal logic to reduce stack usage. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17195) Long chain of tasks created by REPL LOAD shouldn't cause stack corruption.
[ https://issues.apache.org/jira/browse/HIVE-17195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-17195: Status: Open (was: Patch Available) > Long chain of tasks created by REPL LOAD shouldn't cause stack corruption. > -- > > Key: HIVE-17195 > URL: https://issues.apache.org/jira/browse/HIVE-17195 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DAG, DR, Executor, replication > Fix For: 3.0.0 > > Attachments: HIVE-17195.01.patch > > > Currently, long chain REPL LOAD tasks lead to huge recursive calls when try > to traverse the DAG. > For example, getMRTasks, getTezTasks, getSparkTasks and iterateTasks methods > run recursively to traverse the DAG. > Need to modify this traversal logic to reduce stack usage. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17190) Schema changes for bitvectors for unpartitioned tables
[
https://issues.apache.org/jira/browse/HIVE-17190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106094#comment-16106094
]
Hive QA commented on HIVE-17190:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12879420/HIVE-17190.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 104 failed/errored test(s), 4 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[create_merge_compressed]
(batchId=240)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query10]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query11]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query12]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query13]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query15]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query16]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query17]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query18]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query19]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query1] (batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query20]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query21]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query22]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query24]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query25]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query26]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query27]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query28]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query29]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query2] (batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query30]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query31]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query32]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query33]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query34]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query35]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query36]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query37]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query38]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query39]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query3] (batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query40]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query42]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query43]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query44]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query45]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query46]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query47]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query48]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query49]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query4] (batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query50]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query51]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query52]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query53]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query54]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query55]
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query56]
[jira] [Commented] (HIVE-17192) Add InterfaceAudience and InterfaceStability annotations for Stats Collection APIs
[
https://issues.apache.org/jira/browse/HIVE-17192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106083#comment-16106083
]
Hive QA commented on HIVE-17192:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12879417/HIVE-17192.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 11017 tests
executed
*Failed tests:*
{noformat}
TestPerfCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=240)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_13]
(batchId=240)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=168)
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[infer_bucket_sort_map_operators]
(batchId=87)
org.apache.hadoop.hive.metastore.TestHiveMetaStoreStatsMerge.testStatsMerge
(batchId=206)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=179)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6184/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6184/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6184/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12879417 - PreCommit-HIVE-Build
> Add InterfaceAudience and InterfaceStability annotations for Stats Collection
> APIs
> --
>
> Key: HIVE-17192
> URL: https://issues.apache.org/jira/browse/HIVE-17192
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-17192.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16811) Estimate statistics in absence of stats
[
https://issues.apache.org/jira/browse/HIVE-16811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106072#comment-16106072
]
Hive QA commented on HIVE-16811:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12879392/HIVE-16811.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 339 failed/errored test(s), 6 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[insert_overwrite_local_directory_1]
(batchId=240)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_filter]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_groupby]
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_select]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_table]
(batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_annotate_stats_groupby]
(batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[columnStatsUpdateForStatsOptimizer_2]
(batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_47]
(batchId=28)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[cte_4]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[except_distinct]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[explainuser_2]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[insert_into1]
(batchId=141)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[intersect_all]
(batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[intersect_distinct]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[intersect_merge]
(batchId=141)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_nullscan]
(batchId=141)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_stats]
(batchId=141)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_udf]
(batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llapdecider]
(batchId=141)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[multi_count_distinct_null]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge3]
(batchId=141)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_merge4]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[rcfile_createas1]
(batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[rcfile_merge3]
(batchId=143)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[rcfile_merge4]
(batchId=141)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[reduce_deduplicate_distinct]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_10]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_12]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_13]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_14]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_15]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_1]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_3]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_4]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_5]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_6]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_7]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_8]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_groupby]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez2]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketmapjoin2]
(batchId=145)
[jira] [Commented] (HIVE-17102) Example For Vectorized Execution in Hive in Cwiki not Seems to Work
[
https://issues.apache.org/jira/browse/HIVE-17102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106070#comment-16106070
]
Lefty Leverenz commented on HIVE-17102:
---
Also, why does your explain output have table name "alltypesorc" with alias
"vectorizedtable" when your CREATE TABLE statement just names the table
"vectorizedtable"?
> Example For Vectorized Execution in Hive in Cwiki not Seems to Work
> ---
>
> Key: HIVE-17102
> URL: https://issues.apache.org/jira/browse/HIVE-17102
> Project: Hive
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.2.0
>Reporter: anubhav tarar
>Assignee: anubhav tarar
>
> i tried to do vectorized execution in hive by using hive cwiki but example do
> not seems to work
> step1:created a orc table
> hive> create table Addresses (
> > name string,
> > street string,
> > city string,
> > state string,
> > zip int
> > ) stored as orc tblproperties ("orc.compress"="NONE");
> step2:insert the values in table
> hive> insert into Addresses values('anubhav','ggn','ggn','haryana','122001');
> Query ID = hduser_20170716093152_14774003-d2c4-4620-b773-ca17cafd902b
> Total jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Listening for transport dt_socket at address: 5005
> Job running in-process (local Hadoop)
> 2017-07-16 09:31:59,689 Stage-1 map = 100%, reduce = 0%
> Ended Job = job_local1858411694_0004
> Stage-4 is selected by condition resolver.
> Stage-3 is filtered out by condition resolver.
> Stage-5 is filtered out by condition resolver.
> Moving data to:
> hdfs://localhost:54310/user/hive/warehouse/addresses/.hive-staging_hive_2017-07-16_09-31-52_428_7861150459629073282-1/-ext-1
> Loading data to table default.addresses
> Table default.addresses stats: [numFiles=1, numRows=1, totalSize=713,
> rawDataSize=360]
> MapReduce Jobs Launched:
> Stage-Stage-1: HDFS Read: 778 HDFS Write: 818 SUCCESS
> Total MapReduce CPU Time Spent: 0 msec
> step3:query the table with explain command
> hive> set hive.vectorized.execution.enabled = true;
> hive> explain select name from Addresses where zip>1;
> OK
> STAGE DEPENDENCIES:
> Stage-0 is a root stage
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> TableScan
> alias: addresses
> Statistics: Num rows: 1 Data size: 360 Basic stats: COMPLETE Column
> stats: NONE
> Filter Operator
> predicate: (zip > 1) (type: boolean)
> Statistics: Num rows: 1 Data size: 360 Basic stats: COMPLETE
> Column stats: NONE
> Select Operator
> expressions: name (type: string)
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 360 Basic stats: COMPLETE
> Column stats: NONE
> ListSink
> Time taken: 0.081 seconds, Fetched: 20 row(s)
> note:in explain command there is not vectorized reader applied
> reason for failiure is that When Fetch is used in the plan instead of Map, it
> do not vectorize
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17102) Example For Vectorized Execution in Hive in Cwiki not Seems to Work
[
https://issues.apache.org/jira/browse/HIVE-17102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106067#comment-16106067
]
Lefty Leverenz commented on HIVE-17102:
---
[~anubhavtarar], your new example does not show what is supposed to be shown in
that section: "Vectorized execution: true" notation in an explain output. We
need an example showing vectorization as well as your example showing absence
of vectorization.
Can you modify the original example to make it accurate, then give your example
after it? (Page History is available in the ". . ." drop-down menu on the
upper right corner of the page.)
* [Vectorized Query Execution -- Seeing whether vectorization is used for a
query |
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=73630110#VectorizedQueryExecution-Seeingwhethervectorizationisusedforaquery]
> Example For Vectorized Execution in Hive in Cwiki not Seems to Work
> ---
>
> Key: HIVE-17102
> URL: https://issues.apache.org/jira/browse/HIVE-17102
> Project: Hive
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.2.0
>Reporter: anubhav tarar
>Assignee: anubhav tarar
>
> i tried to do vectorized execution in hive by using hive cwiki but example do
> not seems to work
> step1:created a orc table
> hive> create table Addresses (
> > name string,
> > street string,
> > city string,
> > state string,
> > zip int
> > ) stored as orc tblproperties ("orc.compress"="NONE");
> step2:insert the values in table
> hive> insert into Addresses values('anubhav','ggn','ggn','haryana','122001');
> Query ID = hduser_20170716093152_14774003-d2c4-4620-b773-ca17cafd902b
> Total jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Listening for transport dt_socket at address: 5005
> Job running in-process (local Hadoop)
> 2017-07-16 09:31:59,689 Stage-1 map = 100%, reduce = 0%
> Ended Job = job_local1858411694_0004
> Stage-4 is selected by condition resolver.
> Stage-3 is filtered out by condition resolver.
> Stage-5 is filtered out by condition resolver.
> Moving data to:
> hdfs://localhost:54310/user/hive/warehouse/addresses/.hive-staging_hive_2017-07-16_09-31-52_428_7861150459629073282-1/-ext-1
> Loading data to table default.addresses
> Table default.addresses stats: [numFiles=1, numRows=1, totalSize=713,
> rawDataSize=360]
> MapReduce Jobs Launched:
> Stage-Stage-1: HDFS Read: 778 HDFS Write: 818 SUCCESS
> Total MapReduce CPU Time Spent: 0 msec
> step3:query the table with explain command
> hive> set hive.vectorized.execution.enabled = true;
> hive> explain select name from Addresses where zip>1;
> OK
> STAGE DEPENDENCIES:
> Stage-0 is a root stage
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> TableScan
> alias: addresses
> Statistics: Num rows: 1 Data size: 360 Basic stats: COMPLETE Column
> stats: NONE
> Filter Operator
> predicate: (zip > 1) (type: boolean)
> Statistics: Num rows: 1 Data size: 360 Basic stats: COMPLETE
> Column stats: NONE
> Select Operator
> expressions: name (type: string)
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 360 Basic stats: COMPLETE
> Column stats: NONE
> ListSink
> Time taken: 0.081 seconds, Fetched: 20 row(s)
> note:in explain command there is not vectorized reader applied
> reason for failiure is that When Fetch is used in the plan instead of Map, it
> do not vectorize
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16998) Add config to enable HoS DPP only for map-joins
[
https://issues.apache.org/jira/browse/HIVE-16998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106053#comment-16106053
]
Hive QA commented on HIVE-16998:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12879391/HIVE16998.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 11018 tests
executed
*Failed tests:*
{noformat}
TestPerfCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample_islocalmode_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=99)
org.apache.hadoop.hive.metastore.TestHiveMetaStoreStatsMerge.testStatsMerge
(batchId=206)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=179)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testConnection (batchId=241)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeTokenAuth
(batchId=241)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6182/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6182/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6182/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12879391 - PreCommit-HIVE-Build
> Add config to enable HoS DPP only for map-joins
> ---
>
> Key: HIVE-16998
> URL: https://issues.apache.org/jira/browse/HIVE-16998
> Project: Hive
> Issue Type: Sub-task
> Components: Logical Optimizer, Spark
>Reporter: Sahil Takiar
>Assignee: Janaki Lahorani
> Attachments: HIVE16998.1.patch, HIVE16998.2.patch, HIVE16998.3.patch,
> HIVE16998.4.patch, HIVE16998.5.patch
>
>
> HoS DPP will split a given operator tree in two under the following
> conditions: it has detected that the query can benefit from DPP, and the
> filter is not a map-join (see SplitOpTreeForDPP).
> This can hurt performance if the the non-partitioned side of the join
> involves a complex operator tree - e.g. the query {{select count(*) from
> srcpart where srcpart.ds in (select max(srcpart.ds) from srcpart union all
> select min(srcpart.ds) from srcpart)}} will require running the subquery
> twice, once in each Spark job.
> Queries with map-joins don't get split into two operator trees and thus don't
> suffer from this drawback. Thus, it would be nice to have a config key that
> just enables DPP on HoS for map-joins.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16982) WebUI "Show Query" tab prints "UNKNOWN" instead of explaining configuration option
[ https://issues.apache.org/jira/browse/HIVE-16982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106045#comment-16106045 ] Lefty Leverenz commented on HIVE-16982: --- [~klcopp], you can take care of the documentation yourself if you get wiki edit privileges: * [About This Wiki -- How to get permission to edit | https://cwiki.apache.org/confluence/display/Hive/AboutThisWiki#AboutThisWiki-Howtogetpermissiontoedit] > WebUI "Show Query" tab prints "UNKNOWN" instead of explaining configuration > option > -- > > Key: HIVE-16982 > URL: https://issues.apache.org/jira/browse/HIVE-16982 > Project: Hive > Issue Type: Bug > Components: Configuration, Web UI >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Minor > Labels: TODOC3.0, newbie, patch > Fix For: 3.0.0 > > Attachments: HIVE-16982.3.patch > > > In the Hive WebUI / Drilldown: the Show Query tab always displays "UNKNOWN." > If the user wants to see the query plan here, they should set configuration > hive.log.explain.output to true. The user should be made aware of this option: > 1) in WebUI / Drilldown / Show Query and > 2) in HiveConf.java, line 2232. > This configuration's description reads: > "Whether to log explain output for every query > When enabled, will log EXPLAIN EXTENDED output for the query at INFO log4j > log level." > this should be added: > "...and in the WebUI / Show Query tab." -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16982) WebUI "Show Query" tab prints "UNKNOWN" instead of explaining configuration option
[ https://issues.apache.org/jira/browse/HIVE-16982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106040#comment-16106040 ] Lefty Leverenz commented on HIVE-16982: --- Doc note: This changes the description of configuration parameter *hive.log.explain.output*, which is not yet documented in the wiki. It was introduced in release 1.1.0 by HIVE-8600, and ought to be documented here: * [Configuration Properties -- Query and DDL Execution | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-QueryandDDLExecution] * [Getting Started -- Logs | https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-ErrorLogs] * [Language Manual -- Explain | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Explain] Configuration Properties can give just the new description, since the behavior isn't changing. Added a TODOC3.0 label. > WebUI "Show Query" tab prints "UNKNOWN" instead of explaining configuration > option > -- > > Key: HIVE-16982 > URL: https://issues.apache.org/jira/browse/HIVE-16982 > Project: Hive > Issue Type: Bug > Components: Configuration, Web UI >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Minor > Labels: TODOC3.0, newbie, patch > Fix For: 3.0.0 > > Attachments: HIVE-16982.3.patch > > > In the Hive WebUI / Drilldown: the Show Query tab always displays "UNKNOWN." > If the user wants to see the query plan here, they should set configuration > hive.log.explain.output to true. The user should be made aware of this option: > 1) in WebUI / Drilldown / Show Query and > 2) in HiveConf.java, line 2232. > This configuration's description reads: > "Whether to log explain output for every query > When enabled, will log EXPLAIN EXTENDED output for the query at INFO log4j > log level." > this should be added: > "...and in the WebUI / Show Query tab." -- This message was sent by Atlassian JIRA (v6.4.14#64029)
